CLIENT-SIDE EXECUTION FLOW (SF3 + SPESSASYNTH + INDEXEDDB)

This document describes the step-by-step processing chain that takes place inside the Client Browser, from launching the Web DAW application, downloading and buffering .sf3 instrument files, and setting up MIDI channels, to outputting real-time audio.

1. SEQUENCE DIAGRAM

[ USER / UI ]        [ APP / CLIENT ]        [ INDEXEDDB ]        [ SERVER API ]      [ SPESSASYNTH ENGINE ]
      |                     |                      |                    |                       |
1. Open Web Page ---------> | Initial AudioCtx     |                    |                       |
      |                     | Init SpessaSynth -----------------------------------------------> | Connect Destination
      |                     | Fetch Catalog --------------------------->| Get /catalog          |
      |                     |                      |                    |                       |
2. Select Instrument -----> | Read (sf_id, bank, prog)                  |                       |
   (e.g., Pipa)             | Query SF3 Buffer --->| Check Key (sf_id)  |                       |
      |                     |                      | -- (Miss) -------->| Fetch /download/sf_id |
      |                     |                      |                    | Return .sf3 (~4MB)    |
      |                     | <--------------------| Save ArrayBuffer --|                       |
      |                     | Load SF3 Memory ------------------------------------------------> | addSoundFont(buffer)
      |                     |                      |                    |                       |
3. Channel Router --------> | Switch Bank/Program -------------------------------------------> | controllerChange(ch, 0, bank)
      |                     |                      |                    |                       | programChange(ch, prog)
      |                     |                      |                    |                       |
4. Trigger MIDI Key ------> | Raw MIDI Event       |                    |                       |
   (or Timeline Play)       | (noteOn: pitch, vel) -------------------------------------------> | noteOn(ch, pitch, vel)
      |                     |                      |                    |                       | AudioWorklet Synthesis
      |                     | <------------------------------------------------------------------ | Audio Out (User Speakers)

2. DETAILED PROCESSING PHASES

PHASE 1: BOOTSTRAPPING & ENGINE INIT

Web Audio Context Initialization: Upon the user's first interaction with the web page (Mouse Click/Keypress), the application initializes the AudioContext.
SpessaSynth Synthesizer Initialization: The soundfontPlayer.js module instantiates SpessaSynthClass and connects its output directly to audioCtx.destination:

this.synthInstance = new window.SpessaSynthClass(this.audioCtx.destination);

Instrument Catalog Load (Catalog Context): The Frontend dispatches a GET /api/v1/plugins/soundfonts/catalog request to load the condensed_catalog, which contains lookup tables for sf_id, bank, and program.

PHASE 2: `.SF3` ASSET LOADING & CACHING

Triggered when a user selects an instrument via the Synth UI button (or when the AI Copilot spawns a new Track with a designated instrument, e.g., dsk_asian_dreamz):

Query Browser Cache (IndexedDB): The Client calls sfStorage.getBuffer(sfId) to search for the .sf3 file's ArrayBuffer inside the DAW_SoundFont_Cache database.
Handling Cache Hit vs Cache Miss:
Cache Hit (0\text{ms}): Retrieves the ArrayBuffer directly from the browser's RAM/Storage.
Cache Miss:

Sends a GET /api/v1/plugins/soundfonts/download/{sf_id} request to the Server.
Downloads the compressed, optimized .sf3 asset (ultra-lightweight size \sim 3.5 - 5.5\text{ MB}).
Invokes sfStorage.saveBuffer(sfId, arrayBuffer) to store it inside IndexedDB for subsequent visits.

Load Data into SpessaSynth Wasm/JS Memory: Passes the ArrayBuffer to SpessaSynth Engine's SoundFontManager:

await this.synthInstance.soundFontManager.addSoundFont(buffer);

PHASE 3: BANK/PROGRAM ROUTING & MIDI CHANNEL SETUP

This is the most critical phase to resolve No preset found for 0:0:0 errors.

MIDI Channel Assignment:
Melodic Instruments (Piano, Pipa, Strings, Brass, etc.): Allocated to Channels 0 through 8.
Percussion / Drum Kits (Bank 128): Mandatory allocation to Channel 9 (GM Standard Channel 10).
Dispatch Bank Select & Program Change to SpessaSynth Engine: Prior to scheduling any note events, the Client triggers two simultaneous control events:

// 1. Select Bank (Control Change 0)
this.synthInstance.controllerChange(channel, 0, bank);

// 2. Select Program (Program Change)
this.synthInstance.programChange(channel, program);

Example for Pipa (dsk_asian_dreamz): Calls controllerChange(0, 0, 0) and programChange(0, 0). SpessaSynth switches Channel 0's state to the Pipa instrument patch.

PHASE 4: REALTIME SYNTHESIS & AUDIO OUTPUT

Triggered when receiving note-control signals (from a Hardware MIDI Keyboard or Timeline Transport Playback):

Scenario A: User plays a Hardware MIDI Keyboard (e.g., Nektar SE49)

The browser receives a Raw MIDI Event: Web MIDI API captures message [144, 65, 78] (NoteOn, Pitch 65, Velocity 78).
Latency Compensation: Calculates real-time offsets and issues NoteOn to SpessaSynth:

const midiPitch = pitch;
const midiVelocity = Math.floor(velocity * 127);
this.synthInstance.noteOn(channel, midiPitch, midiVelocity);

Key Release: Triggers a NoteOff event:

this.synthInstance.noteOff(channel, midiPitch);

Scenario B: User triggers Play on Timeline / Piano Roll

Transport Controller & Scheduler (PrecisionAudioScheduler): Scans for MIDI notes located within the moving Playhead window.
Note Scheduling:

Converts beat positions to precise audio timing based on BPM tempo (exactAudioTime).
Dispatches noteOn(channel, pitch, velocity) at the exact timestamp T_{\text{start}}.
Dispatches noteOff(channel, pitch) at timestamp T_{\text{start}} + T_{\text{duration}}.
Audio Worklet Audio Rendering: SpessaSynth Engine reads Ogg/WAV sample data inside the .sf3 asset, applies Envelopes (ADSR), Modulators, and Gain Control parameters on the designated Channel, and pushes PCM audio data directly to user speakers with 0\text{ms} latency.

3. 100% RELIABILITY VERIFICATION CHECKLIST

.sf3 files loaded into the browser open without triggering Corrupted File errors.
The sfStorage.getBuffer function successfully stores and retrieves ArrayBuffer data from IndexedDB.
Both controllerChange(channel, 0, bank) and programChange(channel, program) are invoked immediately upon changing instruments on the UI.
Percussion/Drum instruments are persistently allocated to Channel 9.
Console logs confirm: [SonicSF] Switched Channel X -> Bank: B, Program: P.

7.1 KiB Raw Permalink Blame History