144 lines
7.1 KiB
Markdown
144 lines
7.1 KiB
Markdown
# CLIENT-SIDE EXECUTION FLOW (SF3 + SPESSASYNTH + INDEXEDDB)
|
|
|
|
This document describes the step-by-step processing chain that takes place inside the Client Browser, from launching the Web DAW application, downloading and buffering `.sf3` instrument files, and setting up MIDI channels, to outputting real-time audio.
|
|
|
|
---
|
|
|
|
## 1. SEQUENCE DIAGRAM
|
|
|
|
```text
|
|
[ USER / UI ] [ APP / CLIENT ] [ INDEXEDDB ] [ SERVER API ] [ SPESSASYNTH ENGINE ]
|
|
| | | | |
|
|
1. Open Web Page ---------> | Initial AudioCtx | | |
|
|
| | Init SpessaSynth -----------------------------------------------> | Connect Destination
|
|
| | Fetch Catalog --------------------------->| Get /catalog |
|
|
| | | | |
|
|
2. Select Instrument -----> | Read (sf_id, bank, prog) | |
|
|
(e.g., Pipa) | Query SF3 Buffer --->| Check Key (sf_id) | |
|
|
| | | -- (Miss) -------->| Fetch /download/sf_id |
|
|
| | | | Return .sf3 (~4MB) |
|
|
| | <--------------------| Save ArrayBuffer --| |
|
|
| | Load SF3 Memory ------------------------------------------------> | addSoundFont(buffer)
|
|
| | | | |
|
|
3. Channel Router --------> | Switch Bank/Program -------------------------------------------> | controllerChange(ch, 0, bank)
|
|
| | | | | programChange(ch, prog)
|
|
| | | | |
|
|
4. Trigger MIDI Key ------> | Raw MIDI Event | | |
|
|
(or Timeline Play) | (noteOn: pitch, vel) -------------------------------------------> | noteOn(ch, pitch, vel)
|
|
| | | | | AudioWorklet Synthesis
|
|
| | <------------------------------------------------------------------ | Audio Out (User Speakers)
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 2. DETAILED PROCESSING PHASES
|
|
|
|
### PHASE 1: BOOTSTRAPPING & ENGINE INIT
|
|
|
|
* **Web Audio Context Initialization:** Upon the user's first interaction with the web page (Mouse Click/Keypress), the application initializes the `AudioContext`.
|
|
* **SpessaSynth Synthesizer Initialization:** The `soundfontPlayer.js` module instantiates `SpessaSynthClass` and connects its output directly to `audioCtx.destination`:
|
|
```javascript
|
|
this.synthInstance = new window.SpessaSynthClass(this.audioCtx.destination);
|
|
|
|
```
|
|
|
|
|
|
* **Instrument Catalog Load (Catalog Context):** The Frontend dispatches a `GET /api/v1/plugins/soundfonts/catalog` request to load the `condensed_catalog`, which contains lookup tables for `sf_id`, `bank`, and `program`.
|
|
|
|
---
|
|
|
|
### PHASE 2: `.SF3` ASSET LOADING & CACHING
|
|
|
|
Triggered when a user selects an instrument via the Synth UI button (or when the AI Copilot spawns a new Track with a designated instrument, e.g., `dsk_asian_dreamz`):
|
|
|
|
* **Query Browser Cache (IndexedDB):** The Client calls `sfStorage.getBuffer(sfId)` to search for the `.sf3` file's `ArrayBuffer` inside the `DAW_SoundFont_Cache` database.
|
|
* **Handling Cache Hit vs Cache Miss:**
|
|
* **Cache Hit ($0\text{ms}$):** Retrieves the `ArrayBuffer` directly from the browser's RAM/Storage.
|
|
* **Cache Miss:**
|
|
1. Sends a `GET /api/v1/plugins/soundfonts/download/{sf_id}` request to the Server.
|
|
2. Downloads the compressed, optimized `.sf3` asset (ultra-lightweight size $\sim 3.5 - 5.5\text{ MB}$).
|
|
3. Invokes `sfStorage.saveBuffer(sfId, arrayBuffer)` to store it inside IndexedDB for subsequent visits.
|
|
|
|
|
|
|
|
|
|
* **Load Data into SpessaSynth Wasm/JS Memory:** Passes the `ArrayBuffer` to SpessaSynth Engine's `SoundFontManager`:
|
|
```javascript
|
|
await this.synthInstance.soundFontManager.addSoundFont(buffer);
|
|
|
|
```
|
|
|
|
|
|
|
|
---
|
|
|
|
### PHASE 3: BANK/PROGRAM ROUTING & MIDI CHANNEL SETUP
|
|
|
|
This is the most critical phase to resolve `No preset found for 0:0:0` errors.
|
|
|
|
* **MIDI Channel Assignment:**
|
|
* **Melodic Instruments (Piano, Pipa, Strings, Brass, etc.):** Allocated to Channels 0 through 8.
|
|
* **Percussion / Drum Kits (Bank 128):** Mandatory allocation to Channel 9 (GM Standard Channel 10).
|
|
|
|
|
|
* **Dispatch Bank Select & Program Change to SpessaSynth Engine:** Prior to scheduling any note events, the Client triggers two simultaneous control events:
|
|
```javascript
|
|
// 1. Select Bank (Control Change 0)
|
|
this.synthInstance.controllerChange(channel, 0, bank);
|
|
|
|
// 2. Select Program (Program Change)
|
|
this.synthInstance.programChange(channel, program);
|
|
|
|
```
|
|
|
|
|
|
*Example for Pipa (`dsk_asian_dreamz`):* Calls `controllerChange(0, 0, 0)` and `programChange(0, 0)`. SpessaSynth switches Channel 0's state to the Pipa instrument patch.
|
|
|
|
---
|
|
|
|
### PHASE 4: REALTIME SYNTHESIS & AUDIO OUTPUT
|
|
|
|
Triggered when receiving note-control signals (from a Hardware MIDI Keyboard or Timeline Transport Playback):
|
|
|
|
* **Scenario A: User plays a Hardware MIDI Keyboard (e.g., Nektar SE49)**
|
|
1. The browser receives a Raw MIDI Event: Web MIDI API captures message `[144, 65, 78]` (`NoteOn`, `Pitch 65`, `Velocity 78`).
|
|
2. **Latency Compensation:** Calculates real-time offsets and issues `NoteOn` to SpessaSynth:
|
|
```javascript
|
|
const midiPitch = pitch;
|
|
const midiVelocity = Math.floor(velocity * 127);
|
|
this.synthInstance.noteOn(channel, midiPitch, midiVelocity);
|
|
|
|
```
|
|
|
|
|
|
3. **Key Release:** Triggers a `NoteOff` event:
|
|
```javascript
|
|
this.synthInstance.noteOff(channel, midiPitch);
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
* **Scenario B: User triggers Play on Timeline / Piano Roll**
|
|
1. **Transport Controller & Scheduler (`PrecisionAudioScheduler`):** Scans for MIDI notes located within the moving Playhead window.
|
|
2. **Note Scheduling:**
|
|
* Converts beat positions to precise audio timing based on BPM tempo (`exactAudioTime`).
|
|
* Dispatches `noteOn(channel, pitch, velocity)` at the exact timestamp $T_{\text{start}}$.
|
|
* Dispatches `noteOff(channel, pitch)` at timestamp $T_{\text{start}} + T_{\text{duration}}$.
|
|
|
|
|
|
|
|
|
|
* **Audio Worklet Audio Rendering:** SpessaSynth Engine reads Ogg/WAV sample data inside the `.sf3` asset, applies Envelopes (ADSR), Modulators, and Gain Control parameters on the designated Channel, and pushes PCM audio data directly to user speakers with $0\text{ms}$ latency.
|
|
|
|
---
|
|
|
|
## 3. 100% RELIABILITY VERIFICATION CHECKLIST
|
|
|
|
* [ ] `.sf3` files loaded into the browser open without triggering `Corrupted File` errors.
|
|
* [ ] The `sfStorage.getBuffer` function successfully stores and retrieves `ArrayBuffer` data from IndexedDB.
|
|
* [ ] Both `controllerChange(channel, 0, bank)` and `programChange(channel, program)` are invoked immediately upon changing instruments on the UI.
|
|
* [ ] Percussion/Drum instruments are persistently allocated to Channel 9.
|
|
* [ ] Console logs confirm: `[SonicSF] Switched Channel X -> Bank: B, Program: P`. |