diff --git a/.kilo/plans/1785130485229-fluidsynth-wasm-migration-plan.md b/.kilo/plans/1785130485229-fluidsynth-wasm-migration-plan.md new file mode 100644 index 0000000..692a638 --- /dev/null +++ b/.kilo/plans/1785130485229-fluidsynth-wasm-migration-plan.md @@ -0,0 +1,398 @@ +# Technical Analysis & Implementation Plan: SpessaSynth → FluidSynth WASM Migration + +## 1. Key Findings from Codebase Investigation + +### 1.1 Current State (SpessaSynth) +| Aspect | Detail | +|---|---| +| Engine | `spessasynth_lib@4.3.1` (CDN via jsDelivr + importmap) | +| File | `app/static/js/services/soundfontPlayer.js` — IIFE, `window.SonicSF` singleton | +| API surface | 10 methods: `init`, `playNote`, `stopNote`, `stopAll`, `selectInstrument`, `controllerChange`, `programChange`, `pitchBend`, `sustainActive`, `applyAITrackInstrument`, `allocateChannel`, `getChannelState`, `loadSoundFont` (internal) | +| Call sites | 45 in `app.jsx` across piano roll, MIDI keyboard, timeline playback, external MIDI input, AI track generation | +| storage | `app/static/js/services/soundfontStorage.js` — IndexedDB cache, `window.SonicSFStorage` | +| Server render | `app/core/render_engine.py` uses `pyfluidsynth` (C++ FluidSynth) | +| Server SF3 | `app/core/soundfont_converter.py` converts SF2→SF3 via `fluidsynth`/`mscore` CLI | +| API endpoint | `GET /api/v1/plugins/soundfonts/download/{sf_id}` — serves both `.sf2` and `.sf3` | +| Docker | `libfluidsynth3`, `fluidsynth` CLI installed in container | + +### 1.2 Target State (FluidSynth WASM) — Corrected Spec + +**CDN package**: `fluidsynth-wasm` does NOT exist on npm/CDN. +**Real package**: `@enikey87/fluidsynth-emscripten@0.1.1` + +| File | Size | Purpose | +|---|---|---| +| `libfluidsynth-2.3.0-sf3.js` | 130 KB | JS loader, SF3 support, separate WASM | +| `libfluidsynth-2.3.0-sf3.wasm` | 1.7 MB | WASM binary with Ogg/SF3 support | +| `libfluidsynth-2.3.0-sf3-all-in-one.js` | 2.39 MB | JS+WASM embedded (no separate .wasm fetch) | + +**CDN URL** (chosen: separate .wasm for caching): +``` +https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js +https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.wasm +``` + +### 1.3 CRITICAL Architecture Flaw in Spec + +The spec's code (md/39_SF3_NEW.md) passes `wasmModule` and `synthPtr` via `postMessage` to AudioWorklet. **This cannot work** because: +1. AudioWorklet runs in a separate audio thread with an isolated global scope +2. `synthPtr` (a C pointer / integer) refers to WASM linear memory in the main thread's `WebAssembly.Module` instance +3. Loading `libfluidsynth-2.3.0-sf3.js` inside the worklet creates a **separate WASM instance** with its own memory +4. The spec also uses `_fluid_synth_write_float` inside `process()` — this blocks the audio thread if audio buffer underruns occur + +**Correct architecture (per README):** +> "To use libfluidsynth-X.X.X.js in AudioWorklet, load it into AudioWorklet before your worklet JS file." + +The FluidSynth instance must be created **inside** the AudioWorklet. MIDI events + ArrayBuffer data are sent from main thread to worklet via `port.postMessage`. + +### 1.4 `playNote` Signature Complexity + +Current `playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine)` has 8 parameters and 16 call sites. The `synthEngine` object carries `{ soundfont_id, soundfont_bank, soundfont_program }` which must be converted to FluidSynth's `bank_select` + `program_change` before `noteon`. + +### 1.5 Additional `window.SonicSF` Methods Not in Spec + +Must preserve in new implementation: +- `stopNote(channel, pitch)` — MIDI note-off with extra CC 64/120 cleanup +- `controllerChange(channel, controller, value)` — MIDI CC forwarding +- `programChange(channel, program)` — pure JS state tracking +- `pitchBend(channel, value)` — 14-bit bend value +- `sustainActive(channel)` — getter for sustain pedal state +- `allocateChannel(bank)` — channel allocation (percussion = ch 9) +- `getChannelState(channel)` — JS-side { bank, program, isPercussion } state +- `saveToIndexedDB / loadFromIndexedDB` — legacy storage + +--- + +## 2. Implementation Plan + +### Phase 1: Preparation (Infrastructure) + +**Task 1.1 — Create worklet directory** +``` +mkdir -p app/static/js/worklets/ +touch app/static/js/worklets/.gitkeep +``` + +**Task 1.2 — Update `index.html`** +- Remove SpessaSynth importmap + module script (lines 13-25) +- Add FluidSynth WASM CDN script: +```html + +``` + +### Phase 2: AudioWorklet Processor + +**Task 2.1 — Create `app/static/js/worklets/fluidsynth-worklet.js`** + +This file runs inside AudioWorkletGlobalScope. It: +- Receives the FluidSynth WASM module (loaded via `addModule()`) +- Maintains its own `_fluid_synth` instance +- Receives commands from main thread via `port.onmessage`: + - `INIT_SYNTH` — create settings + synth, store `synthPtr` + - `LOAD_SF` — receive ArrayBuffer, write to MEMFS, call `_fluid_synth_sfload` + - `NOTE_ON` — `_fluid_synth_noteon(synthPtr, channel, pitch, velocity)` + - `NOTE_OFF` — `_fluid_synth_noteoff(synthPtr, channel, pitch)` + - `CC` — `_fluid_synth_cc(synthPtr, channel, controller, value)` + - `PROGRAM_CHANGE` — `_fluid_synth_program_change / bank_select` + - `PITCH_BEND` — `_fluid_synth_pitch_bend(synthPtr, channel, value)` + - `ALL_NOTES_OFF` / `ALL_SOUNDS_OFF` + - `SET_GAIN` — `_fluid_synth_set_gain` +- In `process(inputs, outputs)`: + - Get `synthPtr` from closure + - Call `_fluid_synth_write_float(synthPtr, bufferSize, leftPtr, 0, 1, rightPtr, 0, 1)` + - Return `true` to keep processor alive + +Key constraint: The FluidSynth WASM's `_fluid_synth_write_float` needs access to the output channel Float32Array's **byteOffset** relative to the WASM heap. AudioWorklet `output` arrays are not backed by WASM memory. Two solutions: +- **A**: After rendering, copy from WASM heap Float32Array to output channels (less efficient but safe) +- **B**: Load FluidSynth WASM inside worklet, allocate output buffers inside WASM heap (complex) + +**Recommendation: Solution A** — simpler and avoids memory management issues. + +### Phase 3: Rewrite `soundfontPlayer.js` + +**Task 3.1 — Rewrite as IIFE-compatible `window.SonicSF`** + +Keep the IIFE pattern (`window.SonicSF = SonicSF`) to minimize diff in `app.jsx`. Internal implementation uses `FluidSynthWasmWorkletBridge` class. + +```javascript +// Internal state +const _channels = Array.from({ length: 16 }, () => ({ bank: 0, program: 0, isPercussion: false })); +const _sustainStates = new Array(16).fill(false); +let _workletNode = null; +let _audioCtx = null; +let _initialized = false; +let _initPromise = null; +let _currentSfId = null; +let _sfHandleMap = new Map(); // sfId -> sfHandle (integer, tracked on JS side) +let _fluidModule = null; // guarded global, used before worklet takes over +``` + +**Key architectural decision:** + +The FluidSynth instance lives in the AudioWorklet. The main thread `soundfontPlayer.js`: +1. Loads the WASM module only for FS operations (MEMFS file writing) - needed because worklet can't do `fetch()` +2. Creates an `AudioWorkletNode`, registers the worklet +3. Sends ArrayBuffer data + MIDI commands to worklet via `port.postMessage` +4. Tracks JS-side state (channel, bank, program) for methods like `getChannelState`, `allocateChannel` + +**Alternative (simpler, recommended for v1):** +Run FluidSynth entirely on **main thread**, render PCM buffers in a rAF/setInterval loop, feed to a `ScriptProcessorNode` or a simple AudioWorklet that just outputs pre-rendered buffers. This avoids the AudioWorklet WASM complexity. + +**Decision:** Use main-thread FluidSynth + simple AudioWorklet output. Rationale: +- Much simpler implementation +- FluidSynth `_fluid_synth_write_float` is fast enough for real-time rendering +- No need to manage two WASM instances +- Can reuse the spec's API surface directly +- MIDI events can be processed synchronously on main thread + +**Architecture:** +``` +Main thread: + FluidSynth Module (loaded via CDN