SonicForgeStudio/.kilo/plans/1785130485229-fluidsynth-wasm-migration-plan.md

# Technical Analysis & Implementation Plan: SpessaSynth → FluidSynth WASM Migration

## 1. Key Findings from Codebase Investigation

### 1.1 Current State (SpessaSynth)
| Aspect | Detail |
|---|---|
| Engine | `spessasynth_lib@4.3.1` (CDN via jsDelivr + importmap) |
| File | `app/static/js/services/soundfontPlayer.js` — IIFE, `window.SonicSF` singleton |
| API surface | 10 methods: `init`, `playNote`, `stopNote`, `stopAll`, `selectInstrument`, `controllerChange`, `programChange`, `pitchBend`, `sustainActive`, `applyAITrackInstrument`, `allocateChannel`, `getChannelState`, `loadSoundFont` (internal) |
| Call sites | 45 in `app.jsx` across piano roll, MIDI keyboard, timeline playback, external MIDI input, AI track generation |
| storage | `app/static/js/services/soundfontStorage.js` — IndexedDB cache, `window.SonicSFStorage` |
| Server render | `app/core/render_engine.py` uses `pyfluidsynth` (C++ FluidSynth) |
| Server SF3 | `app/core/soundfont_converter.py` converts SF2→SF3 via `fluidsynth`/`mscore` CLI |
| API endpoint | `GET /api/v1/plugins/soundfonts/download/{sf_id}` — serves both `.sf2` and `.sf3` |
| Docker | `libfluidsynth3`, `fluidsynth` CLI installed in container |

### 1.2 Target State (FluidSynth WASM) — Corrected Spec

**CDN package**: `fluidsynth-wasm` does NOT exist on npm/CDN.  
**Real package**: `@enikey87/fluidsynth-emscripten@0.1.1`  

| File | Size | Purpose |
|---|---|---|
| `libfluidsynth-2.3.0-sf3.js` | 130 KB | JS loader, SF3 support, separate WASM |
| `libfluidsynth-2.3.0-sf3.wasm` | 1.7 MB | WASM binary with Ogg/SF3 support |
| `libfluidsynth-2.3.0-sf3-all-in-one.js` | 2.39 MB | JS+WASM embedded (no separate .wasm fetch) |

**CDN URL** (chosen: separate .wasm for caching):  
```
https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js
https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.wasm
```

### 1.3 CRITICAL Architecture Flaw in Spec

The spec's code (md/39_SF3_NEW.md) passes `wasmModule` and `synthPtr` via `postMessage` to AudioWorklet. **This cannot work** because:
1. AudioWorklet runs in a separate audio thread with an isolated global scope
2. `synthPtr` (a C pointer / integer) refers to WASM linear memory in the main thread's `WebAssembly.Module` instance
3. Loading `libfluidsynth-2.3.0-sf3.js` inside the worklet creates a **separate WASM instance** with its own memory
4. The spec also uses `_fluid_synth_write_float` inside `process()` — this blocks the audio thread if audio buffer underruns occur

**Correct architecture (per README):**  
> "To use libfluidsynth-X.X.X.js in AudioWorklet, load it into AudioWorklet before your worklet JS file."

The FluidSynth instance must be created **inside** the AudioWorklet. MIDI events + ArrayBuffer data are sent from main thread to worklet via `port.postMessage`.

### 1.4 `playNote` Signature Complexity

Current `playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine)` has 8 parameters and 16 call sites. The `synthEngine` object carries `{ soundfont_id, soundfont_bank, soundfont_program }` which must be converted to FluidSynth's `bank_select` + `program_change` before `noteon`.

### 1.5 Additional `window.SonicSF` Methods Not in Spec

Must preserve in new implementation:
- `stopNote(channel, pitch)` — MIDI note-off with extra CC 64/120 cleanup
- `controllerChange(channel, controller, value)` — MIDI CC forwarding
- `programChange(channel, program)` — pure JS state tracking
- `pitchBend(channel, value)` — 14-bit bend value
- `sustainActive(channel)` — getter for sustain pedal state
- `allocateChannel(bank)` — channel allocation (percussion = ch 9)
- `getChannelState(channel)` — JS-side { bank, program, isPercussion } state
- `saveToIndexedDB / loadFromIndexedDB` — legacy storage

---

## 2. Implementation Plan

### Phase 1: Preparation (Infrastructure)

**Task 1.1 — Create worklet directory**
```
mkdir -p app/static/js/worklets/
touch app/static/js/worklets/.gitkeep
```

**Task 1.2 — Update `index.html`**
- Remove SpessaSynth importmap + module script (lines 13-25)
- Add FluidSynth WASM CDN script:
```html
<script src="https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js"></script>
```

### Phase 2: AudioWorklet Processor

**Task 2.1 — Create `app/static/js/worklets/fluidsynth-worklet.js`**

This file runs inside AudioWorkletGlobalScope. It:
- Receives the FluidSynth WASM module (loaded via `addModule()`)
- Maintains its own `_fluid_synth` instance
- Receives commands from main thread via `port.onmessage`:
  - `INIT_SYNTH` — create settings + synth, store `synthPtr`
  - `LOAD_SF` — receive ArrayBuffer, write to MEMFS, call `_fluid_synth_sfload`
  - `NOTE_ON` — `_fluid_synth_noteon(synthPtr, channel, pitch, velocity)`
  - `NOTE_OFF` — `_fluid_synth_noteoff(synthPtr, channel, pitch)`
  - `CC` — `_fluid_synth_cc(synthPtr, channel, controller, value)`
  - `PROGRAM_CHANGE` — `_fluid_synth_program_change / bank_select`
  - `PITCH_BEND` — `_fluid_synth_pitch_bend(synthPtr, channel, value)`
  - `ALL_NOTES_OFF` / `ALL_SOUNDS_OFF`
  - `SET_GAIN` — `_fluid_synth_set_gain`
- In `process(inputs, outputs)`:
  - Get `synthPtr` from closure
  - Call `_fluid_synth_write_float(synthPtr, bufferSize, leftPtr, 0, 1, rightPtr, 0, 1)`
  - Return `true` to keep processor alive

Key constraint: The FluidSynth WASM's `_fluid_synth_write_float` needs access to the output channel Float32Array's **byteOffset** relative to the WASM heap. AudioWorklet `output` arrays are not backed by WASM memory. Two solutions:
- **A**: After rendering, copy from WASM heap Float32Array to output channels (less efficient but safe)
- **B**: Load FluidSynth WASM inside worklet, allocate output buffers inside WASM heap (complex)

**Recommendation: Solution A** — simpler and avoids memory management issues.

### Phase 3: Rewrite `soundfontPlayer.js`

**Task 3.1 — Rewrite as IIFE-compatible `window.SonicSF`**

Keep the IIFE pattern (`window.SonicSF = SonicSF`) to minimize diff in `app.jsx`. Internal implementation uses `FluidSynthWasmWorkletBridge` class.

```javascript
// Internal state
const _channels = Array.from({ length: 16 }, () => ({ bank: 0, program: 0, isPercussion: false }));
const _sustainStates = new Array(16).fill(false);
let _workletNode = null;
let _audioCtx = null;
let _initialized = false;
let _initPromise = null;
let _currentSfId = null;
let _sfHandleMap = new Map(); // sfId -> sfHandle (integer, tracked on JS side)
let _fluidModule = null; // guarded global, used before worklet takes over
```

**Key architectural decision:**

The FluidSynth instance lives in the AudioWorklet. The main thread `soundfontPlayer.js`:
1. Loads the WASM module only for FS operations (MEMFS file writing) - needed because worklet can't do `fetch()`
2. Creates an `AudioWorkletNode`, registers the worklet
3. Sends ArrayBuffer data + MIDI commands to worklet via `port.postMessage`
4. Tracks JS-side state (channel, bank, program) for methods like `getChannelState`, `allocateChannel`

**Alternative (simpler, recommended for v1):**  
Run FluidSynth entirely on **main thread**, render PCM buffers in a rAF/setInterval loop, feed to a `ScriptProcessorNode` or a simple AudioWorklet that just outputs pre-rendered buffers. This avoids the AudioWorklet WASM complexity.

**Decision:** Use main-thread FluidSynth + simple AudioWorklet output. Rationale:
- Much simpler implementation
- FluidSynth `_fluid_synth_write_float` is fast enough for real-time rendering
- No need to manage two WASM instances
- Can reuse the spec's API surface directly
- MIDI events can be processed synchronously on main thread

**Architecture:**
```
Main thread:
  FluidSynth Module (loaded via CDN <script> tag)
    └─ _new_fluid_settings, _new_fluid_synth
    └─ MEMFS: FS.writeFile('/soundfonts/...', data)
    └─ _fluid_synth_sfload, _fluid_synth_noteon, etc.
    └─ _fluid_synth_write_float(bufSize, ...) -> Float32Array

AudioWorklet (simple passthrough):
  └─ Receives Float32Array PCM buffers via postMessage
  └─ Outputs to audio destination in process()
```

### 3.2 Method Implementation Details

**`init(audioCtx)`:**
```javascript
async init(audioContext) {
  this.audioCtx = audioContext;
  // resolve createFluidSynthModule() from CDN
  this.fluidModule = await new Promise(resolve => {
    // libfluidsynth-2.3.0-sf3.js exposes createFluidSynthModule()
    resolve(window.createFluidSynthModule());
  });
  // create settings, synth
  const settingsPtr = this.fluidModule._new_fluid_settings();
  this.fluidModule._fluid_settings_setnum(settingsPtr, "synth.sample-rate", this.audioCtx.sampleRate);
  this.fluidModule._fluid_settings_setnum(settingsPtr, "synth.gain", 1.0);
  this.synthPtr = this.fluidModule._new_fluid_synth(settingsPtr);
  // create MEMFS dir
  this.fluidModule.FS.mkdir('/soundfonts');
  // create AudioWorklet node
  this.workletNode = new AudioWorkletNode(this.audioCtx, 'fluidsynth-bridge');
  this.workletNode.connect(this.audioCtx.destination);
  // start render loop
  this._startRenderLoop();
}
```

**`_startRenderLoop()`:**
Uses `requestAnimationFrame` or scheduler to periodically:
1. Check if worklet Node needs more data
2. Call `_fluid_synth_write_float` into a Float32Array
3. Send via `postMessage` to worklet

**`loadSoundFont(sfId)`:**
```javascript
async loadSoundFont(sfId) {
  let buffer = await sfStorage.getBuffer(sfId);
  if (!buffer) {
    const resp = await fetch(`/api/v1/plugins/soundfonts/download/${sfId}`);
    buffer = await resp.arrayBuffer();
    await sfStorage.saveBuffer(sfId, buffer);
  }
  this.fluidModule.FS.writeFile(`/soundfonts/${sfId}.sf3`, new Uint8Array(buffer));
  const sfHandle = this.fluidModule._fluid_synth_sfload(this.synthPtr, `/soundfonts/${sfId}.sf3`, 1);
  // store handle
  return sfHandle !== -1;
}
```

**`playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine)`:**
```javascript
function playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine) {
  // Handle synthEngine (bank/program setup)
  let ch = channel ?? 0;
  let bank = 0, prog = 0;
  if (synthEngine) {
    bank = synthEngine.soundfont_bank ?? 0;
    prog = synthEngine.soundfont_program ?? 0;
    this.fluidModule._fluid_synth_bank_select(this.synthPtr, ch, bank);
    this.fluidModule._fluid_synth_program_change(this.synthPtr, ch, prog);
  } else if (program !== undefined) {
    prog = program;
    this.fluidModule._fluid_synth_program_change(this.synthPtr, ch, prog);
  }
  // Note on
  this.fluidModule._fluid_synth_noteon(this.synthPtr, ch, midiPitch, midiVel);
  // Schedule note off
  if (durationMs > 0 && durationMs < 60000) { // skip held notes
    setTimeout(() => {
      this.fluidModule._fluid_synth_noteoff(this.synthPtr, ch, midiPitch);
    }, durationMs);
  }
}
```

**`stopNote(channel, pitch)`:**  
→ `this.fluidModule._fluid_synth_noteoff(synthPtr, channel, pitch)`  
Also set sustain off + all notes off (preserve current behavior).

**`stopAll()`:**  
→ `_fluid_synth_all_notes_off` + `_fluid_synth_all_sounds_off` for all 16 channels.

**`controllerChange(channel, controller, value)`:**  
→ `this.fluidModule._fluid_synth_cc(synthPtr, channel, controller, value)`  
+ track JS-side state for `_channels[channel].bank`, `_sustainStates[channel]`.

**`pitchBend(channel, value)`:**  
→ `this.fluidModule._fluid_synth_pitch_bend(synthPtr, channel, value)`.

**`selectInstrument(channel, bank, program, sfId)`:**  
→ Load SF if needed, then `_fluid_synth_bank_select` + `_fluid_synth_program_change`.

**`sustainActive(channel)`:**  
→ Return `_sustainStates[channel]` (tracked by `controllerChange` for CC 64).

**`applyAITrackInstrument(bank, program, synthEngine)`:**  
→ Same as current: allocate channel, call `selectInstrument` with `synthEngine.soundfont_id`.

### Phase 4: Bridge AudioWorklet

**Task 4.1 — Create `app/static/js/worklets/fluidsynth-bridge.js`**

Simple AudioWorkletProcessor that receives pre-rendered PCM buffers:
```javascript
class FluidSynthBridge extends AudioWorkletProcessor {
  constructor() {
    super();
    this.audioQueue = [[], []]; // L/R channel queues
    this.port.onmessage = (e) => {
      if (e.data.type === 'PCM_FRAME') {
        this.audioQueue[0].push(...e.data.left);
        this.audioQueue[1].push(...e.data.right);
      }
    };
  }
  process(inputs, outputs) {
    const out = outputs[0];
    if (!out) return true;
    const len = out[0].length;
    const left = this.audioQueue[0].splice(0, len);
    const right = this.audioQueue[1].splice(0, len);
    // Fill output; zero-fill if buffer underrun
    for (let i = 0; i < len; i++) {
      out[0][i] = i < left.length ? left[i] : 0;
      out[1][i] = i < right.length ? right[i] : 0;
    }
    return true;
  }
}
registerProcessor('fluidsynth-bridge', FluidSynthBridge);
```

### Phase 5: Optimize Render Loop

**Task 5.1 — Implement ring-buffer approach**

The render loop must balance latency vs. buffer underruns:
- Render ~512 samples per frame (≈11.6ms at 44.1kHz)
- Queue 3 frames ahead (≈35ms buffer → safe against GC pauses)
- Use `AudioWorkletNode.port.postMessage` with `transferable: true` for zero-copy

Alternative: Use SharedArrayBuffer for lock-free ring buffer (requires COOP/COEP headers).

### Phase 6: Remove SpessaSynth from index.html

**Task 6.1 — Clean up CDN imports**
- Remove: `<script type="importmap">` block (lines 13-19)
- Remove: SpessaSynth module script (lines 20-25)
- Remove: `spessasynth_core` importmap entry

### Phase 7: API Endpoint for WASM asset (optional)

If self-hosting is preferred over CDN, serve `libfluidsynth-2.3.0-sf3.js` and `.wasm` from static files directory.

---

## 3. Files to Modify/Create

| Action | File | Description |
|---|---|---|
| **CREATE** | `app/static/js/worklets/fluidsynth-bridge.js` | Bridge AudioWorkletProcessor (PCM passthrough) |
| **MODIFY** | `app/static/js/services/soundfontPlayer.js` | Rewrite engine from SpessaSynth to FluidSynth WASM; preserve `window.SonicSF` API surface |
| **MODIFY** | `app/templates/index.html` | Replace SpessaSynth CDN imports with `@enikey87/fluidsynth-emscripten` CDN |
| **MODIFY** | `app/static/js/services/soundfontStorage.js` | Minor: expose `sfStorage` as named export + `window.SonicSFStorage` (likely no change needed) |
| **NO CHANGE** | `app/static/js/app.jsx` | Zero changes — `window.SonicSF` API surface preserved |
| **NO CHANGE** | `app/api/v1/plugins.py` | Download endpoint already serves `.sf2`/`.sf3` correctly |
| **NO CHANGE** | `app/core/render_engine.py` | Already uses pyfluidsynth (server-side) |
| **NO CHANGE** | `app/core/soundfont_converter.py` | Already converts SF2→SF3 |
| **NO CHANGE** | Dockerfile | Already has `libfluidsynth3`, `fluidsynth` CLI |

---

## 4. API Signature Mapping

| Old (SpessaSynth) | New (FluidSynth WASM) | Notes |
|---|---|---|
| `_synthInstance.noteOn(ch, pitch, vel)` | `_fluid_synth_noteon(ptr, ch, pitch, vel)` | Direct C function |
| `_synthInstance.noteOff(ch, pitch)` | `_fluid_synth_noteoff(ptr, ch, pitch)` | Direct C function |
| `_synthInstance.controllerChange(ch, cc, val)` | `_fluid_synth_cc(ptr, ch, cc, val)` | Direct C function |
| `_synthInstance.programChange(ch, prog)` | `_fluid_synth_program_change(ptr, ch, prog)` | Direct C function |
| `_synthInstance.pitchBend(ch, val)` | `_fluid_synth_pitch_bend(ptr, ch, val)` | Direct C function |
| `soundBankManager.addSoundBank(buf, id)` | `FS.writeFile(path, uint8arr)` + `_fluid_synth_sfload(ptr, path, 1)` | MEMFS-based |
| `soundBankManager.soundBankList` | JS-side `_sfHandleMap` (Map) | Track manually |
| `controllerChange(ch, 120, 0)` (all sound off) | `_fluid_synth_all_sounds_off(ptr, ch)` | Direct C function |
| `controllerChange(ch, 123, 0)` (all notes off) | `_fluid_synth_all_notes_off(ptr, ch)` | Direct C function |

---

## 5. Risk Assessment

| Risk | Impact | Mitigation |
|---|---|---|
| FluidSynth WASM CPU usage causes audio glitches | Medium | Use triple-buffered rendering at 512-sample blocks; monitor with `performance.now()` |
| `_fluid_synth_write_float` blocks main thread | Medium | Move rendering to a separate rAF loop, not synchronized to audio callback |
| SF3 loading latency on first load | Low | IndexedDB caching already in place; loading ≈2-5MB via MEMFS is sub-100ms |
| CDN availability for `@enikey87/fluidsynth-emscripten` | Low | Package is 17.9MB unpacked; fallback host via jsDelivr + self-host option |
| Backward compatibility with existing MIDI sessions | Medium | All `window.SonicSF` methods preserved with same signatures; `synthEngine` object handled correctly |

---

## 6. Validation

```bash
# 1. Build frontend
cd /home/locpham/SonicForgeStudio
cd app/static/js && npx babel app.jsx --config-file ../../babel.config.json -o app.precompiled.js

# 2. Start server
cd /home/locpham/SonicForgeStudio
docker compose up -d

# 3. Browser tests
#   - Open DevTools → check `window.createFluidSynthModule` exists
#   - Load app → check `SonicSF.init(audioCtx)` creates FluidSynth synth
#   - Select SoundFont instrument → check MEMFS write + sfload succeeds
#   - Play piano roll notes → hear authentic FluidSynth output
#   - Play timeline → notes play with correct timing
#   - External MIDI controller → CC, pitch bend, note on/off work
#   - Transport stop → all notes stop immediately
#   - Sustain pedal → notes sustain on CC 64 >= 64
#   - Stress: play 50+ simultaneous notes → no stuck notes
```

**Server-side parity validation:**
- Render a project with server (`/api/v1/plugins/render`)
- Play same project in client browser
- Compare WAV spectrograms: should be identical (same C++ FluidSynth core)

---

## 7. Open Questions for User

1. **Self-host WASM vs CDN?** — `@enikey87/fluidsynth-emscripten` from jsDelivr (5.9MB) or serve from `app/static/` folder. CDN reduces server load but requires internet. If user wants offline-capable, suggest self-host.

2. **Separate WASM file vs all-in-one?** — `libfluidsynth-2.3.0-sf3.js` (130KB) + `libfluidsynth-2.3.0-sf3.wasm` (1.7MB) allow browser to cache WASM separately. Or `libfluidsynth-2.3.0-sf3-all-in-one.js` (2.39MB) single file but no caching benefit. Recommend separate files.

3. **Main-thread render vs worklet-own-synth?** — The spec's worklet-own-synth approach is theoretically ideal but complex (MIDI events + file data must cross thread boundary). Recommend v1 as main-thread render + simple bridge worklet. Can optimize later.