3dtours/SonicForgeStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b947237c3c6842a5f8013f754afa4aba47bf5648
SonicForgeStudio/.kilo/plans/1785130485229-fluidsynth-wasm-migration-plan.md
T

18 KiB
Raw Blame History

Technical Analysis & Implementation Plan: SpessaSynth → FluidSynth WASM Migration

1. Key Findings from Codebase Investigation

1.1 Current State (SpessaSynth)

Aspect Detail
Engine spessasynth_lib@4.3.1 (CDN via jsDelivr + importmap)
File app/static/js/services/soundfontPlayer.js — IIFE, window.SonicSF singleton
API surface 10 methods: init, playNote, stopNote, stopAll, selectInstrument, controllerChange, programChange, pitchBend, sustainActive, applyAITrackInstrument, allocateChannel, getChannelState, loadSoundFont (internal)
Call sites 45 in app.jsx across piano roll, MIDI keyboard, timeline playback, external MIDI input, AI track generation
storage app/static/js/services/soundfontStorage.js — IndexedDB cache, window.SonicSFStorage
Server render app/core/render_engine.py uses pyfluidsynth (C++ FluidSynth)
Server SF3 app/core/soundfont_converter.py converts SF2→SF3 via fluidsynth/mscore CLI
API endpoint GET /api/v1/plugins/soundfonts/download/{sf_id} — serves both .sf2 and .sf3
Docker libfluidsynth3, fluidsynth CLI installed in container

1.2 Target State (FluidSynth WASM) — Corrected Spec

CDN package: fluidsynth-wasm does NOT exist on npm/CDN.
Real package: @enikey87/fluidsynth-emscripten@0.1.1

File Size Purpose
libfluidsynth-2.3.0-sf3.js 130 KB JS loader, SF3 support, separate WASM
libfluidsynth-2.3.0-sf3.wasm 1.7 MB WASM binary with Ogg/SF3 support
libfluidsynth-2.3.0-sf3-all-in-one.js 2.39 MB JS+WASM embedded (no separate .wasm fetch)

CDN URL (chosen: separate .wasm for caching):

https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js
https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.wasm

1.3 CRITICAL Architecture Flaw in Spec

The spec's code (md/39_SF3_NEW.md) passes wasmModule and synthPtr via postMessage to AudioWorklet. This cannot work because:

  1. AudioWorklet runs in a separate audio thread with an isolated global scope
  2. synthPtr (a C pointer / integer) refers to WASM linear memory in the main thread's WebAssembly.Module instance
  3. Loading libfluidsynth-2.3.0-sf3.js inside the worklet creates a separate WASM instance with its own memory
  4. The spec also uses _fluid_synth_write_float inside process() — this blocks the audio thread if audio buffer underruns occur

Correct architecture (per README):

"To use libfluidsynth-X.X.X.js in AudioWorklet, load it into AudioWorklet before your worklet JS file."

The FluidSynth instance must be created inside the AudioWorklet. MIDI events + ArrayBuffer data are sent from main thread to worklet via port.postMessage.

1.4 playNote Signature Complexity

Current playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine) has 8 parameters and 16 call sites. The synthEngine object carries { soundfont_id, soundfont_bank, soundfont_program } which must be converted to FluidSynth's bank_select + program_change before noteon.

1.5 Additional window.SonicSF Methods Not in Spec

Must preserve in new implementation:

  • stopNote(channel, pitch) — MIDI note-off with extra CC 64/120 cleanup
  • controllerChange(channel, controller, value) — MIDI CC forwarding
  • programChange(channel, program) — pure JS state tracking
  • pitchBend(channel, value) — 14-bit bend value
  • sustainActive(channel) — getter for sustain pedal state
  • allocateChannel(bank) — channel allocation (percussion = ch 9)
  • getChannelState(channel) — JS-side { bank, program, isPercussion } state
  • saveToIndexedDB / loadFromIndexedDB — legacy storage

2. Implementation Plan

Phase 1: Preparation (Infrastructure)

Task 1.1 — Create worklet directory

mkdir -p app/static/js/worklets/
touch app/static/js/worklets/.gitkeep

Task 1.2 — Update index.html

  • Remove SpessaSynth importmap + module script (lines 13-25)
  • Add FluidSynth WASM CDN script:
<script src="https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js"></script>

Phase 2: AudioWorklet Processor

Task 2.1 — Create app/static/js/worklets/fluidsynth-worklet.js

This file runs inside AudioWorkletGlobalScope. It:

  • Receives the FluidSynth WASM module (loaded via addModule())
  • Maintains its own _fluid_synth instance
  • Receives commands from main thread via port.onmessage:
    • INIT_SYNTH — create settings + synth, store synthPtr
    • LOAD_SF — receive ArrayBuffer, write to MEMFS, call _fluid_synth_sfload
    • NOTE_ON — _fluid_synth_noteon(synthPtr, channel, pitch, velocity)
    • NOTE_OFF — _fluid_synth_noteoff(synthPtr, channel, pitch)
    • CC — _fluid_synth_cc(synthPtr, channel, controller, value)
    • PROGRAM_CHANGE — _fluid_synth_program_change / bank_select
    • PITCH_BEND — _fluid_synth_pitch_bend(synthPtr, channel, value)
    • ALL_NOTES_OFF / ALL_SOUNDS_OFF
    • SET_GAIN — _fluid_synth_set_gain
  • In process(inputs, outputs):
    • Get synthPtr from closure
    • Call _fluid_synth_write_float(synthPtr, bufferSize, leftPtr, 0, 1, rightPtr, 0, 1)
    • Return true to keep processor alive

Key constraint: The FluidSynth WASM's _fluid_synth_write_float needs access to the output channel Float32Array's byteOffset relative to the WASM heap. AudioWorklet output arrays are not backed by WASM memory. Two solutions:

  • A: After rendering, copy from WASM heap Float32Array to output channels (less efficient but safe)
  • B: Load FluidSynth WASM inside worklet, allocate output buffers inside WASM heap (complex)

Recommendation: Solution A — simpler and avoids memory management issues.

Phase 3: Rewrite soundfontPlayer.js

Task 3.1 — Rewrite as IIFE-compatible window.SonicSF

Keep the IIFE pattern (window.SonicSF = SonicSF) to minimize diff in app.jsx. Internal implementation uses FluidSynthWasmWorkletBridge class.

// Internal state
const _channels = Array.from({ length: 16 }, () => ({ bank: 0, program: 0, isPercussion: false }));
const _sustainStates = new Array(16).fill(false);
let _workletNode = null;
let _audioCtx = null;
let _initialized = false;
let _initPromise = null;
let _currentSfId = null;
let _sfHandleMap = new Map(); // sfId -> sfHandle (integer, tracked on JS side)
let _fluidModule = null; // guarded global, used before worklet takes over

Key architectural decision:

The FluidSynth instance lives in the AudioWorklet. The main thread soundfontPlayer.js:

  1. Loads the WASM module only for FS operations (MEMFS file writing) - needed because worklet can't do fetch()
  2. Creates an AudioWorkletNode, registers the worklet
  3. Sends ArrayBuffer data + MIDI commands to worklet via port.postMessage
  4. Tracks JS-side state (channel, bank, program) for methods like getChannelState, allocateChannel

Alternative (simpler, recommended for v1):
Run FluidSynth entirely on main thread, render PCM buffers in a rAF/setInterval loop, feed to a ScriptProcessorNode or a simple AudioWorklet that just outputs pre-rendered buffers. This avoids the AudioWorklet WASM complexity.

Decision: Use main-thread FluidSynth + simple AudioWorklet output. Rationale:

  • Much simpler implementation
  • FluidSynth _fluid_synth_write_float is fast enough for real-time rendering
  • No need to manage two WASM instances
  • Can reuse the spec's API surface directly
  • MIDI events can be processed synchronously on main thread

Architecture:

Main thread:
  FluidSynth Module (loaded via CDN <script> tag)
    └─ _new_fluid_settings, _new_fluid_synth
    └─ MEMFS: FS.writeFile('/soundfonts/...', data)
    └─ _fluid_synth_sfload, _fluid_synth_noteon, etc.
    └─ _fluid_synth_write_float(bufSize, ...) -> Float32Array

AudioWorklet (simple passthrough):
  └─ Receives Float32Array PCM buffers via postMessage
  └─ Outputs to audio destination in process()

3.2 Method Implementation Details

init(audioCtx):

async init(audioContext) {
  this.audioCtx = audioContext;
  // resolve createFluidSynthModule() from CDN
  this.fluidModule = await new Promise(resolve => {
    // libfluidsynth-2.3.0-sf3.js exposes createFluidSynthModule()
    resolve(window.createFluidSynthModule());
  });
  // create settings, synth
  const settingsPtr = this.fluidModule._new_fluid_settings();
  this.fluidModule._fluid_settings_setnum(settingsPtr, "synth.sample-rate", this.audioCtx.sampleRate);
  this.fluidModule._fluid_settings_setnum(settingsPtr, "synth.gain", 1.0);
  this.synthPtr = this.fluidModule._new_fluid_synth(settingsPtr);
  // create MEMFS dir
  this.fluidModule.FS.mkdir('/soundfonts');
  // create AudioWorklet node
  this.workletNode = new AudioWorkletNode(this.audioCtx, 'fluidsynth-bridge');
  this.workletNode.connect(this.audioCtx.destination);
  // start render loop
  this._startRenderLoop();
}

_startRenderLoop(): Uses requestAnimationFrame or scheduler to periodically:

  1. Check if worklet Node needs more data
  2. Call _fluid_synth_write_float into a Float32Array
  3. Send via postMessage to worklet

loadSoundFont(sfId):

async loadSoundFont(sfId) {
  let buffer = await sfStorage.getBuffer(sfId);
  if (!buffer) {
    const resp = await fetch(`/api/v1/plugins/soundfonts/download/${sfId}`);
    buffer = await resp.arrayBuffer();
    await sfStorage.saveBuffer(sfId, buffer);
  }
  this.fluidModule.FS.writeFile(`/soundfonts/${sfId}.sf3`, new Uint8Array(buffer));
  const sfHandle = this.fluidModule._fluid_synth_sfload(this.synthPtr, `/soundfonts/${sfId}.sf3`, 1);
  // store handle
  return sfHandle !== -1;
}

playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine):

function playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine) {
  // Handle synthEngine (bank/program setup)
  let ch = channel ?? 0;
  let bank = 0, prog = 0;
  if (synthEngine) {
    bank = synthEngine.soundfont_bank ?? 0;
    prog = synthEngine.soundfont_program ?? 0;
    this.fluidModule._fluid_synth_bank_select(this.synthPtr, ch, bank);
    this.fluidModule._fluid_synth_program_change(this.synthPtr, ch, prog);
  } else if (program !== undefined) {
    prog = program;
    this.fluidModule._fluid_synth_program_change(this.synthPtr, ch, prog);
  }
  // Note on
  this.fluidModule._fluid_synth_noteon(this.synthPtr, ch, midiPitch, midiVel);
  // Schedule note off
  if (durationMs > 0 && durationMs < 60000) { // skip held notes
    setTimeout(() => {
      this.fluidModule._fluid_synth_noteoff(this.synthPtr, ch, midiPitch);
    }, durationMs);
  }
}

stopNote(channel, pitch):
→ this.fluidModule._fluid_synth_noteoff(synthPtr, channel, pitch)
Also set sustain off + all notes off (preserve current behavior).

stopAll():
→ _fluid_synth_all_notes_off + _fluid_synth_all_sounds_off for all 16 channels.

controllerChange(channel, controller, value):
→ this.fluidModule._fluid_synth_cc(synthPtr, channel, controller, value)

  • track JS-side state for _channels[channel].bank, _sustainStates[channel].

pitchBend(channel, value):
→ this.fluidModule._fluid_synth_pitch_bend(synthPtr, channel, value).

selectInstrument(channel, bank, program, sfId):
→ Load SF if needed, then _fluid_synth_bank_select + _fluid_synth_program_change.

sustainActive(channel):
→ Return _sustainStates[channel] (tracked by controllerChange for CC 64).

applyAITrackInstrument(bank, program, synthEngine):
→ Same as current: allocate channel, call selectInstrument with synthEngine.soundfont_id.

Phase 4: Bridge AudioWorklet

Task 4.1 — Create app/static/js/worklets/fluidsynth-bridge.js

Simple AudioWorkletProcessor that receives pre-rendered PCM buffers:

class FluidSynthBridge extends AudioWorkletProcessor {
  constructor() {
    super();
    this.audioQueue = [[], []]; // L/R channel queues
    this.port.onmessage = (e) => {
      if (e.data.type === 'PCM_FRAME') {
        this.audioQueue[0].push(...e.data.left);
        this.audioQueue[1].push(...e.data.right);
      }
    };
  }
  process(inputs, outputs) {
    const out = outputs[0];
    if (!out) return true;
    const len = out[0].length;
    const left = this.audioQueue[0].splice(0, len);
    const right = this.audioQueue[1].splice(0, len);
    // Fill output; zero-fill if buffer underrun
    for (let i = 0; i < len; i++) {
      out[0][i] = i < left.length ? left[i] : 0;
      out[1][i] = i < right.length ? right[i] : 0;
    }
    return true;
  }
}
registerProcessor('fluidsynth-bridge', FluidSynthBridge);

Phase 5: Optimize Render Loop

Task 5.1 — Implement ring-buffer approach

The render loop must balance latency vs. buffer underruns:

  • Render ~512 samples per frame (≈11.6ms at 44.1kHz)
  • Queue 3 frames ahead (≈35ms buffer → safe against GC pauses)
  • Use AudioWorkletNode.port.postMessage with transferable: true for zero-copy

Alternative: Use SharedArrayBuffer for lock-free ring buffer (requires COOP/COEP headers).

Phase 6: Remove SpessaSynth from index.html

Task 6.1 — Clean up CDN imports

  • Remove: <script type="importmap"> block (lines 13-19)
  • Remove: SpessaSynth module script (lines 20-25)
  • Remove: spessasynth_core importmap entry

Phase 7: API Endpoint for WASM asset (optional)

If self-hosting is preferred over CDN, serve libfluidsynth-2.3.0-sf3.js and .wasm from static files directory.

3. Files to Modify/Create

Action File Description
CREATE app/static/js/worklets/fluidsynth-bridge.js Bridge AudioWorkletProcessor (PCM passthrough)
MODIFY app/static/js/services/soundfontPlayer.js Rewrite engine from SpessaSynth to FluidSynth WASM; preserve window.SonicSF API surface
MODIFY app/templates/index.html Replace SpessaSynth CDN imports with @enikey87/fluidsynth-emscripten CDN
MODIFY app/static/js/services/soundfontStorage.js Minor: expose sfStorage as named export + window.SonicSFStorage (likely no change needed)
NO CHANGE app/static/js/app.jsx Zero changes — window.SonicSF API surface preserved
NO CHANGE app/api/v1/plugins.py Download endpoint already serves .sf2/.sf3 correctly
NO CHANGE app/core/render_engine.py Already uses pyfluidsynth (server-side)
NO CHANGE app/core/soundfont_converter.py Already converts SF2→SF3
NO CHANGE Dockerfile Already has libfluidsynth3, fluidsynth CLI

4. API Signature Mapping

Old (SpessaSynth) New (FluidSynth WASM) Notes
_synthInstance.noteOn(ch, pitch, vel) _fluid_synth_noteon(ptr, ch, pitch, vel) Direct C function
_synthInstance.noteOff(ch, pitch) _fluid_synth_noteoff(ptr, ch, pitch) Direct C function
_synthInstance.controllerChange(ch, cc, val) _fluid_synth_cc(ptr, ch, cc, val) Direct C function
_synthInstance.programChange(ch, prog) _fluid_synth_program_change(ptr, ch, prog) Direct C function
_synthInstance.pitchBend(ch, val) _fluid_synth_pitch_bend(ptr, ch, val) Direct C function
soundBankManager.addSoundBank(buf, id) FS.writeFile(path, uint8arr) + _fluid_synth_sfload(ptr, path, 1) MEMFS-based
soundBankManager.soundBankList JS-side _sfHandleMap (Map) Track manually
controllerChange(ch, 120, 0) (all sound off) _fluid_synth_all_sounds_off(ptr, ch) Direct C function
controllerChange(ch, 123, 0) (all notes off) _fluid_synth_all_notes_off(ptr, ch) Direct C function

5. Risk Assessment

Risk Impact Mitigation
FluidSynth WASM CPU usage causes audio glitches Medium Use triple-buffered rendering at 512-sample blocks; monitor with performance.now()
_fluid_synth_write_float blocks main thread Medium Move rendering to a separate rAF loop, not synchronized to audio callback
SF3 loading latency on first load Low IndexedDB caching already in place; loading ≈2-5MB via MEMFS is sub-100ms
CDN availability for @enikey87/fluidsynth-emscripten Low Package is 17.9MB unpacked; fallback host via jsDelivr + self-host option
Backward compatibility with existing MIDI sessions Medium All window.SonicSF methods preserved with same signatures; synthEngine object handled correctly

6. Validation

# 1. Build frontend
cd /home/locpham/SonicForgeStudio
cd app/static/js && npx babel app.jsx --config-file ../../babel.config.json -o app.precompiled.js

# 2. Start server
cd /home/locpham/SonicForgeStudio
docker compose up -d

# 3. Browser tests
#   - Open DevTools → check `window.createFluidSynthModule` exists
#   - Load app → check `SonicSF.init(audioCtx)` creates FluidSynth synth
#   - Select SoundFont instrument → check MEMFS write + sfload succeeds
#   - Play piano roll notes → hear authentic FluidSynth output
#   - Play timeline → notes play with correct timing
#   - External MIDI controller → CC, pitch bend, note on/off work
#   - Transport stop → all notes stop immediately
#   - Sustain pedal → notes sustain on CC 64 >= 64
#   - Stress: play 50+ simultaneous notes → no stuck notes

Server-side parity validation:

  • Render a project with server (/api/v1/plugins/render)
  • Play same project in client browser
  • Compare WAV spectrograms: should be identical (same C++ FluidSynth core)

7. Open Questions for User

  1. Self-host WASM vs CDN? — @enikey87/fluidsynth-emscripten from jsDelivr (5.9MB) or serve from app/static/ folder. CDN reduces server load but requires internet. If user wants offline-capable, suggest self-host.

  2. Separate WASM file vs all-in-one? — libfluidsynth-2.3.0-sf3.js (130KB) + libfluidsynth-2.3.0-sf3.wasm (1.7MB) allow browser to cache WASM separately. Or libfluidsynth-2.3.0-sf3-all-in-one.js (2.39MB) single file but no caching benefit. Recommend separate files.

  3. Main-thread render vs worklet-own-synth? — The spec's worklet-own-synth approach is theoretically ideal but complex (MIDI events + file data must cross thread boundary). Recommend v1 as main-thread render + simple bridge worklet. Can optimize later.

Reference in New Issue View Git Blame Copy Permalink