SonicForgeStudio/.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md

Technical Analysis & Implementation Plan: SF3 + SpessaSynth Client (md/37_SF_CONVERT.md)

1. Assessment Summary

The spec proposes 2 steps:

1. Server: Convert .sf2 → .sf3 (Ogg Vorbis compressed) via mscore CLI
2. Client: Replace oscillator emulation with SpessaSynth library for authentic SoundFont playback

Current State vs Spec

Requirement	Status	Impact
mscore in Dockerfile	❌ Not installed	200MB+ dependency
soundfont_converter.py	❌ Does not exist	Needs creation
GET /soundfonts/download/{sf_id} endpoint	❌ Missing	Blocks client download
spessasynth_lib CDN import in index.html	❌ Not present	Blocks client upgrade
soundfontStorage.js (IndexedDB)	❌ Not created	Needed for caching
SonicSF.init(audioCtx)	❌ Not called anywhere	New integration point
playNote signature compat	⚠️ Spec uses 4 params, codebase uses 6–8	Must bridge
applyAITrackInstrument signature	⚠️ Spec has channel first, code passes it last	Must bridge

Key Risk: mscore Dependency

mscore (MuseScore) pulls in Qt, fontconfig, audio drivers — easily 200-400MB in the container. Alternative approaches:

1. fluidsynth built-in conversion: fluidsynth already installed (libfluidsynth3, pyfluidsynth). Can convert SF2→SF3 via --convert flag or using fluid_synth_sffd_* APIs, but the low-level Python bindings don't expose this.

2. sf2convert/sf2pack: Smaller tools, but less commonly packaged.

3. Python + libsndfile/pydub/ogg: Parse SF2, extract WAV samples, compress to Ogg, rebuild SF3 structure. Complex — would need a SF2 parser and Ogg encoder.

4. Python acousticbrainz-sf2convert: Lightweight Python library specifically for SF2↔SF3.

Recommendation: Install mscore via apt-get install -y mscore --no-install-recommends to minimize deps. If the image grows too much (~1.5GB+), fall back to fluidsynth command-line conversion (fluidsynth --convert).

Key Risk: playNote API Compatibility

The spec's proposed API:

playNote(pitch, velocity=0.8, durationSec=1.0, channel=0)

Current callers use:

playNote(pitch, velocity, durationMs, startTime, program, destinationNode, channel?, synthEngine?)

Strategy: Don't replace. Instead, wrap SpessaSynth inside the existing SonicSF object. Map:

• velocity (0–1) → MIDI velocity (1–127)
• startTime → if in future, use setTimeout for note scheduling (not perfect but adequate for preview)
• destinationNode → SpessaSynth routes to its own internal destination, but can add a gain node stage
• program → internal _channels[channel].program state (as now), but SpessaSynth also gets programChange(ch, program)
• synthEngine → load the right SF3 and set bank/program on SpessaSynth

Backward compat: Keep ALL existing methods. Add SpessaSynth as an optional enhanced engine. If SpessaSynth isn't loaded (CDN fails), fall back to oscillator emulation.

---

2. Implementation Plan

Task 1 — Server: SF2→SF3 Conversion Pipeline

1.1 Docker Dependency

• Dockerfile: Add mscore with --no-install-recommends

  RUN apt-get install -y --no-install-recommends mscore && rm -rf /var/lib/apt/lists/*
  

• Test: docker compose build — measure image size delta

1.2 app/core/soundfont_converter.py

• SoundFontConverter class with:
  • convert_sf2_to_sf3(sf2_path) → runs mscore -o output.sf3 input.sf2
  • batch_convert_all() → walks target_dirs, converts missing .sf3
  • target_dirs = ["/opt/daw_engine/soundfonts", "app/storage/uploads/soundfonts"]
  • Progress logging, error handling, cache validation (skip if .sf3 newer than .sf2)
• Integrate with server startup:
  • In app/main.py startup event, call SoundFontConverter().batch_convert_all() as background task (non-blocking, don't delay boot)

1.3 API Download Endpoint

• app/api/v1/plugins.py: Add:

  @router.get("/soundfonts/download/{sf_id}")
  async def download_soundfont_asset(sf_id: str):
      # Search system_dir, upload_dir for .sf3 or .sf2
      # Return FileResponse
      # Fallback: .sf3 → .sf2 → 404
  

• Use _find_sf2_path from render_engine.py logic (case-insensitive, multi-dir)
• Set media_type="application/octet-stream" and proper Content-Disposition

1.4 Update SoundFontInspector to Also Scan .sf3

• soundfont_inspector.py:80: Change .sf2 filter to ('.sf2', '.sf3')
• Ensure the catalog includes .sf3 files as available instruments

---

Task 2 — Client: SpessaSynth Integration

2.1 index.html — Add SpessaSynth CDN

<script type="module">
  import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
  window.SpessaSynthClass = Synthesizer;
</script>

• Add BEFORE soundfontPlayer.js so the class is available when the player initializes

2.2 app/static/js/services/soundfontStorage.js — IndexedDB Cache

• SoundFontStorage class with openDB(), getBuffer(sfId), saveBuffer(sfId, arrayBuffer)
• Uses indexedDB with DB name "DAW_SoundFont_Cache", store name "sf3_buffers"
• Export singleton sfStorage

2.3 app/static/js/services/soundfontPlayer.js — Dual-Mode Rewrite

Architecture: Keep the existing window.SonicSF as the public API. Internally use SpessaSynth when available, fall back to oscillator when not.

Changes to existing methods:

Method	Change
init(audioCtx)	NEW — creates SpessaSynth instance, triggers default SF load
loadSoundFont(sfId)	REWORK — try IndexedDB → fetch /api/v1/plugins/soundfonts/download/{sfId} → load into SpessaSynth
playNote(...)	ADD SpessaSynth path: if initialized, delegate to synthInstance.noteOn/noteOff; else use oscillator fallback
applyAITrackInstrument(bank, program, synthEngine?)	ADD SpessaSynth path: call controllerChange + programChange on SpessaSynth
stopAll()	ADD SpessaSynth: allNotesOff(channel) or programReset()
controllerChange(ch, cc, val)	ADD SpessaSynth: delegate if initialized
programChange(ch, prog)	ADD SpessaSynth: delegate if initialized

Backward compat guarantee:

• All existing callers continue to work unchanged
• 6-arg playNote(pitch, vel, durMs, startTime, program, destNode) → SpessaSynth ignores startTime (schedules immediately) and destNode (uses internal routing)
• 8-arg playNote(..., channel, synthEngine) → SpessaSynth uses synthEngine.soundfont_id for SF loading, synthEngine.soundfont_program for program selection
• If SpessaSynth not loaded/CDN fails → transparent fallback to existing oscillator code

2.4 app.jsx — Integration Points

• Startup (in a useEffect or the existing audio context initialization):

  if (window.SonicSF && window.SonicSF.init) {
    SonicSF.init(getAudioContext());
  }
  

• Synth selection: After setTrackInstrumentWithProgram / setTrackInstrument, trigger SF load:

  if (synthEngine && synthEngine.type === 'soundfont') {
    SonicSF.loadSoundFont(synthEngine.soundfont_id);
  }
  

• AI track creation: Already calls applyAITrackInstrument — SpessaSynth path handles it

---

Task 3 — Backward Compatibility & Migration

3.1 Fallback Behavior

• If window.SpessaSynthClass is undefined (CDN blocked, offline): fall back to existing oscillator code
• If .sf3 download fails: fall back to oscillator
• If SpessaSynth throws: catch error, log warning, fall back to oscillator

3.2 Testing Matrix

Scenario	Expected
SpessaSynth loaded + SF3 cached	Authentic playback, 0ms load
SpessaSynth loaded + SF3 needs download	Authentic playback after 1-2s load
CDN blocked (offline)	Transparent oscillator fallback
SF3 not available on server	Transparent oscillator fallback
MIDI keyboard + armed track	Authentic or oscillator based on availability

---

3. Files Affected

File	Change
Dockerfile	Add mscore --no-install-recommends
app/core/soundfont_converter.py	NEW — SF2→SF3 convert + batch scan
app/api/v1/plugins.py	Add GET /soundfonts/download/{sf_id}
app/core/soundfont_inspector.py	Accept .sf3 in scan filter
app/main.py	Add startup background conversion
app/templates/index.html	Add SpessaSynth CDN <script type="module">
app/static/js/services/soundfontStorage.js	NEW — IndexedDB cache
app/static/js/services/soundfontPlayer.js	Dual-mode rewrite (SpessaSynth + oscillator fallback)
app/static/js/app.jsx	Add init + SF load calls

4. Validation

# Server-side
docker compose build  # verify mscore installs
docker compose exec web python -c "from app.core.soundfont_converter import SoundFontConverter; print('OK')"
curl -s http://localhost:8000/api/v1/plugins/soundfonts/download/sgm_v2.01 | head -c 4 | file -
docker compose exec web ls -la /opt/daw_engine/soundfonts/*.sf3  # verify conversion

# Client-side
# Open browser → DevTools → check SpessaSynth loaded (window.SpessaSynthClass)
# Select SoundFont → check network tab for .sf3 download
# Play MIDI notes → hear authentic instrument (not oscillator)

5. Open Questions

1. mscore image size: Measure actual delta. If >300MB, consider fluidsynth --convert alternative.
2. CDN reliability: SpessaSynth loaded from jsdelivr CDN — consider bundling or NPM install as fallback.
3. Startup delay: batch_convert_all() could take minutes for large SF2s. Run as background Celery task, not inline startup.
4. SGM_v2.01.sf2 (529MB): Conversion time for this file. Need to handle gracefully (streaming, timeout).