VST3 / SoundFont Engine — Implementation Plan

Current State vs Requirements

Check	Status	Ref
FluidSynth Channel 9 for Drums (bank=128)	✅ Already implemented. `render_engine.py:64` forces `midi_channel=9` when `is_percussion` or `bank==128`.	R1 verify
Catalog cache invalidation on SF upload	✅ Already implemented. `plugins.py:93-95` calls `invalidate_catalog_cache()`.	R2 verify
DecentSampler CWD swap for `.wav` samples	✅ Already implemented. `vst_engine.py:338-341` does `os.chdir(preset_dir)`.	R3 verify
FluidSynth SF2 path uses `static/soundfonts/` hardcoded	❌ `render_engine.py:186-189` hardcodes `static/soundfonts/<id>.sf2`. Must scan system + upload dirs.	Bug
SF2 upload dir (`app/storage/uploads/soundfonts/`) not searched in render_engine	❌ FluidSynth render branch never looks at user uploads.	Bug
`synth_engine` structured metadata not parsed	❌ Render path uses flat fields only. Spec uses `{type, plugin_id, bank, program}`.	Gap
Client preview still oscillator-emulated	⚠️ Acceptable per spec: "Wasm Module / Preview Synth" — no real SF2 in browser. `soundfontPlayer.js` reads flat fields, not `synth_engine`.	Gap
Host asset permissions might deny Docker	⚠️ Non-root users may get `Permission Denied` on mounted `.vst3`/`.sf2`.	R4
Missing VST3/SF2 fallback chain	⚠️ No graceful degraded path if selected instrument is absent.	R5

Tasks

Task A — Environment Validation

Host asset check + permissions
- Verify ls /home/locpham/daw_assets/{vst3,soundfonts,pianobook}/* returns files
- chmod -R 755 /home/locpham/daw_assets
- Files: host paths only
Docker compose mount verification
- docker-compose.yml:13-15 maps:
  - vst3 → /opt/daw_engine/vst3
  - soundfonts → /opt/daw_engine/soundfonts
  - pianobook → /opt/daw_engine/samples/pianobook
Runtime Python deps verification
- docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"
- Check server logs for HAS_PEDALBOARD, HAS_PYFLUIDSYNTH flags in vst_engine.py

Task B — Fix SF2 Path Resolution (Bugfix)

render_engine.py FluidSynth branch — replace hardcoded static/soundfonts/ path
- Current (line 185-189): builds path to app/static/soundfonts/<sf_id>.sf2 only
- Target: search in order:
  1. UPLOAD_SF_DIR = app/storage/uploads/soundfonts/<id>.sf2
  2. SYSTEM_SF_DIR = /opt/daw_engine/soundfonts/<id>.sf2
  3. Match by soundfont_id field in track metadata (not just filename)
- Accept both sf_<id> and bare <id> in instrument_id
- Files: app/core/render_engine.py:185-225
Read soundfont_bank/soundfont_program from track in FluidSynth branch
- Currently read at lines 61-63 (before item loop) — ✅ correct
- Ensure fl.program_select(midi_channel, fid, bank, program) uses them (line 194) — ✅ correct
- Files: app/core/render_engine.py (verify only)

Task C — `synth_engine` Metadata Alignment

render_engine.py: parse synth_engine object

se = track.get("synth_engine", {})
instrument_id = se.get("plugin_id") or track.get("instrument_id", "")
soundfont_bank = se.get("soundfont_bank") or track.get("soundfont_bank", 0)
soundfont_program = se.get("soundfont_program") or track.get("soundfont_program", 0)
instrument_source = se.get("type") or track.get("instrument_source", "soundfont")

Apply before the item loop (around line 57-67)
Keep flat fields as fallback for backward compat
Files: app/core/render_engine.py

app.jsx: write synth_engine alongside flat fields
- In setTrackInstrumentWithProgram() (line 6490): add synth_engine: {type, plugin_id, soundfont_bank, soundfont_program}
- In setTrackInstrument() (line 6540): same
- Type logic:
  - instrumentId starts with sf_ → type: "soundfont"
  - instrumentId is VST name → type: "vst3"
  - null → type: "default"
- Files: app/static/js/app.jsx:6490-6561
aiGateway.js: include synth_engine in track context
- In buildAIPromptContext() (line 218): add synth_engine to track objects
- Files: app/static/js/services/aiGateway.js:218-245

Task D — Graceful Fallback Chain (R5)

render_engine.py: 3-level fallback for missing instruments
- Level 1: Selected VST3/SoundFont
- Level 2: Default GeneralUser_GS.sf2 (or first available .sf2)
- Level 3: Basic oscillator synth (render_midi_events_to_audio)
- Files: app/core/render_engine.py:131-230

Task E — Client Playback Enhancement

soundfontPlayer.js: read synth_engine from track context
- playNote() accepts optional synthEngine param
- Before scheduling: call controllerChange(ch, 0, bank) and programChange(ch, program)
- Use existing oscillator ADSR emulation (no WASM SF2 — scope limit)
- Files: app/static/js/services/soundfontPlayer.js:82-204
app.jsx: pass synth_engine to SonicSF.playNote
- In schedulePianoRollMidi(): read synth_engine from active track, forward it
- Files: app/static/js/app.jsx

Task F — AI Workflow Verification

Catalog API test
- GET /api/v1/plugins/soundfonts/catalog returns {full_catalog, condensed_catalog}
- Condensed ≤ 50 entries
- window.__soundfontCatalog populated on app load
AI prompt injection test
- buildCatalogPromptSection() generates catalog text with bank/program rules
- Submit: "Compose 8 bars of Brass horns and a drum kit"
- Verify returned JSON: Brass has bank:0, program:56, Drums has bank:128, program:0

Task G — End-to-End Test

Manual checklist
- docker compose up --build succeeds
- Synth button shows "Synth" → click → dropdown lists SoundFonts + VSTs
- Select SoundFont → button label updates → instrument presets appear
- Select preset → label reflects exact instrument name
- Draw MIDI notes → Play → oscillator preview (approximate GM sound)
- Export WAV → file plays correct FluidSynth/VST3 instrument
- Select "None (Default Synth)" → oscillator fallback works
- AI generates track with instrument → export plays correct patch
- Upload new .sf2 → appears in dropdown after refresh

Task H — Install Missing Assets (if needed)

If .sf2 absent: copy GeneralUser_GS.sf2 or SGM-V2.01.sf2 to /home/locpham/daw_assets/soundfonts/
If .vst3 absent: place DecentSampler.vst3 in /home/locpham/daw_assets/vst3/
If .dspreset absent: place Pianobook library in /home/locpham/daw_assets/pianobook/

Affected Files

File	Changes
`app/core/render_engine.py`	Fix SF2 path resolution (Task B), parse `synth_engine` (Task C), 3-level fallback (Task D)
`app/static/js/app.jsx`	Write `synth_engine` in setTrackInstrument functions (Task C), pass to SoundFontPlayer (Task E)
`app/static/js/services/soundfontPlayer.js`	Accept `synthEngine` param, dispatch CC/program before note (Task E)
`app/static/js/services/aiGateway.js`	Include `synth_engine` in AI track context (Task C)
`app/api/v1/plugins.py`	No changes needed (cache invalidation already exists)
`app/core/vst_engine.py`	No changes needed (CWD swap already exists)
Host `/home/locpham/daw_assets/*`	`chmod 755`, ensure files exist

Validation

# 1. Build & boot
docker compose up --build -d

# 2. Verify deps
docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"

# 3. Catalog API
curl -s http://localhost:8000/api/v1/plugins/soundfonts/catalog | python -m json.tool | head -60

# 4. Render smoke test — create minimal project JSON and POST /api/v1/plugins/render

Rollback

All changes backward-compatible (flat fields still work if synth_engine absent).

git checkout -- app/core/render_engine.py app/static/js/app.jsx \
  app/static/js/services/soundfontPlayer.js app/static/js/services/aiGateway.js

8.2 KiB Raw Blame History

VST3 / SoundFont Engine — Implementation Plan

Current State vs Requirements

Tasks

Task A — Environment Validation

Task B — Fix SF2 Path Resolution (Bugfix)

Task C — synth_engine Metadata Alignment

Task D — Graceful Fallback Chain (R5)

Task E — Client Playback Enhancement

Task F — AI Workflow Verification

Task G — End-to-End Test

Task H — Install Missing Assets (if needed)

Affected Files

Validation

Rollback

8.2 KiB

Raw Blame History

Task C — `synth_engine` Metadata Alignment