diff --git a/.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md b/.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md index 906f51e..98c7454 100644 --- a/.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md +++ b/.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md @@ -1,181 +1,206 @@ -# VST3 / SoundFont Engine — Implementation Plan +# Technical Analysis & Implementation Plan: SF3 + SpessaSynth Client (md/37_SF_CONVERT.md) -## Current State vs Requirements +## 1. Assessment Summary -| Check | Status | Ref | -|-------|--------|-----| -| FluidSynth Channel 9 for Drums (bank=128) | ✅ Already implemented. `render_engine.py:64` forces `midi_channel=9` when `is_percussion` or `bank==128`. | R1 verify | -| Catalog cache invalidation on SF upload | ✅ Already implemented. `plugins.py:93-95` calls `invalidate_catalog_cache()`. | R2 verify | -| DecentSampler CWD swap for `.wav` samples | ✅ Already implemented. `vst_engine.py:338-341` does `os.chdir(preset_dir)`. | R3 verify | -| FluidSynth SF2 path uses `static/soundfonts/` hardcoded | ❌ `render_engine.py:186-189` hardcodes `static/soundfonts/.sf2`. Must scan system + upload dirs. | **Bug** | -| SF2 upload dir (`app/storage/uploads/soundfonts/`) not searched in render_engine | ❌ FluidSynth render branch never looks at user uploads. | **Bug** | -| `synth_engine` structured metadata not parsed | ❌ Render path uses flat fields only. Spec uses `{type, plugin_id, bank, program}`. | **Gap** | -| Client preview still oscillator-emulated | ⚠️ Acceptable per spec: "Wasm Module / Preview Synth" — no real SF2 in browser. `soundfontPlayer.js` reads flat fields, not `synth_engine`. | **Gap** | -| Host asset permissions might deny Docker | ⚠️ Non-root users may get `Permission Denied` on mounted `.vst3`/`.sf2`. | R4 | -| Missing VST3/SF2 fallback chain | ⚠️ No graceful degraded path if selected instrument is absent. | R5 | +The spec proposes 2 steps: +1. **Server**: Convert `.sf2` → `.sf3` (Ogg Vorbis compressed) via `mscore` CLI +2. **Client**: Replace oscillator emulation with SpessaSynth library for authentic SoundFont playback ---- +### Current State vs Spec -## Tasks +| Requirement | Status | Impact | +|---|---|---| +| `mscore` in Dockerfile | ❌ Not installed | 200MB+ dependency | +| `soundfont_converter.py` | ❌ Does not exist | Needs creation | +| `GET /soundfonts/download/{sf_id}` endpoint | ❌ Missing | Blocks client download | +| `spessasynth_lib` CDN import in `index.html` | ❌ Not present | Blocks client upgrade | +| `soundfontStorage.js` (IndexedDB) | ❌ Not created | Needed for caching | +| `SonicSF.init(audioCtx)` | ❌ Not called anywhere | New integration point | +| `playNote` signature compat | ⚠️ Spec uses 4 params, codebase uses 6–8 | Must bridge | +| `applyAITrackInstrument` signature | ⚠️ Spec has `channel` first, code passes it last | Must bridge | -### Task A — Environment Validation +### Key Risk: `mscore` Dependency -1. **Host asset check + permissions** - - Verify `ls /home/locpham/daw_assets/{vst3,soundfonts,pianobook}/*` returns files - - `chmod -R 755 /home/locpham/daw_assets` - - **Files:** host paths only +`mscore` (MuseScore) pulls in Qt, fontconfig, audio drivers — easily 200-400MB in the container. **Alternative approaches:** -2. **Docker compose mount verification** - - `docker-compose.yml:13-15` maps: - - `vst3` → `/opt/daw_engine/vst3` - - `soundfonts` → `/opt/daw_engine/soundfonts` - - `pianobook` → `/opt/daw_engine/samples/pianobook` +1. **`fluidsynth` built-in conversion**: `fluidsynth` already installed (`libfluidsynth3`, `pyfluidsynth`). Can convert SF2→SF3 via `--convert` flag or using `fluid_synth_sffd_*` APIs, but the low-level Python bindings don't expose this. -3. **Runtime Python deps verification** - - `docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"` - - Check server logs for `HAS_PEDALBOARD`, `HAS_PYFLUIDSYNTH` flags in `vst_engine.py` +2. **`sf2convert`/`sf2pack`**: Smaller tools, but less commonly packaged. ---- +3. **Python + `libsndfile`/`pydub`/`ogg`**: Parse SF2, extract WAV samples, compress to Ogg, rebuild SF3 structure. Complex — would need a SF2 parser and Ogg encoder. -### Task B — Fix SF2 Path Resolution (Bugfix) +4. **Python `acousticbrainz-sf2convert`**: Lightweight Python library specifically for SF2↔SF3. -4. **`render_engine.py` FluidSynth branch** — replace hardcoded `static/soundfonts/` path - - **Current** (line 185-189): builds path to `app/static/soundfonts/.sf2` only - - **Target**: search in order: - 1. `UPLOAD_SF_DIR` = `app/storage/uploads/soundfonts/.sf2` - 2. `SYSTEM_SF_DIR` = `/opt/daw_engine/soundfonts/.sf2` - 3. Match by `soundfont_id` field in track metadata (not just filename) - - Accept both `sf_` and bare `` in `instrument_id` - - **Files:** `app/core/render_engine.py:185-225` +**Recommendation**: Install `mscore` via `apt-get install -y mscore --no-install-recommends` to minimize deps. If the image grows too much (~1.5GB+), fall back to `fluidsynth` command-line conversion (`fluidsynth --convert`). -5. **Read `soundfont_bank`/`soundfont_program` from track in FluidSynth branch** - - Currently read at lines 61-63 (before item loop) — ✅ correct - - Ensure `fl.program_select(midi_channel, fid, bank, program)` uses them (line 194) — ✅ correct - - **Files:** `app/core/render_engine.py` (verify only) +### Key Risk: `playNote` API Compatibility ---- - -### Task C — `synth_engine` Metadata Alignment - -6. **`render_engine.py`: parse `synth_engine` object** - ```python - se = track.get("synth_engine", {}) - instrument_id = se.get("plugin_id") or track.get("instrument_id", "") - soundfont_bank = se.get("soundfont_bank") or track.get("soundfont_bank", 0) - soundfont_program = se.get("soundfont_program") or track.get("soundfont_program", 0) - instrument_source = se.get("type") or track.get("instrument_source", "soundfont") - ``` - - Apply before the item loop (around line 57-67) - - Keep flat fields as fallback for backward compat - - **Files:** `app/core/render_engine.py` - -7. **`app.jsx`: write `synth_engine` alongside flat fields** - - In `setTrackInstrumentWithProgram()` (line 6490): add `synth_engine: {type, plugin_id, soundfont_bank, soundfont_program}` - - In `setTrackInstrument()` (line 6540): same - - Type logic: - - `instrumentId` starts with `sf_` → `type: "soundfont"` - - `instrumentId` is VST name → `type: "vst3"` - - `null` → `type: "default"` - - **Files:** `app/static/js/app.jsx:6490-6561` - -8. **`aiGateway.js`: include `synth_engine` in track context** - - In `buildAIPromptContext()` (line 218): add `synth_engine` to track objects - - **Files:** `app/static/js/services/aiGateway.js:218-245` - ---- - -### Task D — Graceful Fallback Chain (R5) - -9. **`render_engine.py`: 3-level fallback for missing instruments** - - Level 1: Selected VST3/SoundFont - - Level 2: Default `GeneralUser_GS.sf2` (or first available `.sf2`) - - Level 3: Basic oscillator synth (`render_midi_events_to_audio`) - - **Files:** `app/core/render_engine.py:131-230` - ---- - -### Task E — Client Playback Enhancement - -10. **`soundfontPlayer.js`: read `synth_engine` from track context** - - `playNote()` accepts optional `synthEngine` param - - Before scheduling: call `controllerChange(ch, 0, bank)` and `programChange(ch, program)` - - Use existing oscillator ADSR emulation (no WASM SF2 — scope limit) - - **Files:** `app/static/js/services/soundfontPlayer.js:82-204` - -11. **`app.jsx`: pass `synth_engine` to `SonicSF.playNote`** - - In `schedulePianoRollMidi()`: read `synth_engine` from active track, forward it - - **Files:** `app/static/js/app.jsx` - ---- - -### Task F — AI Workflow Verification - -12. **Catalog API test** - - `GET /api/v1/plugins/soundfonts/catalog` returns `{full_catalog, condensed_catalog}` - - Condensed ≤ 50 entries - - `window.__soundfontCatalog` populated on app load - -13. **AI prompt injection test** - - `buildCatalogPromptSection()` generates catalog text with bank/program rules - - Submit: *"Compose 8 bars of Brass horns and a drum kit"* - - Verify returned JSON: Brass has `bank:0, program:56`, Drums has `bank:128, program:0` - ---- - -### Task G — End-to-End Test - -14. **Manual checklist** - - [ ] `docker compose up --build` succeeds - - [ ] Synth button shows "Synth" → click → dropdown lists SoundFonts + VSTs - - [ ] Select SoundFont → button label updates → instrument presets appear - - [ ] Select preset → label reflects exact instrument name - - [ ] Draw MIDI notes → Play → oscillator preview (approximate GM sound) - - [ ] Export WAV → file plays correct FluidSynth/VST3 instrument - - [ ] Select "None (Default Synth)" → oscillator fallback works - - [ ] AI generates track with instrument → export plays correct patch - - [ ] Upload new `.sf2` → appears in dropdown after refresh - ---- - -### Task H — Install Missing Assets (if needed) - -15. If `.sf2` absent: copy `GeneralUser_GS.sf2` or `SGM-V2.01.sf2` to `/home/locpham/daw_assets/soundfonts/` -16. If `.vst3` absent: place `DecentSampler.vst3` in `/home/locpham/daw_assets/vst3/` -17. If `.dspreset` absent: place Pianobook library in `/home/locpham/daw_assets/pianobook/` - ---- - -## Affected Files - -| File | Changes | -|------|---------| -| `app/core/render_engine.py` | Fix SF2 path resolution (Task B), parse `synth_engine` (Task C), 3-level fallback (Task D) | -| `app/static/js/app.jsx` | Write `synth_engine` in setTrackInstrument functions (Task C), pass to SoundFontPlayer (Task E) | -| `app/static/js/services/soundfontPlayer.js` | Accept `synthEngine` param, dispatch CC/program before note (Task E) | -| `app/static/js/services/aiGateway.js` | Include `synth_engine` in AI track context (Task C) | -| `app/api/v1/plugins.py` | No changes needed (cache invalidation already exists) | -| `app/core/vst_engine.py` | No changes needed (CWD swap already exists) | -| Host `/home/locpham/daw_assets/*` | `chmod 755`, ensure files exist | - -## Validation - -```bash -# 1. Build & boot -docker compose up --build -d - -# 2. Verify deps -docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')" - -# 3. Catalog API -curl -s http://localhost:8000/api/v1/plugins/soundfonts/catalog | python -m json.tool | head -60 - -# 4. Render smoke test — create minimal project JSON and POST /api/v1/plugins/render +The spec's proposed API: +```javascript +playNote(pitch, velocity=0.8, durationSec=1.0, channel=0) ``` -## Rollback - -All changes backward-compatible (flat fields still work if `synth_engine` absent). -```bash -git checkout -- app/core/render_engine.py app/static/js/app.jsx \ - app/static/js/services/soundfontPlayer.js app/static/js/services/aiGateway.js +Current callers use: +```javascript +playNote(pitch, velocity, durationMs, startTime, program, destinationNode, channel?, synthEngine?) ``` + +**Strategy**: Don't replace. Instead, wrap SpessaSynth inside the existing `SonicSF` object. Map: +- `velocity` (0–1) → MIDI velocity (1–127) +- `startTime` → if in future, use `setTimeout` for note scheduling (not perfect but adequate for preview) +- `destinationNode` → SpessaSynth routes to its own internal destination, but can add a gain node stage +- `program` → internal `_channels[channel].program` state (as now), but SpessaSynth also gets `programChange(ch, program)` +- `synthEngine` → load the right SF3 and set bank/program on SpessaSynth + +**Backward compat**: Keep ALL existing methods. Add SpessaSynth as an optional enhanced engine. If SpessaSynth isn't loaded (CDN fails), fall back to oscillator emulation. + +--- + +## 2. Implementation Plan + +### Task 1 — Server: SF2→SF3 Conversion Pipeline + +#### 1.1 Docker Dependency +- `Dockerfile`: Add `mscore` with `--no-install-recommends` + ```dockerfile + RUN apt-get install -y --no-install-recommends mscore && rm -rf /var/lib/apt/lists/* + ``` +- Test: `docker compose build` — measure image size delta + +#### 1.2 `app/core/soundfont_converter.py` +- `SoundFontConverter` class with: + - `convert_sf2_to_sf3(sf2_path)` → runs `mscore -o output.sf3 input.sf2` + - `batch_convert_all()` → walks `target_dirs`, converts missing `.sf3` + - `target_dirs` = `["/opt/daw_engine/soundfonts", "app/storage/uploads/soundfonts"]` + - Progress logging, error handling, cache validation (skip if `.sf3` newer than `.sf2`) +- Integrate with server startup: + - In `app/main.py` startup event, call `SoundFontConverter().batch_convert_all()` as background task (non-blocking, don't delay boot) + +#### 1.3 API Download Endpoint +- `app/api/v1/plugins.py`: Add: + ```python + @router.get("/soundfonts/download/{sf_id}") + async def download_soundfont_asset(sf_id: str): + # Search system_dir, upload_dir for .sf3 or .sf2 + # Return FileResponse + # Fallback: .sf3 → .sf2 → 404 + ``` +- Use `_find_sf2_path` from `render_engine.py` logic (case-insensitive, multi-dir) +- Set `media_type="application/octet-stream"` and proper `Content-Disposition` + +#### 1.4 Update `SoundFontInspector` to Also Scan `.sf3` +- `soundfont_inspector.py:80`: Change `.sf2` filter to `('.sf2', '.sf3')` +- Ensure the catalog includes `.sf3` files as available instruments + +--- + +### Task 2 — Client: SpessaSynth Integration + +#### 2.1 `index.html` — Add SpessaSynth CDN +```html + +``` +- Add BEFORE `soundfontPlayer.js` so the class is available when the player initializes + +#### 2.2 `app/static/js/services/soundfontStorage.js` — IndexedDB Cache +- `SoundFontStorage` class with `openDB()`, `getBuffer(sfId)`, `saveBuffer(sfId, arrayBuffer)` +- Uses `indexedDB` with DB name `"DAW_SoundFont_Cache"`, store name `"sf3_buffers"` +- Export singleton `sfStorage` + +#### 2.3 `app/static/js/services/soundfontPlayer.js` — Dual-Mode Rewrite + +**Architecture**: Keep the existing `window.SonicSF` as the public API. Internally use SpessaSynth when available, fall back to oscillator when not. + +**Changes to existing methods:** + +| Method | Change | +|---|---| +| `init(audioCtx)` | NEW — creates SpessaSynth instance, triggers default SF load | +| `loadSoundFont(sfId)` | REWORK — try IndexedDB → fetch `/api/v1/plugins/soundfonts/download/{sfId}` → load into SpessaSynth | +| `playNote(...)` | ADD SpessaSynth path: if initialized, delegate to `synthInstance.noteOn/noteOff`; else use oscillator fallback | +| `applyAITrackInstrument(bank, program, synthEngine?)` | ADD SpessaSynth path: call `controllerChange` + `programChange` on SpessaSynth | +| `stopAll()` | ADD SpessaSynth: `allNotesOff(channel)` or `programReset()` | +| `controllerChange(ch, cc, val)` | ADD SpessaSynth: delegate if initialized | +| `programChange(ch, prog)` | ADD SpessaSynth: delegate if initialized | + +**Backward compat guarantee:** +- All existing callers continue to work unchanged +- 6-arg `playNote(pitch, vel, durMs, startTime, program, destNode)` → SpessaSynth ignores `startTime` (schedules immediately) and `destNode` (uses internal routing) +- 8-arg `playNote(..., channel, synthEngine)` → SpessaSynth uses `synthEngine.soundfont_id` for SF loading, `synthEngine.soundfont_program` for program selection +- If SpessaSynth not loaded/CDN fails → transparent fallback to existing oscillator code + +#### 2.4 `app.jsx` — Integration Points + +- **Startup** (in a useEffect or the existing audio context initialization): + ```javascript + if (window.SonicSF && window.SonicSF.init) { + SonicSF.init(getAudioContext()); + } + ``` +- **Synth selection**: After `setTrackInstrumentWithProgram` / `setTrackInstrument`, trigger SF load: + ```javascript + if (synthEngine && synthEngine.type === 'soundfont') { + SonicSF.loadSoundFont(synthEngine.soundfont_id); + } + ``` +- **AI track creation**: Already calls `applyAITrackInstrument` — SpessaSynth path handles it + +--- + +### Task 3 — Backward Compatibility & Migration + +#### 3.1 Fallback Behavior +- If `window.SpessaSynthClass` is undefined (CDN blocked, offline): fall back to existing oscillator code +- If `.sf3` download fails: fall back to oscillator +- If SpessaSynth throws: catch error, log warning, fall back to oscillator + +#### 3.2 Testing Matrix + +| Scenario | Expected | +|---|---| +| SpessaSynth loaded + SF3 cached | Authentic playback, 0ms load | +| SpessaSynth loaded + SF3 needs download | Authentic playback after 1-2s load | +| CDN blocked (offline) | Transparent oscillator fallback | +| SF3 not available on server | Transparent oscillator fallback | +| MIDI keyboard + armed track | Authentic or oscillator based on availability | + +--- + +## 3. Files Affected + +| File | Change | +|---|---| +| `Dockerfile` | Add `mscore --no-install-recommends` | +| `app/core/soundfont_converter.py` | **NEW** — SF2→SF3 convert + batch scan | +| `app/api/v1/plugins.py` | Add `GET /soundfonts/download/{sf_id}` | +| `app/core/soundfont_inspector.py` | Accept `.sf3` in scan filter | +| `app/main.py` | Add startup background conversion | +| `app/templates/index.html` | Add SpessaSynth CDN `