182 lines
8.2 KiB
Markdown
182 lines
8.2 KiB
Markdown
# VST3 / SoundFont Engine — Implementation Plan
|
|
|
|
## Current State vs Requirements
|
|
|
|
| Check | Status | Ref |
|
|
|-------|--------|-----|
|
|
| FluidSynth Channel 9 for Drums (bank=128) | ✅ Already implemented. `render_engine.py:64` forces `midi_channel=9` when `is_percussion` or `bank==128`. | R1 verify |
|
|
| Catalog cache invalidation on SF upload | ✅ Already implemented. `plugins.py:93-95` calls `invalidate_catalog_cache()`. | R2 verify |
|
|
| DecentSampler CWD swap for `.wav` samples | ✅ Already implemented. `vst_engine.py:338-341` does `os.chdir(preset_dir)`. | R3 verify |
|
|
| FluidSynth SF2 path uses `static/soundfonts/` hardcoded | ❌ `render_engine.py:186-189` hardcodes `static/soundfonts/<id>.sf2`. Must scan system + upload dirs. | **Bug** |
|
|
| SF2 upload dir (`app/storage/uploads/soundfonts/`) not searched in render_engine | ❌ FluidSynth render branch never looks at user uploads. | **Bug** |
|
|
| `synth_engine` structured metadata not parsed | ❌ Render path uses flat fields only. Spec uses `{type, plugin_id, bank, program}`. | **Gap** |
|
|
| Client preview still oscillator-emulated | ⚠️ Acceptable per spec: "Wasm Module / Preview Synth" — no real SF2 in browser. `soundfontPlayer.js` reads flat fields, not `synth_engine`. | **Gap** |
|
|
| Host asset permissions might deny Docker | ⚠️ Non-root users may get `Permission Denied` on mounted `.vst3`/`.sf2`. | R4 |
|
|
| Missing VST3/SF2 fallback chain | ⚠️ No graceful degraded path if selected instrument is absent. | R5 |
|
|
|
|
---
|
|
|
|
## Tasks
|
|
|
|
### Task A — Environment Validation
|
|
|
|
1. **Host asset check + permissions**
|
|
- Verify `ls /home/locpham/daw_assets/{vst3,soundfonts,pianobook}/*` returns files
|
|
- `chmod -R 755 /home/locpham/daw_assets`
|
|
- **Files:** host paths only
|
|
|
|
2. **Docker compose mount verification**
|
|
- `docker-compose.yml:13-15` maps:
|
|
- `vst3` → `/opt/daw_engine/vst3`
|
|
- `soundfonts` → `/opt/daw_engine/soundfonts`
|
|
- `pianobook` → `/opt/daw_engine/samples/pianobook`
|
|
|
|
3. **Runtime Python deps verification**
|
|
- `docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"`
|
|
- Check server logs for `HAS_PEDALBOARD`, `HAS_PYFLUIDSYNTH` flags in `vst_engine.py`
|
|
|
|
---
|
|
|
|
### Task B — Fix SF2 Path Resolution (Bugfix)
|
|
|
|
4. **`render_engine.py` FluidSynth branch** — replace hardcoded `static/soundfonts/` path
|
|
- **Current** (line 185-189): builds path to `app/static/soundfonts/<sf_id>.sf2` only
|
|
- **Target**: search in order:
|
|
1. `UPLOAD_SF_DIR` = `app/storage/uploads/soundfonts/<id>.sf2`
|
|
2. `SYSTEM_SF_DIR` = `/opt/daw_engine/soundfonts/<id>.sf2`
|
|
3. Match by `soundfont_id` field in track metadata (not just filename)
|
|
- Accept both `sf_<id>` and bare `<id>` in `instrument_id`
|
|
- **Files:** `app/core/render_engine.py:185-225`
|
|
|
|
5. **Read `soundfont_bank`/`soundfont_program` from track in FluidSynth branch**
|
|
- Currently read at lines 61-63 (before item loop) — ✅ correct
|
|
- Ensure `fl.program_select(midi_channel, fid, bank, program)` uses them (line 194) — ✅ correct
|
|
- **Files:** `app/core/render_engine.py` (verify only)
|
|
|
|
---
|
|
|
|
### Task C — `synth_engine` Metadata Alignment
|
|
|
|
6. **`render_engine.py`: parse `synth_engine` object**
|
|
```python
|
|
se = track.get("synth_engine", {})
|
|
instrument_id = se.get("plugin_id") or track.get("instrument_id", "")
|
|
soundfont_bank = se.get("soundfont_bank") or track.get("soundfont_bank", 0)
|
|
soundfont_program = se.get("soundfont_program") or track.get("soundfont_program", 0)
|
|
instrument_source = se.get("type") or track.get("instrument_source", "soundfont")
|
|
```
|
|
- Apply before the item loop (around line 57-67)
|
|
- Keep flat fields as fallback for backward compat
|
|
- **Files:** `app/core/render_engine.py`
|
|
|
|
7. **`app.jsx`: write `synth_engine` alongside flat fields**
|
|
- In `setTrackInstrumentWithProgram()` (line 6490): add `synth_engine: {type, plugin_id, soundfont_bank, soundfont_program}`
|
|
- In `setTrackInstrument()` (line 6540): same
|
|
- Type logic:
|
|
- `instrumentId` starts with `sf_` → `type: "soundfont"`
|
|
- `instrumentId` is VST name → `type: "vst3"`
|
|
- `null` → `type: "default"`
|
|
- **Files:** `app/static/js/app.jsx:6490-6561`
|
|
|
|
8. **`aiGateway.js`: include `synth_engine` in track context**
|
|
- In `buildAIPromptContext()` (line 218): add `synth_engine` to track objects
|
|
- **Files:** `app/static/js/services/aiGateway.js:218-245`
|
|
|
|
---
|
|
|
|
### Task D — Graceful Fallback Chain (R5)
|
|
|
|
9. **`render_engine.py`: 3-level fallback for missing instruments**
|
|
- Level 1: Selected VST3/SoundFont
|
|
- Level 2: Default `GeneralUser_GS.sf2` (or first available `.sf2`)
|
|
- Level 3: Basic oscillator synth (`render_midi_events_to_audio`)
|
|
- **Files:** `app/core/render_engine.py:131-230`
|
|
|
|
---
|
|
|
|
### Task E — Client Playback Enhancement
|
|
|
|
10. **`soundfontPlayer.js`: read `synth_engine` from track context**
|
|
- `playNote()` accepts optional `synthEngine` param
|
|
- Before scheduling: call `controllerChange(ch, 0, bank)` and `programChange(ch, program)`
|
|
- Use existing oscillator ADSR emulation (no WASM SF2 — scope limit)
|
|
- **Files:** `app/static/js/services/soundfontPlayer.js:82-204`
|
|
|
|
11. **`app.jsx`: pass `synth_engine` to `SonicSF.playNote`**
|
|
- In `schedulePianoRollMidi()`: read `synth_engine` from active track, forward it
|
|
- **Files:** `app/static/js/app.jsx`
|
|
|
|
---
|
|
|
|
### Task F — AI Workflow Verification
|
|
|
|
12. **Catalog API test**
|
|
- `GET /api/v1/plugins/soundfonts/catalog` returns `{full_catalog, condensed_catalog}`
|
|
- Condensed ≤ 50 entries
|
|
- `window.__soundfontCatalog` populated on app load
|
|
|
|
13. **AI prompt injection test**
|
|
- `buildCatalogPromptSection()` generates catalog text with bank/program rules
|
|
- Submit: *"Compose 8 bars of Brass horns and a drum kit"*
|
|
- Verify returned JSON: Brass has `bank:0, program:56`, Drums has `bank:128, program:0`
|
|
|
|
---
|
|
|
|
### Task G — End-to-End Test
|
|
|
|
14. **Manual checklist**
|
|
- [ ] `docker compose up --build` succeeds
|
|
- [ ] Synth button shows "Synth" → click → dropdown lists SoundFonts + VSTs
|
|
- [ ] Select SoundFont → button label updates → instrument presets appear
|
|
- [ ] Select preset → label reflects exact instrument name
|
|
- [ ] Draw MIDI notes → Play → oscillator preview (approximate GM sound)
|
|
- [ ] Export WAV → file plays correct FluidSynth/VST3 instrument
|
|
- [ ] Select "None (Default Synth)" → oscillator fallback works
|
|
- [ ] AI generates track with instrument → export plays correct patch
|
|
- [ ] Upload new `.sf2` → appears in dropdown after refresh
|
|
|
|
---
|
|
|
|
### Task H — Install Missing Assets (if needed)
|
|
|
|
15. If `.sf2` absent: copy `GeneralUser_GS.sf2` or `SGM-V2.01.sf2` to `/home/locpham/daw_assets/soundfonts/`
|
|
16. If `.vst3` absent: place `DecentSampler.vst3` in `/home/locpham/daw_assets/vst3/`
|
|
17. If `.dspreset` absent: place Pianobook library in `/home/locpham/daw_assets/pianobook/`
|
|
|
|
---
|
|
|
|
## Affected Files
|
|
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `app/core/render_engine.py` | Fix SF2 path resolution (Task B), parse `synth_engine` (Task C), 3-level fallback (Task D) |
|
|
| `app/static/js/app.jsx` | Write `synth_engine` in setTrackInstrument functions (Task C), pass to SoundFontPlayer (Task E) |
|
|
| `app/static/js/services/soundfontPlayer.js` | Accept `synthEngine` param, dispatch CC/program before note (Task E) |
|
|
| `app/static/js/services/aiGateway.js` | Include `synth_engine` in AI track context (Task C) |
|
|
| `app/api/v1/plugins.py` | No changes needed (cache invalidation already exists) |
|
|
| `app/core/vst_engine.py` | No changes needed (CWD swap already exists) |
|
|
| Host `/home/locpham/daw_assets/*` | `chmod 755`, ensure files exist |
|
|
|
|
## Validation
|
|
|
|
```bash
|
|
# 1. Build & boot
|
|
docker compose up --build -d
|
|
|
|
# 2. Verify deps
|
|
docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"
|
|
|
|
# 3. Catalog API
|
|
curl -s http://localhost:8000/api/v1/plugins/soundfonts/catalog | python -m json.tool | head -60
|
|
|
|
# 4. Render smoke test — create minimal project JSON and POST /api/v1/plugins/render
|
|
```
|
|
|
|
## Rollback
|
|
|
|
All changes backward-compatible (flat fields still work if `synth_engine` absent).
|
|
```bash
|
|
git checkout -- app/core/render_engine.py app/static/js/app.jsx \
|
|
app/static/js/services/soundfontPlayer.js app/static/js/services/aiGateway.js
|
|
```
|