207 lines
9.7 KiB
Markdown
207 lines
9.7 KiB
Markdown
# Technical Analysis & Implementation Plan: SF3 + SpessaSynth Client (md/37_SF_CONVERT.md)
|
||
|
||
## 1. Assessment Summary
|
||
|
||
The spec proposes 2 steps:
|
||
1. **Server**: Convert `.sf2` → `.sf3` (Ogg Vorbis compressed) via `mscore` CLI
|
||
2. **Client**: Replace oscillator emulation with SpessaSynth library for authentic SoundFont playback
|
||
|
||
### Current State vs Spec
|
||
|
||
| Requirement | Status | Impact |
|
||
|---|---|---|
|
||
| `mscore` in Dockerfile | ❌ Not installed | 200MB+ dependency |
|
||
| `soundfont_converter.py` | ❌ Does not exist | Needs creation |
|
||
| `GET /soundfonts/download/{sf_id}` endpoint | ❌ Missing | Blocks client download |
|
||
| `spessasynth_lib` CDN import in `index.html` | ❌ Not present | Blocks client upgrade |
|
||
| `soundfontStorage.js` (IndexedDB) | ❌ Not created | Needed for caching |
|
||
| `SonicSF.init(audioCtx)` | ❌ Not called anywhere | New integration point |
|
||
| `playNote` signature compat | ⚠️ Spec uses 4 params, codebase uses 6–8 | Must bridge |
|
||
| `applyAITrackInstrument` signature | ⚠️ Spec has `channel` first, code passes it last | Must bridge |
|
||
|
||
### Key Risk: `mscore` Dependency
|
||
|
||
`mscore` (MuseScore) pulls in Qt, fontconfig, audio drivers — easily 200-400MB in the container. **Alternative approaches:**
|
||
|
||
1. **`fluidsynth` built-in conversion**: `fluidsynth` already installed (`libfluidsynth3`, `pyfluidsynth`). Can convert SF2→SF3 via `--convert` flag or using `fluid_synth_sffd_*` APIs, but the low-level Python bindings don't expose this.
|
||
|
||
2. **`sf2convert`/`sf2pack`**: Smaller tools, but less commonly packaged.
|
||
|
||
3. **Python + `libsndfile`/`pydub`/`ogg`**: Parse SF2, extract WAV samples, compress to Ogg, rebuild SF3 structure. Complex — would need a SF2 parser and Ogg encoder.
|
||
|
||
4. **Python `acousticbrainz-sf2convert`**: Lightweight Python library specifically for SF2↔SF3.
|
||
|
||
**Recommendation**: Install `mscore` via `apt-get install -y mscore --no-install-recommends` to minimize deps. If the image grows too much (~1.5GB+), fall back to `fluidsynth` command-line conversion (`fluidsynth --convert`).
|
||
|
||
### Key Risk: `playNote` API Compatibility
|
||
|
||
The spec's proposed API:
|
||
```javascript
|
||
playNote(pitch, velocity=0.8, durationSec=1.0, channel=0)
|
||
```
|
||
|
||
Current callers use:
|
||
```javascript
|
||
playNote(pitch, velocity, durationMs, startTime, program, destinationNode, channel?, synthEngine?)
|
||
```
|
||
|
||
**Strategy**: Don't replace. Instead, wrap SpessaSynth inside the existing `SonicSF` object. Map:
|
||
- `velocity` (0–1) → MIDI velocity (1–127)
|
||
- `startTime` → if in future, use `setTimeout` for note scheduling (not perfect but adequate for preview)
|
||
- `destinationNode` → SpessaSynth routes to its own internal destination, but can add a gain node stage
|
||
- `program` → internal `_channels[channel].program` state (as now), but SpessaSynth also gets `programChange(ch, program)`
|
||
- `synthEngine` → load the right SF3 and set bank/program on SpessaSynth
|
||
|
||
**Backward compat**: Keep ALL existing methods. Add SpessaSynth as an optional enhanced engine. If SpessaSynth isn't loaded (CDN fails), fall back to oscillator emulation.
|
||
|
||
---
|
||
|
||
## 2. Implementation Plan
|
||
|
||
### Task 1 — Server: SF2→SF3 Conversion Pipeline
|
||
|
||
#### 1.1 Docker Dependency
|
||
- `Dockerfile`: Add `mscore` with `--no-install-recommends`
|
||
```dockerfile
|
||
RUN apt-get install -y --no-install-recommends mscore && rm -rf /var/lib/apt/lists/*
|
||
```
|
||
- Test: `docker compose build` — measure image size delta
|
||
|
||
#### 1.2 `app/core/soundfont_converter.py`
|
||
- `SoundFontConverter` class with:
|
||
- `convert_sf2_to_sf3(sf2_path)` → runs `mscore -o output.sf3 input.sf2`
|
||
- `batch_convert_all()` → walks `target_dirs`, converts missing `.sf3`
|
||
- `target_dirs` = `["/opt/daw_engine/soundfonts", "app/storage/uploads/soundfonts"]`
|
||
- Progress logging, error handling, cache validation (skip if `.sf3` newer than `.sf2`)
|
||
- Integrate with server startup:
|
||
- In `app/main.py` startup event, call `SoundFontConverter().batch_convert_all()` as background task (non-blocking, don't delay boot)
|
||
|
||
#### 1.3 API Download Endpoint
|
||
- `app/api/v1/plugins.py`: Add:
|
||
```python
|
||
@router.get("/soundfonts/download/{sf_id}")
|
||
async def download_soundfont_asset(sf_id: str):
|
||
# Search system_dir, upload_dir for .sf3 or .sf2
|
||
# Return FileResponse
|
||
# Fallback: .sf3 → .sf2 → 404
|
||
```
|
||
- Use `_find_sf2_path` from `render_engine.py` logic (case-insensitive, multi-dir)
|
||
- Set `media_type="application/octet-stream"` and proper `Content-Disposition`
|
||
|
||
#### 1.4 Update `SoundFontInspector` to Also Scan `.sf3`
|
||
- `soundfont_inspector.py:80`: Change `.sf2` filter to `('.sf2', '.sf3')`
|
||
- Ensure the catalog includes `.sf3` files as available instruments
|
||
|
||
---
|
||
|
||
### Task 2 — Client: SpessaSynth Integration
|
||
|
||
#### 2.1 `index.html` — Add SpessaSynth CDN
|
||
```html
|
||
<script type="module">
|
||
import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
|
||
window.SpessaSynthClass = Synthesizer;
|
||
</script>
|
||
```
|
||
- Add BEFORE `soundfontPlayer.js` so the class is available when the player initializes
|
||
|
||
#### 2.2 `app/static/js/services/soundfontStorage.js` — IndexedDB Cache
|
||
- `SoundFontStorage` class with `openDB()`, `getBuffer(sfId)`, `saveBuffer(sfId, arrayBuffer)`
|
||
- Uses `indexedDB` with DB name `"DAW_SoundFont_Cache"`, store name `"sf3_buffers"`
|
||
- Export singleton `sfStorage`
|
||
|
||
#### 2.3 `app/static/js/services/soundfontPlayer.js` — Dual-Mode Rewrite
|
||
|
||
**Architecture**: Keep the existing `window.SonicSF` as the public API. Internally use SpessaSynth when available, fall back to oscillator when not.
|
||
|
||
**Changes to existing methods:**
|
||
|
||
| Method | Change |
|
||
|---|---|
|
||
| `init(audioCtx)` | NEW — creates SpessaSynth instance, triggers default SF load |
|
||
| `loadSoundFont(sfId)` | REWORK — try IndexedDB → fetch `/api/v1/plugins/soundfonts/download/{sfId}` → load into SpessaSynth |
|
||
| `playNote(...)` | ADD SpessaSynth path: if initialized, delegate to `synthInstance.noteOn/noteOff`; else use oscillator fallback |
|
||
| `applyAITrackInstrument(bank, program, synthEngine?)` | ADD SpessaSynth path: call `controllerChange` + `programChange` on SpessaSynth |
|
||
| `stopAll()` | ADD SpessaSynth: `allNotesOff(channel)` or `programReset()` |
|
||
| `controllerChange(ch, cc, val)` | ADD SpessaSynth: delegate if initialized |
|
||
| `programChange(ch, prog)` | ADD SpessaSynth: delegate if initialized |
|
||
|
||
**Backward compat guarantee:**
|
||
- All existing callers continue to work unchanged
|
||
- 6-arg `playNote(pitch, vel, durMs, startTime, program, destNode)` → SpessaSynth ignores `startTime` (schedules immediately) and `destNode` (uses internal routing)
|
||
- 8-arg `playNote(..., channel, synthEngine)` → SpessaSynth uses `synthEngine.soundfont_id` for SF loading, `synthEngine.soundfont_program` for program selection
|
||
- If SpessaSynth not loaded/CDN fails → transparent fallback to existing oscillator code
|
||
|
||
#### 2.4 `app.jsx` — Integration Points
|
||
|
||
- **Startup** (in a useEffect or the existing audio context initialization):
|
||
```javascript
|
||
if (window.SonicSF && window.SonicSF.init) {
|
||
SonicSF.init(getAudioContext());
|
||
}
|
||
```
|
||
- **Synth selection**: After `setTrackInstrumentWithProgram` / `setTrackInstrument`, trigger SF load:
|
||
```javascript
|
||
if (synthEngine && synthEngine.type === 'soundfont') {
|
||
SonicSF.loadSoundFont(synthEngine.soundfont_id);
|
||
}
|
||
```
|
||
- **AI track creation**: Already calls `applyAITrackInstrument` — SpessaSynth path handles it
|
||
|
||
---
|
||
|
||
### Task 3 — Backward Compatibility & Migration
|
||
|
||
#### 3.1 Fallback Behavior
|
||
- If `window.SpessaSynthClass` is undefined (CDN blocked, offline): fall back to existing oscillator code
|
||
- If `.sf3` download fails: fall back to oscillator
|
||
- If SpessaSynth throws: catch error, log warning, fall back to oscillator
|
||
|
||
#### 3.2 Testing Matrix
|
||
|
||
| Scenario | Expected |
|
||
|---|---|
|
||
| SpessaSynth loaded + SF3 cached | Authentic playback, 0ms load |
|
||
| SpessaSynth loaded + SF3 needs download | Authentic playback after 1-2s load |
|
||
| CDN blocked (offline) | Transparent oscillator fallback |
|
||
| SF3 not available on server | Transparent oscillator fallback |
|
||
| MIDI keyboard + armed track | Authentic or oscillator based on availability |
|
||
|
||
---
|
||
|
||
## 3. Files Affected
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `Dockerfile` | Add `mscore --no-install-recommends` |
|
||
| `app/core/soundfont_converter.py` | **NEW** — SF2→SF3 convert + batch scan |
|
||
| `app/api/v1/plugins.py` | Add `GET /soundfonts/download/{sf_id}` |
|
||
| `app/core/soundfont_inspector.py` | Accept `.sf3` in scan filter |
|
||
| `app/main.py` | Add startup background conversion |
|
||
| `app/templates/index.html` | Add SpessaSynth CDN `<script type="module">` |
|
||
| `app/static/js/services/soundfontStorage.js` | **NEW** — IndexedDB cache |
|
||
| `app/static/js/services/soundfontPlayer.js` | Dual-mode rewrite (SpessaSynth + oscillator fallback) |
|
||
| `app/static/js/app.jsx` | Add init + SF load calls |
|
||
|
||
## 4. Validation
|
||
|
||
```bash
|
||
# Server-side
|
||
docker compose build # verify mscore installs
|
||
docker compose exec web python -c "from app.core.soundfont_converter import SoundFontConverter; print('OK')"
|
||
curl -s http://localhost:8000/api/v1/plugins/soundfonts/download/sgm_v2.01 | head -c 4 | file -
|
||
docker compose exec web ls -la /opt/daw_engine/soundfonts/*.sf3 # verify conversion
|
||
|
||
# Client-side
|
||
# Open browser → DevTools → check SpessaSynth loaded (window.SpessaSynthClass)
|
||
# Select SoundFont → check network tab for .sf3 download
|
||
# Play MIDI notes → hear authentic instrument (not oscillator)
|
||
```
|
||
|
||
## 5. Open Questions
|
||
|
||
1. **`mscore` image size**: Measure actual delta. If >300MB, consider `fluidsynth --convert` alternative.
|
||
2. **CDN reliability**: SpessaSynth loaded from jsdelivr CDN — consider bundling or NPM install as fallback.
|
||
3. **Startup delay**: `batch_convert_all()` could take minutes for large SF2s. Run as background Celery task, not inline startup.
|
||
4. **SGM_v2.01.sf2 (529MB)**: Conversion time for this file. Need to handle gracefully (streaming, timeout).
|