# Technical Analysis & Implementation Plan: SF3 + SpessaSynth Client (md/37_SF_CONVERT.md)

## 1. Assessment Summary

The spec proposes 2 steps:
1. **Server**: Convert `.sf2` → `.sf3` (Ogg Vorbis compressed) via `mscore` CLI
2. **Client**: Replace oscillator emulation with SpessaSynth library for authentic SoundFont playback

### Current State vs Spec

| Requirement | Status | Impact |
|---|---|---|
| `mscore` in Dockerfile | ❌ Not installed | 200MB+ dependency |
| `soundfont_converter.py` | ❌ Does not exist | Needs creation |
| `GET /soundfonts/download/{sf_id}` endpoint | ❌ Missing | Blocks client download |
| `spessasynth_lib` CDN import in `index.html` | ❌ Not present | Blocks client upgrade |
| `soundfontStorage.js` (IndexedDB) | ❌ Not created | Needed for caching |
| `SonicSF.init(audioCtx)` | ❌ Not called anywhere | New integration point |
| `playNote` signature compat | ⚠️ Spec uses 4 params, codebase uses 6–8 | Must bridge |
| `applyAITrackInstrument` signature | ⚠️ Spec has `channel` first, code passes it last | Must bridge |

### Key Risk: `mscore` Dependency

`mscore` (MuseScore) pulls in Qt, fontconfig, audio drivers — easily 200-400MB in the container. **Alternative approaches:**

1. **`fluidsynth` built-in conversion**: `fluidsynth` already installed (`libfluidsynth3`, `pyfluidsynth`). Can convert SF2→SF3 via `--convert` flag or using `fluid_synth_sffd_*` APIs, but the low-level Python bindings don't expose this.

2. **`sf2convert`/`sf2pack`**: Smaller tools, but less commonly packaged.

3. **Python + `libsndfile`/`pydub`/`ogg`**: Parse SF2, extract WAV samples, compress to Ogg, rebuild SF3 structure. Complex — would need a SF2 parser and Ogg encoder.

4. **Python `acousticbrainz-sf2convert`**: Lightweight Python library specifically for SF2↔SF3.

**Recommendation**: Install `mscore` via `apt-get install -y mscore --no-install-recommends` to minimize deps. If the image grows too much (~1.5GB+), fall back to `fluidsynth` command-line conversion (`fluidsynth --convert`).

### Key Risk: `playNote` API Compatibility

The spec's proposed API:
```javascript
playNote(pitch, velocity=0.8, durationSec=1.0, channel=0)
```

Current callers use:
```javascript
playNote(pitch, velocity, durationMs, startTime, program, destinationNode, channel?, synthEngine?)
```

**Strategy**: Don't replace. Instead, wrap SpessaSynth inside the existing `SonicSF` object. Map:
- `velocity` (0–1) → MIDI velocity (1–127)
- `startTime` → if in future, use `setTimeout` for note scheduling (not perfect but adequate for preview)
- `destinationNode` → SpessaSynth routes to its own internal destination, but can add a gain node stage
- `program` → internal `_channels[channel].program` state (as now), but SpessaSynth also gets `programChange(ch, program)`
- `synthEngine` → load the right SF3 and set bank/program on SpessaSynth

**Backward compat**: Keep ALL existing methods. Add SpessaSynth as an optional enhanced engine. If SpessaSynth isn't loaded (CDN fails), fall back to oscillator emulation.

---

## 2. Implementation Plan

### Task 1 — Server: SF2→SF3 Conversion Pipeline

#### 1.1 Docker Dependency
- `Dockerfile`: Add `mscore` with `--no-install-recommends`
  ```dockerfile
  RUN apt-get install -y --no-install-recommends mscore && rm -rf /var/lib/apt/lists/*
  ```
- Test: `docker compose build` — measure image size delta

#### 1.2 `app/core/soundfont_converter.py`
- `SoundFontConverter` class with:
  - `convert_sf2_to_sf3(sf2_path)` → runs `mscore -o output.sf3 input.sf2`
  - `batch_convert_all()` → walks `target_dirs`, converts missing `.sf3`
  - `target_dirs` = `["/opt/daw_engine/soundfonts", "app/storage/uploads/soundfonts"]`
  - Progress logging, error handling, cache validation (skip if `.sf3` newer than `.sf2`)
- Integrate with server startup:
  - In `app/main.py` startup event, call `SoundFontConverter().batch_convert_all()` as background task (non-blocking, don't delay boot)

#### 1.3 API Download Endpoint
- `app/api/v1/plugins.py`: Add:
  ```python
  @router.get("/soundfonts/download/{sf_id}")
  async def download_soundfont_asset(sf_id: str):
      # Search system_dir, upload_dir for .sf3 or .sf2
      # Return FileResponse
      # Fallback: .sf3 → .sf2 → 404
  ```
- Use `_find_sf2_path` from `render_engine.py` logic (case-insensitive, multi-dir)
- Set `media_type="application/octet-stream"` and proper `Content-Disposition`

#### 1.4 Update `SoundFontInspector` to Also Scan `.sf3`
- `soundfont_inspector.py:80`: Change `.sf2` filter to `('.sf2', '.sf3')`
- Ensure the catalog includes `.sf3` files as available instruments

---

### Task 2 — Client: SpessaSynth Integration

#### 2.1 `index.html` — Add SpessaSynth CDN
```html
<script type="module">
  import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
  window.SpessaSynthClass = Synthesizer;
</script>
```
- Add BEFORE `soundfontPlayer.js` so the class is available when the player initializes

#### 2.2 `app/static/js/services/soundfontStorage.js` — IndexedDB Cache
- `SoundFontStorage` class with `openDB()`, `getBuffer(sfId)`, `saveBuffer(sfId, arrayBuffer)`
- Uses `indexedDB` with DB name `"DAW_SoundFont_Cache"`, store name `"sf3_buffers"`
- Export singleton `sfStorage`

#### 2.3 `app/static/js/services/soundfontPlayer.js` — Dual-Mode Rewrite

**Architecture**: Keep the existing `window.SonicSF` as the public API. Internally use SpessaSynth when available, fall back to oscillator when not.

**Changes to existing methods:**

| Method | Change |
|---|---|
| `init(audioCtx)` | NEW — creates SpessaSynth instance, triggers default SF load |
| `loadSoundFont(sfId)` | REWORK — try IndexedDB → fetch `/api/v1/plugins/soundfonts/download/{sfId}` → load into SpessaSynth |
| `playNote(...)` | ADD SpessaSynth path: if initialized, delegate to `synthInstance.noteOn/noteOff`; else use oscillator fallback |
| `applyAITrackInstrument(bank, program, synthEngine?)` | ADD SpessaSynth path: call `controllerChange` + `programChange` on SpessaSynth |
| `stopAll()` | ADD SpessaSynth: `allNotesOff(channel)` or `programReset()` |
| `controllerChange(ch, cc, val)` | ADD SpessaSynth: delegate if initialized |
| `programChange(ch, prog)` | ADD SpessaSynth: delegate if initialized |

**Backward compat guarantee:**
- All existing callers continue to work unchanged
- 6-arg `playNote(pitch, vel, durMs, startTime, program, destNode)` → SpessaSynth ignores `startTime` (schedules immediately) and `destNode` (uses internal routing)
- 8-arg `playNote(..., channel, synthEngine)` → SpessaSynth uses `synthEngine.soundfont_id` for SF loading, `synthEngine.soundfont_program` for program selection
- If SpessaSynth not loaded/CDN fails → transparent fallback to existing oscillator code

#### 2.4 `app.jsx` — Integration Points

- **Startup** (in a useEffect or the existing audio context initialization):
  ```javascript
  if (window.SonicSF && window.SonicSF.init) {
    SonicSF.init(getAudioContext());
  }
  ```
- **Synth selection**: After `setTrackInstrumentWithProgram` / `setTrackInstrument`, trigger SF load:
  ```javascript
  if (synthEngine && synthEngine.type === 'soundfont') {
    SonicSF.loadSoundFont(synthEngine.soundfont_id);
  }
  ```
- **AI track creation**: Already calls `applyAITrackInstrument` — SpessaSynth path handles it

---

### Task 3 — Backward Compatibility & Migration

#### 3.1 Fallback Behavior
- If `window.SpessaSynthClass` is undefined (CDN blocked, offline): fall back to existing oscillator code
- If `.sf3` download fails: fall back to oscillator
- If SpessaSynth throws: catch error, log warning, fall back to oscillator

#### 3.2 Testing Matrix

| Scenario | Expected |
|---|---|
| SpessaSynth loaded + SF3 cached | Authentic playback, 0ms load |
| SpessaSynth loaded + SF3 needs download | Authentic playback after 1-2s load |
| CDN blocked (offline) | Transparent oscillator fallback |
| SF3 not available on server | Transparent oscillator fallback |
| MIDI keyboard + armed track | Authentic or oscillator based on availability |

---

## 3. Files Affected

| File | Change |
|---|---|
| `Dockerfile` | Add `mscore --no-install-recommends` |
| `app/core/soundfont_converter.py` | **NEW** — SF2→SF3 convert + batch scan |
| `app/api/v1/plugins.py` | Add `GET /soundfonts/download/{sf_id}` |
| `app/core/soundfont_inspector.py` | Accept `.sf3` in scan filter |
| `app/main.py` | Add startup background conversion |
| `app/templates/index.html` | Add SpessaSynth CDN `<script type="module">` |
| `app/static/js/services/soundfontStorage.js` | **NEW** — IndexedDB cache |
| `app/static/js/services/soundfontPlayer.js` | Dual-mode rewrite (SpessaSynth + oscillator fallback) |
| `app/static/js/app.jsx` | Add init + SF load calls |

## 4. Validation

```bash
# Server-side
docker compose build  # verify mscore installs
docker compose exec web python -c "from app.core.soundfont_converter import SoundFontConverter; print('OK')"
curl -s http://localhost:8000/api/v1/plugins/soundfonts/download/sgm_v2.01 | head -c 4 | file -
docker compose exec web ls -la /opt/daw_engine/soundfonts/*.sf3  # verify conversion

# Client-side
# Open browser → DevTools → check SpessaSynth loaded (window.SpessaSynthClass)
# Select SoundFont → check network tab for .sf3 download
# Play MIDI notes → hear authentic instrument (not oscillator)
```

## 5. Open Questions

1. **`mscore` image size**: Measure actual delta. If >300MB, consider `fluidsynth --convert` alternative.
2. **CDN reliability**: SpessaSynth loaded from jsdelivr CDN — consider bundling or NPM install as fallback.
3. **Startup delay**: `batch_convert_all()` could take minutes for large SF2s. Run as background Celery task, not inline startup.
4. **SGM_v2.01.sf2 (529MB)**: Conversion time for this file. Need to handle gracefully (streaming, timeout).