3dtours/SonicForgeStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: sử dụng soundfont từ server

Browse Source
This commit is contained in:
3dtours
2026-07-26 20:25:31 +07:00
parent 2f2cb3a066
commit 40c531087c
4 changed files with 420 additions and 165 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md
+190 -165
View File
@@ -1,181 +1,206 @@
# VST3 / SoundFont Engine — Implementation Plan # Technical Analysis & Implementation Plan: SF3 + SpessaSynth Client (md/37_SF_CONVERT.md)
## Current State vs Requirements ## 1. Assessment Summary
| Check | Status | Ref | The spec proposes 2 steps:
|-------|--------|-----| 1. **Server**: Convert `.sf2` → `.sf3` (Ogg Vorbis compressed) via `mscore` CLI
| FluidSynth Channel 9 for Drums (bank=128) | ✅ Already implemented. `render_engine.py:64` forces `midi_channel=9` when `is_percussion` or `bank==128`. | R1 verify | 2. **Client**: Replace oscillator emulation with SpessaSynth library for authentic SoundFont playback
| Catalog cache invalidation on SF upload | ✅ Already implemented. `plugins.py:93-95` calls `invalidate_catalog_cache()`. | R2 verify |
| DecentSampler CWD swap for `.wav` samples | ✅ Already implemented. `vst_engine.py:338-341` does `os.chdir(preset_dir)`. | R3 verify |
| FluidSynth SF2 path uses `static/soundfonts/` hardcoded | ❌ `render_engine.py:186-189` hardcodes `static/soundfonts/<id>.sf2`. Must scan system + upload dirs. | **Bug** |
| SF2 upload dir (`app/storage/uploads/soundfonts/`) not searched in render_engine | ❌ FluidSynth render branch never looks at user uploads. | **Bug** |
| `synth_engine` structured metadata not parsed | ❌ Render path uses flat fields only. Spec uses `{type, plugin_id, bank, program}`. | **Gap** |
| Client preview still oscillator-emulated | ⚠️ Acceptable per spec: "Wasm Module / Preview Synth" — no real SF2 in browser. `soundfontPlayer.js` reads flat fields, not `synth_engine`. | **Gap** |
| Host asset permissions might deny Docker | ⚠️ Non-root users may get `Permission Denied` on mounted `.vst3`/`.sf2`. | R4 |
| Missing VST3/SF2 fallback chain | ⚠️ No graceful degraded path if selected instrument is absent. | R5 |
--- ### Current State vs Spec
## Tasks | Requirement | Status | Impact |
|---|---|---|
| `mscore` in Dockerfile | ❌ Not installed | 200MB+ dependency |
| `soundfont_converter.py` | ❌ Does not exist | Needs creation |
| `GET /soundfonts/download/{sf_id}` endpoint | ❌ Missing | Blocks client download |
| `spessasynth_lib` CDN import in `index.html` | ❌ Not present | Blocks client upgrade |
| `soundfontStorage.js` (IndexedDB) | ❌ Not created | Needed for caching |
| `SonicSF.init(audioCtx)` | ❌ Not called anywhere | New integration point |
| `playNote` signature compat | ⚠️ Spec uses 4 params, codebase uses 6–8 | Must bridge |
| `applyAITrackInstrument` signature | ⚠️ Spec has `channel` first, code passes it last | Must bridge |
### Task A — Environment Validation ### Key Risk: `mscore` Dependency
1. **Host asset check + permissions** `mscore` (MuseScore) pulls in Qt, fontconfig, audio drivers — easily 200-400MB in the container. **Alternative approaches:**
- Verify `ls /home/locpham/daw_assets/{vst3,soundfonts,pianobook}/*` returns files
- `chmod -R 755 /home/locpham/daw_assets`
- **Files:** host paths only
2. **Docker compose mount verification** 1. **`fluidsynth` built-in conversion**: `fluidsynth` already installed (`libfluidsynth3`, `pyfluidsynth`). Can convert SF2→SF3 via `--convert` flag or using `fluid_synth_sffd_*` APIs, but the low-level Python bindings don't expose this.
- `docker-compose.yml:13-15` maps:
- `vst3` → `/opt/daw_engine/vst3`
- `soundfonts` → `/opt/daw_engine/soundfonts`
- `pianobook` → `/opt/daw_engine/samples/pianobook`
3. **Runtime Python deps verification** 2. **`sf2convert`/`sf2pack`**: Smaller tools, but less commonly packaged.
- `docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"`
- Check server logs for `HAS_PEDALBOARD`, `HAS_PYFLUIDSYNTH` flags in `vst_engine.py`
--- 3. **Python + `libsndfile`/`pydub`/`ogg`**: Parse SF2, extract WAV samples, compress to Ogg, rebuild SF3 structure. Complex — would need a SF2 parser and Ogg encoder.
### Task B — Fix SF2 Path Resolution (Bugfix) 4. **Python `acousticbrainz-sf2convert`**: Lightweight Python library specifically for SF2↔SF3.
4. **`render_engine.py` FluidSynth branch** — replace hardcoded `static/soundfonts/` path **Recommendation**: Install `mscore` via `apt-get install -y mscore --no-install-recommends` to minimize deps. If the image grows too much (~1.5GB+), fall back to `fluidsynth` command-line conversion (`fluidsynth --convert`).
- **Current** (line 185-189): builds path to `app/static/soundfonts/<sf_id>.sf2` only
- **Target**: search in order:
1. `UPLOAD_SF_DIR` = `app/storage/uploads/soundfonts/<id>.sf2`
2. `SYSTEM_SF_DIR` = `/opt/daw_engine/soundfonts/<id>.sf2`
3. Match by `soundfont_id` field in track metadata (not just filename)
- Accept both `sf_<id>` and bare `<id>` in `instrument_id`
- **Files:** `app/core/render_engine.py:185-225`
5. **Read `soundfont_bank`/`soundfont_program` from track in FluidSynth branch** ### Key Risk: `playNote` API Compatibility
- Currently read at lines 61-63 (before item loop) — ✅ correct
- Ensure `fl.program_select(midi_channel, fid, bank, program)` uses them (line 194) — ✅ correct
- **Files:** `app/core/render_engine.py` (verify only)
--- The spec's proposed API:
```javascript
### Task C — `synth_engine` Metadata Alignment playNote(pitch, velocity=0.8, durationSec=1.0, channel=0)
6. **`render_engine.py`: parse `synth_engine` object**
```python
se = track.get("synth_engine", {})
instrument_id = se.get("plugin_id") or track.get("instrument_id", "")
soundfont_bank = se.get("soundfont_bank") or track.get("soundfont_bank", 0)
soundfont_program = se.get("soundfont_program") or track.get("soundfont_program", 0)
instrument_source = se.get("type") or track.get("instrument_source", "soundfont")
```
- Apply before the item loop (around line 57-67)
- Keep flat fields as fallback for backward compat
- **Files:** `app/core/render_engine.py`
7. **`app.jsx`: write `synth_engine` alongside flat fields**
- In `setTrackInstrumentWithProgram()` (line 6490): add `synth_engine: {type, plugin_id, soundfont_bank, soundfont_program}`
- In `setTrackInstrument()` (line 6540): same
- Type logic:
- `instrumentId` starts with `sf_` → `type: "soundfont"`
- `instrumentId` is VST name → `type: "vst3"`
- `null` → `type: "default"`
- **Files:** `app/static/js/app.jsx:6490-6561`
8. **`aiGateway.js`: include `synth_engine` in track context**
- In `buildAIPromptContext()` (line 218): add `synth_engine` to track objects
- **Files:** `app/static/js/services/aiGateway.js:218-245`
---
### Task D — Graceful Fallback Chain (R5)
9. **`render_engine.py`: 3-level fallback for missing instruments**
- Level 1: Selected VST3/SoundFont
- Level 2: Default `GeneralUser_GS.sf2` (or first available `.sf2`)
- Level 3: Basic oscillator synth (`render_midi_events_to_audio`)
- **Files:** `app/core/render_engine.py:131-230`
---
### Task E — Client Playback Enhancement
10. **`soundfontPlayer.js`: read `synth_engine` from track context**
- `playNote()` accepts optional `synthEngine` param
- Before scheduling: call `controllerChange(ch, 0, bank)` and `programChange(ch, program)`
- Use existing oscillator ADSR emulation (no WASM SF2 — scope limit)
- **Files:** `app/static/js/services/soundfontPlayer.js:82-204`
11. **`app.jsx`: pass `synth_engine` to `SonicSF.playNote`**
- In `schedulePianoRollMidi()`: read `synth_engine` from active track, forward it
- **Files:** `app/static/js/app.jsx`
---
### Task F — AI Workflow Verification
12. **Catalog API test**
- `GET /api/v1/plugins/soundfonts/catalog` returns `{full_catalog, condensed_catalog}`
- Condensed ≤ 50 entries
- `window.__soundfontCatalog` populated on app load
13. **AI prompt injection test**
- `buildCatalogPromptSection()` generates catalog text with bank/program rules
- Submit: *"Compose 8 bars of Brass horns and a drum kit"*
- Verify returned JSON: Brass has `bank:0, program:56`, Drums has `bank:128, program:0`
---
### Task G — End-to-End Test
14. **Manual checklist**
- [ ] `docker compose up --build` succeeds
- [ ] Synth button shows "Synth" → click → dropdown lists SoundFonts + VSTs
- [ ] Select SoundFont → button label updates → instrument presets appear
- [ ] Select preset → label reflects exact instrument name
- [ ] Draw MIDI notes → Play → oscillator preview (approximate GM sound)
- [ ] Export WAV → file plays correct FluidSynth/VST3 instrument
- [ ] Select "None (Default Synth)" → oscillator fallback works
- [ ] AI generates track with instrument → export plays correct patch
- [ ] Upload new `.sf2` → appears in dropdown after refresh
---
### Task H — Install Missing Assets (if needed)
15. If `.sf2` absent: copy `GeneralUser_GS.sf2` or `SGM-V2.01.sf2` to `/home/locpham/daw_assets/soundfonts/`
16. If `.vst3` absent: place `DecentSampler.vst3` in `/home/locpham/daw_assets/vst3/`
17. If `.dspreset` absent: place Pianobook library in `/home/locpham/daw_assets/pianobook/`
---
## Affected Files
| File | Changes |
|------|---------|
| `app/core/render_engine.py` | Fix SF2 path resolution (Task B), parse `synth_engine` (Task C), 3-level fallback (Task D) |
| `app/static/js/app.jsx` | Write `synth_engine` in setTrackInstrument functions (Task C), pass to SoundFontPlayer (Task E) |
| `app/static/js/services/soundfontPlayer.js` | Accept `synthEngine` param, dispatch CC/program before note (Task E) |
| `app/static/js/services/aiGateway.js` | Include `synth_engine` in AI track context (Task C) |
| `app/api/v1/plugins.py` | No changes needed (cache invalidation already exists) |
| `app/core/vst_engine.py` | No changes needed (CWD swap already exists) |
| Host `/home/locpham/daw_assets/*` | `chmod 755`, ensure files exist |
## Validation
```bash
# 1. Build & boot
docker compose up --build -d
# 2. Verify deps
docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"
# 3. Catalog API
curl -s http://localhost:8000/api/v1/plugins/soundfonts/catalog | python -m json.tool | head -60
# 4. Render smoke test — create minimal project JSON and POST /api/v1/plugins/render
``` ```
## Rollback Current callers use:
```javascript
All changes backward-compatible (flat fields still work if `synth_engine` absent). playNote(pitch, velocity, durationMs, startTime, program, destinationNode, channel?, synthEngine?)
```bash
git checkout -- app/core/render_engine.py app/static/js/app.jsx \
app/static/js/services/soundfontPlayer.js app/static/js/services/aiGateway.js
``` ```
**Strategy**: Don't replace. Instead, wrap SpessaSynth inside the existing `SonicSF` object. Map:
- `velocity` (0–1) → MIDI velocity (1–127)
- `startTime` → if in future, use `setTimeout` for note scheduling (not perfect but adequate for preview)
- `destinationNode` → SpessaSynth routes to its own internal destination, but can add a gain node stage
- `program` → internal `_channels[channel].program` state (as now), but SpessaSynth also gets `programChange(ch, program)`
- `synthEngine` → load the right SF3 and set bank/program on SpessaSynth
**Backward compat**: Keep ALL existing methods. Add SpessaSynth as an optional enhanced engine. If SpessaSynth isn't loaded (CDN fails), fall back to oscillator emulation.
---
## 2. Implementation Plan
### Task 1 — Server: SF2→SF3 Conversion Pipeline
#### 1.1 Docker Dependency
- `Dockerfile`: Add `mscore` with `--no-install-recommends`
```dockerfile
RUN apt-get install -y --no-install-recommends mscore && rm -rf /var/lib/apt/lists/*
```
- Test: `docker compose build` — measure image size delta
#### 1.2 `app/core/soundfont_converter.py`
- `SoundFontConverter` class with:
- `convert_sf2_to_sf3(sf2_path)` → runs `mscore -o output.sf3 input.sf2`
- `batch_convert_all()` → walks `target_dirs`, converts missing `.sf3`
- `target_dirs` = `["/opt/daw_engine/soundfonts", "app/storage/uploads/soundfonts"]`
- Progress logging, error handling, cache validation (skip if `.sf3` newer than `.sf2`)
- Integrate with server startup:
- In `app/main.py` startup event, call `SoundFontConverter().batch_convert_all()` as background task (non-blocking, don't delay boot)
#### 1.3 API Download Endpoint
- `app/api/v1/plugins.py`: Add:
```python
@router.get("/soundfonts/download/{sf_id}")
async def download_soundfont_asset(sf_id: str):
# Search system_dir, upload_dir for .sf3 or .sf2
# Return FileResponse
# Fallback: .sf3 → .sf2 → 404
```
- Use `_find_sf2_path` from `render_engine.py` logic (case-insensitive, multi-dir)
- Set `media_type="application/octet-stream"` and proper `Content-Disposition`
#### 1.4 Update `SoundFontInspector` to Also Scan `.sf3`
- `soundfont_inspector.py:80`: Change `.sf2` filter to `('.sf2', '.sf3')`
- Ensure the catalog includes `.sf3` files as available instruments
---
### Task 2 — Client: SpessaSynth Integration
#### 2.1 `index.html` — Add SpessaSynth CDN
```html
<script type="module">
import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
window.SpessaSynthClass = Synthesizer;
</script>
```
- Add BEFORE `soundfontPlayer.js` so the class is available when the player initializes
#### 2.2 `app/static/js/services/soundfontStorage.js` — IndexedDB Cache
- `SoundFontStorage` class with `openDB()`, `getBuffer(sfId)`, `saveBuffer(sfId, arrayBuffer)`
- Uses `indexedDB` with DB name `"DAW_SoundFont_Cache"`, store name `"sf3_buffers"`
- Export singleton `sfStorage`
#### 2.3 `app/static/js/services/soundfontPlayer.js` — Dual-Mode Rewrite
**Architecture**: Keep the existing `window.SonicSF` as the public API. Internally use SpessaSynth when available, fall back to oscillator when not.
**Changes to existing methods:**
| Method | Change |
|---|---|
| `init(audioCtx)` | NEW — creates SpessaSynth instance, triggers default SF load |
| `loadSoundFont(sfId)` | REWORK — try IndexedDB → fetch `/api/v1/plugins/soundfonts/download/{sfId}` → load into SpessaSynth |
| `playNote(...)` | ADD SpessaSynth path: if initialized, delegate to `synthInstance.noteOn/noteOff`; else use oscillator fallback |
| `applyAITrackInstrument(bank, program, synthEngine?)` | ADD SpessaSynth path: call `controllerChange` + `programChange` on SpessaSynth |
| `stopAll()` | ADD SpessaSynth: `allNotesOff(channel)` or `programReset()` |
| `controllerChange(ch, cc, val)` | ADD SpessaSynth: delegate if initialized |
| `programChange(ch, prog)` | ADD SpessaSynth: delegate if initialized |
**Backward compat guarantee:**
- All existing callers continue to work unchanged
- 6-arg `playNote(pitch, vel, durMs, startTime, program, destNode)` → SpessaSynth ignores `startTime` (schedules immediately) and `destNode` (uses internal routing)
- 8-arg `playNote(..., channel, synthEngine)` → SpessaSynth uses `synthEngine.soundfont_id` for SF loading, `synthEngine.soundfont_program` for program selection
- If SpessaSynth not loaded/CDN fails → transparent fallback to existing oscillator code
#### 2.4 `app.jsx` — Integration Points
- **Startup** (in a useEffect or the existing audio context initialization):
```javascript
if (window.SonicSF && window.SonicSF.init) {
SonicSF.init(getAudioContext());
}
```
- **Synth selection**: After `setTrackInstrumentWithProgram` / `setTrackInstrument`, trigger SF load:
```javascript
if (synthEngine && synthEngine.type === 'soundfont') {
SonicSF.loadSoundFont(synthEngine.soundfont_id);
}
```
- **AI track creation**: Already calls `applyAITrackInstrument` — SpessaSynth path handles it
---
### Task 3 — Backward Compatibility & Migration
#### 3.1 Fallback Behavior
- If `window.SpessaSynthClass` is undefined (CDN blocked, offline): fall back to existing oscillator code
- If `.sf3` download fails: fall back to oscillator
- If SpessaSynth throws: catch error, log warning, fall back to oscillator
#### 3.2 Testing Matrix
| Scenario | Expected |
|---|---|
| SpessaSynth loaded + SF3 cached | Authentic playback, 0ms load |
| SpessaSynth loaded + SF3 needs download | Authentic playback after 1-2s load |
| CDN blocked (offline) | Transparent oscillator fallback |
| SF3 not available on server | Transparent oscillator fallback |
| MIDI keyboard + armed track | Authentic or oscillator based on availability |
---
## 3. Files Affected
| File | Change |
|---|---|
| `Dockerfile` | Add `mscore --no-install-recommends` |
| `app/core/soundfont_converter.py` | **NEW** — SF2→SF3 convert + batch scan |
| `app/api/v1/plugins.py` | Add `GET /soundfonts/download/{sf_id}` |
| `app/core/soundfont_inspector.py` | Accept `.sf3` in scan filter |
| `app/main.py` | Add startup background conversion |
| `app/templates/index.html` | Add SpessaSynth CDN `<script type="module">` |
| `app/static/js/services/soundfontStorage.js` | **NEW** — IndexedDB cache |
| `app/static/js/services/soundfontPlayer.js` | Dual-mode rewrite (SpessaSynth + oscillator fallback) |
| `app/static/js/app.jsx` | Add init + SF load calls |
## 4. Validation
```bash
# Server-side
docker compose build # verify mscore installs
docker compose exec web python -c "from app.core.soundfont_converter import SoundFontConverter; print('OK')"
curl -s http://localhost:8000/api/v1/plugins/soundfonts/download/sgm_v2.01 | head -c 4 | file -
docker compose exec web ls -la /opt/daw_engine/soundfonts/*.sf3 # verify conversion
# Client-side
# Open browser → DevTools → check SpessaSynth loaded (window.SpessaSynthClass)
# Select SoundFont → check network tab for .sf3 download
# Play MIDI notes → hear authentic instrument (not oscillator)
```
## 5. Open Questions
1. **`mscore` image size**: Measure actual delta. If >300MB, consider `fluidsynth --convert` alternative.
2. **CDN reliability**: SpessaSynth loaded from jsdelivr CDN — consider bundling or NPM install as fallback.
3. **Startup delay**: `batch_convert_all()` could take minutes for large SF2s. Run as background Celery task, not inline startup.
4. **SGM_v2.01.sf2 (529MB)**: Conversion time for this file. Need to handle gracefully (streaming, timeout).
app/storage/sonicforge.db
BIN
View File
Binary file not shown.
md/37_FIXVST3_SUGGEST.md
+86
View File
@@ -0,0 +1,86 @@
Nguyên nhân xuất hiện thông báo lỗi từ SpessaSynth Core:
`basic_synthesizer_core.ts:169 No preset found for 0:0:0! Did you forget to add a sound bank?`
Thông số `0:0:0` trong thông báo đại diện cho `Bank MSB : Bank LSB : Program Number` (cấu hình mặc định ban đầu của kênh MIDI). Lỗi này xảy ra do **3 nguyên nhân chính** sau:
---
### 1. Phân tích nguyên nhân kỹ thuật
1. **Chưa gửi lệnh `programChange` & `controllerChange` sang SpessaSynth khi chọn "Pipa"**:
* Khi chọn nhạc cụ "Pipa" trên UI, ứng dụng mới chỉ cập nhật giao diện mà **chưa kích hoạt lệnh đổi tiếng** (`synthInstance.programChange(channel, program)`) sang kênh MIDI tương ứng của SpessaSynth.
* Khi gõ phím trên `SE49` (nhận dữ liệu Raw MIDI `[144, 65, 78]`), SpessaSynth vẫn truy xuất bộ tiếng ở vị trí mặc định là `Bank 0 : Program 0` (thường là Grand Piano theo chuẩn General MIDI).
* Trong SoundFont `DSK_Asian_DreamZ`, vị trí `0:0:0` không tồn tại hoặc không chứa thông tin nốt, khiến SpessaSynth thông báo không tìm thấy preset.
2. **Sai chỉ số Bank / Program của nhạc cụ "Pipa"**:
* Tệp SoundFont `DSK_Asian_DreamZ` là bộ tiếng tùy chỉnh (Non-General MIDI). 8 nhạc cụ bên trong (*Pipa, Pipa Tremolo, Luan, Guzhen, Erhu, Koto, Ban Di, Percussion*) được đánh số `Program` riêng biệt.
* Nếu UI gửi `program = 0` nhưng thực tế trong file SoundFont, tiếng Pipa nằm ở `program = 1` hoặc `bank = 1`, SpessaSynth sẽ không tìm thấy preset tương ứng.
3. **Lỗi Header/Mapping khi convert từ `.sf2` sang `.sf3**`:
* Một số công cụ nén (như `mscore` CLI) khi convert các SoundFont tùy chỉnh có cấu trúc phi chuẩn có thể làm mất hoặc biến đổi bảng thông tin Preset Header.
---
### 2. Các bước khắc phục
#### **Bước 1: Gọi `programChange` & `controllerChange` ngay khi chọn nhạc cụ trên UI**
Đảm bảo khi người dùng chọn nhạc cụ trên giao diện, hàm chọn Bank/Program được kích hoạt trên kênh MIDI phát tiếng:
```javascript
// Khi người dùng chọn "Pipa" trên Dropdown Synth UI
async function onSelectTrackInstrument(trackChannel, sfId, bank, program) {
// 1. Nạp file .sf3 vào SpessaSynth (nếu chưa nạp)
await soundFontPlayerInstance.loadSoundFont(sfId);
// 2. BẮT BUỘC: Gửi lệnh đổi Bank (CC 0) và Program Change sang SpessaSynth
if (soundFontPlayerInstance.synthInstance) {
soundFontPlayerInstance.synthInstance.controllerChange(trackChannel, 0, bank);
soundFontPlayerInstance.synthInstance.programChange(trackChannel, program);
console.log(`[SonicSF] Switched Channel ${trackChannel} -> Bank: ${bank}, Program: ${program}`);
}
}
```
#### **Bước 2: Kiểm tra chính xác chỉ số Bank & Program của Pipa từ Catalog API**
Sử dụng API `GET /api/v1/plugins/soundfonts/catalog` (từ mô-đun `SoundFontInspector` đã xây dựng) để tra cứu vị trí chính xác của "Pipa":
```json
"dsk_asian_dreamz": {
"soundfont_id": "dsk_asian_dreamz",
"instruments": [
{ "bank": 0, "program": 0, "name": "Pipa" },
{ "bank": 0, "program": 1, "name": "Pipa Tremolo" },
{ "bank": 0, "program": 6, "name": "Ban Di" }
]
}
```
*Lưu ý:* Nếu kết quả trả về tiếng Pipa nằm ở `program: 1` hoặc `bank: 1`, hãy truyền đúng thông số này vào hàm `programChange`.
#### **Bước 3: Kiểm tra danh sách Presets mà SpessaSynth đọc được từ file `.sf3**`
Để đảm bảo quá trình convert `.sf3` không làm hỏng dữ liệu Preset Header, bạn hãy log danh sách preset sau khi nạp tệp:
```javascript
// Thêm log kiểm tra sau khi addSoundFont vào SpessaSynth
try {
await this.synthInstance.soundFontManager.addSoundFont(buffer);
// In danh sách các preset đọc được ra console để kiểm tra
const loadedSF = this.synthInstance.soundFontManager.soundFonts[0];
console.log("[SonicSF] Loaded Presets in SF3:", loadedSF.presets);
} catch (e) {
console.error("[SonicSF] Error parsing SF3:", e);
}
```
* Nếu `loadedSF.presets` rỗng (`[]`), file `.sf3` đã bị hỏng khi nén. Bạn hãy thử nạp lại file `.sf2` gốc chưa nén để đối chiếu.
md/38_FLOWCLIENT.md
+144
View File
@@ -0,0 +1,144 @@
# CLIENT-SIDE EXECUTION FLOW (SF3 + SPESSASYNTH + INDEXEDDB)
This document describes the step-by-step processing chain that takes place inside the Client Browser, from launching the Web DAW application, downloading and buffering `.sf3` instrument files, and setting up MIDI channels, to outputting real-time audio.
---
## 1. SEQUENCE DIAGRAM
```text
[ USER / UI ] [ APP / CLIENT ] [ INDEXEDDB ] [ SERVER API ] [ SPESSASYNTH ENGINE ]
| | | | |
1. Open Web Page ---------> | Initial AudioCtx | | |
| | Init SpessaSynth -----------------------------------------------> | Connect Destination
| | Fetch Catalog --------------------------->| Get /catalog |
| | | | |
2. Select Instrument -----> | Read (sf_id, bank, prog) | |
(e.g., Pipa) | Query SF3 Buffer --->| Check Key (sf_id) | |
| | | -- (Miss) -------->| Fetch /download/sf_id |
| | | | Return .sf3 (~4MB) |
| | <--------------------| Save ArrayBuffer --| |
| | Load SF3 Memory ------------------------------------------------> | addSoundFont(buffer)
| | | | |
3. Channel Router --------> | Switch Bank/Program -------------------------------------------> | controllerChange(ch, 0, bank)
| | | | | programChange(ch, prog)
| | | | |
4. Trigger MIDI Key ------> | Raw MIDI Event | | |
(or Timeline Play) | (noteOn: pitch, vel) -------------------------------------------> | noteOn(ch, pitch, vel)
| | | | | AudioWorklet Synthesis
| | <------------------------------------------------------------------ | Audio Out (User Speakers)
```
---
## 2. DETAILED PROCESSING PHASES
### PHASE 1: BOOTSTRAPPING & ENGINE INIT
* **Web Audio Context Initialization:** Upon the user's first interaction with the web page (Mouse Click/Keypress), the application initializes the `AudioContext`.
* **SpessaSynth Synthesizer Initialization:** The `soundfontPlayer.js` module instantiates `SpessaSynthClass` and connects its output directly to `audioCtx.destination`:
```javascript
this.synthInstance = new window.SpessaSynthClass(this.audioCtx.destination);
```
* **Instrument Catalog Load (Catalog Context):** The Frontend dispatches a `GET /api/v1/plugins/soundfonts/catalog` request to load the `condensed_catalog`, which contains lookup tables for `sf_id`, `bank`, and `program`.
---
### PHASE 2: `.SF3` ASSET LOADING & CACHING
Triggered when a user selects an instrument via the Synth UI button (or when the AI Copilot spawns a new Track with a designated instrument, e.g., `dsk_asian_dreamz`):
* **Query Browser Cache (IndexedDB):** The Client calls `sfStorage.getBuffer(sfId)` to search for the `.sf3` file's `ArrayBuffer` inside the `DAW_SoundFont_Cache` database.
* **Handling Cache Hit vs Cache Miss:**
* **Cache Hit ($0\text{ms}$):** Retrieves the `ArrayBuffer` directly from the browser's RAM/Storage.
* **Cache Miss:**
1. Sends a `GET /api/v1/plugins/soundfonts/download/{sf_id}` request to the Server.
2. Downloads the compressed, optimized `.sf3` asset (ultra-lightweight size $\sim 3.5 - 5.5\text{ MB}$).
3. Invokes `sfStorage.saveBuffer(sfId, arrayBuffer)` to store it inside IndexedDB for subsequent visits.
* **Load Data into SpessaSynth Wasm/JS Memory:** Passes the `ArrayBuffer` to SpessaSynth Engine's `SoundFontManager`:
```javascript
await this.synthInstance.soundFontManager.addSoundFont(buffer);
```
---
### PHASE 3: BANK/PROGRAM ROUTING & MIDI CHANNEL SETUP
This is the most critical phase to resolve `No preset found for 0:0:0` errors.
* **MIDI Channel Assignment:**
* **Melodic Instruments (Piano, Pipa, Strings, Brass, etc.):** Allocated to Channels 0 through 8.
* **Percussion / Drum Kits (Bank 128):** Mandatory allocation to Channel 9 (GM Standard Channel 10).
* **Dispatch Bank Select & Program Change to SpessaSynth Engine:** Prior to scheduling any note events, the Client triggers two simultaneous control events:
```javascript
// 1. Select Bank (Control Change 0)
this.synthInstance.controllerChange(channel, 0, bank);
// 2. Select Program (Program Change)
this.synthInstance.programChange(channel, program);
```
*Example for Pipa (`dsk_asian_dreamz`):* Calls `controllerChange(0, 0, 0)` and `programChange(0, 0)`. SpessaSynth switches Channel 0's state to the Pipa instrument patch.
---
### PHASE 4: REALTIME SYNTHESIS & AUDIO OUTPUT
Triggered when receiving note-control signals (from a Hardware MIDI Keyboard or Timeline Transport Playback):
* **Scenario A: User plays a Hardware MIDI Keyboard (e.g., Nektar SE49)**
1. The browser receives a Raw MIDI Event: Web MIDI API captures message `[144, 65, 78]` (`NoteOn`, `Pitch 65`, `Velocity 78`).
2. **Latency Compensation:** Calculates real-time offsets and issues `NoteOn` to SpessaSynth:
```javascript
const midiPitch = pitch;
const midiVelocity = Math.floor(velocity * 127);
this.synthInstance.noteOn(channel, midiPitch, midiVelocity);
```
3. **Key Release:** Triggers a `NoteOff` event:
```javascript
this.synthInstance.noteOff(channel, midiPitch);
```
* **Scenario B: User triggers Play on Timeline / Piano Roll**
1. **Transport Controller & Scheduler (`PrecisionAudioScheduler`):** Scans for MIDI notes located within the moving Playhead window.
2. **Note Scheduling:**
* Converts beat positions to precise audio timing based on BPM tempo (`exactAudioTime`).
* Dispatches `noteOn(channel, pitch, velocity)` at the exact timestamp $T_{\text{start}}$.
* Dispatches `noteOff(channel, pitch)` at timestamp $T_{\text{start}} + T_{\text{duration}}$.
* **Audio Worklet Audio Rendering:** SpessaSynth Engine reads Ogg/WAV sample data inside the `.sf3` asset, applies Envelopes (ADSR), Modulators, and Gain Control parameters on the designated Channel, and pushes PCM audio data directly to user speakers with $0\text{ms}$ latency.
---
## 3. 100% RELIABILITY VERIFICATION CHECKLIST
* [ ] `.sf3` files loaded into the browser open without triggering `Corrupted File` errors.
* [ ] The `sfStorage.getBuffer` function successfully stores and retrieves `ArrayBuffer` data from IndexedDB.
* [ ] Both `controllerChange(channel, 0, bank)` and `programChange(channel, program)` are invoked immediately upon changing instruments on the UI.
* [ ] Percussion/Drum instruments are persistently allocated to Channel 9.
* [ ] Console logs confirm: `[SonicSF] Switched Channel X -> Bank: B, Program: P`.