3dtours/SonicForgeStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: sử dụng synth cho MIDI KEys

Browse Source
This commit is contained in:
3dtours
2026-07-26 17:36:36 +07:00
parent bfd7146bfc
commit 44e0a6d736
5 changed files with 641 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kilo/plans/1785058528193-vst3-soundfont-engine-plan.md
+181
View File
@@ -0,0 +1,181 @@
# VST3 / SoundFont Engine — Implementation Plan
## Current State vs Requirements
| Check | Status | Ref |
|-------|--------|-----|
| FluidSynth Channel 9 for Drums (bank=128) | ✅ Already implemented. `render_engine.py:64` forces `midi_channel=9` when `is_percussion` or `bank==128`. | R1 verify |
| Catalog cache invalidation on SF upload | ✅ Already implemented. `plugins.py:93-95` calls `invalidate_catalog_cache()`. | R2 verify |
| DecentSampler CWD swap for `.wav` samples | ✅ Already implemented. `vst_engine.py:338-341` does `os.chdir(preset_dir)`. | R3 verify |
| FluidSynth SF2 path uses `static/soundfonts/` hardcoded | ❌ `render_engine.py:186-189` hardcodes `static/soundfonts/<id>.sf2`. Must scan system + upload dirs. | **Bug** |
| SF2 upload dir (`app/storage/uploads/soundfonts/`) not searched in render_engine | ❌ FluidSynth render branch never looks at user uploads. | **Bug** |
| `synth_engine` structured metadata not parsed | ❌ Render path uses flat fields only. Spec uses `{type, plugin_id, bank, program}`. | **Gap** |
| Client preview still oscillator-emulated | ⚠️ Acceptable per spec: "Wasm Module / Preview Synth" — no real SF2 in browser. `soundfontPlayer.js` reads flat fields, not `synth_engine`. | **Gap** |
| Host asset permissions might deny Docker | ⚠️ Non-root users may get `Permission Denied` on mounted `.vst3`/`.sf2`. | R4 |
| Missing VST3/SF2 fallback chain | ⚠️ No graceful degraded path if selected instrument is absent. | R5 |
---
## Tasks
### Task A — Environment Validation
1. **Host asset check + permissions**
- Verify `ls /home/locpham/daw_assets/{vst3,soundfonts,pianobook}/*` returns files
- `chmod -R 755 /home/locpham/daw_assets`
- **Files:** host paths only
2. **Docker compose mount verification**
- `docker-compose.yml:13-15` maps:
- `vst3` → `/opt/daw_engine/vst3`
- `soundfonts` → `/opt/daw_engine/soundfonts`
- `pianobook` → `/opt/daw_engine/samples/pianobook`
3. **Runtime Python deps verification**
- `docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"`
- Check server logs for `HAS_PEDALBOARD`, `HAS_PYFLUIDSYNTH` flags in `vst_engine.py`
---
### Task B — Fix SF2 Path Resolution (Bugfix)
4. **`render_engine.py` FluidSynth branch** — replace hardcoded `static/soundfonts/` path
- **Current** (line 185-189): builds path to `app/static/soundfonts/<sf_id>.sf2` only
- **Target**: search in order:
1. `UPLOAD_SF_DIR` = `app/storage/uploads/soundfonts/<id>.sf2`
2. `SYSTEM_SF_DIR` = `/opt/daw_engine/soundfonts/<id>.sf2`
3. Match by `soundfont_id` field in track metadata (not just filename)
- Accept both `sf_<id>` and bare `<id>` in `instrument_id`
- **Files:** `app/core/render_engine.py:185-225`
5. **Read `soundfont_bank`/`soundfont_program` from track in FluidSynth branch**
- Currently read at lines 61-63 (before item loop) — ✅ correct
- Ensure `fl.program_select(midi_channel, fid, bank, program)` uses them (line 194) — ✅ correct
- **Files:** `app/core/render_engine.py` (verify only)
---
### Task C — `synth_engine` Metadata Alignment
6. **`render_engine.py`: parse `synth_engine` object**
```python
se = track.get("synth_engine", {})
instrument_id = se.get("plugin_id") or track.get("instrument_id", "")
soundfont_bank = se.get("soundfont_bank") or track.get("soundfont_bank", 0)
soundfont_program = se.get("soundfont_program") or track.get("soundfont_program", 0)
instrument_source = se.get("type") or track.get("instrument_source", "soundfont")
```
- Apply before the item loop (around line 57-67)
- Keep flat fields as fallback for backward compat
- **Files:** `app/core/render_engine.py`
7. **`app.jsx`: write `synth_engine` alongside flat fields**
- In `setTrackInstrumentWithProgram()` (line 6490): add `synth_engine: {type, plugin_id, soundfont_bank, soundfont_program}`
- In `setTrackInstrument()` (line 6540): same
- Type logic:
- `instrumentId` starts with `sf_` → `type: "soundfont"`
- `instrumentId` is VST name → `type: "vst3"`
- `null` → `type: "default"`
- **Files:** `app/static/js/app.jsx:6490-6561`
8. **`aiGateway.js`: include `synth_engine` in track context**
- In `buildAIPromptContext()` (line 218): add `synth_engine` to track objects
- **Files:** `app/static/js/services/aiGateway.js:218-245`
---
### Task D — Graceful Fallback Chain (R5)
9. **`render_engine.py`: 3-level fallback for missing instruments**
- Level 1: Selected VST3/SoundFont
- Level 2: Default `GeneralUser_GS.sf2` (or first available `.sf2`)
- Level 3: Basic oscillator synth (`render_midi_events_to_audio`)
- **Files:** `app/core/render_engine.py:131-230`
---
### Task E — Client Playback Enhancement
10. **`soundfontPlayer.js`: read `synth_engine` from track context**
- `playNote()` accepts optional `synthEngine` param
- Before scheduling: call `controllerChange(ch, 0, bank)` and `programChange(ch, program)`
- Use existing oscillator ADSR emulation (no WASM SF2 — scope limit)
- **Files:** `app/static/js/services/soundfontPlayer.js:82-204`
11. **`app.jsx`: pass `synth_engine` to `SonicSF.playNote`**
- In `schedulePianoRollMidi()`: read `synth_engine` from active track, forward it
- **Files:** `app/static/js/app.jsx`
---
### Task F — AI Workflow Verification
12. **Catalog API test**
- `GET /api/v1/plugins/soundfonts/catalog` returns `{full_catalog, condensed_catalog}`
- Condensed ≤ 50 entries
- `window.__soundfontCatalog` populated on app load
13. **AI prompt injection test**
- `buildCatalogPromptSection()` generates catalog text with bank/program rules
- Submit: *"Compose 8 bars of Brass horns and a drum kit"*
- Verify returned JSON: Brass has `bank:0, program:56`, Drums has `bank:128, program:0`
---
### Task G — End-to-End Test
14. **Manual checklist**
- [ ] `docker compose up --build` succeeds
- [ ] Synth button shows "Synth" → click → dropdown lists SoundFonts + VSTs
- [ ] Select SoundFont → button label updates → instrument presets appear
- [ ] Select preset → label reflects exact instrument name
- [ ] Draw MIDI notes → Play → oscillator preview (approximate GM sound)
- [ ] Export WAV → file plays correct FluidSynth/VST3 instrument
- [ ] Select "None (Default Synth)" → oscillator fallback works
- [ ] AI generates track with instrument → export plays correct patch
- [ ] Upload new `.sf2` → appears in dropdown after refresh
---
### Task H — Install Missing Assets (if needed)
15. If `.sf2` absent: copy `GeneralUser_GS.sf2` or `SGM-V2.01.sf2` to `/home/locpham/daw_assets/soundfonts/`
16. If `.vst3` absent: place `DecentSampler.vst3` in `/home/locpham/daw_assets/vst3/`
17. If `.dspreset` absent: place Pianobook library in `/home/locpham/daw_assets/pianobook/`
---
## Affected Files
| File | Changes |
|------|---------|
| `app/core/render_engine.py` | Fix SF2 path resolution (Task B), parse `synth_engine` (Task C), 3-level fallback (Task D) |
| `app/static/js/app.jsx` | Write `synth_engine` in setTrackInstrument functions (Task C), pass to SoundFontPlayer (Task E) |
| `app/static/js/services/soundfontPlayer.js` | Accept `synthEngine` param, dispatch CC/program before note (Task E) |
| `app/static/js/services/aiGateway.js` | Include `synth_engine` in AI track context (Task C) |
| `app/api/v1/plugins.py` | No changes needed (cache invalidation already exists) |
| `app/core/vst_engine.py` | No changes needed (CWD swap already exists) |
| Host `/home/locpham/daw_assets/*` | `chmod 755`, ensure files exist |
## Validation
```bash
# 1. Build & boot
docker compose up --build -d
# 2. Verify deps
docker compose exec web python -c "import pedalboard, fluidsynth; from sf2utils.sf2parse import Sf2File; print('OK')"
# 3. Catalog API
curl -s http://localhost:8000/api/v1/plugins/soundfonts/catalog | python -m json.tool | head -60
# 4. Render smoke test — create minimal project JSON and POST /api/v1/plugins/render
```
## Rollback
All changes backward-compatible (flat fields still work if `synth_engine` absent).
```bash
git checkout -- app/core/render_engine.py app/static/js/app.jsx \
app/static/js/services/soundfontPlayer.js app/static/js/services/aiGateway.js
```
app/static/js/app.precompiled.js
+10 -10
View File
File diff suppressed because one or more lines are too long
app/storage/sonicforge.db
BIN
View File
Binary file not shown.
md/36_USE_VST3.md
+128
View File
@@ -0,0 +1,128 @@
# OPERATION GUIDE & AUDIO PLAYBACK WORKFLOW FOR MIDI TRACKS (SOUNDFONT / VST3)
This document describes in detail the user interface interaction workflow when using the Synth button and explains the underlying technical architecture required for MIDI Notes on a Track to output audio via a selected SoundFont or VST3 Plugin.
---
## 1. User Interface Workflow Description
### Activating the Instrument Selection Menu
* On the Track Control Panel (the left-side pane of Track 01), the user clicks the **🎵 Synth: BAN-DI** button (or the orange Synth button below it).
* A dropdown selection menu appears directly underneath the button.
### Instrument Menu Layout
* **None (Default Synth):** Uses the application's default synthesizer (a simple Oscillator Synth).
* **SOUNDFONTS:** Displays a list of SoundFont (`.sf2`) soundbanks loaded into the system (e.g., `SoundFont_DSK_Asia`, `SoundFont_SGM_v2`, `weedsgm3`).
* **VST INSTRUMENTS:** Displays a list of native 64-bit Linux VST3 Plugins (e.g., `Vital`, `DecentSampler`, `libSurge XT`).
### Selection & State Update Operations
* The user clicks to select an instrument (e.g., selecting `SoundFont_DSK_Asia` or `Vital`).
* The menu closes, and the button label updates to reflect the chosen instrument (e.g., **🎵 DSK_Asia** or **🎵 Vital**).
* The instrument configuration payload is directly assigned to the Track State object (`session.tracks[0].synth_engine`).
---
## 2. Technical Execution Flow for MIDI Note Audio Output (SoundFont / VST3)
To ensure that the purple MIDI note bars on the Timeline or Piano Roll play back audio accurately using the chosen instrument, the system processes tasks across two primary workflows:
```text
+------------------------------------+
| User selects SoundFont / VST3 |
+-----------------+------------------+
|
+--------------------------+--------------------------+
| |
v v
[ 1. Real-time Client Preview ] [ 2. Server-side Offline Export ]
(Audio Playback in Browser) (High-Quality WAV Rendering)
| |
+--------------------+--------------------+ +------------+------------+
| | | |
v v v v
(If SoundFont) (If VST3) (If SoundFont) (If VST3)
FluidSynth Wasm / Load Wasm Module / PyFluidSynth C-API Python Pedalboard
SoundfontPlayer.js AudioWorklet Synth Dispatches Bank/Program Loads .vst3 binary
Dispatches programChange Preview Synth Renders Audio Buffer Renders PCM Buffer
| | | |
+-----------------+-----------------+ +------------+------------+
| |
v v
AudioContext Destination Audio Export Output File
(User Speakers) (Downloaded WAV File)
```
### A. Real-time Client Playback (Browser Audio Preview)
When the user clicks the Play button or clicks a key on the Piano Roll:
1. **Audio Routing Update:**
* The client reads the instrument parameters from `track.synth_engine`.
* **If SoundFont (`.sf2`) is selected:** The client dispatches `controllerChange(channel, 0, bank)` and `programChange(channel, program)` configuration calls to the `soundfontPlayer.js` module (running FluidSynth WebAssembly).
* **If VST3 Plugin (`Vital`, `DecentSampler`, etc.) is selected:** Because browsers cannot natively run Linux `.vst3`/`.so` binary executables directly, the client uses an equivalent WebAssembly Synth or Preview Synth to output real-time audio with $0\text{ ms}$ latency.
2. **Note Scheduling:**
* The Transport driver (`PrecisionAudioScheduler`) scans for MIDI notes located within the moving Playhead range.
* Each MIDI note includes: `pitch` (0–127), `start_beat` (start position), `duration_beats` (length), and `velocity` (keypress intensity 0.0–1.0).
* The scheduler converts beat timing to absolute time in seconds (`exactAudioTime`) and dispatches audio events:
* `noteOn(pitch, velocity, exactAudioTime)`
* `noteOff(pitch, exactAudioTime + durationSec)`
* Audio signals generated by the WebAssembly Engine travel through `Track Gain Node` $\rightarrow$ `Track Pan Node` $\rightarrow$ `Master Bus` $\rightarrow$ `AudioContext.destination` (User Speakers).
### B. Server-side Offline Render (High-Quality WAV Export)
When the user exports a track (Bounce Track / Export WAV), the Python Backend on the server receives the project's JSON payload:
1. **Reading Track Instrument Metadata:**
```json
{
"track_id": "track_01",
"synth_engine": {
"type": "VST3",
"plugin_id": "Vital",
"soundfont_bank": 0,
"soundfont_program": 0
}
}
```
2. **Rendering SoundFont (`.sf2`) Instruments:**
* `render_engine.py` initializes a FluidSynth instance.
* Calls `fl.program_select(channel, sf_id, bank, program)`.
* Feeds the list of MIDI notes directly to FluidSynth to render an Audio Buffer.
3. **Rendering VST3 (`.vst3`) Instruments:**
* `vst_engine.py` invokes `pedalboard.VST3Plugin("/opt/daw_engine/vst3/Vital.vst3")`.
* If DecentSampler is selected, it loads the corresponding Pianobook sample preset file (`.dspreset`).
* Converts all MIDI Notes into an array of `pedalboard.Message` events:
* Inserts `control_change` (Bank Select) and `program_change` events at timestamp $0.0\text{ s}$.
* Inserts `note_on` and `note_off` events matching the pitch and duration parameters of each note.
* Feeds the MIDI message stream into the VST3 instance to generate a high-fidelity Float32 PCM audio stream.
* Mixes down the Track PCM Audio Buffers into the Master Mix and creates the final `.wav` output file.
---
## 3. Instrument Selection Checklist
To ensure that selecting an instrument via the Synth button produces audio output successfully:
* [ ] **Track is Unmuted:** Verify that the Mute button `[M]` is not active (orange/red) and that the Solo button `[S]` on other tracks is not muting the current track.
* [ ] **MIDI Notes in Valid Key Range:** Some instruments (such as Bass or Horns) operate within constrained pitch boundaries (e.g., C1 to C5). Ensure the notes drawn on the Piano Roll fall within the playable range of the selected SoundFont or VST3 instrument.
* [ ] **VST3 / SoundFont Files Ready on Server:** Confirm that the `.vst3` binary files are placed inside `/opt/daw_engine/vst3/` and `.sf2` files are present in `/opt/daw_engine/soundfonts/`.
* [ ] **Appropriate Volume / Gain Settings:** Verify that the Track 01 Volume slider is configured to $0\text{ dB}$ to avoid signal clipping or silent playback.
md/37_SF_CONVERT.md
+322
View File
@@ -0,0 +1,322 @@
# TECHNICAL SPECIFICATION: CLIENT SOUNDFONT OPTIMIZATION USING SF3 AND SPESSASYNTH
This document details a two-step technical workflow to upgrade the real-time client audio playback experience:
1. **Server Asset Conversion:** Converts original `.sf2` files into compressed `.sf3` (Ogg Vorbis) format, reducing file size from $30 - 150\text{ MB}$ down to just $3 - 6\text{ MB}$ ($\sim 85-90\%$ compression).
2. **Client Engine Upgrade:** Replaces the oscillator emulation logic inside `soundfontPlayer.js` with the SpessaSynth library (Web Audio API / AudioWorklet Engine), achieving $100\%$ authentic audio rendering relative to the server exporter with initial load times of only $1 - 2\text{ seconds}$.
---
## STEP 1: AUTOMATED SF2 TO SF3 ASSET CONVERSION ON SERVER
### 1.1 Technical Principles of the `.sf3` Format
* `.sf2` files store raw uncompressed PCM Float/Int audio samples (Raw Uncompressed Audio).
* `.sf3` files preserve the complete Header, Preset, and Instrument Mapping structure of SF2, but compress raw WAV sample streams using the Ogg Vorbis compression algorithm.
* Human ears cannot distinguish quality differences between `.sf2` and `.sf3` playback, but the reduced footprint ensures exceptionally fast browser downloads.
### 1.2 Installing Conversion Utilities in Server Docker (`Dockerfile`)
Append `mscore` (MuseScore CLI) or `sf2pack` packages to the `Dockerfile`:
```dockerfile
# Dockerfile
RUN apt-get update && apt-get install -y \
mscore \
vorbis-tools \
&& rm -rf /var/lib/apt/lists/*
```
### 1.3 Python Automated SoundFont Converter Module (`app/core/soundfont_converter.py`)
Creates a Python module to automatically scan `.sf2` files within system/upload directories and generate parallel `.sf3` converted files:
```python
import os
import subprocess
import logging
logger = logging.getLogger(__name__)
class SoundFontConverter:
def __init__(self, target_dirs=None):
if target_dirs is None:
self.target_dirs = [
"/opt/daw_engine/soundfonts",
"app/storage/uploads/soundfonts"
]
else:
self.target_dirs = target_dirs
def convert_sf2_to_sf3(self, sf2_path: str) -> str:
"""
Converts a single .sf2 file to .sf3 using MuseScore CLI.
Returns the path to the converted .sf3 file.
"""
if not os.path.exists(sf2_path):
raise FileNotFoundError(f"Source SF2 file not found: {sf2_path}")
sf3_path = os.path.splitext(sf2_path)[0] + ".sf3"
# Check if already converted and up-to-date
if os.path.exists(sf3_path) and os.path.getmtime(sf3_path) >= os.path.getmtime(sf2_path):
return sf3_path
try:
logger.info(f"Converting '{sf2_path}' -> '{sf3_path}'...")
# Command: mscore -o output.sf3 input.sf2
cmd = ["mscore", "-o", sf3_path, sf2_path]
result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
if result.returncode == 0 and os.path.exists(sf3_path):
logger.info(f"Successfully created SF3 asset: {sf3_path} ({os.path.getsize(sf3_path) / (1024*1024):.2f} MB)")
return sf3_path
else:
logger.error(f"SF2 to SF3 conversion failed: {result.stderr}")
return sf2_path # Fallback to original SF2
except Exception as e:
logger.error(f"Error executing SF2 conversion: {str(e)}")
return sf2_path
def batch_convert_all(self):
"""
Scans all target directories and converts any missing .sf3 files.
"""
for sdir in self.target_dirs:
if not os.path.exists(sdir):
continue
for root, _, files in os.walk(sdir):
for file in files:
if file.lower().endswith('.sf2'):
full_sf2_path = os.path.join(root, file)
self.convert_sf2_to_sf3(full_sf2_path)
```
### 1.4 API Endpoint Serving `.sf3` Files to Clients (`app/api/v1/plugins.py`)
Provides a static download route serving optimized `.sf3` assets:
```python
@router.get("/soundfonts/download/{sf_id}")
async def download_soundfont_asset(sf_id: str):
"""
Returns the optimized .sf3 file if available, otherwise falls back to .sf2.
"""
sf3_path = f"/opt/daw_engine/soundfonts/{sf_id}.sf3"
sf2_path = f"/opt/daw_engine/soundfonts/{sf_id}.sf2"
if os.path.exists(sf3_path):
return FileResponse(sf3_path, media_type="application/octet-stream", filename=f"{sf_id}.sf3")
elif os.path.exists(sf2_path):
return FileResponse(sf2_path, media_type="application/octet-stream", filename=f"{sf_id}.sf2")
else:
raise HTTPException(status_code=404, detail="SoundFont asset not found")
```
---
## STEP 2: UPGRADING CLIENT PLAYER USING SPESSASYNTH
SpessaSynth (`spessasynth_lib`) is a next-generation JavaScript SoundFont Synthesizer written entirely using the Web Audio API & AudioWorklet. It supports direct loading of `.sf3` files without requiring complex C/Wasm compilation wrappers.
### 2.1 Integrating the SpessaSynth Library into Frontend
Add the npm package or embed the ES Module script directly inside `index.html`:
```html
<!-- index.html -->
<script type="module">
import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
window.SpessaSynthClass = Synthesizer;
</script>
```
### 2.2 Client Storage Optimization (`IndexedDB`)
Caches downloaded `.sf3` files inside `IndexedDB` so that upon reopening the browser, the application loads audio buffers instantly in $0\text{ms}$ without re-fetching from the server.
```javascript
// app/static/js/services/soundfontStorage.js
class SoundFontStorage {
constructor() {
this.dbName = "DAW_SoundFont_Cache";
this.storeName = "sf3_buffers";
}
async openDB() {
return new Promise((resolve, reject) => {
const request = indexedDB.open(this.dbName, 1);
request.onupgradeneeded = (e) => {
const db = e.target.result;
if (!db.objectStoreNames.contains(this.storeName)) {
db.createObjectStore(this.storeName);
}
};
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
}
async getBuffer(sfId) {
const db = await this.openDB();
return new Promise((resolve) => {
const tx = db.transaction(this.storeName, "readonly");
const store = tx.objectStore(this.storeName);
const req = store.get(sfId);
req.onsuccess = () => resolve(req.result || null);
req.onerror = () => resolve(null);
});
}
async saveBuffer(sfId, arrayBuffer) {
const db = await this.openDB();
return new Promise((resolve, reject) => {
const tx = db.transaction(this.storeName, "readwrite");
const store = tx.objectStore(this.storeName);
const req = store.put(arrayBuffer, sfId);
req.onsuccess = () => resolve(true);
req.onerror = () => reject(req.error);
});
}
}
export const sfStorage = new SoundFontStorage();
```
### 2.3 Comprehensive Upgrade of `soundfontPlayer.js`
Replaces oscillator emulation loops with the SpessaSynth Engine:
```javascript
// app/static/js/services/soundfontPlayer.js
import { sfStorage } from './soundfontStorage.js';
class RealSoundFontPlayer {
constructor() {
this.audioCtx = null;
this.synthInstance = null;
this.currentSfId = null;
this.isInitialized = false;
}
async init(audioContext) {
if (this.isInitialized) return;
this.audioCtx = audioContext;
if (window.SpessaSynthClass) {
// Initialize SpessaSynth Synthesizer routed to Web Audio Destination
this.synthInstance = new window.SpessaSynthClass(this.audioCtx.destination);
this.isInitialized = true;
console.log("[SonicSF] SpessaSynth Engine Initialized successfully.");
} else {
console.warn("[SonicSF] SpessaSynth library not loaded. Falling back to basic audio.");
}
}
/**
* Loads .sf3 file from IndexedDB Cache or Server API
*/
async loadSoundFont(sfId = "generaluser_gs") {
if (!this.isInitialized) return;
if (this.currentSfId === sfId) return;
console.log(`[SonicSF] Loading SoundFont asset: ${sfId}...`);
// 1. Try fetching from IndexedDB Cache
let buffer = await sfStorage.getBuffer(sfId);
if (!buffer) {
// 2. If missing, download .sf3 asset from Server (~4MB footprint)
try {
const response = await fetch(`/api/v1/plugins/soundfonts/download/${sfId}`);
if (!response.ok) throw new Error("Network download failed");
buffer = await response.arrayBuffer();
// Save to IndexedDB for instant future loads
await sfStorage.saveBuffer(sfId, buffer);
} catch (err) {
console.error(`[SonicSF] Failed to load SoundFont '${sfId}':`, err);
return;
}
}
// 3. Load .sf3 ArrayBuffer into SpessaSynth Engine
try {
await this.synthInstance.soundFontManager.addSoundFont(buffer);
this.currentSfId = sfId;
console.log(`[SonicSF] SoundFont '${sfId}' loaded into Wasm/JS memory.`);
} catch (e) {
console.error("[SonicSF] Error parsing SF3 buffer in SpessaSynth:", e);
}
}
/**
* Configures MIDI Channel, Bank, Program
*/
applyAITrackInstrument(channel, bank, program) {
if (!this.synthInstance) return;
// Bank Select (CC 0)
this.synthInstance.controllerChange(channel, 0, bank);
// Program Change
this.synthInstance.programChange(channel, program);
}
/**
* Plays a MIDI note in real time with 100% authentic instrument sound
*/
playNote(pitch, velocity = 0.8, durationSec = 1.0, channel = 0) {
if (!this.synthInstance) return;
const midiPitch = Math.min(127, Math.max(0, pitch));
const midiVelocity = Math.floor(velocity * 127);
// Note On
this.synthInstance.noteOn(channel, midiPitch, midiVelocity);
// Note Off scheduled by duration
setTimeout(() => {
this.synthInstance.noteOff(channel, midiPitch);
}, durationSec * 1000);
}
}
export const soundFontPlayerInstance = new RealSoundFontPlayer();
```
---
## UI INTEGRATION WORKFLOW (`app.jsx`)
1. **Application Startup:**
* When the user clicks on the web page or triggers Transport Play, call `soundFontPlayerInstance.init(audioCtx)` and trigger a background fetch for the default General SoundFont (`generaluser_gs.sf3`).
2. **When User Selects Instrument via Synth Button:**
* Read `sf_id` from the selected instrument object.
* Call `await soundFontPlayerInstance.loadSoundFont(sf_id)`.
* Call `soundFontPlayerInstance.applyAITrackInstrument(channel, bank, program)`.
3. **When Playing Piano Roll / Timeline:**
* Every emitted MIDI note invokes `soundFontPlayerInstance.playNote(pitch, velocity, durationSec, channel)`.
* Audio signals pass through Envelopes, Modulators, and Standard General MIDI Sample Mapping via SpessaSynth $\rightarrow$ outputs $100\%$ authentic instrument audio matching the server WAV export engine.
---
## POST-OPTIMIZATION PERFORMANCE COMPARISON
| Metric | Before Optimization (SF2 + Oscillator) | After Optimization (SF3 + SpessaSynth) |
| --- | --- | --- |
| **Asset Download Size** | $35\text{ MB} - 140\text{ MB}$ (Extremely Heavy) | 🟢 $3.5\text{ MB} - 5.5\text{ MB}$ (Ultra Light) |
| **Initial Load Time** | $10 - 25\text{ seconds}$ | ⚡ $1 - 2\text{ seconds}$ |
| **Subsequent Load Time** | $10 - 25\text{ seconds}$ | ⚡ $0\text{ seconds}$ (Retrieved from IndexedDB Cache) |
| **Preview Fidelity** | 🔴 Crude Emulated Waveform (Oscillator) | 🟢 $100\%$ Authentic SoundFont Rendering |
| **Keypress Latency** | $0\text{ms}$ | ⚡ $0\text{ms}$ (Runs on AudioWorklet) |