diff --git a/app/storage/sonicforge.db b/app/storage/sonicforge.db index 98f37b7..a3e82e7 100644 Binary files a/app/storage/sonicforge.db and b/app/storage/sonicforge.db differ diff --git a/md/34_VST_LINUX.md b/md/34_VST_LINUX.md index e69de29..7d8d7bf 100644 --- a/md/34_VST_LINUX.md +++ b/md/34_VST_LINUX.md @@ -0,0 +1,344 @@ +# TECHNICAL PLAN: DECENT SAMPLER + PIANOBOOK INSTALLATION AND SOUNDFONT MAPPING EXTRACTION FOR AI AGENT + +--- + +## 1. Executive Overview + +The system needs to fulfill two core requirements: + +* **Integrate DecentSampler & Pianobook on Linux Server:** +* Install `DecentSampler.vst3` (Linux 64-bit) into the Docker Server environment. +* Structure the Pianobook sample library directories (`.dspreset` + `.wav` files). +* Integrate Preset Loading into Python `pedalboard` for offline rendering steps. + + +* **Resolve the AI Agent's "Instrument Information Blindness" regarding SoundFont (`.sf2`):** +* *Current State:* The AI Agent and the system only load raw `.sf2` files without knowing what instruments are contained within (which Bank, which Program/Patch number, or what the instrument names are). +* *Solution:* +* Build a **SoundFont Inspection Engine** using Python (`sf2utils`) to scan `.sf2` files upon upload/scan and extract the instrument catalog table (Bank, Program/Preset ID, Instrument Name). +* Export a Catalog table (`soundfont_catalog.json`) and pass this context to the AI Agent. +* Update the AI Tool Schema so the AI accurately passes `soundfont_id`, `bank`, and `program` (MIDI Program Change) when creating a Track. + + + + + +--- + +## 2. DecentSampler + Pianobook Installation Plan (Server Backend) + +### 2.1 Installing DecentSampler Linux Native VST3 in Docker + +1. **Download DecentSampler Linux VST3:** +* Download the official Linux 64-bit build from the DecentSampler website (`DecentSampler_Linux_x64.tar.gz` or `.vst3` file). + + +2. **Server Directory Structure:** +```text +/opt/daw_engine/ +├── vst3/ +│ └── DecentSampler.vst3/ <-- Native Linux VST3 Binary +├── soundfonts/ +│ ├── GeneralUser_GS.sf2 +│ └── SGM-V2.01.sf2 +└── samples/ + └── pianobook/ <-- Pianobook Sample Libraries + ├── salamander_grand_piano/ + │ ├── salamander_piano.dspreset + │ └── samples/ (*.wav) + └── acoustic_guitar/ + ├── guitar.dspreset + └── samples/ (*.wav) + +``` + + +3. **Additions to `Dockerfile`:** +```dockerfile +# Install audio system dependencies +RUN apt-get update && apt-get install -y \ + libgl1-mesa-glx \ + libfreetype6 \ + libcurl4 \ + && rm -rf /var/lib/apt/lists/* + +# Copy DecentSampler VST3 to Server +COPY ./vst_plugins/DecentSampler.vst3 /opt/daw_engine/vst3/DecentSampler.vst3 + +``` + + + +### 2.2 Integrating DecentSampler into the Python Engine (`app/core/vst_engine.py`) + +The `pedalboard` library supports loading VST3 plugins and preset files for DecentSampler: + +```python +import os +from pedalboard import VST3Plugin + +class DecentSamplerManager: + def __init__(self, vst_path="/opt/daw_engine/vst3/DecentSampler.vst3"): + self.vst_path = vst_path + + def create_decent_sampler_instance(self, dspreset_path: str) -> VST3Plugin: + """ + Instantiates VST3 DecentSampler and loads the Pianobook sample preset (.dspreset) file. + """ + if not os.path.exists(self.vst_path): + raise FileNotFoundError(f"DecentSampler VST3 not found at {self.vst_path}") + + plugin = VST3Plugin(self.vst_path) + + # Load the Pianobook preset file into DecentSampler VST3 + if os.path.exists(dspreset_path): + plugin.load_preset(dspreset_path) + + return plugin + +``` + +--- + +## 3. Designing the SoundFont Inspection Engine (Bank/Program Extraction) + +Every SoundFont (`.sf2`) file is a collection of Presets (or Programs). For the AI Agent to know what sound presets exist inside the `.sf2` file, the Backend must scan and parse the `.sf2` file. + +### 3.1 Adding Python SoundFont Inspection Libraries + +Add to `requirements.txt`: + +```text +sf2utils>=0.9.0 +mido>=1.3.0 + +``` + +### 3.2 Building the SoundFont Metadata Inspection Service (`app/core/soundfont_inspector.py`) + +```python +import os +import json +from sf2utils.sf2parse import Sf2File + +class SoundFontInspector: + def __init__(self, sf_dir="/opt/daw_engine/soundfonts"): + self.sf_dir = sf_dir + + def inspect_sf2_file(self, filepath: str) -> dict: + """ + Parses an .sf2 file and returns a complete instrument catalog (Bank, Program, Instrument Name). + """ + if not os.path.exists(filepath): + return {} + + sf_name = os.path.basename(filepath) + sf_id = os.path.splitext(sf_name)[0].lower() + + instruments = [] + with open(filepath, 'rb') as f: + sf2 = Sf2File(f) + for preset in sf2.presets: + # Ignore EOP (End of Header) preset + if preset.name.strip() == "EOP" or (preset.bank == 128 and preset.preset == 127): + continue + + instruments.append({ + "bank": preset.bank, # Bank number (0 = General MIDI Standard, 128 = Percussion/Drums) + "program": preset.preset, # Program/Patch number (0-127) + "name": preset.name.strip(), # Instrument name (e.g. "Stereo Grand", "Violin", "Brass Section") + "is_percussion": (preset.bank == 128) + }) + + return { + "soundfont_id": sf_id, + "filename": sf_name, + "total_instruments": len(instruments), + "instruments": instruments + } + + def generate_full_catalog(self, output_json_path="/opt/daw_engine/soundfont_catalog.json"): + """ + Scans all .sf2 files in the directory and builds a Catalog JSON for the AI Agent. + """ + catalog = {} + for root, dirs, files in os.walk(self.sf_dir): + for file in files: + if file.endswith(('.sf2', '.SF2')): + full_path = os.path.join(root, file) + sf_info = self.inspect_sf2_file(full_path) + catalog[sf_info["soundfont_id"]] = sf_info + + with open(output_json_path, 'w', encoding='utf-8') as f: + json.dump(catalog, f, ensure_ascii=False, indent=2) + + return catalog + +``` + +### 3.3 Catalog File Output Structure (`soundfont_catalog.json`) + +This JSON file serves as an Instrument Dictionary for the AI Agent: + +```json +{ + "generaluser_gs": { + "soundfont_id": "generaluser_gs", + "filename": "GeneralUser_GS.sf2", + "total_instruments": 128, + "instruments": [ + { "bank": 0, "program": 0, "name": "Stereo Grand Piano", "is_percussion": false }, + { "bank": 0, "program": 19, "name": "Church Organ", "is_percussion": false }, + { "bank": 0, "program": 40, "name": "Violin Ensemble", "is_percussion": false }, + { "bank": 0, "program": 56, "name": "Trumpet", "is_percussion": false }, + { "bank": 128, "program": 0, "name": "Standard Drum Kit", "is_percussion": true } + ] + } +} + +``` + +--- + +## 4. Guiding the AI Agent in Instrument Selection & Accurate Note Loading + +When the user types: *"Create a Piano track and a Strings section track for 8 bars"*, the AI Agent needs to know precisely which soundfont, bank, and program numbers to assign to the Tracks. + +### 4.1 Updating the AI Tool Function Spec (`tools_spec.json`) + +Add `soundfont_bank` and `soundfont_program` fields to the Tool Schema sent to the AI: + +```json +{ + "type": "function", + "function": { + "name": "generate_multitrack_midi", + "description": "Generates multi-track MIDI data along with appropriate SoundFont Program configurations.", + "parameters": { + "type": "object", + "properties": { + "tracks": { + "type": "array", + "items": { + "type": "object", + "properties": { + "track_name": { "type": "string" }, + "instrument_type": { "type": "string", "enum": ["PIANO", "STRINGS", "BRASS", "SYNTH", "DRUMS"] }, + "soundfont_id": { + "type": "string", + "description": "ID of the SoundFont to use (e.g. 'generaluser_gs')" + }, + "soundfont_bank": { + "type": "integer", + "default": 0, + "description": "MIDI Bank code (0 for melodic instruments, 128 for Drums)" + }, + "soundfont_program": { + "type": "integer", + "description": "MIDI Program Number (0-127) corresponding to the instrument name in the Catalog" + }, + "notes": { "type": "array", "items": { "type": "object" } } + }, + "required": ["track_name", "soundfont_id", "soundfont_bank", "soundfont_program", "notes"] + } + } + } + } + } +} + +``` + +### 4.2 Injecting the Catalog into the AI Agent's Context Prompt (Prompt Template) + +Before sending the user query to the LLM, the system reads `soundfont_catalog.json` and injects a condensed catalog table into the System Instruction: + +```python +# System Context Prompt Injector +def build_ai_system_instruction(catalog_data: dict) -> str: + sf_summary = [] + for sf_id, sf_info in catalog_data.items(): + sf_summary.append(f"SoundFont ID: '{sf_id}' (File: {sf_info['filename']}):") + for inst in sf_info['instruments'][:20]: # Inject primary instrument lists + sf_summary.append( + f" - [{inst['name']}]: bank={inst['bank']}, program={inst['program']}" + ) + + catalog_context = "\n".join(sf_summary) + + system_instruction = f""" +You are an AI Copilot for a DAW. Below is the Catalog of available SoundFonts on the system: + +{catalog_context} + +MANDATORY RULES WHEN CREATING TRACKS: +1. When creating any track, you MUST look up the catalog above and fill in the correct `soundfont_id`, `soundfont_bank`, and `soundfont_program`. +2. Example: If the user requests "Piano", select soundfont_id="generaluser_gs", soundfont_bank=0, soundfont_program=0 ("Stereo Grand Piano"). +3. If the user requests "Violin/Strings", select soundfont_bank=0, soundfont_program=40 ("Violin Ensemble"). +4. If the user requests "Drums", select soundfont_bank=128, soundfont_program=0 ("Standard Drum Kit"). +""" + return system_instruction + +``` + +### 4.3 Applying Program Change on Client & Server Render Layers + +#### A. Client Browser Side (FluidSynth Wasm / SoundFont Player) + +Upon receiving JSON from the AI, the Frontend invokes the Bank and Program selection function to trigger the correct sound: + +```javascript +// Client-side Javascript (spessasynth / fluidsynth.wasm) +function applyAITrackInstrument(trackId, soundfontBank, soundfontProgram) { + const channel = getTrackMIDIChannel(trackId); + + // Send MIDI Bank Select (CC 0) + synthInstance.controllerChange(channel, 0, soundfontBank); + + // Send MIDI Program Change + synthInstance.programChange(channel, soundfontProgram); +} + +``` + +#### B. Server Offline Render Side (`app/core/render_engine.py`) + +When rendering to a WAV file, Python inserts a MIDI Program Change event ahead of the Track's note sequence: + +```python +import mido + +def create_midi_track_with_program(notes_data, bank=0, program=0): + midi_track = mido.MidiTrack() + + # 1. Insert Bank Select (Control Change 0) + midi_track.append(mido.Message('control_change', channel=0, control=0, value=bank, time=0)) + + # 2. Insert Program Change (Instrument Sound Selection) + midi_track.append(mido.Message('program_change', channel=0, program=program, time=0)) + + # 3. Insert MIDI notes generated by AI + for note in notes_data: + start_tick = int(note['start_beat'] * 480) # 480 ticks per beat + dur_tick = int(note['duration_beats'] * 480) + pitch = int(note['pitch']) + vel = int(note['velocity'] * 127) + + midi_track.append(mido.Message('note_on', note=pitch, velocity=vel, time=start_tick)) + midi_track.append(mido.Message('note_off', note=pitch, velocity=0, time=dur_tick)) + + return midi_track + +``` + +--- + +## 5. Action Checklist + +* [ ] **Step 1:** Download Linux 64-bit `DecentSampler.vst3` and copy it into `/opt/daw_engine/vst3/`. +* [ ] **Step 2:** Download Pianobook sound libraries (e.g. Salamander Grand Piano) and extract them to `/opt/daw_engine/samples/pianobook/`. +* [ ] **Step 3:** Add `sf2utils` to `requirements.txt` and install it in Docker. +* [ ] **Step 4:** Create `app/core/soundfont_inspector.py` to automatically scan all `.sf2` files in the project and generate `soundfont_catalog.json`. +* [ ] **Step 5:** Build API Endpoint `GET /api/v1/plugins/soundfonts/catalog` returning the extracted instrument catalog. +* [ ] **Step 6:** Update the Prompt Template and AI Tool Schema to support `soundfont_bank` & `soundfont_program`. +* [ ] **Step 7:** End-to-End Verification: Type Prompt *"Generate 8 bars of Brass horn music"* $\rightarrow$ AI reads Catalog and selects Program `56` $\rightarrow$ Web Audio & Server Render produce the correct Brass horn instrument sound. \ No newline at end of file