SonicForgeStudio/md/34_VST_LINUX.md

TECHNICAL PLAN: DECENT SAMPLER + PIANOBOOK INSTALLATION AND SOUNDFONT MAPPING EXTRACTION FOR AI AGENT

---

1. Executive Overview

The system needs to fulfill two core requirements:

• Integrate DecentSampler & Pianobook on Linux Server:

• Install DecentSampler.vst3 (Linux 64-bit) into the Docker Server environment.

• Structure the Pianobook sample library directories (.dspreset + .wav files).

• Integrate Preset Loading into Python pedalboard for offline rendering steps.

• Resolve the AI Agent's "Instrument Information Blindness" regarding SoundFont (.sf2):

• Current State: The AI Agent and the system only load raw .sf2 files without knowing what instruments are contained within (which Bank, which Program/Patch number, or what the instrument names are).

• Solution:

• Build a SoundFont Inspection Engine using Python (sf2utils) to scan .sf2 files upon upload/scan and extract the instrument catalog table (Bank, Program/Preset ID, Instrument Name).

• Export a Catalog table (soundfont_catalog.json) and pass this context to the AI Agent.

• Update the AI Tool Schema so the AI accurately passes soundfont_id, bank, and program (MIDI Program Change) when creating a Track.

---

2. DecentSampler + Pianobook Installation Plan (Server Backend)

2.1 Installing DecentSampler Linux Native VST3 in Docker

1. Download DecentSampler Linux VST3:

• Download the official Linux 64-bit build from the DecentSampler website (DecentSampler_Linux_x64.tar.gz or .vst3 file).

2. Server Directory Structure:

/opt/daw_engine/
├── vst3/
│   └── DecentSampler.vst3/          <-- Native Linux VST3 Binary
├── soundfonts/
│   ├── GeneralUser_GS.sf2
│   └── SGM-V2.01.sf2
└── samples/
    └── pianobook/                   <-- Pianobook Sample Libraries
        ├── salamander_grand_piano/
        │   ├── salamander_piano.dspreset
        │   └── samples/ (*.wav)
        └── acoustic_guitar/
            ├── guitar.dspreset
            └── samples/ (*.wav)

3. Additions to Dockerfile:

# Install audio system dependencies
RUN apt-get update && apt-get install -y \
    libgl1-mesa-glx \
    libfreetype6 \
    libcurl4 \
    && rm -rf /var/lib/apt/lists/*

# Copy DecentSampler VST3 to Server
COPY ./vst_plugins/DecentSampler.vst3 /opt/daw_engine/vst3/DecentSampler.vst3

2.2 Integrating DecentSampler into the Python Engine (app/core/vst_engine.py)

The pedalboard library supports loading VST3 plugins and preset files for DecentSampler:

import os
from pedalboard import VST3Plugin

class DecentSamplerManager:
    def __init__(self, vst_path="/opt/daw_engine/vst3/DecentSampler.vst3"):
        self.vst_path = vst_path

    def create_decent_sampler_instance(self, dspreset_path: str) -> VST3Plugin:
        """
        Instantiates VST3 DecentSampler and loads the Pianobook sample preset (.dspreset) file.
        """
        if not os.path.exists(self.vst_path):
            raise FileNotFoundError(f"DecentSampler VST3 not found at {self.vst_path}")
        
        plugin = VST3Plugin(self.vst_path)
        
        # Load the Pianobook preset file into DecentSampler VST3
        if os.path.exists(dspreset_path):
            plugin.load_preset(dspreset_path)
            
        return plugin

---

3. Designing the SoundFont Inspection Engine (Bank/Program Extraction)

Every SoundFont (.sf2) file is a collection of Presets (or Programs). For the AI Agent to know what sound presets exist inside the .sf2 file, the Backend must scan and parse the .sf2 file.

3.1 Adding Python SoundFont Inspection Libraries

Add to requirements.txt:

sf2utils>=0.9.0
mido>=1.3.0

3.2 Building the SoundFont Metadata Inspection Service (app/core/soundfont_inspector.py)

import os
import json
from sf2utils.sf2parse import Sf2File

class SoundFontInspector:
    def __init__(self, sf_dir="/opt/daw_engine/soundfonts"):
        self.sf_dir = sf_dir

    def inspect_sf2_file(self, filepath: str) -> dict:
        """
        Parses an .sf2 file and returns a complete instrument catalog (Bank, Program, Instrument Name).
        """
        if not os.path.exists(filepath):
            return {}

        sf_name = os.path.basename(filepath)
        sf_id = os.path.splitext(sf_name)[0].lower()

        instruments = []
        with open(filepath, 'rb') as f:
            sf2 = Sf2File(f)
            for preset in sf2.presets:
                # Ignore EOP (End of Header) preset
                if preset.name.strip() == "EOP" or (preset.bank == 128 and preset.preset == 127):
                    continue
                
                instruments.append({
                    "bank": preset.bank,               # Bank number (0 = General MIDI Standard, 128 = Percussion/Drums)
                    "program": preset.preset,          # Program/Patch number (0-127)
                    "name": preset.name.strip(),       # Instrument name (e.g. "Stereo Grand", "Violin", "Brass Section")
                    "is_percussion": (preset.bank == 128)
                })

        return {
            "soundfont_id": sf_id,
            "filename": sf_name,
            "total_instruments": len(instruments),
            "instruments": instruments
        }

    def generate_full_catalog(self, output_json_path="/opt/daw_engine/soundfont_catalog.json"):
        """
        Scans all .sf2 files in the directory and builds a Catalog JSON for the AI Agent.
        """
        catalog = {}
        for root, dirs, files in os.walk(self.sf_dir):
            for file in files:
                if file.endswith(('.sf2', '.SF2')):
                    full_path = os.path.join(root, file)
                    sf_info = self.inspect_sf2_file(full_path)
                    catalog[sf_info["soundfont_id"]] = sf_info

        with open(output_json_path, 'w', encoding='utf-8') as f:
            json.dump(catalog, f, ensure_ascii=False, indent=2)

        return catalog

3.3 Catalog File Output Structure (soundfont_catalog.json)

This JSON file serves as an Instrument Dictionary for the AI Agent:

{
  "generaluser_gs": {
    "soundfont_id": "generaluser_gs",
    "filename": "GeneralUser_GS.sf2",
    "total_instruments": 128,
    "instruments": [
      { "bank": 0, "program": 0, "name": "Stereo Grand Piano", "is_percussion": false },
      { "bank": 0, "program": 19, "name": "Church Organ", "is_percussion": false },
      { "bank": 0, "program": 40, "name": "Violin Ensemble", "is_percussion": false },
      { "bank": 0, "program": 56, "name": "Trumpet", "is_percussion": false },
      { "bank": 128, "program": 0, "name": "Standard Drum Kit", "is_percussion": true }
    ]
  }
}

---

4. Guiding the AI Agent in Instrument Selection & Accurate Note Loading

When the user types: "Create a Piano track and a Strings section track for 8 bars", the AI Agent needs to know precisely which soundfont, bank, and program numbers to assign to the Tracks.

4.1 Updating the AI Tool Function Spec (tools_spec.json)

Add soundfont_bank and soundfont_program fields to the Tool Schema sent to the AI:

{
  "type": "function",
  "function": {
    "name": "generate_multitrack_midi",
    "description": "Generates multi-track MIDI data along with appropriate SoundFont Program configurations.",
    "parameters": {
      "type": "object",
      "properties": {
        "tracks": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "track_name": { "type": "string" },
              "instrument_type": { "type": "string", "enum": ["PIANO", "STRINGS", "BRASS", "SYNTH", "DRUMS"] },
              "soundfont_id": { 
                "type": "string", 
                "description": "ID of the SoundFont to use (e.g. 'generaluser_gs')" 
              },
              "soundfont_bank": { 
                "type": "integer", 
                "default": 0, 
                "description": "MIDI Bank code (0 for melodic instruments, 128 for Drums)" 
              },
              "soundfont_program": { 
                "type": "integer", 
                "description": "MIDI Program Number (0-127) corresponding to the instrument name in the Catalog" 
              },
              "notes": { "type": "array", "items": { "type": "object" } }
            },
            "required": ["track_name", "soundfont_id", "soundfont_bank", "soundfont_program", "notes"]
          }
        }
      }
    }
  }
}

4.2 Injecting the Catalog into the AI Agent's Context Prompt (Prompt Template)

Before sending the user query to the LLM, the system reads soundfont_catalog.json and injects a condensed catalog table into the System Instruction:

# System Context Prompt Injector
def build_ai_system_instruction(catalog_data: dict) -> str:
    sf_summary = []
    for sf_id, sf_info in catalog_data.items():
        sf_summary.append(f"SoundFont ID: '{sf_id}' (File: {sf_info['filename']}):")
        for inst in sf_info['instruments'][:20]: # Inject primary instrument lists
            sf_summary.append(
                f"  - [{inst['name']}]: bank={inst['bank']}, program={inst['program']}"
            )

    catalog_context = "\n".join(sf_summary)

    system_instruction = f"""
You are an AI Copilot for a DAW. Below is the Catalog of available SoundFonts on the system:

{catalog_context}

MANDATORY RULES WHEN CREATING TRACKS:
1. When creating any track, you MUST look up the catalog above and fill in the correct `soundfont_id`, `soundfont_bank`, and `soundfont_program`.
2. Example: If the user requests "Piano", select soundfont_id="generaluser_gs", soundfont_bank=0, soundfont_program=0 ("Stereo Grand Piano").
3. If the user requests "Violin/Strings", select soundfont_bank=0, soundfont_program=40 ("Violin Ensemble").
4. If the user requests "Drums", select soundfont_bank=128, soundfont_program=0 ("Standard Drum Kit").
"""
    return system_instruction

4.3 Applying Program Change on Client & Server Render Layers

A. Client Browser Side (FluidSynth Wasm / SoundFont Player)

Upon receiving JSON from the AI, the Frontend invokes the Bank and Program selection function to trigger the correct sound:

// Client-side Javascript (spessasynth / fluidsynth.wasm)
function applyAITrackInstrument(trackId, soundfontBank, soundfontProgram) {
    const channel = getTrackMIDIChannel(trackId);
    
    // Send MIDI Bank Select (CC 0)
    synthInstance.controllerChange(channel, 0, soundfontBank);
    
    // Send MIDI Program Change
    synthInstance.programChange(channel, soundfontProgram);
}

B. Server Offline Render Side (app/core/render_engine.py)

When rendering to a WAV file, Python inserts a MIDI Program Change event ahead of the Track's note sequence:

import mido

def create_midi_track_with_program(notes_data, bank=0, program=0):
    midi_track = mido.MidiTrack()
    
    # 1. Insert Bank Select (Control Change 0)
    midi_track.append(mido.Message('control_change', channel=0, control=0, value=bank, time=0))
    
    # 2. Insert Program Change (Instrument Sound Selection)
    midi_track.append(mido.Message('program_change', channel=0, program=program, time=0))
    
    # 3. Insert MIDI notes generated by AI
    for note in notes_data:
        start_tick = int(note['start_beat'] * 480) # 480 ticks per beat
        dur_tick = int(note['duration_beats'] * 480)
        pitch = int(note['pitch'])
        vel = int(note['velocity'] * 127)
        
        midi_track.append(mido.Message('note_on', note=pitch, velocity=vel, time=start_tick))
        midi_track.append(mido.Message('note_off', note=pitch, velocity=0, time=dur_tick))
        
    return midi_track

---

5. Action Checklist

• Step 1: Download Linux 64-bit DecentSampler.vst3 and copy it into /opt/daw_engine/vst3/.
• Step 2: Download Pianobook sound libraries (e.g. Salamander Grand Piano) and extract them to /opt/daw_engine/samples/pianobook/.
• Step 3: Add sf2utils to requirements.txt and install it in Docker.
• Step 4: Create app/core/soundfont_inspector.py to automatically scan all .sf2 files in the project and generate soundfont_catalog.json.
• Step 5: Build API Endpoint GET /api/v1/plugins/soundfonts/catalog returning the extracted instrument catalog.
• Step 6: Update the Prompt Template and AI Tool Schema to support soundfont_bank & soundfont_program.
• Step 7: End-to-End Verification: Type Prompt "Generate 8 bars of Brass horn music" \rightarrow AI reads Catalog and selects Program 56 \rightarrow Web Audio & Server Render produce the correct Brass horn instrument sound.