SonicForgeStudio/md/35.1_VST_FIX.md

INTEGRATION & OPERATIONAL GUIDE: SOUNDFONT & VST3 ENGINE SYSTEM

This document outlines the workflow for connecting and operating the designed technical methods and modules across the entire DAW system, clearly categorized by system integration steps.

---

1. System Environment & Storage Setup

A. Server & Docker Directory Structure

• System SoundFont Directory (/opt/daw_engine/soundfonts/): Stores system default .sf2 files (e.g., GeneralUser_GS.sf2, SGM-V2.01.sf2).
• User Upload Directory (app/storage/uploads/soundfonts/): Stores .sf2 files uploaded by users via the web interface.
• VST3 & Pianobook Directories (/opt/daw_engine/vst3/, /opt/daw_engine/samples/pianobook/): Contains the DecentSampler.vst3 binary along with the directory structure holding .dspreset sample files and samples/*.wav subdirectories.

B. System Dependencies

Ensure the Dockerfile/Virtualenv has installed the libcurl4 system library (mandatory for DecentSampler) and the Python package sf2utils>=0.9.0.

---

2. SoundFont Catalog Operational Lifecycle

A. First Startup (Lazy Initialization)

• When the Server boots, the Catalog is not generated immediately to prevent slowing down the app boot time.

• When the Frontend dispatches its first request to the API Endpoint GET /api/v1/plugins/soundfonts/catalog, the Backend triggers SoundFontInspector to simultaneously scan both system and upload directories.

• The extracted data is categorized into 2 versions:

• Full Catalog: Designed for the UI to display the complete list of instruments.

• Condensed Catalog: A summary (maximum 40–50 representative instruments categorized under General MIDI groups such as Piano, Brass, Drums, etc.) specifically tailored for the AI Agent.

• The parsed data is cached in memory (Memory Cache) for subsequent queries.

B. Cache Invalidation on User Upload

• Once the upload handling endpoint successfully saves an uploaded .sf2 file to the upload directory:
• Automatically invokes the invalidate_catalog_cache() method to purge the memory cache.
• Triggers a Background Task calling the catalog initialization function to incrementally scan the new file without blocking the user's HTTP response.

---

3. AI Copilot Integration Workflow (AI Gateway & System Prompt)

A. Initial Instrument Catalog Load (Frontend Startup)

• As soon as the Web application launches (app.jsx), the Frontend proactively calls the API to fetch the Catalog.
• Extracts the condensed_catalog section and persists it into the application's global state (Global State).

B. Automated Prompt Context Injection

• When a user submits an interaction command to the AI:
• The System Instruction generator reads the condensed_catalog and converts it into a concise text description of available instruments (including name, bank code, and program code).
• Enforces the rule that the AI must assign soundfont_bank: 0 for melodic instruments and soundfont_bank: 128 for Drum Kits.

C. Function Calling Schema Definition

• When dispatching requests to the LLM, the Tools list configuring generate_multitrack_midi includes 3 mandatory fields for every Track: soundfont_id, soundfont_bank, and soundfont_program.

---

4. Real-time Client-Side Instrument Switching (Browser Playback)

A. Listening for AI Responses

• When the AI successfully completes a Function Call and returns a JSON payload containing musical notes alongside soundfont_bank & soundfont_program parameters for each Track:
• The Client allocates each Track to a corresponding MIDI Channel (Channels 0 through 8 for standard instruments, fixed Channel 9 for Drum Kits).

B. Applying Real-Time Program Changes

• Calls the applyAITrackInstrument method on the Client's SoundFont Player module.
• The module dispatches a Control Change (CC 0) signal to select the Bank, followed by a Program Change event to the designated MIDI channel to immediately play the newly selected instrument sound inside the browser.

---

5. Server-Side Offline Render Workflow (Audio Export)

When a user clicks "Export WAV" or "Bounce Track", the processing pipeline on the Server executes as follows:

A. Reading Track Metadata

Extracts soundfont_bank and soundfont_program parameters from the Track metadata received in the project's JSON payload.

B. MIDI Channel Routing & FluidSynth Rendering

• Channel Rules: If soundfont_bank == 128 or the track is marked as percussion (is_percussion), rigidly assigns midi_channel = 9 (Channel 10 under the General MIDI standard). Otherwise, assigns free channels from 0 to 8.
• Executes program_select settings on the FluidSynth Instance targeting the correct channel, bank, and program before feeding the note sequence into the audio rendering buffer.

C. Rendering Pianobook (.dspreset)

• If a Track selects a Pianobook instrument source:
• Calls DecentSamplerManager passing the absolute file path to the .dspreset file.
• The manager automatically changes the Current Working Directory (CWD) temporarily to the parent folder of the .dspreset file, loads the preset into VST3, and subsequently restores the original working directory to prevent "Sample Not Found" errors on relative .wav sample files.

D. Rendering VST3 via Pedalboard

Prior to passing the MIDI note array into the VST3 Plugin, inserts 2 initialization MIDI messages at timestamp 0.0\text{s}:

• A control_change message (Control 0, Value = bank).
• A program_change message (Program = program).

---

6. Verification & Testing Workflow

• Catalog API Verification: Use Postman or a browser to call GET /api/v1/plugins/soundfonts/catalog, confirming that the returned payload contains both full_catalog and condensed_catalog.
• AI Response Verification: Input the command "Compose 8 bars of Brass horns and a drum kit" \rightarrow Inspect the returned JSON from the AI to verify that the Brass track has program: 56, bank: 0 and the Drums track has program: 0, bank: 128.
• Audio Output Verification: Export the WAV file and listen to confirm that the Brass horn and Drum sounds are rendered using the correct instrument patches.