fix: đã sửa lỗi bị mất không hiển thị midi và vẽ cclane

2026-07-26 10:37:41 +07:00
parent d20fafb75e
commit 3ae48f2f2f
4 changed files with 438 additions and 0 deletions
@@ -0,0 +1,115 @@
+# INTEGRATION & OPERATIONAL GUIDE: SOUNDFONT & VST3 ENGINE SYSTEM
+
+This document outlines the workflow for connecting and operating the designed technical methods and modules across the entire DAW system, clearly categorized by system integration steps.
+
+---
+
+## 1. System Environment & Storage Setup
+
+### A. Server & Docker Directory Structure
+
+* **System SoundFont Directory (`/opt/daw_engine/soundfonts/`):** Stores system default `.sf2` files (e.g., `GeneralUser_GS.sf2`, `SGM-V2.01.sf2`).
+* **User Upload Directory (`app/storage/uploads/soundfonts/`):** Stores `.sf2` files uploaded by users via the web interface.
+* **VST3 & Pianobook Directories (`/opt/daw_engine/vst3/`, `/opt/daw_engine/samples/pianobook/`):** Contains the `DecentSampler.vst3` binary along with the directory structure holding `.dspreset` sample files and `samples/*.wav` subdirectories.
+
+### B. System Dependencies
+
+Ensure the `Dockerfile`/`Virtualenv` has installed the `libcurl4` system library (mandatory for DecentSampler) and the Python package `sf2utils>=0.9.0`.
+
+---
+
+## 2. SoundFont Catalog Operational Lifecycle
+
+### A. First Startup (Lazy Initialization)
+
+* When the Server boots, the Catalog is not generated immediately to prevent slowing down the app boot time.
+* When the Frontend dispatches its first request to the API Endpoint `GET /api/v1/plugins/soundfonts/catalog`, the Backend triggers `SoundFontInspector` to simultaneously scan both system and upload directories.
+* The extracted data is categorized into 2 versions:
+* **Full Catalog:** Designed for the UI to display the complete list of instruments.
+* **Condensed Catalog:** A summary (maximum 40–50 representative instruments categorized under General MIDI groups such as Piano, Brass, Drums, etc.) specifically tailored for the AI Agent.
+
+
+* The parsed data is cached in memory (Memory Cache) for subsequent queries.
+
+### B. Cache Invalidation on User Upload
+
+* Once the upload handling endpoint successfully saves an uploaded `.sf2` file to the upload directory:
+* Automatically invokes the `invalidate_catalog_cache()` method to purge the memory cache.
+* Triggers a Background Task calling the catalog initialization function to incrementally scan the new file without blocking the user's HTTP response.
+
+
+
+---
+
+## 3. AI Copilot Integration Workflow (AI Gateway & System Prompt)
+
+### A. Initial Instrument Catalog Load (Frontend Startup)
+
+* As soon as the Web application launches (`app.jsx`), the Frontend proactively calls the API to fetch the Catalog.
+* Extracts the `condensed_catalog` section and persists it into the application's global state (Global State).
+
+### B. Automated Prompt Context Injection
+
+* When a user submits an interaction command to the AI:
+* The System Instruction generator reads the `condensed_catalog` and converts it into a concise text description of available instruments (including name, bank code, and program code).
+* Enforces the rule that the AI must assign `soundfont_bank: 0` for melodic instruments and `soundfont_bank: 128` for Drum Kits.
+
+
+
+### C. Function Calling Schema Definition
+
+* When dispatching requests to the LLM, the Tools list configuring `generate_multitrack_midi` includes 3 mandatory fields for every Track: `soundfont_id`, `soundfont_bank`, and `soundfont_program`.
+
+---
+
+## 4. Real-time Client-Side Instrument Switching (Browser Playback)
+
+### A. Listening for AI Responses
+
+* When the AI successfully completes a Function Call and returns a JSON payload containing musical notes alongside `soundfont_bank` & `soundfont_program` parameters for each Track:
+* The Client allocates each Track to a corresponding MIDI Channel (Channels 0 through 8 for standard instruments, fixed Channel 9 for Drum Kits).
+
+
+
+### B. Applying Real-Time Program Changes
+
+* Calls the `applyAITrackInstrument` method on the Client's SoundFont Player module.
+* The module dispatches a Control Change (CC 0) signal to select the Bank, followed by a Program Change event to the designated MIDI channel to immediately play the newly selected instrument sound inside the browser.
+
+---
+
+## 5. Server-Side Offline Render Workflow (Audio Export)
+
+When a user clicks "Export WAV" or "Bounce Track", the processing pipeline on the Server executes as follows:
+
+### A. Reading Track Metadata
+
+Extracts `soundfont_bank` and `soundfont_program` parameters from the Track metadata received in the project's JSON payload.
+
+### B. MIDI Channel Routing & FluidSynth Rendering
+
+* **Channel Rules:** If `soundfont_bank == 128` or the track is marked as percussion (`is_percussion`), rigidly assigns `midi_channel = 9` (Channel 10 under the General MIDI standard). Otherwise, assigns free channels from 0 to 8.
+* Executes `program_select` settings on the FluidSynth Instance targeting the correct channel, bank, and program before feeding the note sequence into the audio rendering buffer.
+
+### C. Rendering Pianobook (`.dspreset`)
+
+* If a Track selects a Pianobook instrument source:
+* Calls `DecentSamplerManager` passing the absolute file path to the `.dspreset` file.
+* The manager automatically changes the Current Working Directory (CWD) temporarily to the parent folder of the `.dspreset` file, loads the preset into VST3, and subsequently restores the original working directory to prevent "Sample Not Found" errors on relative `.wav` sample files.
+
+
+
+### D. Rendering VST3 via Pedalboard
+
+Prior to passing the MIDI note array into the VST3 Plugin, inserts 2 initialization MIDI messages at timestamp $0.0\text{s}$:
+
+* A `control_change` message (Control 0, Value = bank).
+* A `program_change` message (Program = program).
+
+---
+
+## 6. Verification & Testing Workflow
+
+* **Catalog API Verification:** Use Postman or a browser to call `GET /api/v1/plugins/soundfonts/catalog`, confirming that the returned payload contains both `full_catalog` and `condensed_catalog`.
+* **AI Response Verification:** Input the command *"Compose 8 bars of Brass horns and a drum kit"* $\rightarrow$ Inspect the returned JSON from the AI to verify that the Brass track has `program: 56`, `bank: 0` and the Drums track has `program: 0`, `bank: 128`.
+* **Audio Output Verification:** Export the WAV file and listen to confirm that the Brass horn and Drum sounds are rendered using the correct instrument patches.