fix: đã sửa lỗi bị mất không hiển thị midi và vẽ cclane
This commit is contained in:
@@ -0,0 +1,115 @@
|
||||
# INTEGRATION & OPERATIONAL GUIDE: SOUNDFONT & VST3 ENGINE SYSTEM
|
||||
|
||||
This document outlines the workflow for connecting and operating the designed technical methods and modules across the entire DAW system, clearly categorized by system integration steps.
|
||||
|
||||
---
|
||||
|
||||
## 1. System Environment & Storage Setup
|
||||
|
||||
### A. Server & Docker Directory Structure
|
||||
|
||||
* **System SoundFont Directory (`/opt/daw_engine/soundfonts/`):** Stores system default `.sf2` files (e.g., `GeneralUser_GS.sf2`, `SGM-V2.01.sf2`).
|
||||
* **User Upload Directory (`app/storage/uploads/soundfonts/`):** Stores `.sf2` files uploaded by users via the web interface.
|
||||
* **VST3 & Pianobook Directories (`/opt/daw_engine/vst3/`, `/opt/daw_engine/samples/pianobook/`):** Contains the `DecentSampler.vst3` binary along with the directory structure holding `.dspreset` sample files and `samples/*.wav` subdirectories.
|
||||
|
||||
### B. System Dependencies
|
||||
|
||||
Ensure the `Dockerfile`/`Virtualenv` has installed the `libcurl4` system library (mandatory for DecentSampler) and the Python package `sf2utils>=0.9.0`.
|
||||
|
||||
---
|
||||
|
||||
## 2. SoundFont Catalog Operational Lifecycle
|
||||
|
||||
### A. First Startup (Lazy Initialization)
|
||||
|
||||
* When the Server boots, the Catalog is not generated immediately to prevent slowing down the app boot time.
|
||||
* When the Frontend dispatches its first request to the API Endpoint `GET /api/v1/plugins/soundfonts/catalog`, the Backend triggers `SoundFontInspector` to simultaneously scan both system and upload directories.
|
||||
* The extracted data is categorized into 2 versions:
|
||||
* **Full Catalog:** Designed for the UI to display the complete list of instruments.
|
||||
* **Condensed Catalog:** A summary (maximum 40–50 representative instruments categorized under General MIDI groups such as Piano, Brass, Drums, etc.) specifically tailored for the AI Agent.
|
||||
|
||||
|
||||
* The parsed data is cached in memory (Memory Cache) for subsequent queries.
|
||||
|
||||
### B. Cache Invalidation on User Upload
|
||||
|
||||
* Once the upload handling endpoint successfully saves an uploaded `.sf2` file to the upload directory:
|
||||
* Automatically invokes the `invalidate_catalog_cache()` method to purge the memory cache.
|
||||
* Triggers a Background Task calling the catalog initialization function to incrementally scan the new file without blocking the user's HTTP response.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. AI Copilot Integration Workflow (AI Gateway & System Prompt)
|
||||
|
||||
### A. Initial Instrument Catalog Load (Frontend Startup)
|
||||
|
||||
* As soon as the Web application launches (`app.jsx`), the Frontend proactively calls the API to fetch the Catalog.
|
||||
* Extracts the `condensed_catalog` section and persists it into the application's global state (Global State).
|
||||
|
||||
### B. Automated Prompt Context Injection
|
||||
|
||||
* When a user submits an interaction command to the AI:
|
||||
* The System Instruction generator reads the `condensed_catalog` and converts it into a concise text description of available instruments (including name, bank code, and program code).
|
||||
* Enforces the rule that the AI must assign `soundfont_bank: 0` for melodic instruments and `soundfont_bank: 128` for Drum Kits.
|
||||
|
||||
|
||||
|
||||
### C. Function Calling Schema Definition
|
||||
|
||||
* When dispatching requests to the LLM, the Tools list configuring `generate_multitrack_midi` includes 3 mandatory fields for every Track: `soundfont_id`, `soundfont_bank`, and `soundfont_program`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Real-time Client-Side Instrument Switching (Browser Playback)
|
||||
|
||||
### A. Listening for AI Responses
|
||||
|
||||
* When the AI successfully completes a Function Call and returns a JSON payload containing musical notes alongside `soundfont_bank` & `soundfont_program` parameters for each Track:
|
||||
* The Client allocates each Track to a corresponding MIDI Channel (Channels 0 through 8 for standard instruments, fixed Channel 9 for Drum Kits).
|
||||
|
||||
|
||||
|
||||
### B. Applying Real-Time Program Changes
|
||||
|
||||
* Calls the `applyAITrackInstrument` method on the Client's SoundFont Player module.
|
||||
* The module dispatches a Control Change (CC 0) signal to select the Bank, followed by a Program Change event to the designated MIDI channel to immediately play the newly selected instrument sound inside the browser.
|
||||
|
||||
---
|
||||
|
||||
## 5. Server-Side Offline Render Workflow (Audio Export)
|
||||
|
||||
When a user clicks "Export WAV" or "Bounce Track", the processing pipeline on the Server executes as follows:
|
||||
|
||||
### A. Reading Track Metadata
|
||||
|
||||
Extracts `soundfont_bank` and `soundfont_program` parameters from the Track metadata received in the project's JSON payload.
|
||||
|
||||
### B. MIDI Channel Routing & FluidSynth Rendering
|
||||
|
||||
* **Channel Rules:** If `soundfont_bank == 128` or the track is marked as percussion (`is_percussion`), rigidly assigns `midi_channel = 9` (Channel 10 under the General MIDI standard). Otherwise, assigns free channels from 0 to 8.
|
||||
* Executes `program_select` settings on the FluidSynth Instance targeting the correct channel, bank, and program before feeding the note sequence into the audio rendering buffer.
|
||||
|
||||
### C. Rendering Pianobook (`.dspreset`)
|
||||
|
||||
* If a Track selects a Pianobook instrument source:
|
||||
* Calls `DecentSamplerManager` passing the absolute file path to the `.dspreset` file.
|
||||
* The manager automatically changes the Current Working Directory (CWD) temporarily to the parent folder of the `.dspreset` file, loads the preset into VST3, and subsequently restores the original working directory to prevent "Sample Not Found" errors on relative `.wav` sample files.
|
||||
|
||||
|
||||
|
||||
### D. Rendering VST3 via Pedalboard
|
||||
|
||||
Prior to passing the MIDI note array into the VST3 Plugin, inserts 2 initialization MIDI messages at timestamp $0.0\text{s}$:
|
||||
|
||||
* A `control_change` message (Control 0, Value = bank).
|
||||
* A `program_change` message (Program = program).
|
||||
|
||||
---
|
||||
|
||||
## 6. Verification & Testing Workflow
|
||||
|
||||
* **Catalog API Verification:** Use Postman or a browser to call `GET /api/v1/plugins/soundfonts/catalog`, confirming that the returned payload contains both `full_catalog` and `condensed_catalog`.
|
||||
* **AI Response Verification:** Input the command *"Compose 8 bars of Brass horns and a drum kit"* $\rightarrow$ Inspect the returned JSON from the AI to verify that the Brass track has `program: 56`, `bank: 0` and the Drums track has `program: 0`, `bank: 128`.
|
||||
* **Audio Output Verification:** Export the WAV file and listen to confirm that the Brass horn and Drum sounds are rendered using the correct instrument patches.
|
||||
@@ -0,0 +1,130 @@
|
||||
# ASSESSMENT REPORT & OPTIMIZATION PLAN: LINUX VST3 & SOUNDFONT MAPPING ENGINE
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Review
|
||||
|
||||
Your plan closely aligns with the current codebase status and correctly identifies key bottlenecks (such as missing `libcurl4`, missing `sf2utils`, hardcoded `program_select(0, fid, 0, 0)` calls in `render_engine.py`, and missing schema fields in the AI Tool Schema).
|
||||
|
||||
However, to guarantee stable system operation within the Docker Linux environment and prevent failures during End-to-End execution, the plan requires the 5 critical technical refinements detailed below.
|
||||
|
||||
---
|
||||
|
||||
## 2. 5 Mandatory Technical Refinements
|
||||
|
||||
### 💡 Refinement 1: AI Prompt Context Size Control (Avoiding Token Overflow)
|
||||
|
||||
* **Problem in Previous Plan:** Injecting the entire `soundfont_catalog.json` file into the AI System Instruction. A full SoundFont file (like GeneralUser GS or SGM-V2.01) can contain hundreds to thousands of presets/notes, causing LLM Token Limit overflows, inflating costs, and degrading response latency.
|
||||
* **Solution:**
|
||||
* Implement `get_condensed_catalog_summary()` within `SoundFontInspector` to extract only a condensed catalog (categorized into core instrument groups: Piano, Organ, Guitar, Bass, Strings, Ensemble, Brass, Reed, Pipe, Synth Lead, Synth Pad, Drum Kit).
|
||||
* Inject a maximum of 40–50 of the most common instruments along with their representative bank and program codes into the AI Prompt.
|
||||
|
||||
|
||||
|
||||
### 💡 Refinement 2: MIDI Channel Handling for Percussion Kits (Bank 128 / Percussion)
|
||||
|
||||
* **Problem in Previous Plan:** Defaulting to `channel=0` for all tracks when invoking `program_select(0, fid, bank, prog)`. In General MIDI and SoundFont (`.sf2`) standards, Drum/Percussion sounds (Bank 128) must reside on MIDI Channel 9 (the 10th channel, 0-based index 9).
|
||||
* **Solution:**
|
||||
* In `render_engine.py`, if `soundfont_bank == 128` or `is_percussion == True`, automatically assign that track's MIDI Channel to `channel = 9` for both FluidSynth rendering and `mido` message generation.
|
||||
|
||||
|
||||
|
||||
### 3. Refinement 3: Catalog Refresh on User SoundFont Upload (Cache Invalidation)
|
||||
|
||||
* **Problem in Previous Plan:** The `GET /api/v1/plugins/soundfonts/catalog` endpoint scans the catalog only once or upon application startup. When a user uploads a new `.sf2` file via `POST /api/v1/audio/upload-soundfont`, the AI remains unaware of the newly added file.
|
||||
* **Solution:**
|
||||
* Implement a Cache Invalidation mechanism: Upon successfully saving an uploaded `.sf2` file, automatically invoke `SoundFontInspector.generate_full_catalog()` to update the `soundfont_catalog.json` file.
|
||||
|
||||
|
||||
|
||||
### 💡 Refinement 4: Handling Relative Sample Paths for Pianobook `.dspreset` Files
|
||||
|
||||
* **Problem in Previous Plan:** Pianobook `.dspreset` files contain relative path links pointing to subfolder `samples/*.wav` files. When DecentSampler VST3 loads a `.dspreset` file via `pedalboard`, if the Working Directory is not set to the folder containing the `.dspreset` file, the VST3 engine triggers a "Sample Not Found" error (resulting in silence).
|
||||
* **Solution:**
|
||||
* Before invoking `plugin.load_preset(dspreset_path)`, ensure an absolute path (`os.path.abspath(dspreset_path)`) is passed and temporarily switch the Working Directory or properly configure the Root Sample Directory for DecentSampler.
|
||||
|
||||
|
||||
|
||||
### 💡 Refinement 5: Robust Error Handling in `SoundFontInspector`
|
||||
|
||||
* **Problem in Previous Plan:** If a user uploads a corrupted or malformed `.sf2` file, the `sf2utils` library may throw an exception, crashing the entire Catalog scanning workflow.
|
||||
* **Solution:**
|
||||
* Wrap each `.sf2` file processing block inside a `try...except` block in `SoundFontInspector`. If a file is corrupted, log a warning and skip that specific file instead of interrupting the complete scan process.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Updated Execution Plan
|
||||
|
||||
### Task A: SoundFont Inspection Engine (`sf2utils`)
|
||||
|
||||
* [x] **A1:** Add `sf2utils>=0.9.0` to `requirements.txt`.
|
||||
* [x] **A2:** Create `app/core/soundfont_inspector.py`:
|
||||
* Add `inspect_sf2_file(filepath)` wrapped in a `try...except` block.
|
||||
* Add `generate_full_catalog(output_json_path)` scanning both `/opt/daw_engine/soundfonts` and `app/storage/uploads/soundfonts`.
|
||||
* Add `get_condensed_catalog_summary()` to build a condensed summary for the AI Context Prompt.
|
||||
|
||||
|
||||
* [x] **A3:** Create API Endpoint `GET /api/v1/plugins/soundfonts/catalog` in `app/api/v1/plugins.py`:
|
||||
* Return Full Catalog for Frontend UI and Condensed Catalog for AI Agent.
|
||||
* Integrate cache refresh functionality triggered upon new `.sf2` file uploads.
|
||||
|
||||
|
||||
|
||||
### Task B: DecentSampler + Pianobook Support
|
||||
|
||||
* [x] **B1:** Update `Dockerfile`:
|
||||
* Add `libcurl4` to the `apt-get install` package list.
|
||||
* Pre-create directory structures `/opt/daw_engine/vst3/` and `/opt/daw_engine/samples/pianobook/`.
|
||||
|
||||
|
||||
* [x] **B2:** Create local host directory structure `vst_plugins/` and `samples/pianobook/` (Update `.gitignore`).
|
||||
* [x] **B3:** Update `app/core/vst_engine.py`:
|
||||
* Add `DecentSamplerManager` supporting `.dspreset` loading using absolute paths.
|
||||
|
||||
|
||||
* [x] **B4:** Integrate Pianobook rendering into `app/core/render_engine.py` when a Track selects a Pianobook instrument.
|
||||
|
||||
### Task C: AI Tool Schema & Prompt Injection
|
||||
|
||||
* [x] **C1:** Update `DEFAULT_TOOLS` in `app/static/js/services/aiGateway.js`:
|
||||
* Add 3 properties to the `generate_multitrack_midi` schema: `soundfont_id` (string), `soundfont_bank` (integer), `soundfont_program` (integer).
|
||||
|
||||
|
||||
* [x] **C2:** Inject condensed instrument catalog into System Instruction within `aiGateway.js`.
|
||||
* [x] **C3:** Load Catalog automatically upon Frontend application startup (`app.jsx`).
|
||||
|
||||
### Task D: Server Render — Program Change & Channel Mapping
|
||||
|
||||
* [x] **D1:** Update `render_engine.py`:
|
||||
* Read `soundfont_bank` and `soundfont_program` from Track metadata.
|
||||
* MIDI channel rules: If `soundfont_bank == 128` (Drums), automatically assign `channel = 9` (GM Standard Channel 10). Otherwise, assign channels from 0 through 8.
|
||||
|
||||
|
||||
* [x] **D2:** Update FluidSynth render path:
|
||||
```python
|
||||
midi_channel = 9 if (bank == 128 or track.get("is_percussion")) else target_channel
|
||||
fl.program_select(midi_channel, fid, bank, prog)
|
||||
|
||||
```
|
||||
|
||||
|
||||
* [x] **D3:** Update VST3/Pedalboard render path:
|
||||
* Insert `CONTROL_CHANGE` (CC 0 for Bank) and `PROGRAM_CHANGE` events into the note sequence prior to rendering the audio buffer.
|
||||
|
||||
|
||||
|
||||
### Task E: Client SoundFont Player — Program Change
|
||||
|
||||
* [x] **E1:** Update `app/static/js/services/soundfontPlayer.js`:
|
||||
* Add `programChange(channel, program)` and `controllerChange(channel, controller, value)` methods.
|
||||
|
||||
|
||||
* [x] **E2:** Add `applyAITrackInstrument(trackId, bank, program)` function to dynamically switch instrument sounds in real time when AI generates new Tracks on the UI.
|
||||
|
||||
### Task F: Validation & Testing
|
||||
|
||||
* [x] **F1:** Test Catalog API: `GET /api/v1/plugins/soundfonts/catalog`.
|
||||
* [x] **F2:** Test AI Generation: Input prompt *"Compose 8 bars of Brass horns and a drum kit"* $\rightarrow$ Verify AI returns JSON with `program=56` (Brass) and `bank=128` (Drums).
|
||||
* [x] **F3:** Test Server Render: Export WAV $\rightarrow$ Listen to output audio file to verify correct Brass horn and Drum sound execution.
|
||||
Reference in New Issue
Block a user