feat: bổ sung MIDI
This commit is contained in:
@@ -0,0 +1,237 @@
|
||||
Here is the clean, nicely formatted Markdown version of the technical specification document:
|
||||
|
||||
# TECHNICAL INSTALLATION & INTEGRATION GUIDE FOR SOUNDFONT / VSTI IN DAW
|
||||
|
||||
This document provides a detailed technical architecture model for integrating SoundFonts, WebAssembly Plugins (Client), and Native VSTi/AU (Server). It clearly delineates components pre-installed by the Developer (Coder) versus those open for User uploads and additions.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architectural Distribution Overview (Developer vs. User)
|
||||
|
||||
| Plugin / Asset Category | Processing Location | Installed By | Storage & Management Method | Security & Safety Profile |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Default SoundFont (`.sf2`)** | Client (Wasm) | Coder | Static Assets hosted on Web Server / CDN | Extremely High |
|
||||
| **User Custom SoundFont (`.sf2`)** | Client (Wasm) | User | Browser `IndexedDB` or User Cloud Storage | Extremely High (Runs inside Wasm Sandbox) |
|
||||
| **WebAssembly Synths (WAMs)** | Client (JS/Wasm) | Coder | Bundled within Frontend Source Code | Extremely High |
|
||||
| **Core Server VSTi (Vital, Surge...)** | Server (Python) | Coder | System Directory inside Docker/Linux Container | High (Controlled binary footprint) |
|
||||
| **User Custom VST3 / Preset** | Server (Python) | User (Restricted) | Stores `.vst3` files or `.fxp`/`.json` on Container | High Security Risk (Requires Sandboxing) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Client-Side Integration Tech (Browser / WebAssembly)
|
||||
|
||||
The Client-Side handles zero-latency real-time composition and audio previews.
|
||||
|
||||
### 2.1 Coder Pre-bundled Assets
|
||||
|
||||
* **Static SoundFont Hosting:**
|
||||
* The developer places standard `.sf2` files (such as `GeneralUser_GS.sf2`) into the `public/soundfonts/` directory or hosts them via CDN.
|
||||
* Upon application startup, default SoundFonts are queried via REST API:
|
||||
```http
|
||||
GET /api/v1/assets/default-soundfonts
|
||||
|
||||
```
|
||||
|
||||
|
||||
```json
|
||||
[
|
||||
{ "id": "sf_generaluser", "name": "GeneralUser GS v1.471", "size_mb": 31.2, "url": "/soundfonts/GeneralUser.sf2" },
|
||||
{ "id": "sf_sso", "name": "Sonatina Symphonic Orchestra", "size_mb": 95.0, "url": "/soundfonts/SSO.sf2" }
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
* **FluidSynth WebAssembly Engine Integration:**
|
||||
* Compiles FluidSynth C/C++ code into WebAssembly (`fluidsynth.wasm` + `fluidsynth.js`) using Emscripten.
|
||||
* Alternatively, leverages open JavaScript wrappers such as `@soundfont/player` or `SpessaSynth`.
|
||||
|
||||
|
||||
|
||||
### 2.2 Allowing User Custom SoundFont (`.sf2`) Uploads
|
||||
|
||||
Delivers a flexible user experience without overloading server storage:
|
||||
|
||||
* **Upload Mechanism & Local Cache (`IndexedDB`):**
|
||||
* Users drag and drop `.sf2` files directly into the DAW interface.
|
||||
* JavaScript reads the file as an `ArrayBuffer` via the `FileReader` API.
|
||||
* The file persists directly within the browser's local `IndexedDB` cache for immediate reuse across sessions without re-uploading to the server.
|
||||
|
||||
|
||||
* **Dynamic Injection into WebAssembly Memory:**
|
||||
```javascript
|
||||
// Client-side JavaScript snippet
|
||||
async function loadUserSoundFont(fileBuffer) {
|
||||
const uint8Array = new Uint8Array(fileBuffer);
|
||||
// Write buffer straight into Emscripten FluidSynth Virtual File System (MEMFS)
|
||||
Module.FS.writeFile('/user_font.sf2', uint8Array);
|
||||
|
||||
// Call Wasm C-function to load bank
|
||||
const sfont_id = Module._fluid_synth_sfload(synthInstance, '/user_font.sf2', 1);
|
||||
console.log(`User SoundFont loaded successfully with ID: ${sfont_id}`);
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Server-Side Integration Tech (Python Backend Engine)
|
||||
|
||||
The Server-Side executes high-resolution offline WAV rendering when an operator triggers the Export / Bounce workflow.
|
||||
|
||||
### 3.1 Server Environment Installed by Coder
|
||||
|
||||
The developer configures the Server environment (or Docker Container) with pre-installed Native C++ libraries and Python utilities.
|
||||
|
||||
1. **Server Base `Dockerfile` Configuration:**
|
||||
```dockerfile
|
||||
FROM python:3.10-slim
|
||||
|
||||
# Install Linux audio libraries
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fluidsynth \
|
||||
libfluidsynth-dev \
|
||||
libasound2-dev \
|
||||
libjack-jackd2-dev \
|
||||
build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Initialize directories for Native VST3 and system SoundFonts
|
||||
RUN mkdir -p /opt/daw_engine/vst3 \
|
||||
&& mkdir -p /opt/daw_engine/soundfonts
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
```
|
||||
|
||||
|
||||
2. **Pre-installing Native VST3 Plugins:**
|
||||
Places 64-bit Linux `.vst3` binary builds of open-source synths inside `/opt/daw_engine/vst3/`:
|
||||
* `/opt/daw_engine/vst3/Vital.vst3`
|
||||
* `/opt/daw_engine/vst3/Surge XT.vst3`
|
||||
* `/opt/daw_engine/vst3/Dexed.vst3`
|
||||
|
||||
|
||||
3. **Python Backend Integration via Spotify `pedalboard`:**
|
||||
```python
|
||||
# render_engine/vst_loader.py
|
||||
import os
|
||||
from pedalboard import VST3Plugin, Pedalboard
|
||||
|
||||
class PluginManager:
|
||||
def __init__(self, vst_dir="/opt/daw_engine/vst3"):
|
||||
self.vst_dir = vst_dir
|
||||
self.available_plugins = self._scan_plugins()
|
||||
|
||||
def _scan_plugins(self):
|
||||
plugins = {}
|
||||
for root, dirs, files in os.walk(self.vst_dir):
|
||||
for file in files:
|
||||
if file.endswith(".vst3") or file.endswith(".so"):
|
||||
plugin_path = os.path.join(root, file)
|
||||
plugin_name = os.path.splitext(file)[0]
|
||||
plugins[plugin_name] = plugin_path
|
||||
return plugins
|
||||
|
||||
def load_vst(self, plugin_name: str, preset_data: dict = None) -> VST3Plugin:
|
||||
if plugin_name not in self.available_plugins:
|
||||
raise FileNotFoundError(f"VST3 Plugin '{plugin_name}' not found on server.")
|
||||
|
||||
path = self.available_plugins[plugin_name]
|
||||
vst_instance = VST3Plugin(path)
|
||||
|
||||
# Inject parameters if provided
|
||||
if preset_data:
|
||||
for param_name, param_value in preset_data.items():
|
||||
setattr(vst_instance, param_name, param_value)
|
||||
|
||||
return vst_instance
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 3.2 Handling User Custom Plugins / Presets
|
||||
|
||||
#### Option 1: User Presets / Patches Uploads (**RECOMMENDED - Safe**)
|
||||
|
||||
* **Implementation:** The backend locks native VST3 installations to common open engines (Vital, Dexed, Surge XT). Users upload lightweight preset patches like `.vitalbank`, `.syx` (DX7 patches), `.fxp`, or JSON parameter states.
|
||||
* **Workflow:**
|
||||
1. User selects the Vital Synth on the Client UI.
|
||||
2. User clicks "Import Preset" $\rightarrow$ Uploads a `.vital` file or JSON parameter bundle.
|
||||
3. Server parses JSON parameters and injects them directly into the VST3 instance via `pedalboard` during render execution.
|
||||
|
||||
|
||||
* **Benefits:** Absolutely safe, minimal footprint, zero security vulnerabilities to the host infrastructure.
|
||||
|
||||
#### Option 2: User Native Binary VST3 Uploads (**HIGH RISK - Requires Isolation**)
|
||||
|
||||
* **Risk:** A `.vst3` file contains executable machine code (`.so` Shared Object on Linux). Accepting arbitrary uploads grants 100% vector exposure to Remote Code Execution (RCE) attacks.
|
||||
* **Technical Mitigation (If Mandatory):**
|
||||
* **Sandboxing Isolation:** Every user Export/Render request runs inside an isolated, short-lived container (Ephemeral Docker / Firejail / gVisor) stripped of `root` privileges and completely isolated from external internet interfaces.
|
||||
* **Time-To-Live (TTL):** User `.vst3` binaries persist inside temporary directories `/tmp/user_sessions/{user_id}/` and purge automatically upon render job completion.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4. API Specification for SoundFonts & Plugins
|
||||
|
||||
### 4.1 OpenAPI Endpoint Spec for Frontend
|
||||
|
||||
```yaml
|
||||
/api/v1/plugins/available:
|
||||
get:
|
||||
summary: Query available VSTi engines and SoundFont resources on the Server
|
||||
responses:
|
||||
200:
|
||||
content:
|
||||
application/json:
|
||||
example:
|
||||
vst_instruments:
|
||||
- id: "vst_vital"
|
||||
name: "Vital Wavetable Synth"
|
||||
type: "VST3"
|
||||
has_native_support: true
|
||||
- id: "vst_dexed"
|
||||
name: "Dexed FM Synth"
|
||||
type: "VST3"
|
||||
has_native_support: true
|
||||
soundfonts:
|
||||
- id: "sf_generaluser"
|
||||
name: "GeneralUser GS"
|
||||
file: "GeneralUser.sf2"
|
||||
|
||||
/api/v1/projects/render:
|
||||
post:
|
||||
summary: Trigger offline DAW Project rendering to WAV on the Server
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ProjectSchema'
|
||||
responses:
|
||||
200:
|
||||
description: Returns the URL pointing to the rendered WAV file
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Development Team Best Practices Summary
|
||||
|
||||
* **SoundFont (`.sf2`):**
|
||||
* **For Users:** Encourage unrestricted local uploads on the Client (Browser). Store assets in `IndexedDB` to ensure optimal real-time performance without straining server resources.
|
||||
* **For Developers:** Supply 1–2 default General MIDI (GM) SoundFont banks (`GeneralUser_GS.sf2`) bundled on both Client and Server.
|
||||
|
||||
|
||||
* **VSTi Instruments:**
|
||||
* **For Developers:** Pre-install top open-source Linux-native synths on the Server (Vital, Surge XT, Dexed, OB-Xd).
|
||||
* **For Users:** Do **not** allow direct `.vst3` binary uploads to the production server. Instead, permit users to upload Presets / Patches / JSON parameters for the supported synth models. This guarantees 100% security while saving storage and network bandwidth.
|
||||
Reference in New Issue
Block a user