3dtours/SonicForgeStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: thêm soundfont và VSTi cho MIDI

Browse Source
This commit is contained in:
3dtours
2026-07-23 17:47:46 +07:00
parent 0b2382573f
commit 225f23516f
16 changed files with 1232 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
md/33_MIDI_AI.md
+367
View File
@@ -0,0 +1,367 @@
# TECHNICAL SPECIFICATION: AI MIDI GENERATOR TOOL & PROMPT PRESET SYSTEM
---
## 1. System Overview
The AI Copilot system integrated within the DAW provides two core capabilities:
* **AI Tool Call (Function Calling):** Receives user requests, triggers the automated music generation engine, and returns a list of `Tracks` and `MIDIItems` complying with the JSON schema structure for direct placement onto the DAW Timeline.
* **Prompt Template Engine (Preset Library):** Manages a prompt template directory categorized by genre, mood, and song structure. Upon detecting specific keywords (e.g., *"epic orchestra"*), the engine automatically looks up and expands the raw user prompt into a System Context Standard Prompt containing music theory definitions (pitch range, scales, rhythm patterns, chord progressions, and voicing) prior to dispatching to the LLM.
```text
+-----------------------------------------------------------------------------------+
| USER INTERFACE |
| |
| [ Prompt Bar / Copilot UI ] <---> [ Prompt Template Preset Manager (CRUD) ] |
| | | |
| | Input: "Write 8 bars of | Matched Preset: |
| | epic orchestra MIDI notes..." | "Epic Orchestra Intro Spec" |
| v v |
| +---------------------------------------------------------------------------+ |
| | Prompt Context Expander Engine | |
| +--------------------------------------------------+------------------------+ |
| | |
+------------------------------------------------------|----------------------------+
| Extended Prompt Payload
v
+-----------------------------------------------------------------------------------+
| AI LLM ENGINE |
| |
| Tool Calling Execution: `generate_multitrack_midi()` |
| Output: Structured JSON Payload (Multi-track 8-bar notes) |
+------------------------------------------------------|----------------------------+
| Validated JSON Output
v
+-----------------------------------------------------------------------------------+
| DAW CLIENT STATE ENGINE |
| |
| - Parses JSON Payload |
| - Spawns/Finds Tracks (Strings, Brass, Synth, Percussion, Drums) |
| - Injects `MIDIItem` into `Main Session` / `Section Store` |
| - Re-renders Canvas & Piano Roll UI |
+-----------------------------------------------------------------------------------+
```
---
## 2. Prompt Template & Preset Engine
To eliminate the need to re-type lengthy system instructions, the application provides a Preset Manager stored as JSON format in `LocalStorage` / `IndexedDB` on the client side or within the server database.
### 2.1 Prompt Preset Schema (`preset_schema.json`)
```json
{
"id": "preset_epic_orchestra_intro",
"name": "Epic Orchestra Intro (8 Bars)",
"keywords": ["epic orchestra", "epic orchestral", "hoành tráng", "nhạc phim epic"],
"category": "Orchestral / Film Score",
"default_bars": 8,
"default_bpm": 130,
"default_scale": "C Minor",
"system_instruction_template": "You are a professional Epic Orchestral film composer. Create a powerful, dramatic 8-bar intro composition.\nThe required structure to return via the `generate_multitrack_midi` tool consists of 5 tracks:\n1. Strings Ensemble: Plays staccato 16th notes in the low register (C2, G2) driving the rhythm (Ostinato).\n2. Brass Section: Plays the main swelling melodic theme (Horn/Trumpet swells) in the C3-C5 range.\n3. Epic Percussion / Taiko: Hits heavily on beats 1 and 3, featuring a snare roll accent at bars 4 and 8.\n4. Synth Bass/Pad: Holds smooth octave foundation layers (Legato).\n5. Orchestral Drums/Cymbals: Crashes on bar 1 and bar 5.\nEnsure the duration of each track is precisely 8 bars (32 beats).",
"is_user_defined": false,
"created_at": "2026-07-23T16:00:00Z"
}
```
### 2.2 Intent Detection & Auto-Expansion Flow
When a user submits a prompt message inside the AI Copilot UI:
```javascript
class PromptTemplateManager {
constructor() {
this.presets = [];
this.loadPresets();
}
// Loads preset list from LocalStorage or API
async loadPresets() {
const localData = localStorage.getItem('daw_ai_prompt_presets');
if (localData) {
this.presets = JSON.parse(localData);
} else {
this.presets = DEFAULT_PRESETS; // Developer default fallback templates
this.savePresets();
}
}
// Matches queries against preset keywords automatically
matchPreset(userQuery) {
const queryLower = userQuery.toLowerCase();
for (const preset of this.presets) {
const hasKeyword = preset.keywords.some(kw => queryLower.includes(kw.toLowerCase()));
if (hasKeyword) {
return preset;
}
}
return null; // Fallback to raw user prompt if no keyword matches
}
// CRUD methods for user-defined presets
saveUserPreset(presetObject) {
const index = this.presets.findIndex(p => p.id === presetObject.id);
if (index >= 0) {
this.presets[index] = presetObject;
} else {
this.presets.push(presetObject);
}
this.savePresets();
}
savePresets() {
localStorage.setItem('daw_ai_prompt_presets', JSON.stringify(this.presets));
}
}
```
---
## 3. Function Calling Specification
Defines the Tool/Function schema passed to the LLM API (OpenAI / Local LLM) to enforce strict structured JSON output.
### 3.1 Function Tool Schema (`tools_spec.json`)
```json
{
"type": "function",
"function": {
"name": "generate_multitrack_midi",
"description": "Generates multi-track MIDI data based on genre, bar duration, and requested instruments list.",
"parameters": {
"type": "object",
"properties": {
"composition_title": {
"type": "string",
"description": "Title of the musical piece (e.g., Epic Orchestra Intro 8-Bars)"
},
"bpm": {
"type": "integer",
"default": 120
},
"total_bars": {
"type": "integer",
"default": 8
},
"tracks": {
"type": "array",
"description": "Array of instrument tracks along with their corresponding MIDI notes",
"items": {
"type": "object",
"properties": {
"track_name": {
"type": "string",
"description": "Track name (e.g., String Ensemble, Epic Brass, Taiko Drums)"
},
"instrument_type": {
"type": "string",
"enum": ["STRINGS", "BRASS", "SYNTH", "PERCUSSION", "DRUMS"]
},
"notes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"pitch": {
"type": "integer",
"description": "MIDI note pitch from 0 to 127 (e.g., C4 = 60, C3 = 48)"
},
"start_beat": {
"type": "number",
"description": "Note start position in beats (from 0.0 to total_bars * 4.0)"
},
"duration_beats": {
"type": "number",
"description": "Note length in beats (e.g., Quarter note = 1.0, Eighth note = 0.5)"
},
"velocity": {
"type": "number",
"description": "Keypress velocity intensity from 0.0 to 1.0",
"default": 0.8
}
},
"required": ["pitch", "start_beat", "duration_beats", "velocity"]
}
}
},
"required": ["track_name", "instrument_type", "notes"]
}
}
},
"required": ["composition_title", "bpm", "total_bars", "tracks"]
}
}
}
```
---
## 4. AI Response Payload Example
Below is an example JSON payload returned by the AI following the execution of `generate_multitrack_midi` for an 8-bar Epic Orchestra request:
```json
{
"tool_call": "generate_multitrack_midi",
"result": {
"composition_title": "Epic Orchestra Intro",
"bpm": 130,
"total_bars": 8,
"tracks": [
{
"track_name": "String Ensemble (Staccato)",
"instrument_type": "STRINGS",
"notes": [
{ "pitch": 36, "start_beat": 0.0, "duration_beats": 0.25, "velocity": 0.9 },
{ "pitch": 36, "start_beat": 0.5, "duration_beats": 0.25, "velocity": 0.85 },
{ "pitch": 48, "start_beat": 1.0, "duration_beats": 0.25, "velocity": 0.95 },
{ "pitch": 36, "start_beat": 1.5, "duration_beats": 0.25, "velocity": 0.8 }
]
},
{
"track_name": "French Horns & Brass",
"instrument_type": "BRASS",
"notes": [
{ "pitch": 60, "start_beat": 0.0, "duration_beats": 2.0, "velocity": 0.95 },
{ "pitch": 63, "start_beat": 2.0, "duration_beats": 2.0, "velocity": 0.9 },
{ "pitch": 67, "start_beat": 4.0, "duration_beats": 4.0, "velocity": 1.0 }
]
},
{
"track_name": "Epic Synth Lead",
"instrument_type": "SYNTH",
"notes": [
{ "pitch": 72, "start_beat": 4.0, "duration_beats": 1.0, "velocity": 0.85 },
{ "pitch": 75, "start_beat": 5.0, "duration_beats": 1.0, "velocity": 0.85 }
]
},
{
"track_name": "Taiko & Percussion",
"instrument_type": "PERCUSSION",
"notes": [
{ "pitch": 36, "start_beat": 0.0, "duration_beats": 0.5, "velocity": 1.0 },
{ "pitch": 36, "start_beat": 2.0, "duration_beats": 0.5, "velocity": 0.95 },
{ "pitch": 38, "start_beat": 3.5, "duration_beats": 0.25, "velocity": 0.8 }
]
},
{
"track_name": "Orchestral Cymbals",
"instrument_type": "DRUMS",
"notes": [
{ "pitch": 49, "start_beat": 0.0, "duration_beats": 4.0, "velocity": 0.9 },
{ "pitch": 49, "start_beat": 16.0, "duration_beats": 4.0, "velocity": 1.0 }
]
}
]
}
}
```
---
## 5. DAW State Ingestion Logic
When the client receives the AI JSON payload, the `ingestAIGeneratedMIDI()` function executes the following pipeline:
1. Spawns or matches corresponding tracks within `main_session`.
2. Generates 8-bar `MIDIItem` objects containing the note lists.
3. Triggers UI canvas and timeline re-renders.
```javascript
function ingestAIGeneratedMIDI(aiPayload, sessionState) {
const { composition_title, bpm, total_bars, tracks } = aiPayload.result;
// 1. Update project BPM if specified
if (bpm) sessionState.metadata.bpm = bpm;
// 2. Iterate through generated tracks
tracks.forEach((aiTrack) => {
// Match existing track or instantiate a new one
let targetTrack = sessionState.main_session.tracks.find(
t => t.name.toLowerCase() === aiTrack.track_name.toLowerCase()
);
if (!targetTrack) {
targetTrack = {
id: `track_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`,
name: aiTrack.track_name,
type: "MIDI",
volume_db: 0.0,
pan: 0.0,
mute: false,
solo: false,
items: []
};
sessionState.main_session.tracks.push(targetTrack);
}
// 3. Create 8-bar MIDIItem
const newMidiItem = {
id: `item_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`,
name: `${composition_title} - ${aiTrack.track_name}`,
type: "MIDI_ITEM",
start_bar: 0.0, // Placed at Timeline start or active Playhead position
duration_bars: total_bars,
clip_start_offset_bars: 0.0,
source_data: {
total_buffer_bars: total_bars,
notes: aiTrack.notes.map((note, index) => ({
id: `note_ai_${Date.now()}_${index}`,
pitch: note.pitch,
start_beat: note.start_beat,
duration_beats: note.duration_beats,
velocity: note.velocity,
pan: 0.0
}))
}
};
// 4. Append Item to Track
targetTrack.items.push(newMidiItem);
});
// 5. Trigger Timeline & Piano Roll UI Re-render Event
window.dispatchEvent(new CustomEvent('DAW_STATE_UPDATED', { detail: sessionState }));
}
```
---
## 6. Prompt Preset Manager UI Layout
```text
+-------------------------------------------------------------------------------+
| AI PROMPT PRESET MANAGER [ + New ]|
+-------------------------------------------------------------------------------+
| SEARCH: [ epic orchestra ] FILTER: [ Orchestral v ]|
| |
| Preset Name Keywords matched Default Bars Actions |
| --------------------------------------------------------------------------- |
| [★] Epic Orchestra Intro epic, orchestra, tráng 8 Bars [Edit][Del] |
| [ ] Pop Piano Chords pop, piano, chord 4 Bars [Edit][Del] |
| [ ] Cyberpunk Synth Synth synth, synthwave, 80s 8 Bars [Edit][Del] |
| |
+-------------------------------------------------------------------------------+
| EDIT PRESET: Epic Orchestra Intro |
| |
| Keyword Triggers (comma-separated): |
| [ epic orchestra, hoành tráng, nhạc phim epic ] |
| |
| System Instruction / Music Rules: |
| +---------------------------------------------------------------------------+ |
| | Compose an 8-bar epic orchestral theme featuring Brass, staccato | |
| | Strings, Taiko percussion, and background Synth bass... | |
| +---------------------------------------------------------------------------+ |
| [ CANCEL ] [ SAVE ]
+-------------------------------------------------------------------------------+
```