367 lines
15 KiB
Markdown
367 lines
15 KiB
Markdown
# TECHNICAL SPECIFICATION: AI MIDI GENERATOR TOOL & PROMPT PRESET SYSTEM
|
|
|
|
---
|
|
|
|
## 1. System Overview
|
|
|
|
The AI Copilot system integrated within the DAW provides two core capabilities:
|
|
|
|
* **AI Tool Call (Function Calling):** Receives user requests, triggers the automated music generation engine, and returns a list of `Tracks` and `MIDIItems` complying with the JSON schema structure for direct placement onto the DAW Timeline.
|
|
* **Prompt Template Engine (Preset Library):** Manages a prompt template directory categorized by genre, mood, and song structure. Upon detecting specific keywords (e.g., *"epic orchestra"*), the engine automatically looks up and expands the raw user prompt into a System Context Standard Prompt containing music theory definitions (pitch range, scales, rhythm patterns, chord progressions, and voicing) prior to dispatching to the LLM.
|
|
|
|
```text
|
|
+-----------------------------------------------------------------------------------+
|
|
| USER INTERFACE |
|
|
| |
|
|
| [ Prompt Bar / Copilot UI ] <---> [ Prompt Template Preset Manager (CRUD) ] |
|
|
| | | |
|
|
| | Input: "Write 8 bars of | Matched Preset: |
|
|
| | epic orchestra MIDI notes..." | "Epic Orchestra Intro Spec" |
|
|
| v v |
|
|
| +---------------------------------------------------------------------------+ |
|
|
| | Prompt Context Expander Engine | |
|
|
| +--------------------------------------------------+------------------------+ |
|
|
| | |
|
|
+------------------------------------------------------|----------------------------+
|
|
| Extended Prompt Payload
|
|
v
|
|
+-----------------------------------------------------------------------------------+
|
|
| AI LLM ENGINE |
|
|
| |
|
|
| Tool Calling Execution: `generate_multitrack_midi()` |
|
|
| Output: Structured JSON Payload (Multi-track 8-bar notes) |
|
|
+------------------------------------------------------|----------------------------+
|
|
| Validated JSON Output
|
|
v
|
|
+-----------------------------------------------------------------------------------+
|
|
| DAW CLIENT STATE ENGINE |
|
|
| |
|
|
| - Parses JSON Payload |
|
|
| - Spawns/Finds Tracks (Strings, Brass, Synth, Percussion, Drums) |
|
|
| - Injects `MIDIItem` into `Main Session` / `Section Store` |
|
|
| - Re-renders Canvas & Piano Roll UI |
|
|
+-----------------------------------------------------------------------------------+
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Prompt Template & Preset Engine
|
|
|
|
To eliminate the need to re-type lengthy system instructions, the application provides a Preset Manager stored as JSON format in `LocalStorage` / `IndexedDB` on the client side or within the server database.
|
|
|
|
### 2.1 Prompt Preset Schema (`preset_schema.json`)
|
|
|
|
```json
|
|
{
|
|
"id": "preset_epic_orchestra_intro",
|
|
"name": "Epic Orchestra Intro (8 Bars)",
|
|
"keywords": ["epic orchestra", "epic orchestral", "hoành tráng", "nhạc phim epic"],
|
|
"category": "Orchestral / Film Score",
|
|
"default_bars": 8,
|
|
"default_bpm": 130,
|
|
"default_scale": "C Minor",
|
|
"system_instruction_template": "You are a professional Epic Orchestral film composer. Create a powerful, dramatic 8-bar intro composition.\nThe required structure to return via the `generate_multitrack_midi` tool consists of 5 tracks:\n1. Strings Ensemble: Plays staccato 16th notes in the low register (C2, G2) driving the rhythm (Ostinato).\n2. Brass Section: Plays the main swelling melodic theme (Horn/Trumpet swells) in the C3-C5 range.\n3. Epic Percussion / Taiko: Hits heavily on beats 1 and 3, featuring a snare roll accent at bars 4 and 8.\n4. Synth Bass/Pad: Holds smooth octave foundation layers (Legato).\n5. Orchestral Drums/Cymbals: Crashes on bar 1 and bar 5.\nEnsure the duration of each track is precisely 8 bars (32 beats).",
|
|
"is_user_defined": false,
|
|
"created_at": "2026-07-23T16:00:00Z"
|
|
}
|
|
|
|
```
|
|
|
|
### 2.2 Intent Detection & Auto-Expansion Flow
|
|
|
|
When a user submits a prompt message inside the AI Copilot UI:
|
|
|
|
```javascript
|
|
class PromptTemplateManager {
|
|
constructor() {
|
|
this.presets = [];
|
|
this.loadPresets();
|
|
}
|
|
|
|
// Loads preset list from LocalStorage or API
|
|
async loadPresets() {
|
|
const localData = localStorage.getItem('daw_ai_prompt_presets');
|
|
if (localData) {
|
|
this.presets = JSON.parse(localData);
|
|
} else {
|
|
this.presets = DEFAULT_PRESETS; // Developer default fallback templates
|
|
this.savePresets();
|
|
}
|
|
}
|
|
|
|
// Matches queries against preset keywords automatically
|
|
matchPreset(userQuery) {
|
|
const queryLower = userQuery.toLowerCase();
|
|
for (const preset of this.presets) {
|
|
const hasKeyword = preset.keywords.some(kw => queryLower.includes(kw.toLowerCase()));
|
|
if (hasKeyword) {
|
|
return preset;
|
|
}
|
|
}
|
|
return null; // Fallback to raw user prompt if no keyword matches
|
|
}
|
|
|
|
// CRUD methods for user-defined presets
|
|
saveUserPreset(presetObject) {
|
|
const index = this.presets.findIndex(p => p.id === presetObject.id);
|
|
if (index >= 0) {
|
|
this.presets[index] = presetObject;
|
|
} else {
|
|
this.presets.push(presetObject);
|
|
}
|
|
this.savePresets();
|
|
}
|
|
|
|
savePresets() {
|
|
localStorage.setItem('daw_ai_prompt_presets', JSON.stringify(this.presets));
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Function Calling Specification
|
|
|
|
Defines the Tool/Function schema passed to the LLM API (OpenAI / Local LLM) to enforce strict structured JSON output.
|
|
|
|
### 3.1 Function Tool Schema (`tools_spec.json`)
|
|
|
|
```json
|
|
{
|
|
"type": "function",
|
|
"function": {
|
|
"name": "generate_multitrack_midi",
|
|
"description": "Generates multi-track MIDI data based on genre, bar duration, and requested instruments list.",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"composition_title": {
|
|
"type": "string",
|
|
"description": "Title of the musical piece (e.g., Epic Orchestra Intro 8-Bars)"
|
|
},
|
|
"bpm": {
|
|
"type": "integer",
|
|
"default": 120
|
|
},
|
|
"total_bars": {
|
|
"type": "integer",
|
|
"default": 8
|
|
},
|
|
"tracks": {
|
|
"type": "array",
|
|
"description": "Array of instrument tracks along with their corresponding MIDI notes",
|
|
"items": {
|
|
"type": "object",
|
|
"properties": {
|
|
"track_name": {
|
|
"type": "string",
|
|
"description": "Track name (e.g., String Ensemble, Epic Brass, Taiko Drums)"
|
|
},
|
|
"instrument_type": {
|
|
"type": "string",
|
|
"enum": ["STRINGS", "BRASS", "SYNTH", "PERCUSSION", "DRUMS"]
|
|
},
|
|
"notes": {
|
|
"type": "array",
|
|
"items": {
|
|
"type": "object",
|
|
"properties": {
|
|
"pitch": {
|
|
"type": "integer",
|
|
"description": "MIDI note pitch from 0 to 127 (e.g., C4 = 60, C3 = 48)"
|
|
},
|
|
"start_beat": {
|
|
"type": "number",
|
|
"description": "Note start position in beats (from 0.0 to total_bars * 4.0)"
|
|
},
|
|
"duration_beats": {
|
|
"type": "number",
|
|
"description": "Note length in beats (e.g., Quarter note = 1.0, Eighth note = 0.5)"
|
|
},
|
|
"velocity": {
|
|
"type": "number",
|
|
"description": "Keypress velocity intensity from 0.0 to 1.0",
|
|
"default": 0.8
|
|
}
|
|
},
|
|
"required": ["pitch", "start_beat", "duration_beats", "velocity"]
|
|
}
|
|
}
|
|
},
|
|
"required": ["track_name", "instrument_type", "notes"]
|
|
}
|
|
}
|
|
},
|
|
"required": ["composition_title", "bpm", "total_bars", "tracks"]
|
|
}
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 4. AI Response Payload Example
|
|
|
|
Below is an example JSON payload returned by the AI following the execution of `generate_multitrack_midi` for an 8-bar Epic Orchestra request:
|
|
|
|
```json
|
|
{
|
|
"tool_call": "generate_multitrack_midi",
|
|
"result": {
|
|
"composition_title": "Epic Orchestra Intro",
|
|
"bpm": 130,
|
|
"total_bars": 8,
|
|
"tracks": [
|
|
{
|
|
"track_name": "String Ensemble (Staccato)",
|
|
"instrument_type": "STRINGS",
|
|
"notes": [
|
|
{ "pitch": 36, "start_beat": 0.0, "duration_beats": 0.25, "velocity": 0.9 },
|
|
{ "pitch": 36, "start_beat": 0.5, "duration_beats": 0.25, "velocity": 0.85 },
|
|
{ "pitch": 48, "start_beat": 1.0, "duration_beats": 0.25, "velocity": 0.95 },
|
|
{ "pitch": 36, "start_beat": 1.5, "duration_beats": 0.25, "velocity": 0.8 }
|
|
]
|
|
},
|
|
{
|
|
"track_name": "French Horns & Brass",
|
|
"instrument_type": "BRASS",
|
|
"notes": [
|
|
{ "pitch": 60, "start_beat": 0.0, "duration_beats": 2.0, "velocity": 0.95 },
|
|
{ "pitch": 63, "start_beat": 2.0, "duration_beats": 2.0, "velocity": 0.9 },
|
|
{ "pitch": 67, "start_beat": 4.0, "duration_beats": 4.0, "velocity": 1.0 }
|
|
]
|
|
},
|
|
{
|
|
"track_name": "Epic Synth Lead",
|
|
"instrument_type": "SYNTH",
|
|
"notes": [
|
|
{ "pitch": 72, "start_beat": 4.0, "duration_beats": 1.0, "velocity": 0.85 },
|
|
{ "pitch": 75, "start_beat": 5.0, "duration_beats": 1.0, "velocity": 0.85 }
|
|
]
|
|
},
|
|
{
|
|
"track_name": "Taiko & Percussion",
|
|
"instrument_type": "PERCUSSION",
|
|
"notes": [
|
|
{ "pitch": 36, "start_beat": 0.0, "duration_beats": 0.5, "velocity": 1.0 },
|
|
{ "pitch": 36, "start_beat": 2.0, "duration_beats": 0.5, "velocity": 0.95 },
|
|
{ "pitch": 38, "start_beat": 3.5, "duration_beats": 0.25, "velocity": 0.8 }
|
|
]
|
|
},
|
|
{
|
|
"track_name": "Orchestral Cymbals",
|
|
"instrument_type": "DRUMS",
|
|
"notes": [
|
|
{ "pitch": 49, "start_beat": 0.0, "duration_beats": 4.0, "velocity": 0.9 },
|
|
{ "pitch": 49, "start_beat": 16.0, "duration_beats": 4.0, "velocity": 1.0 }
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 5. DAW State Ingestion Logic
|
|
|
|
When the client receives the AI JSON payload, the `ingestAIGeneratedMIDI()` function executes the following pipeline:
|
|
|
|
1. Spawns or matches corresponding tracks within `main_session`.
|
|
2. Generates 8-bar `MIDIItem` objects containing the note lists.
|
|
3. Triggers UI canvas and timeline re-renders.
|
|
|
|
```javascript
|
|
function ingestAIGeneratedMIDI(aiPayload, sessionState) {
|
|
const { composition_title, bpm, total_bars, tracks } = aiPayload.result;
|
|
|
|
// 1. Update project BPM if specified
|
|
if (bpm) sessionState.metadata.bpm = bpm;
|
|
|
|
// 2. Iterate through generated tracks
|
|
tracks.forEach((aiTrack) => {
|
|
// Match existing track or instantiate a new one
|
|
let targetTrack = sessionState.main_session.tracks.find(
|
|
t => t.name.toLowerCase() === aiTrack.track_name.toLowerCase()
|
|
);
|
|
|
|
if (!targetTrack) {
|
|
targetTrack = {
|
|
id: `track_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`,
|
|
name: aiTrack.track_name,
|
|
type: "MIDI",
|
|
volume_db: 0.0,
|
|
pan: 0.0,
|
|
mute: false,
|
|
solo: false,
|
|
items: []
|
|
};
|
|
sessionState.main_session.tracks.push(targetTrack);
|
|
}
|
|
|
|
// 3. Create 8-bar MIDIItem
|
|
const newMidiItem = {
|
|
id: `item_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`,
|
|
name: `${composition_title} - ${aiTrack.track_name}`,
|
|
type: "MIDI_ITEM",
|
|
start_bar: 0.0, // Placed at Timeline start or active Playhead position
|
|
duration_bars: total_bars,
|
|
clip_start_offset_bars: 0.0,
|
|
source_data: {
|
|
total_buffer_bars: total_bars,
|
|
notes: aiTrack.notes.map((note, index) => ({
|
|
id: `note_ai_${Date.now()}_${index}`,
|
|
pitch: note.pitch,
|
|
start_beat: note.start_beat,
|
|
duration_beats: note.duration_beats,
|
|
velocity: note.velocity,
|
|
pan: 0.0
|
|
}))
|
|
}
|
|
};
|
|
|
|
// 4. Append Item to Track
|
|
targetTrack.items.push(newMidiItem);
|
|
});
|
|
|
|
// 5. Trigger Timeline & Piano Roll UI Re-render Event
|
|
window.dispatchEvent(new CustomEvent('DAW_STATE_UPDATED', { detail: sessionState }));
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Prompt Preset Manager UI Layout
|
|
|
|
```text
|
|
+-------------------------------------------------------------------------------+
|
|
| AI PROMPT PRESET MANAGER [ + New ]|
|
|
+-------------------------------------------------------------------------------+
|
|
| SEARCH: [ epic orchestra ] FILTER: [ Orchestral v ]|
|
|
| |
|
|
| Preset Name Keywords matched Default Bars Actions |
|
|
| --------------------------------------------------------------------------- |
|
|
| [★] Epic Orchestra Intro epic, orchestra, tráng 8 Bars [Edit][Del] |
|
|
| [ ] Pop Piano Chords pop, piano, chord 4 Bars [Edit][Del] |
|
|
| [ ] Cyberpunk Synth Synth synth, synthwave, 80s 8 Bars [Edit][Del] |
|
|
| |
|
|
+-------------------------------------------------------------------------------+
|
|
| EDIT PRESET: Epic Orchestra Intro |
|
|
| |
|
|
| Keyword Triggers (comma-separated): |
|
|
| [ epic orchestra, hoành tráng, nhạc phim epic ] |
|
|
| |
|
|
| System Instruction / Music Rules: |
|
|
| +---------------------------------------------------------------------------+ |
|
|
| | Compose an 8-bar epic orchestral theme featuring Brass, staccato | |
|
|
| | Strings, Taiko percussion, and background Synth bass... | |
|
|
| +---------------------------------------------------------------------------+ |
|
|
| [ CANCEL ] [ SAVE ]
|
|
+-------------------------------------------------------------------------------+
|
|
|
|
``` |