# TECHNICAL SPECIFICATION: AI MIDI GENERATOR TOOL & PROMPT PRESET SYSTEM --- ## 1. System Overview The AI Copilot system integrated within the DAW provides two core capabilities: * **AI Tool Call (Function Calling):** Receives user requests, triggers the automated music generation engine, and returns a list of `Tracks` and `MIDIItems` complying with the JSON schema structure for direct placement onto the DAW Timeline. * **Prompt Template Engine (Preset Library):** Manages a prompt template directory categorized by genre, mood, and song structure. Upon detecting specific keywords (e.g., *"epic orchestra"*), the engine automatically looks up and expands the raw user prompt into a System Context Standard Prompt containing music theory definitions (pitch range, scales, rhythm patterns, chord progressions, and voicing) prior to dispatching to the LLM. ```text +-----------------------------------------------------------------------------------+ | USER INTERFACE | | | | [ Prompt Bar / Copilot UI ] <---> [ Prompt Template Preset Manager (CRUD) ] | | | | | | | Input: "Write 8 bars of | Matched Preset: | | | epic orchestra MIDI notes..." | "Epic Orchestra Intro Spec" | | v v | | +---------------------------------------------------------------------------+ | | | Prompt Context Expander Engine | | | +--------------------------------------------------+------------------------+ | | | | +------------------------------------------------------|----------------------------+ | Extended Prompt Payload v +-----------------------------------------------------------------------------------+ | AI LLM ENGINE | | | | Tool Calling Execution: `generate_multitrack_midi()` | | Output: Structured JSON Payload (Multi-track 8-bar notes) | +------------------------------------------------------|----------------------------+ | Validated JSON Output v +-----------------------------------------------------------------------------------+ | DAW CLIENT STATE ENGINE | | | | - Parses JSON Payload | | - Spawns/Finds Tracks (Strings, Brass, Synth, Percussion, Drums) | | - Injects `MIDIItem` into `Main Session` / `Section Store` | | - Re-renders Canvas & Piano Roll UI | +-----------------------------------------------------------------------------------+ ``` --- ## 2. Prompt Template & Preset Engine To eliminate the need to re-type lengthy system instructions, the application provides a Preset Manager stored as JSON format in `LocalStorage` / `IndexedDB` on the client side or within the server database. ### 2.1 Prompt Preset Schema (`preset_schema.json`) ```json { "id": "preset_epic_orchestra_intro", "name": "Epic Orchestra Intro (8 Bars)", "keywords": ["epic orchestra", "epic orchestral", "hoành tráng", "nhạc phim epic"], "category": "Orchestral / Film Score", "default_bars": 8, "default_bpm": 130, "default_scale": "C Minor", "system_instruction_template": "You are a professional Epic Orchestral film composer. Create a powerful, dramatic 8-bar intro composition.\nThe required structure to return via the `generate_multitrack_midi` tool consists of 5 tracks:\n1. Strings Ensemble: Plays staccato 16th notes in the low register (C2, G2) driving the rhythm (Ostinato).\n2. Brass Section: Plays the main swelling melodic theme (Horn/Trumpet swells) in the C3-C5 range.\n3. Epic Percussion / Taiko: Hits heavily on beats 1 and 3, featuring a snare roll accent at bars 4 and 8.\n4. Synth Bass/Pad: Holds smooth octave foundation layers (Legato).\n5. Orchestral Drums/Cymbals: Crashes on bar 1 and bar 5.\nEnsure the duration of each track is precisely 8 bars (32 beats).", "is_user_defined": false, "created_at": "2026-07-23T16:00:00Z" } ``` ### 2.2 Intent Detection & Auto-Expansion Flow When a user submits a prompt message inside the AI Copilot UI: ```javascript class PromptTemplateManager { constructor() { this.presets = []; this.loadPresets(); } // Loads preset list from LocalStorage or API async loadPresets() { const localData = localStorage.getItem('daw_ai_prompt_presets'); if (localData) { this.presets = JSON.parse(localData); } else { this.presets = DEFAULT_PRESETS; // Developer default fallback templates this.savePresets(); } } // Matches queries against preset keywords automatically matchPreset(userQuery) { const queryLower = userQuery.toLowerCase(); for (const preset of this.presets) { const hasKeyword = preset.keywords.some(kw => queryLower.includes(kw.toLowerCase())); if (hasKeyword) { return preset; } } return null; // Fallback to raw user prompt if no keyword matches } // CRUD methods for user-defined presets saveUserPreset(presetObject) { const index = this.presets.findIndex(p => p.id === presetObject.id); if (index >= 0) { this.presets[index] = presetObject; } else { this.presets.push(presetObject); } this.savePresets(); } savePresets() { localStorage.setItem('daw_ai_prompt_presets', JSON.stringify(this.presets)); } } ``` --- ## 3. Function Calling Specification Defines the Tool/Function schema passed to the LLM API (OpenAI / Local LLM) to enforce strict structured JSON output. ### 3.1 Function Tool Schema (`tools_spec.json`) ```json { "type": "function", "function": { "name": "generate_multitrack_midi", "description": "Generates multi-track MIDI data based on genre, bar duration, and requested instruments list.", "parameters": { "type": "object", "properties": { "composition_title": { "type": "string", "description": "Title of the musical piece (e.g., Epic Orchestra Intro 8-Bars)" }, "bpm": { "type": "integer", "default": 120 }, "total_bars": { "type": "integer", "default": 8 }, "tracks": { "type": "array", "description": "Array of instrument tracks along with their corresponding MIDI notes", "items": { "type": "object", "properties": { "track_name": { "type": "string", "description": "Track name (e.g., String Ensemble, Epic Brass, Taiko Drums)" }, "instrument_type": { "type": "string", "enum": ["STRINGS", "BRASS", "SYNTH", "PERCUSSION", "DRUMS"] }, "notes": { "type": "array", "items": { "type": "object", "properties": { "pitch": { "type": "integer", "description": "MIDI note pitch from 0 to 127 (e.g., C4 = 60, C3 = 48)" }, "start_beat": { "type": "number", "description": "Note start position in beats (from 0.0 to total_bars * 4.0)" }, "duration_beats": { "type": "number", "description": "Note length in beats (e.g., Quarter note = 1.0, Eighth note = 0.5)" }, "velocity": { "type": "number", "description": "Keypress velocity intensity from 0.0 to 1.0", "default": 0.8 } }, "required": ["pitch", "start_beat", "duration_beats", "velocity"] } } }, "required": ["track_name", "instrument_type", "notes"] } } }, "required": ["composition_title", "bpm", "total_bars", "tracks"] } } } ``` --- ## 4. AI Response Payload Example Below is an example JSON payload returned by the AI following the execution of `generate_multitrack_midi` for an 8-bar Epic Orchestra request: ```json { "tool_call": "generate_multitrack_midi", "result": { "composition_title": "Epic Orchestra Intro", "bpm": 130, "total_bars": 8, "tracks": [ { "track_name": "String Ensemble (Staccato)", "instrument_type": "STRINGS", "notes": [ { "pitch": 36, "start_beat": 0.0, "duration_beats": 0.25, "velocity": 0.9 }, { "pitch": 36, "start_beat": 0.5, "duration_beats": 0.25, "velocity": 0.85 }, { "pitch": 48, "start_beat": 1.0, "duration_beats": 0.25, "velocity": 0.95 }, { "pitch": 36, "start_beat": 1.5, "duration_beats": 0.25, "velocity": 0.8 } ] }, { "track_name": "French Horns & Brass", "instrument_type": "BRASS", "notes": [ { "pitch": 60, "start_beat": 0.0, "duration_beats": 2.0, "velocity": 0.95 }, { "pitch": 63, "start_beat": 2.0, "duration_beats": 2.0, "velocity": 0.9 }, { "pitch": 67, "start_beat": 4.0, "duration_beats": 4.0, "velocity": 1.0 } ] }, { "track_name": "Epic Synth Lead", "instrument_type": "SYNTH", "notes": [ { "pitch": 72, "start_beat": 4.0, "duration_beats": 1.0, "velocity": 0.85 }, { "pitch": 75, "start_beat": 5.0, "duration_beats": 1.0, "velocity": 0.85 } ] }, { "track_name": "Taiko & Percussion", "instrument_type": "PERCUSSION", "notes": [ { "pitch": 36, "start_beat": 0.0, "duration_beats": 0.5, "velocity": 1.0 }, { "pitch": 36, "start_beat": 2.0, "duration_beats": 0.5, "velocity": 0.95 }, { "pitch": 38, "start_beat": 3.5, "duration_beats": 0.25, "velocity": 0.8 } ] }, { "track_name": "Orchestral Cymbals", "instrument_type": "DRUMS", "notes": [ { "pitch": 49, "start_beat": 0.0, "duration_beats": 4.0, "velocity": 0.9 }, { "pitch": 49, "start_beat": 16.0, "duration_beats": 4.0, "velocity": 1.0 } ] } ] } } ``` --- ## 5. DAW State Ingestion Logic When the client receives the AI JSON payload, the `ingestAIGeneratedMIDI()` function executes the following pipeline: 1. Spawns or matches corresponding tracks within `main_session`. 2. Generates 8-bar `MIDIItem` objects containing the note lists. 3. Triggers UI canvas and timeline re-renders. ```javascript function ingestAIGeneratedMIDI(aiPayload, sessionState) { const { composition_title, bpm, total_bars, tracks } = aiPayload.result; // 1. Update project BPM if specified if (bpm) sessionState.metadata.bpm = bpm; // 2. Iterate through generated tracks tracks.forEach((aiTrack) => { // Match existing track or instantiate a new one let targetTrack = sessionState.main_session.tracks.find( t => t.name.toLowerCase() === aiTrack.track_name.toLowerCase() ); if (!targetTrack) { targetTrack = { id: `track_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`, name: aiTrack.track_name, type: "MIDI", volume_db: 0.0, pan: 0.0, mute: false, solo: false, items: [] }; sessionState.main_session.tracks.push(targetTrack); } // 3. Create 8-bar MIDIItem const newMidiItem = { id: `item_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`, name: `${composition_title} - ${aiTrack.track_name}`, type: "MIDI_ITEM", start_bar: 0.0, // Placed at Timeline start or active Playhead position duration_bars: total_bars, clip_start_offset_bars: 0.0, source_data: { total_buffer_bars: total_bars, notes: aiTrack.notes.map((note, index) => ({ id: `note_ai_${Date.now()}_${index}`, pitch: note.pitch, start_beat: note.start_beat, duration_beats: note.duration_beats, velocity: note.velocity, pan: 0.0 })) } }; // 4. Append Item to Track targetTrack.items.push(newMidiItem); }); // 5. Trigger Timeline & Piano Roll UI Re-render Event window.dispatchEvent(new CustomEvent('DAW_STATE_UPDATED', { detail: sessionState })); } ``` --- ## 6. Prompt Preset Manager UI Layout ```text +-------------------------------------------------------------------------------+ | AI PROMPT PRESET MANAGER [ + New ]| +-------------------------------------------------------------------------------+ | SEARCH: [ epic orchestra ] FILTER: [ Orchestral v ]| | | | Preset Name Keywords matched Default Bars Actions | | --------------------------------------------------------------------------- | | [★] Epic Orchestra Intro epic, orchestra, tráng 8 Bars [Edit][Del] | | [ ] Pop Piano Chords pop, piano, chord 4 Bars [Edit][Del] | | [ ] Cyberpunk Synth Synth synth, synthwave, 80s 8 Bars [Edit][Del] | | | +-------------------------------------------------------------------------------+ | EDIT PRESET: Epic Orchestra Intro | | | | Keyword Triggers (comma-separated): | | [ epic orchestra, hoành tráng, nhạc phim epic ] | | | | System Instruction / Music Rules: | | +---------------------------------------------------------------------------+ | | | Compose an 8-bar epic orchestral theme featuring Brass, staccato | | | | Strings, Taiko percussion, and background Synth bass... | | | +---------------------------------------------------------------------------+ | | [ CANCEL ] [ SAVE ] +-------------------------------------------------------------------------------+ ```