fix(export): MIDI export qua server endpoint thay blob download (WebView2 chặn) + velocity 1..127, bỏ rest-track
This commit is contained in:
@@ -0,0 +1,373 @@
|
||||
# DESIGN SPECIFICATION & IMPLEMENTATION ROADMAP: FLOATING VIRTUAL MIDI KEYBOARD & CHORDS PANEL ENGINE
|
||||
|
||||
This document specifies the technical architecture, UI/UX design, data flow diagrams, and interaction matrices for integrating two core features into the DAW system:
|
||||
|
||||
1. **Floating Virtual MIDI Keyboard:** A draggable floating virtual keyboard window that receives input from computer QWERTY keys, supporting both live preview and real-time recording of MIDI notes directly into the Timeline / Piano Roll.
|
||||
2. **Comprehensive Scale & Chords System / Insert Chords Panel:** A multi-genre chord theory engine, automated chord progression insertion panel, custom chord builder/storage, and internet/AI chord lookup system.
|
||||
|
||||
---
|
||||
|
||||
## I. SYSTEM ARCHITECTURE OVERVIEW
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CLIENT FRONTEND STUDIO │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────┐ ┌─────────────────────────────────────┐ │
|
||||
│ │ Menu: Tools -> Virtual MIDI Keyboard │ │ Menu: Insert -> Chords panel │ │
|
||||
│ │ Shortcut: [F2] │ │ Shortcut: [Shift + K] │ │
|
||||
│ └────────────────────┬────────────────────┘ └──────────────────┬──────────────────┘ │
|
||||
└────────────────────────┼─────────────────────────────────────────────────┼──────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────────┐ ┌────────────────────────────────────────┐
|
||||
│ FLOATING VIRTUAL MIDI KEYBOARD (UI OVERLAY) │ │ INSERT CHORDS PANEL (MODAL / SIDEBAR) │
|
||||
│ - Visual 25/49-Keybed Rendering │ │ - Style Catalog (Pop, Jazz, Epic...) │
|
||||
│ - QWERTY Key Mapping Engine │ │ - Custom Chord Builder & LocalStorage │
|
||||
│ - Octave Shift (-4 to +4) & Velocity Adjuster │ │ - Internet / AI Chord Search Engine │
|
||||
└────────────────────────┬─────────────────────────┘ └─────────────────┬──────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ UNIFIED MIDI ROUTER & DISPATCHER │
|
||||
│ - Normalizes to `UnifiedMidiEvent`: { command, channel, pitch, velocity, timestamp } │
|
||||
└────────────────────────┬─────────────────────────────────────────────────┬──────────────────────┘
|
||||
│ │
|
||||
├─────────────────────────────────┐ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────────────────────────────────────┐ ┌────────────────────────────────────────────┐
|
||||
│ REAL-TIME SYNTH ENGINE (CLIENT WASM / BRIDGE) │ │ CLIENT MIDI RECORDER & TIMELINE INGESTION │
|
||||
│ - Real-time Audio Preview (< 5ms Latency) │ │ - Live Recording to active `MIDIItem` │
|
||||
│ - FluidSynth WASM / Native VSTi Bridge │ │ - Direct Canvas Redraw on Piano Roll │
|
||||
└──────────────────────────────────────────────────┘ └────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## II. PART I: FLOATING VIRTUAL MIDI KEYBOARD SYSTEM
|
||||
|
||||
### 1. Activation & Floating Window Management
|
||||
|
||||
* **Activation Triggers:**
|
||||
* **System Shortcut:** **F2** (Toggle On/Off).
|
||||
* **Menu Bar:** **Tools ▾ -> Virtual MIDI Keyboard**.
|
||||
|
||||
|
||||
* **Window Properties:**
|
||||
* **Floating & Draggable:** Allows dynamic positioning across MAIN SESSION, SECTION-TAB, and PIANO ROLL TAB views.
|
||||
* **Always-on-Top / Z-Index Isolation:** Renders above all Timeline Canvases without capturing global Transport hotkeys (**Space** for Play/Stop, **R** for Record).
|
||||
* **State Persistence:** Saves coordinates ($x, y$), active toggle state, current octave offset, and default velocity to `localStorage`.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 2. Virtual Keyboard User Interface Layout
|
||||
|
||||
```text
|
||||
┌────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 🎹 VIRTUAL MIDI KEYBOARD [–] [X] │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ [ Octave: -1 ] [ Octave: +1 ] │ Octave Shift: 4 (C4-C6) │ Velocity: [ 100 ] [Slider---|] │
|
||||
│ [ Channel: 1 ▾ ] │ Transpose: 0 semitones │ Scale Highlight: [ C Minor ▾ ] │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | │
|
||||
│ | |W| |E | |T| |Y| |U | |2| |3 | |5| |6| |7 | | | | | | | | | | | │
|
||||
│ | |_| |_| | |_| |_| |_| | |_| |_| | |_| |_| |_| | |_| |_| | |_| |_| |_| | │
|
||||
│ | A | S | D | F | G | H | J | K | Q | W | E | R | T | Y | U | I | O | P | | | | │
|
||||
│ └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘ │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
#### Keyboard Control Bar Components:
|
||||
|
||||
* **Octave Down / Up (`[Oct -]`, `[Oct +]`):** Shifts pitch octave range (Shortcuts: **Shift + Z** / **Shift + X** or **[** / **]**).
|
||||
* **Velocity Slider:** Adjusts note velocity from **1 to 127** (Default: **100**).
|
||||
* **Target MIDI Channel:** Selects MIDI Channel from **1 to 16** (Defaults automatically to the currently selected/armed track's channel).
|
||||
* **Scale Highlight Toggle:** Visual keybed guide highlighting keys that belong to the active musical scale.
|
||||
|
||||
---
|
||||
|
||||
### 3. Computer QWERTY Keyboard Mapping Matrix
|
||||
|
||||
Uses two rows of QWERTY keys to span two continuous octaves:
|
||||
|
||||
#### Lower Octave (Root Base):
|
||||
|
||||
| Computer Key | Relative Pitch | Note Name | Key Type |
|
||||
| --- | --- | --- | --- |
|
||||
| **Z** | $\text{Root} + 0$ | C | White Key |
|
||||
| **S** | $\text{Root} + 1$ | C# / Db | Black Key |
|
||||
| **X** | $\text{Root} + 2$ | D | White Key |
|
||||
| **D** | $\text{Root} + 3$ | D# / Eb | Black Key |
|
||||
| **C** | $\text{Root} + 4$ | E | White Key |
|
||||
| **V** | $\text{Root} + 5$ | F | White Key |
|
||||
| **G** | $\text{Root} + 6$ | F# / Gb | Black Key |
|
||||
| **B** | $\text{Root} + 7$ | G | White Key |
|
||||
| **H** | $\text{Root} + 8$ | G# / Ab | Black Key |
|
||||
| **N** | $\text{Root} + 9$ | A | White Key |
|
||||
| **J** | $\text{Root} + 10$ | A# / Bb | Black Key |
|
||||
| **M** | $\text{Root} + 11$ | B | White Key |
|
||||
|
||||
#### Upper Octave ($+12$ Semitones):
|
||||
|
||||
| Computer Key | Relative Pitch | Note Name | Key Type |
|
||||
| --- | --- | --- | --- |
|
||||
| **Q** | $\text{Root} + 12$ | C (+1 Oct) | White Key |
|
||||
| **2** | $\text{Root} + 13$ | C# (+1 Oct) | Black Key |
|
||||
| **W** | $\text{Root} + 14$ | D (+1 Oct) | White Key |
|
||||
| **3** | $\text{Root} + 15$ | D# (+1 Oct) | Black Key |
|
||||
| **E** | $\text{Root} + 16$ | E (+1 Oct) | White Key |
|
||||
| **R** | $\text{Root} + 17$ | F (+1 Oct) | White Key |
|
||||
| **5** | $\text{Root} + 18$ | F# (+1 Oct) | Black Key |
|
||||
| **T** | $\text{Root} + 19$ | G (+1 Oct) | White Key |
|
||||
| **6** | $\text{Root} + 20$ | G# (+1 Oct) | Black Key |
|
||||
| **Y** | $\text{Root} + 21$ | A (+1 Oct) | White Key |
|
||||
| **7** | $\text{Root} + 22$ | A# (+1 Oct) | Black Key |
|
||||
| **U** | $\text{Root} + 23$ | B (+1 Oct) | White Key |
|
||||
| **I** | $\text{Root} + 24$ | C (+2 Oct) | White Key |
|
||||
|
||||
---
|
||||
|
||||
### 4. Audio Playback & Recording Event Pipeline
|
||||
|
||||
#### Key Press Handler (`KeyDown` Event):
|
||||
|
||||
1. Verifies `e.repeat` is false to prevent event flood on key holds.
|
||||
2. If the active DOM element is a text input field (`<input>`, `<textarea>`), passes through and skips the piano handler.
|
||||
3. Computes the absolute MIDI Pitch value:
|
||||
|
||||
$$Pitch = (Octave + 1) \times 12 + PitchOffset$$
|
||||
|
||||
4. Highlights the corresponding visual key on the Virtual Keyboard UI.
|
||||
5. Dispatches the event to the `UnifiedMidiRouter`:
|
||||
|
||||
```javascript
|
||||
unifiedMidiRouter.dispatchMidiEvent({
|
||||
command: 'NOTE_ON',
|
||||
channel: activeTrackChannel,
|
||||
pitch: calculatedPitch,
|
||||
velocity: currentVirtualVelocity,
|
||||
sourceType: 'VIRTUAL_KEYBOARD'
|
||||
});
|
||||
|
||||
```
|
||||
|
||||
6. **Live Recording Mode (If Transport Record is ACTIVE):**
|
||||
The `ClientMIDIRecorder` captures the `NOTE_ON` event, creates a note instance in the active `MIDIItem` on the armed track, and invokes a `requestAnimationFrame` loop to render real-time red/orange preview blocks on the Timeline and Piano Roll canvases.
|
||||
|
||||
#### Key Release Handler (`KeyUp` Event):
|
||||
|
||||
1. Removes key highlight from the Virtual Keyboard UI.
|
||||
2. Dispatches a `NOTE_OFF` event to `UnifiedMidiRouter`.
|
||||
3. `ClientMIDIRecorder` calculates total note duration:
|
||||
|
||||
$$Duration = Beat_{\text{release}} - Beat_{\text{press}}$$
|
||||
|
||||
and writes the finalized note data to the clip's state array.
|
||||
|
||||
---
|
||||
|
||||
## III. PART II: COMPREHENSIVE SCALE & CHORD SYSTEM & CHORDS PANEL
|
||||
|
||||
### 1. Chords Panel Trigger
|
||||
|
||||
* **Activation Triggers:**
|
||||
* **Menu Bar:** **Insert ▾ -> Chords panel**.
|
||||
* **System Shortcut:** **Shift + K** or clicking the 🎼 **Chords** icon on the Piano Roll Toolbar.
|
||||
|
||||
|
||||
* **Display Format:**
|
||||
* Centered modal window or sliding dockable panel positioned on the right side of the Piano Roll.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 2. Style-Based Chord Catalog
|
||||
|
||||
Contains pre-coded chord progressions categorized by musical genre and emotional mood:
|
||||
|
||||
```text
|
||||
┌────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 🎼 INSERT CHORDS PANEL [X] │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ [ Filter Style: Pop / Ballad ▾ ] [ Key: C ▾ ] [ Scale: Major ▾ ] [ Search: _______ ] │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ PRESET PROGRESSIONS: │
|
||||
│ ┌────────────────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 🌟 Pop Classic Emotional (I - V - vi - IV) │ │
|
||||
│ │ Chords: Cmaj -> Gmaj -> Am -> Fmaj | Voicing: Root Position | Rhythm: 1/2 Beat │ │
|
||||
│ │ [ Preview Sound ] [ Insert to Timeline ] [ Favorite ★ ] │ │
|
||||
│ ├────────────────────────────────────────────────────────────────────────────────────┤ │
|
||||
│ │ 🎷 Jazz Neo-Soul Smooth (ii7 - V7 - Imaj7 - VI7) │ │
|
||||
│ │ Chords: Dm7 -> G7 -> Cmaj7 -> A7 | Voicing: Drop-2 | Rhythm: Syncopated │ │
|
||||
│ │ [ Preview Sound ] [ Insert to Timeline ] [ Favorite ★ ] │ │
|
||||
│ ├────────────────────────────────────────────────────────────────────────────────────┤ │
|
||||
│ │ 🎬 Epic Film Score Rising (i - VI - III - VII) │ │
|
||||
│ │ Chords: Cm -> Ab -> Eb -> Bb | Voicing: Power Octaves | Rhythm: 8th Arp │ │
|
||||
│ │ [ Preview Sound ] [ Insert to Timeline ] [ Favorite ★ ] │ │
|
||||
│ └────────────────────────────────────────────────────────────────────────────────────┘ │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ TAB: [ Preset Library ] [ Custom Chord Builder ] [ Internet / AI Search ] │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
#### Music Style Catalog Matrix:
|
||||
|
||||
| Genre / Style | Standard Progression (Roman Numerals) | Transposed Example (Root Key C) | Voicing & Rhythmic Characteristics |
|
||||
| --- | --- | --- | --- |
|
||||
| **Pop / Ballad** | $I - V - vi - IV$ | $C - G - Am - F$ | Sustained Whole Notes, Triad Voicings |
|
||||
| **Jazz / Neo-Soul** | $ii^7 - V^7 - I^{\text{maj}7} - VI^7$ | $Dm^7 - G^7 - C^{\text{maj}7} - A^7$ | Drop-2 Voicings, Off-beat Syncopation |
|
||||
| **Cinematic / Epic** | $i - VI - III - VII$ | $Cm - Ab - Eb - Bb$ | Low-octave bass root, Octave layering, 8th-note arpeggiation |
|
||||
| **EDM / Future Bass** | $vi - IV - I - V$ | $Am - F - C - G$ | 16th-note synth stabs, Inverted voicings |
|
||||
| **Lofi Chillhop** | $i^7 - iv^7 - v^7 - i^7$ | $Cm^7 - Fm^7 - Gm^7 - Cm^7$ | Add9 / Min7 Extensions, Micro-timing jitter |
|
||||
| **R&B / Soul** | $I^{\text{maj}7} - iii^7 - IV^{\text{maj}7} - V^{13}$ | $C^{\text{maj}7} - Em^7 - F^{\text{maj}7} - G^{13}$ | Rolled Strumming, 9th/11th Extensions |
|
||||
|
||||
---
|
||||
|
||||
### 3. Custom Chord Builder & Storage
|
||||
|
||||
Enables users to create and store custom chord progressions persistently for subsequent sessions:
|
||||
|
||||
#### Custom Builder Input Fields:
|
||||
|
||||
* **Progression Name:** e.g., *"Sad Indie Pop Progression 2026"*.
|
||||
* **Genre Category:** Dropdown selection or free-text tag.
|
||||
* **Chord Tokens:**
|
||||
* Roman Numeral input: `I - vi - IV - V`
|
||||
* Direct Chord Name input: `Cmaj7 - Am9 - Fadd9 - G13`
|
||||
|
||||
|
||||
* **Rhythm Pattern:** Whole Note (Sustained), Quarter Stabs, Arpeggiated, Strummed.
|
||||
|
||||
#### Persistence Engine:
|
||||
|
||||
* Persists to `localStorage` under the key `daw_user_custom_chords`.
|
||||
* Dispatches a custom event `CUSTOM_CHORD_SAVED` to update the Chords Panel list in real time without refreshing the page.
|
||||
|
||||
---
|
||||
|
||||
### 4. Internet & AI Chord Search Engine
|
||||
|
||||
Allows users to look up chord progressions for any song or style query directly through an integrated AI Gateway or web endpoint:
|
||||
|
||||
```text
|
||||
┌────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 🔍 SEARCH CHORDS VIA INTERNET & AI GATEWAY │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Search: [ Hotel California - Eagles / Melancholy Lofi Progression... ] [ Search ] │
|
||||
├────────────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ AI / SEARCH RESULTS: │
|
||||
│ │
|
||||
│ 🎵 Song: Hotel California (Eagles) - Key: B Minor │
|
||||
│ Progression: Bm -> F#7 -> A -> E7 -> G -> D -> Em -> F#7 │
|
||||
│ AI Analysis: "Classic Flamenco-influenced rock progression using secondary dominants"│
|
||||
│ │
|
||||
│ [ 🔊 Listen Preview ] [ ➕ Save to My Library ] [ 🎹 Insert to Piano Roll ] │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
#### Query Flow & Data Extraction:
|
||||
|
||||
1. User enters a song title or style description into the search bar.
|
||||
2. The client submits a request to `/api/v1/ai/search-chords` or dispatches an AI Gateway call with the following Tool Schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search_or_generate_chords",
|
||||
"description": "Searches for existing chord progressions or generates custom progressions matching a target style.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"song_or_style_title": { "type": "string" },
|
||||
"detected_key": { "type": "string" },
|
||||
"chords_list": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"chord_name": { "type": "string" },
|
||||
"roman_numeral": { "type": "string" },
|
||||
"pitches": { "type": "array", "items": { "type": "integer" } },
|
||||
"duration_beats": { "type": "number", "default": 4.0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
3. The API/Backend returns structured MIDI pitch numbers ($0 \to 127$).
|
||||
4. The client inserts the parsed notes directly into the Piano Roll Canvas.
|
||||
|
||||
---
|
||||
|
||||
### 5. Chord Insertion Engine
|
||||
|
||||
When clicking **[ Insert to Timeline / Piano Roll ]**:
|
||||
|
||||
#### Insertion Algorithm:
|
||||
|
||||
1. Let $Beat_{\text{insert}}$ represent the Playhead timestamp (or clicked beat index on the Piano Roll).
|
||||
2. For each chord $C_k$ in the progression sequence:
|
||||
* Compute starting beat position:
|
||||
|
||||
|
||||
|
||||
$$StartBeat(C_k) = Beat_{\text{insert}} + \sum_{i=0}^{k-1} Duration(C_i)$$
|
||||
|
||||
* Convert chord symbols to MIDI pitch arrays (e.g., $C^{\text{maj}7}$ at Key C4, where $\text{Pitch} = 60$):
|
||||
|
||||
$$Pitches(C^{\text{maj}7}) = [60, 64, 67, 71] \quad (C4, E4, G4, B4)$$
|
||||
|
||||
* Apply chosen Voicing/Inversion rules:
|
||||
* **Root Position:** Standard ascending pitch order.
|
||||
* **First Inversion:** Transpose root pitch up $+12$ semitones.
|
||||
* **Drop-2 Voicing:** Transpose the second-highest note down $-12$ semitones.
|
||||
|
||||
|
||||
* Generate `MIDINote` objects:
|
||||
|
||||
```javascript
|
||||
pitches.forEach(pitch => {
|
||||
newNotes.push({
|
||||
id: `note_chord_${Date.now()}_${pitch}`,
|
||||
pitch: pitch,
|
||||
start_beat: startBeat(C_k),
|
||||
duration_beats: duration(C_k),
|
||||
velocity: 0.8,
|
||||
pan: 0.0
|
||||
});
|
||||
});
|
||||
|
||||
```
|
||||
|
||||
3. **State Mutation & Canvas Redraw:**
|
||||
* If an active `MIDIItem` is open: Injects `newNotes` into `item.source_data.notes`.
|
||||
* Dispatches the `DAW_STATE_UPDATED` event to trigger an immediate canvas redraw across both the Piano Roll and Timeline views.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## IV. VERIFICATION & TEST PLAN (E2E CHECKLIST)
|
||||
|
||||
| Test Item | Action | Expected Result |
|
||||
| --- | --- | --- |
|
||||
| **1. Toggle Virtual Keyboard (F2)** | Press **F2** or select **Tools -> Virtual MIDI Keyboard**. | The Floating Keyboard window opens overlaid on the DAW view; pressing **F2** again closes/hides the window. |
|
||||
| **2. QWERTY Preview Playback** | Press **Z, S, X, D, C, V, G, B** on the computer keyboard. | Virtual keys light up; audio from the active track's synth engine triggers immediately with latency **< 5ms**. |
|
||||
| **3. Live Recording to Timeline** | Arm track with **[R]** $\rightarrow$ Click **Record + Play** on Transport $\rightarrow$ Play QWERTY keys. | Notes render in real time on the Timeline canvas; clicking **Stop** creates a finalized `MIDIItem` containing the recorded notes. |
|
||||
| **4. Open Chords Panel** | Select **Insert -> Chords panel** (or press **Shift + K**). | Chords Panel opens displaying progression presets for Pop, Jazz, Epic, and EDM styles. |
|
||||
| **5. Insert Progression into Piano Roll** | Select *Jazz Neo-Soul* progression $\rightarrow$ Click **[ Insert to Piano Roll ]**. | $Dm^7 - G^7 - C^{\text{maj}7} - A^7$ sequence inserts at the Playhead location with calculated durations. |
|
||||
| **6. Custom Chord Builder** | Open Custom Builder tab $\rightarrow$ Enter `Cmaj7 - Am9 - Fadd9` $\rightarrow$ Click **Save**. | New progression saves to `localStorage` and appears under the user's custom catalog. |
|
||||
| **7. AI / Online Chord Lookup** | Open Internet Search tab $\rightarrow$ Query *"Hotel California"* $\rightarrow$ Click **Search**. | Resolves progression ($Bm - F\#7 - A - E7...$) with preview playback and direct insertion actions. |
|
||||
Reference in New Issue
Block a user