3dtours/SonicForgeStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
SonicForgeStudio/virtual_keyboard_and_chords_panel_spec.md
T

24 KiB
Raw Permalink Blame History

DESIGN SPECIFICATION & IMPLEMENTATION ROADMAP: FLOATING VIRTUAL MIDI KEYBOARD & CHORDS PANEL ENGINE

This document specifies the technical architecture, UI/UX design, data flow diagrams, and interaction matrices for integrating two core features into the DAW system:

  1. Floating Virtual MIDI Keyboard: A draggable floating virtual keyboard window that receives input from computer QWERTY keys, supporting both live preview and real-time recording of MIDI notes directly into the Timeline / Piano Roll.
  2. Comprehensive Scale & Chords System / Insert Chords Panel: A multi-genre chord theory engine, automated chord progression insertion panel, custom chord builder/storage, and internet/AI chord lookup system.

I. SYSTEM ARCHITECTURE OVERVIEW

┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                   CLIENT FRONTEND STUDIO                                        │
│                                                                                                 │
│   ┌─────────────────────────────────────────┐         ┌─────────────────────────────────────┐   │
│   │ Menu: Tools -> Virtual MIDI Keyboard    │         │ Menu: Insert -> Chords panel        │   │
│   │ Shortcut: [F2]                          │         │ Shortcut: [Shift + K]               │   │
│   └────────────────────┬────────────────────┘         └──────────────────┬──────────────────┘   │
└────────────────────────┼─────────────────────────────────────────────────┼──────────────────────┘
                         │                                                 │
                         ▼                                                 ▼
┌──────────────────────────────────────────────────┐     ┌────────────────────────────────────────┐
│ FLOATING VIRTUAL MIDI KEYBOARD (UI OVERLAY)      │     │ INSERT CHORDS PANEL (MODAL / SIDEBAR)   │
│ - Visual 25/49-Keybed Rendering                  │     │ - Style Catalog (Pop, Jazz, Epic...)   │
│ - QWERTY Key Mapping Engine                      │     │ - Custom Chord Builder & LocalStorage  │
│ - Octave Shift (-4 to +4) & Velocity Adjuster    │     │ - Internet / AI Chord Search Engine    │
└────────────────────────┬─────────────────────────┘     └─────────────────┬──────────────────────┘
                         │                                                 │
                         ▼                                                 ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                UNIFIED MIDI ROUTER & DISPATCHER                                 │
│ - Normalizes to `UnifiedMidiEvent`: { command, channel, pitch, velocity, timestamp }            │
└────────────────────────┬─────────────────────────────────────────────────┬──────────────────────┘
                         │                                                 │
                         ├─────────────────────────────────┐               │
                         ▼                                 ▼               ▼
┌──────────────────────────────────────────────────┐ ┌────────────────────────────────────────────┐
│ REAL-TIME SYNTH ENGINE (CLIENT WASM / BRIDGE)    │ │ CLIENT MIDI RECORDER & TIMELINE INGESTION  │
│ - Real-time Audio Preview (< 5ms Latency)        │ │ - Live Recording to active `MIDIItem`      │
│ - FluidSynth WASM / Native VSTi Bridge           │ │ - Direct Canvas Redraw on Piano Roll       │
└──────────────────────────────────────────────────┘ └────────────────────────────────────────────┘

II. PART I: FLOATING VIRTUAL MIDI KEYBOARD SYSTEM

1. Activation & Floating Window Management

  • Activation Triggers:

  • System Shortcut: F2 (Toggle On/Off).

  • Menu Bar: Tools ▾ -> Virtual MIDI Keyboard.

  • Window Properties:

  • Floating & Draggable: Allows dynamic positioning across MAIN SESSION, SECTION-TAB, and PIANO ROLL TAB views.

  • Always-on-Top / Z-Index Isolation: Renders above all Timeline Canvases without capturing global Transport hotkeys (Space for Play/Stop, R for Record).

  • State Persistence: Saves coordinates (x, y), active toggle state, current octave offset, and default velocity to localStorage.

2. Virtual Keyboard User Interface Layout

┌────────────────────────────────────────────────────────────────────────────────────────┐
│ 🎹 VIRTUAL MIDI KEYBOARD                                                       [–] [X] │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ [ Octave: -1 ] [ Octave: +1 ] │ Octave Shift: 4 (C4-C6) │ Velocity: [ 100 ] [Slider---|]  │
│ [ Channel: 1 ▾ ]              │ Transpose: 0 semitones  │ Scale Highlight: [ C Minor ▾ ] │
├────────────────────────────────────────────────────────────────────────────────────────┤
│  |   | | |   |   | | | | |   |   | | |   |   | | | | |   |   | | |   |   | | | | |   | │
│  |   |W| |E  |   |T| |Y| |U  |   |2| |3  |   |5| |6| |7  |   | | |   |   | | | | |   | │
│  |   |_| |_| |   |_| |_| |_| |   |_| |_| |   |_| |_| |_| |   |_| |_| |   |_| |_| |_| | │
│  | A | S | D | F | G | H | J | K | Q | W | E | R | T | Y | U | I | O | P |   |   |   | │
│  └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘ │
└────────────────────────────────────────────────────────────────────────────────────────┘

Keyboard Control Bar Components:

  • Octave Down / Up ([Oct -], [Oct +]): Shifts pitch octave range (Shortcuts: Shift + Z / Shift + X or [ / ]).
  • Velocity Slider: Adjusts note velocity from 1 to 127 (Default: 100).
  • Target MIDI Channel: Selects MIDI Channel from 1 to 16 (Defaults automatically to the currently selected/armed track's channel).
  • Scale Highlight Toggle: Visual keybed guide highlighting keys that belong to the active musical scale.

3. Computer QWERTY Keyboard Mapping Matrix

Uses two rows of QWERTY keys to span two continuous octaves:

Lower Octave (Root Base):

Computer Key Relative Pitch Note Name Key Type
Z \text{Root} + 0 C White Key
S \text{Root} + 1 C# / Db Black Key
X \text{Root} + 2 D White Key
D \text{Root} + 3 D# / Eb Black Key
C \text{Root} + 4 E White Key
V \text{Root} + 5 F White Key
G \text{Root} + 6 F# / Gb Black Key
B \text{Root} + 7 G White Key
H \text{Root} + 8 G# / Ab Black Key
N \text{Root} + 9 A White Key
J \text{Root} + 10 A# / Bb Black Key
M \text{Root} + 11 B White Key

Upper Octave (+12 Semitones):

Computer Key Relative Pitch Note Name Key Type
Q \text{Root} + 12 C (+1 Oct) White Key
2 \text{Root} + 13 C# (+1 Oct) Black Key
W \text{Root} + 14 D (+1 Oct) White Key
3 \text{Root} + 15 D# (+1 Oct) Black Key
E \text{Root} + 16 E (+1 Oct) White Key
R \text{Root} + 17 F (+1 Oct) White Key
5 \text{Root} + 18 F# (+1 Oct) Black Key
T \text{Root} + 19 G (+1 Oct) White Key
6 \text{Root} + 20 G# (+1 Oct) Black Key
Y \text{Root} + 21 A (+1 Oct) White Key
7 \text{Root} + 22 A# (+1 Oct) Black Key
U \text{Root} + 23 B (+1 Oct) White Key
I \text{Root} + 24 C (+2 Oct) White Key

4. Audio Playback & Recording Event Pipeline

Key Press Handler (KeyDown Event):

  1. Verifies e.repeat is false to prevent event flood on key holds.
  2. If the active DOM element is a text input field (<input>, <textarea>), passes through and skips the piano handler.
  3. Computes the absolute MIDI Pitch value:
Pitch = (Octave + 1) \times 12 + PitchOffset
  1. Highlights the corresponding visual key on the Virtual Keyboard UI.
  2. Dispatches the event to the UnifiedMidiRouter:
unifiedMidiRouter.dispatchMidiEvent({
  command: 'NOTE_ON',
  channel: activeTrackChannel,
  pitch: calculatedPitch,
  velocity: currentVirtualVelocity,
  sourceType: 'VIRTUAL_KEYBOARD'
});
  1. Live Recording Mode (If Transport Record is ACTIVE): The ClientMIDIRecorder captures the NOTE_ON event, creates a note instance in the active MIDIItem on the armed track, and invokes a requestAnimationFrame loop to render real-time red/orange preview blocks on the Timeline and Piano Roll canvases.

Key Release Handler (KeyUp Event):

  1. Removes key highlight from the Virtual Keyboard UI.
  2. Dispatches a NOTE_OFF event to UnifiedMidiRouter.
  3. ClientMIDIRecorder calculates total note duration:
Duration = Beat_{\text{release}} - Beat_{\text{press}}

and writes the finalized note data to the clip's state array.

III. PART II: COMPREHENSIVE SCALE & CHORD SYSTEM & CHORDS PANEL

1. Chords Panel Trigger

  • Activation Triggers:

  • Menu Bar: Insert ▾ -> Chords panel.

  • System Shortcut: Shift + K or clicking the 🎼 Chords icon on the Piano Roll Toolbar.

  • Display Format:

  • Centered modal window or sliding dockable panel positioned on the right side of the Piano Roll.

2. Style-Based Chord Catalog

Contains pre-coded chord progressions categorized by musical genre and emotional mood:

┌────────────────────────────────────────────────────────────────────────────────────────┐
│ 🎼 INSERT CHORDS PANEL                                                         [X]     │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ [ Filter Style: Pop / Ballad ▾ ] [ Key: C ▾ ] [ Scale: Major ▾ ] [ Search: _______ ]  │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ PRESET PROGRESSIONS:                                                                   │
│ ┌────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ 🌟 Pop Classic Emotional (I - V - vi - IV)                                         │ │
│ │    Chords: Cmaj -> Gmaj -> Am -> Fmaj | Voicing: Root Position | Rhythm: 1/2 Beat   │ │
│ │    [ Preview Sound ] [ Insert to Timeline ] [ Favorite ★ ]                         │ │
│ ├────────────────────────────────────────────────────────────────────────────────────┤ │
│ │ 🎷 Jazz Neo-Soul Smooth (ii7 - V7 - Imaj7 - VI7)                                   │ │
│ │    Chords: Dm7 -> G7 -> Cmaj7 -> A7 | Voicing: Drop-2 | Rhythm: Syncopated        │ │
│ │    [ Preview Sound ] [ Insert to Timeline ] [ Favorite ★ ]                         │ │
│ ├────────────────────────────────────────────────────────────────────────────────────┤ │
│ │ 🎬 Epic Film Score Rising (i - VI - III - VII)                                     │ │
│ │    Chords: Cm -> Ab -> Eb -> Bb | Voicing: Power Octaves | Rhythm: 8th Arp         │ │
│ │    [ Preview Sound ] [ Insert to Timeline ] [ Favorite ★ ]                         │ │
│ └────────────────────────────────────────────────────────────────────────────────────┘ │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ TAB: [ Preset Library ]  [ Custom Chord Builder ]  [ Internet / AI Search ]            │
└────────────────────────────────────────────────────────────────────────────────────────┘

Music Style Catalog Matrix:

Genre / Style Standard Progression (Roman Numerals) Transposed Example (Root Key C) Voicing & Rhythmic Characteristics
Pop / Ballad I - V - vi - IV C - G - Am - F Sustained Whole Notes, Triad Voicings
Jazz / Neo-Soul ii^7 - V^7 - I^{\text{maj}7} - VI^7 Dm^7 - G^7 - C^{\text{maj}7} - A^7 Drop-2 Voicings, Off-beat Syncopation
Cinematic / Epic i - VI - III - VII Cm - Ab - Eb - Bb Low-octave bass root, Octave layering, 8th-note arpeggiation
EDM / Future Bass vi - IV - I - V Am - F - C - G 16th-note synth stabs, Inverted voicings
Lofi Chillhop i^7 - iv^7 - v^7 - i^7 Cm^7 - Fm^7 - Gm^7 - Cm^7 Add9 / Min7 Extensions, Micro-timing jitter
R&B / Soul I^{\text{maj}7} - iii^7 - IV^{\text{maj}7} - V^{13} C^{\text{maj}7} - Em^7 - F^{\text{maj}7} - G^{13} Rolled Strumming, 9th/11th Extensions

3. Custom Chord Builder & Storage

Enables users to create and store custom chord progressions persistently for subsequent sessions:

Custom Builder Input Fields:

  • Progression Name: e.g., "Sad Indie Pop Progression 2026".

  • Genre Category: Dropdown selection or free-text tag.

  • Chord Tokens:

  • Roman Numeral input: I - vi - IV - V

  • Direct Chord Name input: Cmaj7 - Am9 - Fadd9 - G13

  • Rhythm Pattern: Whole Note (Sustained), Quarter Stabs, Arpeggiated, Strummed.

Persistence Engine:

  • Persists to localStorage under the key daw_user_custom_chords.
  • Dispatches a custom event CUSTOM_CHORD_SAVED to update the Chords Panel list in real time without refreshing the page.

4. Internet & AI Chord Search Engine

Allows users to look up chord progressions for any song or style query directly through an integrated AI Gateway or web endpoint:

┌────────────────────────────────────────────────────────────────────────────────────────┐
│ 🔍 SEARCH CHORDS VIA INTERNET & AI GATEWAY                                              │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ Search: [ Hotel California - Eagles / Melancholy Lofi Progression...      ] [ Search ] │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ AI / SEARCH RESULTS:                                                                   │
│                                                                                        │
│ 🎵 Song: Hotel California (Eagles) - Key: B Minor                                      │
│    Progression: Bm -> F#7 -> A -> E7 -> G -> D -> Em -> F#7                            │
│    AI Analysis: "Classic Flamenco-influenced rock progression using secondary dominants"│
│                                                                                        │
│    [ 🔊 Listen Preview ]  [ ➕ Save to My Library ]  [ 🎹 Insert to Piano Roll ]      │
└────────────────────────────────────────────────────────────────────────────────────────┘

Query Flow & Data Extraction:

  1. User enters a song title or style description into the search bar.
  2. The client submits a request to /api/v1/ai/search-chords or dispatches an AI Gateway call with the following Tool Schema:
{
  "type": "function",
  "function": {
    "name": "search_or_generate_chords",
    "description": "Searches for existing chord progressions or generates custom progressions matching a target style.",
    "parameters": {
      "type": "object",
      "properties": {
        "song_or_style_title": { "type": "string" },
        "detected_key": { "type": "string" },
        "chords_list": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "chord_name": { "type": "string" },
              "roman_numeral": { "type": "string" },
              "pitches": { "type": "array", "items": { "type": "integer" } },
              "duration_beats": { "type": "number", "default": 4.0 }
            }
          }
        }
      }
    }
  }
}
  1. The API/Backend returns structured MIDI pitch numbers (0 \to 127).
  2. The client inserts the parsed notes directly into the Piano Roll Canvas.

5. Chord Insertion Engine

When clicking [ Insert to Timeline / Piano Roll ]:

Insertion Algorithm:

  1. Let Beat_{\text{insert}} represent the Playhead timestamp (or clicked beat index on the Piano Roll).
  2. For each chord C_k in the progression sequence:
  • Compute starting beat position:
StartBeat(C_k) = Beat_{\text{insert}} + \sum_{i=0}^{k-1} Duration(C_i)
  • Convert chord symbols to MIDI pitch arrays (e.g., C^{\text{maj}7} at Key C4, where \text{Pitch} = 60):
Pitches(C^{\text{maj}7}) = [60, 64, 67, 71] \quad (C4, E4, G4, B4)

  • Apply chosen Voicing/Inversion rules:

  • Root Position: Standard ascending pitch order.

  • First Inversion: Transpose root pitch up +12 semitones.

  • Drop-2 Voicing: Transpose the second-highest note down -12 semitones.

  • Generate MIDINote objects:

pitches.forEach(pitch => {
  newNotes.push({
    id: `note_chord_${Date.now()}_${pitch}`,
    pitch: pitch,
    start_beat: startBeat(C_k),
    duration_beats: duration(C_k),
    velocity: 0.8,
    pan: 0.0
  });
});
  1. State Mutation & Canvas Redraw:
  • If an active MIDIItem is open: Injects newNotes into item.source_data.notes.
  • Dispatches the DAW_STATE_UPDATED event to trigger an immediate canvas redraw across both the Piano Roll and Timeline views.

IV. VERIFICATION & TEST PLAN (E2E CHECKLIST)

Test Item Action Expected Result
1. Toggle Virtual Keyboard (F2) Press F2 or select Tools -> Virtual MIDI Keyboard. The Floating Keyboard window opens overlaid on the DAW view; pressing F2 again closes/hides the window.
2. QWERTY Preview Playback Press Z, S, X, D, C, V, G, B on the computer keyboard. Virtual keys light up; audio from the active track's synth engine triggers immediately with latency < 5ms.
3. Live Recording to Timeline Arm track with [R] \rightarrow Click Record + Play on Transport \rightarrow Play QWERTY keys. Notes render in real time on the Timeline canvas; clicking Stop creates a finalized MIDIItem containing the recorded notes.
4. Open Chords Panel Select Insert -> Chords panel (or press Shift + K). Chords Panel opens displaying progression presets for Pop, Jazz, Epic, and EDM styles.
5. Insert Progression into Piano Roll Select Jazz Neo-Soul progression \rightarrow Click [ Insert to Piano Roll ]. Dm^7 - G^7 - C^{\text{maj}7} - A^7 sequence inserts at the Playhead location with calculated durations.
6. Custom Chord Builder Open Custom Builder tab \rightarrow Enter Cmaj7 - Am9 - Fadd9 \rightarrow Click Save. New progression saves to localStorage and appears under the user's custom catalog.
7. AI / Online Chord Lookup Open Internet Search tab \rightarrow Query "Hotel California" \rightarrow Click Search. Resolves progression (Bm - F\#7 - A - E7...) with preview playback and direct insertion actions.
Reference in New Issue View Git Blame Copy Permalink