feat: bổ sung MIDI
This commit is contained in:
@@ -0,0 +1,760 @@
|
||||
Here is the complete translation and conversion of the document into a clean, professionally formatted Markdown layout:
|
||||
|
||||
# ARCHITECTURAL, TECHNICAL, AND ALGORITHMIC SPECIFICATION
|
||||
|
||||
## Hybrid Web-Based Digital Audio Workstation (DAW) with Nested Section Architecture and Non-Destructive Timeline Mechanics
|
||||
|
||||
---
|
||||
|
||||
### 1. System Overview & Architecture Design
|
||||
|
||||
#### 1.1 High-Level Architecture Topology
|
||||
|
||||
The system follows a hybrid Client-Server architecture designed for real-time Web-based audio production, composition, and high-performance offline DSP rendering.
|
||||
|
||||
* **Frontend Client (HTML5 / Vanilla JS / Web Audio API / HTML5 Canvas)**
|
||||
* **UI Layer:** HTML5 Canvas / Web Components for high-FPS multi-lane timeline rendering, Piano Roll canvas, Sample Editor, and Sub-Tab navigation.
|
||||
* **Audio Engine Layer:** Web Audio API `AudioContext` graph, Custom `AudioWorklet` Processors (WebAssembly/JS) for real-time synthesis, playback scheduling, sample playback, and latency-compensated signal routing.
|
||||
* **State Management Engine:** Immutable/Reactive Central State Store handling Session tree hierarchy, Section Store registries, Undo/Redo stack, and view-state context isolation.
|
||||
|
||||
|
||||
* **Backend Server (Python Engine)**
|
||||
* **RESTful / WebSocket API:** Event-driven client communication layer (FastAPI or AIOHTTP).
|
||||
* **DSP / Rendering Engine:** Python-based audio processing (`numpy`, `scipy`, `pyo`, `pedalboard`) for offline stem bouncing, high-fidelity export, sample processing, and optional VST/VSTi hosting/bridging.
|
||||
|
||||
|
||||
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| FRONTEND (HTML5/JS) |
|
||||
| |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | UI & View State System | |
|
||||
| | +---------------------+ +----------------------+ +--------------------+ | |
|
||||
| | | Main Session Canvas | | Section-Tab View | | Piano Roll View | | |
|
||||
| | +---------------------+ +----------------------+ +--------------------+ | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | Central Data State Store | |
|
||||
| | [Project Model] ---> [Section Store] ---> [Item Clip Metadata] | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | Audio & Clock Engine | |
|
||||
| | +------------------------+ +------------------+ +---------------------+ | |
|
||||
| | | Precision Scheduler | | Web Audio Graph | | AudioWorklet Synth | | |
|
||||
| | | (Lookahead Timer) | | AudioNode Router | | / WebAssembly Core | | |
|
||||
| | +------------------------+ +------------------+ +---------------------+ | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
+------------------------------------------^----------------------------------------+
|
||||
| WebSocket / REST API
|
||||
+------------------------------------------v----------------------------------------+
|
||||
| BACKEND SERVER (PYTHON) |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | FastAPI / WebSocket Handler | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | DSP Engine (Pedalboard / Numpy / Scipy) - Offline Render, Audio Export | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | VST / VSTi Hosting Bridge & Plugin State Persistence | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Detailed Data Schemas (JSON Specification)
|
||||
|
||||
#### 2.1 Project Root Schema (`project_schema.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "DAWProject",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"project_id": { "type": "string", "format": "uuid" },
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": { "type": "string" },
|
||||
"bpm": { "type": "number", "minimum": 20.0, "maximum": 999.0, "default": 120.0 },
|
||||
"time_signature_numerator": { "type": "integer", "default": 4 },
|
||||
"time_signature_denominator": { "type": "integer", "default": 4 },
|
||||
"sample_rate": { "type": "integer", "default": 44100 }
|
||||
},
|
||||
"required": ["title", "bpm", "time_signature_numerator", "time_signature_denominator", "sample_rate"]
|
||||
},
|
||||
"main_session": { "$ref": "#/definitions/SessionContainer" },
|
||||
"section_store": {
|
||||
"type": "object",
|
||||
"description": "Auxiliary registry mapping section_id to sub-session containers",
|
||||
"additionalProperties": { "$ref": "#/definitions/SessionContainer" }
|
||||
}
|
||||
},
|
||||
"required": ["project_id", "metadata", "main_session", "section_store"],
|
||||
"definitions": {
|
||||
"SessionContainer": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"is_root": { "type": "boolean" },
|
||||
"length_bars": { "type": "number", "description": "Computed or manually set total length in bars" },
|
||||
"auto_compute_length": { "type": "boolean", "default": true },
|
||||
"tracks": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/Track" }
|
||||
}
|
||||
},
|
||||
"required": ["id", "is_root", "tracks"]
|
||||
},
|
||||
"Track": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["AUDIO", "MIDI", "SECTION"] },
|
||||
"volume_db": { "type": "number", "default": 0.0 },
|
||||
"pan": { "type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0 },
|
||||
"mute": { "type": "boolean", "default": false },
|
||||
"solo": { "type": "boolean", "default": false },
|
||||
"fx_chain": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/FXPlugin" }
|
||||
},
|
||||
"synth_engine": { "$ref": "#/definitions/SynthPlugin" },
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/TimelineItem" }
|
||||
}
|
||||
},
|
||||
"required": ["id", "name", "type", "items"]
|
||||
},
|
||||
"TimelineItem": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["AUDIO_ITEM", "MIDI_ITEM", "SECTION_ITEM"] },
|
||||
"start_bar": { "type": "number", "description": "Global timeline position where the item starts" },
|
||||
"duration_bars": { "type": "number", "description": "Visible duration on the track timeline in bars" },
|
||||
"clip_start_offset_bars": { "type": "number", "description": "Internal start offset inside the source buffer/item" },
|
||||
"source_data": {
|
||||
"type": "object",
|
||||
"oneOf": [
|
||||
{ "$ref": "#/definitions/AudioSourceData" },
|
||||
{ "$ref": "#/definitions/MIDISourceData" },
|
||||
{ "$ref": "#/definitions/SectionSourceData" }
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": ["id", "type", "start_bar", "duration_bars", "clip_start_offset_bars", "source_data"]
|
||||
},
|
||||
"AudioSourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"audio_file_url": { "type": "string" },
|
||||
"sample_rate": { "type": "integer" },
|
||||
"channels": { "type": "integer" },
|
||||
"gain": { "type": "number", "default": 1.0 }
|
||||
},
|
||||
"required": ["audio_file_url"]
|
||||
},
|
||||
"MIDISourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"total_buffer_bars": { "type": "number", "default": 8.0 },
|
||||
"notes": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/MIDINote" }
|
||||
}
|
||||
},
|
||||
"required": ["total_buffer_bars", "notes"]
|
||||
},
|
||||
"SectionSourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"referenced_section_id": { "type": "string", "description": "Pointer to section_store key" }
|
||||
},
|
||||
"required": ["referenced_section_id"]
|
||||
},
|
||||
"MIDINote": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
|
||||
"start_beat": { "type": "number", "description": "Beat offset relative to the start of the source buffer (bar 0)" },
|
||||
"duration_beats": { "type": "number" },
|
||||
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0, "default": 0.8 },
|
||||
"pan": { "type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0 }
|
||||
},
|
||||
"required": ["id", "pitch", "start_beat", "duration_beats", "velocity"]
|
||||
},
|
||||
"FXPlugin": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"plugin_id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"bypass": { "type": "boolean", "default": false },
|
||||
"parameters": { "type": "object" }
|
||||
}
|
||||
},
|
||||
"SynthPlugin": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"plugin_id": { "type": "string" },
|
||||
"preset_id": { "type": "string" },
|
||||
"parameters": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. UI, Tab Navigation & View State Management
|
||||
|
||||
#### 3.1 Tab Context Model, Pinning Rules & Close Prevention Hierarchy
|
||||
|
||||
The application manages view tabs dynamically while maintaining strict lifecycle integrity:
|
||||
|
||||
* **Main Session Tab (Fixed / Pinned):** Always pinned at index 0 (`is_closeable: false`). It cannot be closed under any circumstances.
|
||||
* **Sub-Tabs (Section-Tab, Piano Roll Tab, Audio Sample Editor Sub-Tab):** Dynamic views (`is_closeable: true`).
|
||||
* **Parent-Child Tab Dependency Rules:**
|
||||
* A Section-Tab represents an intermediate sub-session.
|
||||
* When a user opens a child item (e.g., a `MIDIItem` or `AudioItem` inside a Section-Tab) into a Piano Roll Tab or Audio Sample Editor Sub-Tab, a parent-child context lineage is registered.
|
||||
* **Close Block Rule:** A Section-Tab cannot be closed while any of its child items are currently open in active sub-tabs. Attempting to close the parent Section-Tab displays a block notice highlighting open child editors.
|
||||
|
||||
|
||||
|
||||
```text
|
||||
+---------------------------------------+
|
||||
| Tab Navigation Controller |
|
||||
+-------------------+-------------------+
|
||||
|
|
||||
+--------------------------------+--------------------------------+
|
||||
| (Pinned / Uncloseable) | (Dynamic / Closable) | (Dynamic / Closable)
|
||||
+--------v--------+ +--------v--------+ +--------v--------+
|
||||
| MAIN SESSION | | SECTION TAB | | PIANO ROLL TAB |
|
||||
| (Root Context) | | (Sub-Session) | | (Item Context) |
|
||||
| | | [Parent Context] | [Child Context]|
|
||||
+-----------------+ +--------+--------+ +--------+--------+
|
||||
| |
|
||||
+---- Depends on child closure ---+
|
||||
|
||||
```
|
||||
|
||||
##### State Object Schema with Tab Dependency Tracking:
|
||||
|
||||
```json
|
||||
{
|
||||
"active_tab_id": "tab_pr_1",
|
||||
"open_tabs": [
|
||||
{
|
||||
"tab_id": "tab_root",
|
||||
"title": "MAIN SESSION",
|
||||
"type": "MAIN_SESSION",
|
||||
"target_id": "main",
|
||||
"is_closeable": false,
|
||||
"parent_tab_id": null
|
||||
},
|
||||
{
|
||||
"tab_id": "tab_sec_1",
|
||||
"title": "Section: Verse 1",
|
||||
"type": "SECTION_TAB",
|
||||
"target_id": "Section_01",
|
||||
"is_closeable": true,
|
||||
"parent_tab_id": "tab_root"
|
||||
},
|
||||
{
|
||||
"tab_id": "tab_pr_1",
|
||||
"title": "Piano Roll: Bassline",
|
||||
"type": "PIANO_ROLL",
|
||||
"target_id": "ItemMIDI_Bassline",
|
||||
"is_closeable": true,
|
||||
"parent_tab_id": "tab_sec_1"
|
||||
}
|
||||
],
|
||||
"piano_roll_state": {
|
||||
"target_item_id": "ItemMIDI_Bassline",
|
||||
"viewport_start_bar": 0.0,
|
||||
"viewport_bar_width": 8.0,
|
||||
"scroll_y_pitch": 60,
|
||||
"snap_resolution": "1/16",
|
||||
"note_selection": []
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 3.2 Piano Roll View Canvas Layout & Interaction Spec
|
||||
|
||||
* **Top Navigation Rule Pane (Bars/Beats Bar):**
|
||||
* Displays bars from $0$ to $N$ (where $N = \text{total\_buffer\_bars}$, e.g., 8 bars).
|
||||
* Highlights active clip visibility bounds (e.g., Bar 4.0 to Bar 6.0 shaded with active overlay, exterior bars dimmed).
|
||||
|
||||
|
||||
* **Left Piano Keybed:**
|
||||
* Anchored vertically, spans pitches $0$ (C-1) through $127$ (G9).
|
||||
* Draws standard 88 key / 128 key pattern with distinct black key visually offset bars and pitch labeling ($C3$, $C4$, etc.).
|
||||
|
||||
|
||||
* **Note Grid Canvas (Right Pane):**
|
||||
* Synced to vertical pitch scroll and horizontal beat zoom.
|
||||
* **Row Background Rendering:** Black key rows are assigned darker background fill color `#1A1A1E`, white key rows use `#25252A`.
|
||||
* **Snap Grid Lines:** Rendered dynamically based on selected snap mode: Free, 1/1 Bar, 1/2 Beat, 1/4 Beat, 1/8 Beat, 1/16 Beat, 1/32 Beat.
|
||||
|
||||
|
||||
* **Bottom Controller Pane (CC / Velocity / Pan Lane):**
|
||||
* Synchronized horizontally with note grid.
|
||||
* Displays vertical stem bars per note representing properties (Velocity, Pan). Allows click-and-drag line shaping or direct stem adjustment.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 4. Audio & Synth Engine Routing Architecture (Web Audio API)
|
||||
|
||||
#### 4.1 Real-Time Signal Flow Graph
|
||||
|
||||
```text
|
||||
[MIDI Scheduler] ---> [AudioWorklet / Virtual Synth Engine]
|
||||
|
|
||||
v (Audio Buffer / Stream)
|
||||
[Audio Sample Playback Node] ----> [Track Channel FX Chain]
|
||||
|
|
||||
v
|
||||
[Track Gain / Pan Node]
|
||||
|
|
||||
v
|
||||
+---------------------+---------------------+
|
||||
| |
|
||||
v (If inside Section) v (If Direct Track)
|
||||
[Section Sub-Mix Bus] [Main Master Mixer Bus]
|
||||
| |
|
||||
+-------------------->----------------------+
|
||||
|
|
||||
v
|
||||
[Web Audio Destination]
|
||||
|
||||
```
|
||||
|
||||
#### 4.2 Web Audio Node Architecture Specifications
|
||||
|
||||
* **AudioTrack Node Structure:**
|
||||
```javascript
|
||||
TrackAudioGraph = {
|
||||
inputNode: GainNode,
|
||||
fxChain: [ BiquadFilterNode, DelayNode, ConvolverNode ],
|
||||
panNode: StereoPannerNode,
|
||||
outputGainNode: GainNode,
|
||||
connect(destination) { ... }
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
* **Section Bus Graph Routing:**
|
||||
* Each Section in Section-tab Store instantiates an intermediate `GainNode` sub-mixer (`SectionBus`).
|
||||
* Tracks within the Section connect their final outputs to `SectionBus`.
|
||||
* When a `SectionItem` is placed on a Main Session track, the `SectionBus` output is routed into the Main Session track's input node, preserving non-destructive DSP processing hierarchies.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 5. Core Mathematical & Technical Algorithms
|
||||
|
||||
#### 5.1 Algorithm 1: Non-Destructive Item Slicing & Offset Playback Math
|
||||
|
||||
##### Mathematical Formulation
|
||||
|
||||
Let:
|
||||
|
||||
* $T_{\text{global}}$ = Current global playback time in seconds on the main timeline.
|
||||
* $\text{BPM}$ = Beats Per Minute of the project.
|
||||
* $\text{TS}_{\text{num}}$ = Time Signature Numerator (e.g., 4 beats per bar).
|
||||
* $S_{\text{item}}$ = Item start position in global bars ($\text{start\_bar}$).
|
||||
* $L_{\text{item}}$ = Item visible length on timeline in bars ($\text{duration\_bars}$).
|
||||
* $O_{\text{item}}$ = Source internal start offset in bars ($\text{clip\_start\_offset\_bars}$).
|
||||
|
||||
Bar to Time Conversion Factor:
|
||||
|
||||
$$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$
|
||||
|
||||
$$\text{SecondsPerBar} = \text{SecondsPerBeat} \times \text{TS}_{\text{num}}$$
|
||||
|
||||
Item Global Time Bounds:
|
||||
|
||||
$$T_{\text{start}} = S_{\text{item}} \times \text{SecondsPerBar}$$
|
||||
|
||||
$$T_{\text{end}} = (S_{\text{item}} + L_{\text{item}}) \times \text{SecondsPerBar}$$
|
||||
|
||||
Active Playback Slicing Condition: An item is active if and only if:
|
||||
|
||||
$$T_{\text{start}} \le T_{\text{global}} < T_{\text{end}}$$
|
||||
|
||||
Local Item Buffer Time Mapping ($T_{\text{local}}$): When $T_{\text{global}}$ falls within $[T_{\text{start}}, T_{\text{end}}]$, the corresponding time $T_{\text{local\_bars}}$ relative to the internal source clip buffer (0 to $\text{BufferLength}$) is:
|
||||
|
||||
$$T_{\text{local\_bars}} = \frac{T_{\text{global}} - T_{\text{start}}}{\text{SecondsPerBar}} + O_{\text{item}}$$
|
||||
|
||||
MIDI Note Slicing & Filtering Rule: For a MIDI note $N$ inside the item source with start beat $N_{\text{start\_beat}}$ and length $N_{\text{dur\_beat}}$ (converted to internal bar metric $N_{\text{bar\_start}} = \frac{N_{\text{start\_beat}}}{\text{TS}_{\text{num}}}$, $N_{\text{bar\_dur}} = \frac{N_{\text{dur\_beat}}}{\text{TS}_{\text{num}}}$):
|
||||
|
||||
The note is triggered during main playback if and only if:
|
||||
|
||||
$$N_{\text{bar\_start}} \ge O_{\text{item}} \quad \text{AND} \quad N_{\text{bar\_start}} < (O_{\text{item}} + L_{\text{item}})$$
|
||||
|
||||
##### Pseudocode Implementation
|
||||
|
||||
```javascript
|
||||
function getActiveMIDINotesForPlayback(item, currentGlobalBar, timeSigNum) {
|
||||
const itemStartBar = item.start_bar;
|
||||
const itemEndBar = item.start_bar + item.duration_bars;
|
||||
const offsetBar = item.clip_start_offset_bars;
|
||||
|
||||
// Check if playback cursor is inside visible item clip
|
||||
if (currentGlobalBar < itemStartBar || currentGlobalBar >= itemEndBar) {
|
||||
return []; // Item inactive
|
||||
}
|
||||
|
||||
const activeNotes = [];
|
||||
const internalWindowStartBar = offsetBar;
|
||||
const internalWindowEndBar = offsetBar + item.duration_bars;
|
||||
|
||||
for (const note of item.source_data.notes) {
|
||||
const noteStartBar = note.start_beat / timeSigNum;
|
||||
const noteEndBar = noteStartBar + (note.duration_beats / timeSigNum);
|
||||
|
||||
// Filter notes outside the non-destructive visible window
|
||||
if (noteStartBar >= internalWindowStartBar && noteStartBar < internalWindowEndBar) {
|
||||
// Calculate playback time relative to global session
|
||||
const relativeBarInItem = noteStartBar - internalWindowStartBar;
|
||||
const targetGlobalBar = itemStartBar + relativeBarInItem;
|
||||
|
||||
activeNotes.push({
|
||||
note: note,
|
||||
scheduledGlobalBar: targetGlobalBar
|
||||
});
|
||||
}
|
||||
}
|
||||
return activeNotes;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.2 Algorithm 2: Dynamic Section Length Calculation Algorithm
|
||||
|
||||
When `auto_compute_length` is enabled for a Section, its total duration in bars $L_{\text{section}}$ is dynamically evaluated from the boundary bounds of all child items across all tracks inside that Section.
|
||||
|
||||
##### Mathematical Formulation
|
||||
|
||||
Let $T$ be the set of tracks in the section, and $I(t)$ be the set of items in track $t$.
|
||||
|
||||
$$L_{\text{section}} = \max_{t \in T} \left( \max_{i \in I(t)} \left( i.\text{start\_bar} + i.\text{duration\_bars} \right) \right)$$
|
||||
|
||||
If $I(t)$ is empty for all $t$, then $L_{\text{section}} = 4.0$ (default baseline minimum).
|
||||
|
||||
##### Implementation Architecture
|
||||
|
||||
```javascript
|
||||
function recomputeSectionLength(sectionContainer) {
|
||||
if (!sectionContainer.auto_compute_length) {
|
||||
return sectionContainer.length_bars;
|
||||
}
|
||||
|
||||
let maxEndBar = 0.0;
|
||||
|
||||
for (const track of sectionContainer.tracks) {
|
||||
for (const item of track.items) {
|
||||
const itemEndBar = item.start_bar + item.duration_bars;
|
||||
if (itemEndBar > maxEndBar) {
|
||||
maxEndBar = itemEndBar;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Enforce baseline grid quantization rounding (e.g. minimum 1 bar)
|
||||
const computedLength = Math.max(1.0, Math.ceil(maxEndBar));
|
||||
sectionContainer.length_bars = computedLength;
|
||||
|
||||
return computedLength;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.3 Algorithm 3: Piano Roll Grid Mapping & Quantization Math
|
||||
|
||||
##### Grid Coordinate Transformation Formulae
|
||||
|
||||
Let:
|
||||
|
||||
* $X_{\text{px}}$ = Pixel X-coordinate on Piano Roll Canvas.
|
||||
* $Y_{\text{px}}$ = Pixel Y-coordinate on Piano Roll Canvas.
|
||||
* $\text{Zoom}_x$ = Pixels per Beat.
|
||||
* $\text{NoteHeight}$ = Height in pixels per pitch key row (e.g., 18px).
|
||||
* $\text{Scroll}_x$ = Horizontal scroll offset in beats.
|
||||
* $\text{Scroll}_y$ = Vertical scroll top note pitch (e.g., pitch 127 down to 0).
|
||||
|
||||
Beat to Canvas Pixel Conversion:
|
||||
|
||||
$$X_{\text{px}} = (\text{Beat} - \text{Scroll}_x) \times \text{Zoom}_x$$
|
||||
|
||||
$$\text{Beat} = \frac{X_{\text{px}}}{\text{Zoom}_x} + \text{Scroll}_x$$
|
||||
|
||||
Pitch to Canvas Pixel Conversion:
|
||||
|
||||
$$Y_{\text{px}} = (127 - \text{Pitch} - \text{Scroll}_y) \times \text{NoteHeight}$$
|
||||
|
||||
$$\text{Pitch} = 127 - \left\lfloor \frac{Y_{\text{px}}}{\text{NoteHeight}} \right\rfloor - \text{Scroll}_y$$
|
||||
|
||||
##### Quantization (Snap To Grid) Math
|
||||
|
||||
Let $Q$ be the snap unit in beats (e.g., $1/4 \text{ bar} = 1.0 \text{ beat}$, $1/16 \text{ note} = 0.25 \text{ beat}$). Given raw unquantized beat $B_{\text{raw}}$:
|
||||
|
||||
$$B_{\text{quantized}} = \text{round}\left(\frac{B_{\text{raw}}}{Q}\right) \times Q$$
|
||||
|
||||
#### 5.4 Algorithm 4: Tab Close Dependency & Lifecycle Validation Algorithm
|
||||
|
||||
This algorithm validates whether a tab close request can be fulfilled, enforcing the fixed Main Session constraint and preventing parent Section tab closures while child editor sub-tabs remain active.
|
||||
|
||||
```javascript
|
||||
function requestCloseTab(tabIdToClose, stateStore) {
|
||||
const targetTab = stateStore.open_tabs.find(tab => tab.tab_id === tabIdToClose);
|
||||
if (!targetTab) {
|
||||
return { success: false, reason: "TAB_NOT_FOUND" };
|
||||
}
|
||||
|
||||
// 1. Rule: Main Session cannot be closed
|
||||
if (!targetTab.is_closeable || targetTab.type === 'MAIN_SESSION') {
|
||||
return { success: false, reason: "CANNOT_CLOSE_MAIN_SESSION" };
|
||||
}
|
||||
|
||||
// 2. Rule: Section Tab cannot be closed if child tabs are active
|
||||
if (targetTab.type === 'SECTION_TAB') {
|
||||
const activeChildTabs = stateStore.open_tabs.filter(
|
||||
tab => tab.parent_tab_id === targetTab.tab_id
|
||||
);
|
||||
|
||||
if (activeChildTabs.length > 0) {
|
||||
return {
|
||||
success: false,
|
||||
reason: "SECTION_HAS_ACTIVE_CHILD_EDITORS",
|
||||
activeChildTabs: activeChildTabs.map(t => ({ id: t.tab_id, title: t.title }))
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Execution: Perform clean tab shutdown and update active context
|
||||
const updatedTabs = stateStore.open_tabs.filter(tab => tab.tab_id !== tabIdToClose);
|
||||
|
||||
// Fallback active tab selection if current active tab is being closed
|
||||
let nextActiveTabId = stateStore.active_tab_id;
|
||||
if (stateStore.active_tab_id === tabIdToClose) {
|
||||
// Fallback to parent tab, or default to main session (index 0)
|
||||
nextActiveTabId = targetTab.parent_tab_id || updatedTabs[0].tab_id;
|
||||
}
|
||||
|
||||
stateStore.open_tabs = updatedTabs;
|
||||
stateStore.active_tab_id = nextActiveTabId;
|
||||
|
||||
return { success: true, nextActiveTabId: nextActiveTabId };
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.5 Algorithm 5: Sample-Accurate Lookahead MIDI & Audio Scheduler
|
||||
|
||||
Web Audio API timing operates on a high-precision hardware audio clock (`audioContext.currentTime`). JavaScript timers (`setTimeout`/`setInterval`) lack frame accuracy. The Lookahead Scheduler combines JS interval ticks with Web Audio precision scheduling.
|
||||
|
||||
```text
|
||||
Lookahead Window (e.g. 100ms)
|
||||
|-------------------------------------------|
|
||||
| AudioContext Time: 10.0s |
|
||||
| Schedule horizon: 10.1s |
|
||||
| |
|
||||
| [Event 1 @ 10.02s] -> Scheduled in WebAudio
|
||||
| [Event 2 @ 10.08s] -> Scheduled in WebAudio
|
||||
|___________________________________________|
|
||||
|
||||
```
|
||||
|
||||
##### Scheduler Specification
|
||||
|
||||
```javascript
|
||||
class PrecisionAudioScheduler {
|
||||
constructor(audioCtx, lookaheadMs = 25.0, scheduleAheadTimeSec = 0.1) {
|
||||
this.audioCtx = audioCtx;
|
||||
this.lookaheadMs = lookaheadMs; // Frequency of timer evaluation
|
||||
this.scheduleAheadTime = scheduleAheadTimeSec; // How far ahead to queue WebAudio events
|
||||
this.nextNoteBeat = 0.0;
|
||||
this.currentBeat = 0.0;
|
||||
this.bpm = 120.0;
|
||||
this.timerId = null;
|
||||
}
|
||||
|
||||
beatToTime(beat) {
|
||||
const secondsPerBeat = 60.0 / this.bpm;
|
||||
return beat * secondsPerBeat;
|
||||
}
|
||||
|
||||
timeToBeat(timeSec) {
|
||||
const secondsPerBeat = 60.0 / this.bpm;
|
||||
return timeSec / secondsPerBeat;
|
||||
}
|
||||
|
||||
schedulerTick(activeSession) {
|
||||
const currentTime = this.audioCtx.currentTime;
|
||||
const horizonTime = currentTime + this.scheduleAheadTime;
|
||||
|
||||
// Traverse session items and find notes falling within [currentTime, horizonTime]
|
||||
const pendingEvents = activeSession.getEventsInTimeRange(
|
||||
this.timeToBeat(currentTime),
|
||||
this.timeToBeat(horizonTime)
|
||||
);
|
||||
|
||||
for (const evt of pendingEvents) {
|
||||
if (!evt.scheduled) {
|
||||
const preciseAudioTime = currentTime + this.beatToTime(evt.targetBeat - this.currentBeat);
|
||||
this.triggerWebAudioEvent(evt, preciseAudioTime);
|
||||
evt.scheduled = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
triggerWebAudioEvent(evt, exactAudioTime) {
|
||||
if (evt.type === 'MIDI_NOTE_ON') {
|
||||
const synthNode = evt.trackSynthNode;
|
||||
synthNode.noteOn(evt.note.pitch, evt.note.velocity, exactAudioTime);
|
||||
synthNode.noteOff(evt.note.pitch, exactAudioTime + this.beatToTime(evt.note.duration_beats));
|
||||
} else if (evt.type === 'AUDIO_CLIP') {
|
||||
const sourceNode = this.audioCtx.createBufferSource();
|
||||
sourceNode.buffer = evt.audioBuffer;
|
||||
sourceNode.connect(evt.trackGainNode);
|
||||
sourceNode.start(exactAudioTime, evt.offsetSec, evt.durationSec);
|
||||
}
|
||||
}
|
||||
|
||||
start(session) {
|
||||
this.timerId = setInterval(() => this.schedulerTick(session), this.lookaheadMs);
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (this.timerId) clearInterval(this.timerId);
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.6 Algorithm 6: Playhead UI Rendering Sync Loop
|
||||
|
||||
UI Playhead rendering uses `requestAnimationFrame` and queries `audioContext.currentTime` directly to prevent visual jitter or lag.
|
||||
|
||||
$$\text{Current Beat UI} = \frac{\text{audioCtx.currentTime} - \text{PlaybackStartTimeSec}}{\text{SecondsPerBeat}}$$
|
||||
|
||||
$$\text{Pixel Position X} = (\text{Current Beat UI} - \text{ViewportStartBeat}) \times \text{Zoom}_x$$
|
||||
|
||||
---
|
||||
|
||||
### 6. Backend Python Server Architecture & Offline Render Spec
|
||||
|
||||
#### 6.1 Server Architecture Framework
|
||||
|
||||
* **Framework:** FastAPI with Async WebSocket endpoints for real-time state synchronization.
|
||||
* **DSP Engine:** `pedalboard` (Spotify's Python Audio Processing Library) and `numpy` for multi-track mixing, high-quality audio resampling, and plugin hosting.
|
||||
|
||||
#### 6.2 Python Offline Stem Bouncing Engine Specification (`render_engine.py`)
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
from pedalboard import Pedalboard, Gain, Reverb, Compressor
|
||||
import soundfile as sf
|
||||
|
||||
class PythonRenderEngine:
|
||||
def __init__(self, sample_rate=44100):
|
||||
self.sample_rate = sample_rate
|
||||
|
||||
def bars_to_samples(self, bars: float, bpm: float, time_sig_num: int) -> int:
|
||||
seconds_per_beat = 60.0 / bpm
|
||||
seconds_per_bar = seconds_per_beat * time_sig_num
|
||||
return int(bars * seconds_per_bar * self.sample_rate)
|
||||
|
||||
def render_project(self, project_json: dict, output_filepath: str):
|
||||
bpm = project_json["metadata"]["bpm"]
|
||||
time_sig_num = project_json["metadata"]["time_signature_numerator"]
|
||||
main_session = project_json["main_session"]
|
||||
|
||||
# 1. Compute total project samples
|
||||
total_bars = main_session.get("length_bars", 16.0)
|
||||
total_samples = self.bars_to_samples(total_bars, bpm, time_sig_num)
|
||||
|
||||
# Stereo Master Buffer
|
||||
master_buffer = np.zeros((2, total_samples), dtype=np.float32)
|
||||
|
||||
# 2. Iterate and process main tracks
|
||||
for track in main_session["tracks"]:
|
||||
track_type = track["type"]
|
||||
track_buffer = np.zeros((2, total_samples), dtype=np.float32)
|
||||
|
||||
for item in track["items"]:
|
||||
start_sample = self.bars_to_samples(item["start_bar"], bpm, time_sig_num)
|
||||
dur_samples = self.bars_to_samples(item["duration_bars"], bpm, time_sig_num)
|
||||
offset_sample = self.bars_to_samples(item["clip_start_offset_bars"], bpm, time_sig_num)
|
||||
|
||||
if item["type"] == "AUDIO_ITEM":
|
||||
# Load audio source sample array
|
||||
audio_data, sr = sf.read(item["source_data"]["audio_file_url"], dtype='float32')
|
||||
audio_data = audio_data.T # Shape: (channels, samples)
|
||||
|
||||
# Apply non-destructive trimming offset
|
||||
sliced_audio = audio_data[:, offset_sample : offset_sample + dur_samples]
|
||||
|
||||
# Accumulate into track buffer with bounds checks
|
||||
end_sample = min(start_sample + sliced_audio.shape[1], total_samples)
|
||||
actual_len = end_sample - start_sample
|
||||
track_buffer[:, start_sample:end_sample] += sliced_audio[:, :actual_len]
|
||||
|
||||
# Apply Track Gain and FX Chain via Pedalboard
|
||||
board = Pedalboard([Gain(gain_db=track.get("volume_db", 0.0))])
|
||||
processed_track = board(track_buffer, sample_rate=self.sample_rate)
|
||||
|
||||
# Mix down to Master
|
||||
master_buffer += processed_track
|
||||
|
||||
# 3. Write final output file
|
||||
sf.write(output_filepath, master_buffer.T, self.sample_rate)
|
||||
return output_filepath
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Execution Context & Sub-Tab Lifecycle Matrix
|
||||
|
||||
| Context Tab Type | Scope Identifier | View Boundaries | Is Closeable | Close Dependency Conditions | Audio Routing Target |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **MAIN SESSION** | Root | Full Master Timeline ($0 \to N$ Bars) | No | Pinned permanently; cannot be closed | WebAudio Hardware Destination |
|
||||
| **SECTION TAB** | Section_ID | Dynamic Section Bounds ($0 \to L_{\text{section}}$) | Yes | Blocked if any child editor sub-tabs are open | Target Section Bus Gain Node |
|
||||
| **PIANO ROLL** | MIDIItem_ID | Item Source Length Bounds ($0 \to N_{\text{buffer}}$) | Yes | Can close freely; notifies parent Section tab | Track Instrument Synth Engine |
|
||||
| **SAMPLE EDITOR** | AudioItem_ID | Sample Buffer Waveform ($0 \to T_{\text{sample}}$) | Yes | Can close freely; notifies parent Section tab | Track Audio Node Router |
|
||||
|
||||
---
|
||||
|
||||
### 8. Summary of Non-Destructive Slice & Tab Lifecycle Validation
|
||||
|
||||
* **Tab Close Prevention Test:**
|
||||
1. `MAIN SESSION` close request is rejected immediately (`CANNOT_CLOSE_MAIN_SESSION`).
|
||||
2. `Section_01` tab has an active child editor tab (`Piano Roll: Bassline`).
|
||||
3. Request to close `Section_01` tab returns `SECTION_HAS_ACTIVE_CHILD_EDITORS`.
|
||||
4. User closes `Piano Roll: Bassline` tab first.
|
||||
5. Subsequent close request for `Section_01` succeeds and cleans up UI context.
|
||||
|
||||
|
||||
* **8-Bar Source with 2-Bar Visible Crop Test:**
|
||||
1. Given `MIDIItem` length = 8 bars ($0 \dots 8$).
|
||||
2. User drags left boundary to Bar 4 and right boundary to Bar 6.
|
||||
3. `start_bar = 4.0` (Global Session Placement), `duration_bars = 2.0`, `clip_start_offset_bars = 4.0`.
|
||||
4. Transport reaches global Bar 4.0 $\to$ scheduler evaluates internal bounds $[4.0, 6.0)$ and triggers only visible notes while preserving complete 8-bar non-destructive source.
|
||||
Reference in New Issue
Block a user