feat: bổ sung MIDI
This commit is contained in:
@@ -0,0 +1,426 @@
|
||||
# TECHNICAL SPECIFICATION: CLIENT-SIDE REAL-TIME RECORDING ENGINE
|
||||
|
||||
## Browser-Based Microphone & Hardware MIDI Keyboard Recording Module
|
||||
|
||||
---
|
||||
|
||||
## 1. System Overview
|
||||
|
||||
The Client-side recording module enables the DAW to capture live audio signals directly from Microphone/Line-in interfaces (via the Web MediaDevices API) and keypress events from Hardware MIDI Keyboards/Controllers (via the Web MIDI API) in real time. The module operates with low latency and includes hardware latency compensation.
|
||||
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| CLIENT BROWSER |
|
||||
| |
|
||||
| +-------------------------+ +-----------------------------+ |
|
||||
| | Hardware MIDI Keyboard | | Live Microphone / Line-In | |
|
||||
| +------------+------------+ +--------------+--------------+ |
|
||||
| | | |
|
||||
| v (Web MIDI API) v (getUserMedia) |
|
||||
| +------------+------------+ +--------------+--------------+ |
|
||||
| | Web MIDI Input Handler | | MediaStreamAudioSourceNode | |
|
||||
| +------------+------------+ +--------------+--------------+ |
|
||||
| | | |
|
||||
| +------------------+ | |
|
||||
| | | v |
|
||||
| v v +--------------+--------------+ |
|
||||
| +------------+-----+ +---------+-----------+ | Track Input Gain Node | |
|
||||
| | Event Clock / | | WebAudio Virtual | +--------------+--------------+ |
|
||||
| | Latency Engine | | Synth Engine | | |
|
||||
| +------------+-----+ +---------+-----------+ +--------+--------+ |
|
||||
| | | | | |
|
||||
| v v v v |
|
||||
| +------------+-----+ (Live Sound) +-------+-------+ +-------+------+ |
|
||||
| | Recorded MIDI | | AudioWorklet | | Monitoring | |
|
||||
| | Buffer | | Ring-Buffer | | Switch | |
|
||||
| +------------+-----+ | Recorder | +-------+------+ |
|
||||
| | +-------+-------+ | |
|
||||
| v | v |
|
||||
| [ Timeline MIDI ] v [ Master Mix ] |
|
||||
| [ Item Creation ] +-------+-------+ |
|
||||
| | Float32 PCM | |
|
||||
| | Audio Buffer | |
|
||||
| +-------+-------+ |
|
||||
| | |
|
||||
| v |
|
||||
| [ Timeline Audio] |
|
||||
| [ Item Creation ] |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Hardware I/O & API Contracts
|
||||
|
||||
### 2.1 MediaDevices (Microphone Capture)
|
||||
|
||||
**Permission Request:** Uses `navigator.mediaDevices.getUserMedia` configured to disable automatic browser processing DSP algorithms to capture pure, unprocessed audio signals:
|
||||
|
||||
```javascript
|
||||
const audioConstraints = {
|
||||
audio: {
|
||||
deviceId: selectedDeviceId ? { exact: selectedDeviceId } : undefined,
|
||||
echoCancellation: false, // Disables echo cancellation to prevent instrument sound distortion
|
||||
noiseSuppression: false, // Disables automatic noise suppression to preserve full frequency range
|
||||
autoGainControl: false, // Disables Automatic Gain Control (AGC)
|
||||
latency: 0 // Requests minimal latency from OS audio driver
|
||||
}
|
||||
};
|
||||
|
||||
```
|
||||
|
||||
### 2.2 Web MIDI API Integration
|
||||
|
||||
**Device Enumeration & Listener Assignment:**
|
||||
|
||||
* Uses `navigator.requestMIDIAccess({ sysex: false })` to scan for USB-connected keyboard devices.
|
||||
* **Timestamp Precision:** Obtains event timestamps from `MIDIMessageEvent.timeStamp` (as a `DOMHighResTimeStamp` in microseconds) and synchronizes them with `AudioContext.currentTime`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recording Lifecycle & State Machine
|
||||
|
||||
```text
|
||||
[IDLE] ───► (User Arms Track) ───► [ARMED] ───► (Press Rec + Play) ───► [COUNT-IN / PRE-ROLL]
|
||||
|
|
||||
[STOP & COMMIT] ◄─── (Press Stop) ◄─── [RECORDING IN PROGRESS] ◄───────────────+
|
||||
|
||||
```
|
||||
|
||||
* **Arming Phase (Record Enable):**
|
||||
* The user selects an input source and activates the Arm (R) button on the target track.
|
||||
* Initializes the input level meter (VU Meter Canvas) to display input volume levels in real time.
|
||||
|
||||
|
||||
* **Pre-Roll / Count-In Phase:**
|
||||
* Transport triggers the metronome count-in (e.g., 1 Bar = 4 beats). The metronome plays click sounds based on project BPM.
|
||||
* The engine does not write data to the Timeline yet, but begins reading the input buffer to prepare memory buffers.
|
||||
|
||||
|
||||
* **Recording Phase:**
|
||||
* Once the transport passes the Start Bar boundary, incoming MIDI key events or PCM Float32 audio samples are written into the active recording buffer memory.
|
||||
* Canvas UI displays real-time visual feedback, rendering waveforms or MIDI note blocks dynamically.
|
||||
|
||||
|
||||
* **Stop & Commit Phase:**
|
||||
* Pressing Stop halts the recording process.
|
||||
* Converts temporary memory buffers into a structured `MIDIItem` or `AudioItem`.
|
||||
* Inserts the new Item onto the target track within the Main Session or Section tab.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Structures
|
||||
|
||||
### 4.1 Live MIDI Event Buffer Element Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
|
||||
"start_beat": { "type": "number", "description": "Start position in beats on the timeline" },
|
||||
"duration_beats": { "type": "number", "description": "Keypress duration in beats" },
|
||||
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0 },
|
||||
"channel": { "type": "integer", "default": 0 }
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### 4.2 Recording Track Input Configuration State
|
||||
|
||||
```json
|
||||
{
|
||||
"track_id": "track_midi_01",
|
||||
"is_armed": true,
|
||||
"monitoring_enabled": true,
|
||||
"input_source": {
|
||||
"device_type": "MIDI_KEYBOARD",
|
||||
"device_id": "midi_input_usb_keyboard_0",
|
||||
"channel": 1
|
||||
},
|
||||
"input_gain_db": 0.0,
|
||||
"latency_offset_ms": 12.5
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Core Algorithms & Latency Compensation
|
||||
|
||||
### 5.1 Algorithm 1: Hardware Latency Compensation Formula
|
||||
|
||||
When recording, the physical moment a key is pressed or sound enters the microphone is inherently delayed relative to speaker output due to input buffers ($L_{\text{input}}$) and output buffers ($L_{\text{output}}$).
|
||||
|
||||
#### Mathematical Formulation
|
||||
|
||||
Let:
|
||||
|
||||
* $T_{\text{audio\_ctx}}$ = Current timestamp in seconds on the `AudioContext` clock (`audioCtx.currentTime`).
|
||||
* $T_{\text{rec\_start}}$ = Recording start timestamp in seconds.
|
||||
* $\text{BPM}$ = Song tempo (Beats Per Minute).
|
||||
* $\text{TS}_{\text{num}}$ = Time Signature Numerator (beats per bar).
|
||||
* $\text{Bar}_{\text{start}}$ = Target timeline start bar for recording.
|
||||
* $L_{\text{comp}}$ = Total hardware latency offset ($L_{\text{input}} + L_{\text{output}} + L_{\text{user\_offset}}$) in seconds.
|
||||
|
||||
**Actual Elapsed Audio Time ($T_{\text{elapsed}}$):**
|
||||
|
||||
$$T_{\text{elapsed}} = \max\left(0, T_{\text{audio\_ctx}} - T_{\text{rec\_start}} - L_{\text{comp}}\right)$$
|
||||
|
||||
**Audio Time to Beat Conversion ($\text{Beat}_{\text{current}}$):**
|
||||
|
||||
$$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$
|
||||
|
||||
$$\text{Beat}_{\text{current}} = \frac{T_{\text{elapsed}}}{\text{SecondsPerBeat}} + \left(\text{Bar}_{\text{start}} \times \text{TS}_{\text{num}}\right)$$
|
||||
|
||||
**Timeline Placement Mapping:**
|
||||
|
||||
$$\text{StartBeat}_{\text{item}} = \text{Beat}_{\text{current}}$$
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Algorithm 2: AudioWorklet PCM Ring-Buffer Processor
|
||||
|
||||
To prevent audio glitches or missing PCM frames when the browser's main thread is processing heavy UI renders, microphone recording runs inside an `AudioWorkletProcessor`:
|
||||
|
||||
```javascript
|
||||
// public/processors/pcm-recorder-processor.js
|
||||
class PCMRecorderProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.bufferSize = 4096;
|
||||
this.buffer = new Float32Array(this.bufferSize);
|
||||
this.bufferIndex = 0;
|
||||
}
|
||||
|
||||
process(inputs, outputs, parameters) {
|
||||
const input = inputs[0];
|
||||
if (input && input.length > 0) {
|
||||
const inputChannel = input[0]; // Mono Channel 0
|
||||
|
||||
for (let i = 0; i < inputChannel.length; i++) {
|
||||
this.buffer[this.bufferIndex++] = inputChannel[i];
|
||||
|
||||
// When Ring-Buffer fills, send Float32Array to Main Thread
|
||||
if (this.bufferIndex >= this.bufferSize) {
|
||||
this.port.postMessage({
|
||||
type: 'PCM_DATA',
|
||||
buffer: this.buffer.slice(0, this.bufferSize)
|
||||
});
|
||||
this.bufferIndex = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
return true; // Keep worklet active
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor('pcm-recorder-processor', PCMRecorderProcessor);
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5.3 Algorithm 3: Client MIDIRecorder Class Implementation
|
||||
|
||||
```javascript
|
||||
class ClientMIDIRecorder {
|
||||
constructor(audioContext, bpm = 120, timeSigNumerator = 4) {
|
||||
this.audioCtx = audioContext;
|
||||
this.bpm = bpm;
|
||||
this.timeSigNum = timeSigNumerator;
|
||||
this.isRecording = false;
|
||||
|
||||
this.activeNotes = new Map(); // Store pitch -> { noteId, startBeat, velocity }
|
||||
this.recordedNotes = [];
|
||||
this.recStartAudioTime = 0.0;
|
||||
this.recStartBar = 0.0;
|
||||
|
||||
// Compute round-trip browser latency
|
||||
this.latencyCompSec = (this.audioCtx.baseLatency || 0) + (this.audioCtx.outputLatency || 0);
|
||||
}
|
||||
|
||||
start(startBar = 0.0) {
|
||||
this.isRecording = true;
|
||||
this.recordedNotes = [];
|
||||
this.activeNotes.clear();
|
||||
this.recStartBar = startBar;
|
||||
this.recStartAudioTime = this.audioCtx.currentTime;
|
||||
|
||||
this.bindMIDIInputs();
|
||||
}
|
||||
|
||||
bindMIDIInputs() {
|
||||
if (navigator.requestMIDIAccess) {
|
||||
navigator.requestMIDIAccess().then(midiAccess => {
|
||||
for (let input of midiAccess.inputs.values()) {
|
||||
input.onmidimessage = (event) => this.handleMIDIMessage(event);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
handleMIDIMessage(event) {
|
||||
if (!this.isRecording) return;
|
||||
|
||||
const [status, pitch, velocity] = event.data;
|
||||
const command = status >> 4;
|
||||
|
||||
// Apply latency compensation formula
|
||||
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
|
||||
const secondsPerBeat = 60.0 / this.bpm;
|
||||
const currentBeat = (currentTimeSec / secondsPerBeat) + (this.recStartBar * this.timeSigNum);
|
||||
|
||||
// Command 0x9: Note On
|
||||
if (command === 0x9 && velocity > 0) {
|
||||
const noteId = `rec_${Date.now()}_${pitch}`;
|
||||
this.activeNotes.set(pitch, {
|
||||
id: noteId,
|
||||
pitch: pitch,
|
||||
start_beat: currentBeat,
|
||||
velocity: velocity / 127.0
|
||||
});
|
||||
}
|
||||
// Command 0x8: Note Off (or Note On with velocity = 0)
|
||||
else if (command === 0x8 || (command === 0x9 && velocity === 0)) {
|
||||
if (this.activeNotes.has(pitch)) {
|
||||
const note = this.activeNotes.get(pitch);
|
||||
const durationBeats = Math.max(0.125, currentBeat - note.start_beat); // Min 1/32 note
|
||||
|
||||
this.recordedNotes.push({
|
||||
id: note.id,
|
||||
pitch: note.pitch,
|
||||
start_beat: note.start_beat,
|
||||
duration_beats: durationBeats,
|
||||
velocity: note.velocity,
|
||||
pan: 0.0
|
||||
});
|
||||
|
||||
this.activeNotes.delete(pitch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
stop() {
|
||||
this.isRecording = false;
|
||||
|
||||
// Flush remaining active keypresses when stop is triggered
|
||||
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
|
||||
const currentBeat = (currentTimeSec / (60.0 / this.bpm)) + (this.recStartBar * this.timeSigNum);
|
||||
|
||||
for (let [pitch, note] of this.activeNotes.entries()) {
|
||||
this.recordedNotes.push({
|
||||
id: note.id,
|
||||
pitch: note.pitch,
|
||||
start_beat: note.start_beat,
|
||||
duration_beats: Math.max(0.25, currentBeat - note.start_beat),
|
||||
velocity: note.velocity,
|
||||
pan: 0.0
|
||||
});
|
||||
}
|
||||
this.activeNotes.clear();
|
||||
return this.recordedNotes;
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5.4 Algorithm 4: Client AudioRecorder & AudioBuffer Splicing Class Implementation
|
||||
|
||||
```javascript
|
||||
class ClientAudioRecorder {
|
||||
constructor(audioContext) {
|
||||
this.audioCtx = audioContext;
|
||||
this.mediaStream = null;
|
||||
this.sourceNode = null;
|
||||
this.workletNode = null;
|
||||
this.pcmChunks = [];
|
||||
this.isRecording = false;
|
||||
}
|
||||
|
||||
async initializeInput(deviceId = null) {
|
||||
const constraints = {
|
||||
audio: {
|
||||
deviceId: deviceId ? { exact: deviceId } : undefined,
|
||||
echoCancellation: false,
|
||||
noiseSuppression: false,
|
||||
autoGainControl: false
|
||||
}
|
||||
};
|
||||
this.mediaStream = await navigator.mediaDevices.getUserMedia(constraints);
|
||||
this.sourceNode = this.audioCtx.createMediaStreamSource(this.mediaStream);
|
||||
}
|
||||
|
||||
async start(destinationTrackGainNode, enableMonitoring = true) {
|
||||
this.pcmChunks = [];
|
||||
this.isRecording = true;
|
||||
|
||||
// Load Worklet Processor Module
|
||||
await this.audioCtx.audioWorklet.addModule('/processors/pcm-recorder-processor.js');
|
||||
this.workletNode = new AudioWorkletNode(this.audioCtx, 'pcm-recorder-processor');
|
||||
|
||||
// Receive PCM data streams from AudioWorklet
|
||||
this.workletNode.port.onmessage = (event) => {
|
||||
if (this.isRecording && event.data.type === 'PCM_DATA') {
|
||||
this.pcmChunks.push(new Float32Array(event.data.buffer));
|
||||
}
|
||||
};
|
||||
|
||||
// Route Audio Nodes
|
||||
this.sourceNode.connect(this.workletNode);
|
||||
|
||||
// Enable Live Input Monitoring if requested
|
||||
if (enableMonitoring) {
|
||||
this.sourceNode.connect(destinationTrackGainNode);
|
||||
}
|
||||
}
|
||||
|
||||
async stop() {
|
||||
this.isRecording = false;
|
||||
|
||||
if (this.sourceNode && this.workletNode) {
|
||||
this.sourceNode.disconnect(this.workletNode);
|
||||
}
|
||||
|
||||
// Concatenate PCM Float32Array chunks into a single AudioBuffer
|
||||
const totalSamples = this.pcmChunks.reduce((sum, chunk) => sum + chunk.length, 0);
|
||||
if (totalSamples === 0) return null;
|
||||
|
||||
const audioBuffer = this.audioCtx.createBuffer(1, totalSamples, this.audioCtx.sampleRate);
|
||||
const channelData = audioBuffer.getChannelData(0);
|
||||
|
||||
let offset = 0;
|
||||
for (const chunk of this.pcmChunks) {
|
||||
channelData.set(chunk, offset);
|
||||
offset += chunk.length;
|
||||
}
|
||||
|
||||
return audioBuffer; // Return compiled AudioBuffer for timeline insertion
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. UI Components & User Interactions
|
||||
|
||||
* **Track Header Arming Controls:**
|
||||
* **[R] Button (Arm Track):** Highlights red when armed for recording on the target track.
|
||||
* **[I] Button (Input Monitor):** Toggles live monitoring for incoming Microphone or Synth audio during performance.
|
||||
* **Input Selector Dropdown:** Allows selection of available Microphone devices or USB Hardware MIDI Keyboards.
|
||||
|
||||
|
||||
* **Real-time VU Meter Component:**
|
||||
* Displays input signal gain level from $-60\text{ dB}$ to $0\text{ dB}$. Displays red clipping indicators when signal levels exceed $0\text{ dBFS}$.
|
||||
|
||||
|
||||
* **Live Waveform & MIDI Preview Rendering:**
|
||||
* **Microphone Recording:** The canvas UI renders incoming waveform signals progressing along the Playhead position in real time.
|
||||
* **MIDI Performance:** Rectangular note blocks (green/orange) appear at note-on trigger events and extend until key release (note-off).
|
||||
Reference in New Issue
Block a user