426 lines
17 KiB
Markdown
426 lines
17 KiB
Markdown
# TECHNICAL SPECIFICATION: CLIENT-SIDE REAL-TIME RECORDING ENGINE
|
|
|
|
## Browser-Based Microphone & Hardware MIDI Keyboard Recording Module
|
|
|
|
---
|
|
|
|
## 1. System Overview
|
|
|
|
The Client-side recording module enables the DAW to capture live audio signals directly from Microphone/Line-in interfaces (via the Web MediaDevices API) and keypress events from Hardware MIDI Keyboards/Controllers (via the Web MIDI API) in real time. The module operates with low latency and includes hardware latency compensation.
|
|
|
|
```text
|
|
+-----------------------------------------------------------------------------------+
|
|
| CLIENT BROWSER |
|
|
| |
|
|
| +-------------------------+ +-----------------------------+ |
|
|
| | Hardware MIDI Keyboard | | Live Microphone / Line-In | |
|
|
| +------------+------------+ +--------------+--------------+ |
|
|
| | | |
|
|
| v (Web MIDI API) v (getUserMedia) |
|
|
| +------------+------------+ +--------------+--------------+ |
|
|
| | Web MIDI Input Handler | | MediaStreamAudioSourceNode | |
|
|
| +------------+------------+ +--------------+--------------+ |
|
|
| | | |
|
|
| +------------------+ | |
|
|
| | | v |
|
|
| v v +--------------+--------------+ |
|
|
| +------------+-----+ +---------+-----------+ | Track Input Gain Node | |
|
|
| | Event Clock / | | WebAudio Virtual | +--------------+--------------+ |
|
|
| | Latency Engine | | Synth Engine | | |
|
|
| +------------+-----+ +---------+-----------+ +--------+--------+ |
|
|
| | | | | |
|
|
| v v v v |
|
|
| +------------+-----+ (Live Sound) +-------+-------+ +-------+------+ |
|
|
| | Recorded MIDI | | AudioWorklet | | Monitoring | |
|
|
| | Buffer | | Ring-Buffer | | Switch | |
|
|
| +------------+-----+ | Recorder | +-------+------+ |
|
|
| | +-------+-------+ | |
|
|
| v | v |
|
|
| [ Timeline MIDI ] v [ Master Mix ] |
|
|
| [ Item Creation ] +-------+-------+ |
|
|
| | Float32 PCM | |
|
|
| | Audio Buffer | |
|
|
| +-------+-------+ |
|
|
| | |
|
|
| v |
|
|
| [ Timeline Audio] |
|
|
| [ Item Creation ] |
|
|
+-----------------------------------------------------------------------------------+
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Hardware I/O & API Contracts
|
|
|
|
### 2.1 MediaDevices (Microphone Capture)
|
|
|
|
**Permission Request:** Uses `navigator.mediaDevices.getUserMedia` configured to disable automatic browser processing DSP algorithms to capture pure, unprocessed audio signals:
|
|
|
|
```javascript
|
|
const audioConstraints = {
|
|
audio: {
|
|
deviceId: selectedDeviceId ? { exact: selectedDeviceId } : undefined,
|
|
echoCancellation: false, // Disables echo cancellation to prevent instrument sound distortion
|
|
noiseSuppression: false, // Disables automatic noise suppression to preserve full frequency range
|
|
autoGainControl: false, // Disables Automatic Gain Control (AGC)
|
|
latency: 0 // Requests minimal latency from OS audio driver
|
|
}
|
|
};
|
|
|
|
```
|
|
|
|
### 2.2 Web MIDI API Integration
|
|
|
|
**Device Enumeration & Listener Assignment:**
|
|
|
|
* Uses `navigator.requestMIDIAccess({ sysex: false })` to scan for USB-connected keyboard devices.
|
|
* **Timestamp Precision:** Obtains event timestamps from `MIDIMessageEvent.timeStamp` (as a `DOMHighResTimeStamp` in microseconds) and synchronizes them with `AudioContext.currentTime`.
|
|
|
|
---
|
|
|
|
## 3. Recording Lifecycle & State Machine
|
|
|
|
```text
|
|
[IDLE] ───► (User Arms Track) ───► [ARMED] ───► (Press Rec + Play) ───► [COUNT-IN / PRE-ROLL]
|
|
|
|
|
[STOP & COMMIT] ◄─── (Press Stop) ◄─── [RECORDING IN PROGRESS] ◄───────────────+
|
|
|
|
```
|
|
|
|
* **Arming Phase (Record Enable):**
|
|
* The user selects an input source and activates the Arm (R) button on the target track.
|
|
* Initializes the input level meter (VU Meter Canvas) to display input volume levels in real time.
|
|
|
|
|
|
* **Pre-Roll / Count-In Phase:**
|
|
* Transport triggers the metronome count-in (e.g., 1 Bar = 4 beats). The metronome plays click sounds based on project BPM.
|
|
* The engine does not write data to the Timeline yet, but begins reading the input buffer to prepare memory buffers.
|
|
|
|
|
|
* **Recording Phase:**
|
|
* Once the transport passes the Start Bar boundary, incoming MIDI key events or PCM Float32 audio samples are written into the active recording buffer memory.
|
|
* Canvas UI displays real-time visual feedback, rendering waveforms or MIDI note blocks dynamically.
|
|
|
|
|
|
* **Stop & Commit Phase:**
|
|
* Pressing Stop halts the recording process.
|
|
* Converts temporary memory buffers into a structured `MIDIItem` or `AudioItem`.
|
|
* Inserts the new Item onto the target track within the Main Session or Section tab.
|
|
|
|
|
|
|
|
---
|
|
|
|
## 4. Data Structures
|
|
|
|
### 4.1 Live MIDI Event Buffer Element Schema
|
|
|
|
```json
|
|
{
|
|
"type": "object",
|
|
"properties": {
|
|
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
|
|
"start_beat": { "type": "number", "description": "Start position in beats on the timeline" },
|
|
"duration_beats": { "type": "number", "description": "Keypress duration in beats" },
|
|
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0 },
|
|
"channel": { "type": "integer", "default": 0 }
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
### 4.2 Recording Track Input Configuration State
|
|
|
|
```json
|
|
{
|
|
"track_id": "track_midi_01",
|
|
"is_armed": true,
|
|
"monitoring_enabled": true,
|
|
"input_source": {
|
|
"device_type": "MIDI_KEYBOARD",
|
|
"device_id": "midi_input_usb_keyboard_0",
|
|
"channel": 1
|
|
},
|
|
"input_gain_db": 0.0,
|
|
"latency_offset_ms": 12.5
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Core Algorithms & Latency Compensation
|
|
|
|
### 5.1 Algorithm 1: Hardware Latency Compensation Formula
|
|
|
|
When recording, the physical moment a key is pressed or sound enters the microphone is inherently delayed relative to speaker output due to input buffers ($L_{\text{input}}$) and output buffers ($L_{\text{output}}$).
|
|
|
|
#### Mathematical Formulation
|
|
|
|
Let:
|
|
|
|
* $T_{\text{audio\_ctx}}$ = Current timestamp in seconds on the `AudioContext` clock (`audioCtx.currentTime`).
|
|
* $T_{\text{rec\_start}}$ = Recording start timestamp in seconds.
|
|
* $\text{BPM}$ = Song tempo (Beats Per Minute).
|
|
* $\text{TS}_{\text{num}}$ = Time Signature Numerator (beats per bar).
|
|
* $\text{Bar}_{\text{start}}$ = Target timeline start bar for recording.
|
|
* $L_{\text{comp}}$ = Total hardware latency offset ($L_{\text{input}} + L_{\text{output}} + L_{\text{user\_offset}}$) in seconds.
|
|
|
|
**Actual Elapsed Audio Time ($T_{\text{elapsed}}$):**
|
|
|
|
$$T_{\text{elapsed}} = \max\left(0, T_{\text{audio\_ctx}} - T_{\text{rec\_start}} - L_{\text{comp}}\right)$$
|
|
|
|
**Audio Time to Beat Conversion ($\text{Beat}_{\text{current}}$):**
|
|
|
|
$$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$
|
|
|
|
$$\text{Beat}_{\text{current}} = \frac{T_{\text{elapsed}}}{\text{SecondsPerBeat}} + \left(\text{Bar}_{\text{start}} \times \text{TS}_{\text{num}}\right)$$
|
|
|
|
**Timeline Placement Mapping:**
|
|
|
|
$$\text{StartBeat}_{\text{item}} = \text{Beat}_{\text{current}}$$
|
|
|
|
---
|
|
|
|
### 5.2 Algorithm 2: AudioWorklet PCM Ring-Buffer Processor
|
|
|
|
To prevent audio glitches or missing PCM frames when the browser's main thread is processing heavy UI renders, microphone recording runs inside an `AudioWorkletProcessor`:
|
|
|
|
```javascript
|
|
// public/processors/pcm-recorder-processor.js
|
|
class PCMRecorderProcessor extends AudioWorkletProcessor {
|
|
constructor() {
|
|
super();
|
|
this.bufferSize = 4096;
|
|
this.buffer = new Float32Array(this.bufferSize);
|
|
this.bufferIndex = 0;
|
|
}
|
|
|
|
process(inputs, outputs, parameters) {
|
|
const input = inputs[0];
|
|
if (input && input.length > 0) {
|
|
const inputChannel = input[0]; // Mono Channel 0
|
|
|
|
for (let i = 0; i < inputChannel.length; i++) {
|
|
this.buffer[this.bufferIndex++] = inputChannel[i];
|
|
|
|
// When Ring-Buffer fills, send Float32Array to Main Thread
|
|
if (this.bufferIndex >= this.bufferSize) {
|
|
this.port.postMessage({
|
|
type: 'PCM_DATA',
|
|
buffer: this.buffer.slice(0, this.bufferSize)
|
|
});
|
|
this.bufferIndex = 0;
|
|
}
|
|
}
|
|
}
|
|
return true; // Keep worklet active
|
|
}
|
|
}
|
|
|
|
registerProcessor('pcm-recorder-processor', PCMRecorderProcessor);
|
|
|
|
```
|
|
|
|
---
|
|
|
|
### 5.3 Algorithm 3: Client MIDIRecorder Class Implementation
|
|
|
|
```javascript
|
|
class ClientMIDIRecorder {
|
|
constructor(audioContext, bpm = 120, timeSigNumerator = 4) {
|
|
this.audioCtx = audioContext;
|
|
this.bpm = bpm;
|
|
this.timeSigNum = timeSigNumerator;
|
|
this.isRecording = false;
|
|
|
|
this.activeNotes = new Map(); // Store pitch -> { noteId, startBeat, velocity }
|
|
this.recordedNotes = [];
|
|
this.recStartAudioTime = 0.0;
|
|
this.recStartBar = 0.0;
|
|
|
|
// Compute round-trip browser latency
|
|
this.latencyCompSec = (this.audioCtx.baseLatency || 0) + (this.audioCtx.outputLatency || 0);
|
|
}
|
|
|
|
start(startBar = 0.0) {
|
|
this.isRecording = true;
|
|
this.recordedNotes = [];
|
|
this.activeNotes.clear();
|
|
this.recStartBar = startBar;
|
|
this.recStartAudioTime = this.audioCtx.currentTime;
|
|
|
|
this.bindMIDIInputs();
|
|
}
|
|
|
|
bindMIDIInputs() {
|
|
if (navigator.requestMIDIAccess) {
|
|
navigator.requestMIDIAccess().then(midiAccess => {
|
|
for (let input of midiAccess.inputs.values()) {
|
|
input.onmidimessage = (event) => this.handleMIDIMessage(event);
|
|
}
|
|
});
|
|
}
|
|
}
|
|
|
|
handleMIDIMessage(event) {
|
|
if (!this.isRecording) return;
|
|
|
|
const [status, pitch, velocity] = event.data;
|
|
const command = status >> 4;
|
|
|
|
// Apply latency compensation formula
|
|
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
|
|
const secondsPerBeat = 60.0 / this.bpm;
|
|
const currentBeat = (currentTimeSec / secondsPerBeat) + (this.recStartBar * this.timeSigNum);
|
|
|
|
// Command 0x9: Note On
|
|
if (command === 0x9 && velocity > 0) {
|
|
const noteId = `rec_${Date.now()}_${pitch}`;
|
|
this.activeNotes.set(pitch, {
|
|
id: noteId,
|
|
pitch: pitch,
|
|
start_beat: currentBeat,
|
|
velocity: velocity / 127.0
|
|
});
|
|
}
|
|
// Command 0x8: Note Off (or Note On with velocity = 0)
|
|
else if (command === 0x8 || (command === 0x9 && velocity === 0)) {
|
|
if (this.activeNotes.has(pitch)) {
|
|
const note = this.activeNotes.get(pitch);
|
|
const durationBeats = Math.max(0.125, currentBeat - note.start_beat); // Min 1/32 note
|
|
|
|
this.recordedNotes.push({
|
|
id: note.id,
|
|
pitch: note.pitch,
|
|
start_beat: note.start_beat,
|
|
duration_beats: durationBeats,
|
|
velocity: note.velocity,
|
|
pan: 0.0
|
|
});
|
|
|
|
this.activeNotes.delete(pitch);
|
|
}
|
|
}
|
|
}
|
|
|
|
stop() {
|
|
this.isRecording = false;
|
|
|
|
// Flush remaining active keypresses when stop is triggered
|
|
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
|
|
const currentBeat = (currentTimeSec / (60.0 / this.bpm)) + (this.recStartBar * this.timeSigNum);
|
|
|
|
for (let [pitch, note] of this.activeNotes.entries()) {
|
|
this.recordedNotes.push({
|
|
id: note.id,
|
|
pitch: note.pitch,
|
|
start_beat: note.start_beat,
|
|
duration_beats: Math.max(0.25, currentBeat - note.start_beat),
|
|
velocity: note.velocity,
|
|
pan: 0.0
|
|
});
|
|
}
|
|
this.activeNotes.clear();
|
|
return this.recordedNotes;
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
### 5.4 Algorithm 4: Client AudioRecorder & AudioBuffer Splicing Class Implementation
|
|
|
|
```javascript
|
|
class ClientAudioRecorder {
|
|
constructor(audioContext) {
|
|
this.audioCtx = audioContext;
|
|
this.mediaStream = null;
|
|
this.sourceNode = null;
|
|
this.workletNode = null;
|
|
this.pcmChunks = [];
|
|
this.isRecording = false;
|
|
}
|
|
|
|
async initializeInput(deviceId = null) {
|
|
const constraints = {
|
|
audio: {
|
|
deviceId: deviceId ? { exact: deviceId } : undefined,
|
|
echoCancellation: false,
|
|
noiseSuppression: false,
|
|
autoGainControl: false
|
|
}
|
|
};
|
|
this.mediaStream = await navigator.mediaDevices.getUserMedia(constraints);
|
|
this.sourceNode = this.audioCtx.createMediaStreamSource(this.mediaStream);
|
|
}
|
|
|
|
async start(destinationTrackGainNode, enableMonitoring = true) {
|
|
this.pcmChunks = [];
|
|
this.isRecording = true;
|
|
|
|
// Load Worklet Processor Module
|
|
await this.audioCtx.audioWorklet.addModule('/processors/pcm-recorder-processor.js');
|
|
this.workletNode = new AudioWorkletNode(this.audioCtx, 'pcm-recorder-processor');
|
|
|
|
// Receive PCM data streams from AudioWorklet
|
|
this.workletNode.port.onmessage = (event) => {
|
|
if (this.isRecording && event.data.type === 'PCM_DATA') {
|
|
this.pcmChunks.push(new Float32Array(event.data.buffer));
|
|
}
|
|
};
|
|
|
|
// Route Audio Nodes
|
|
this.sourceNode.connect(this.workletNode);
|
|
|
|
// Enable Live Input Monitoring if requested
|
|
if (enableMonitoring) {
|
|
this.sourceNode.connect(destinationTrackGainNode);
|
|
}
|
|
}
|
|
|
|
async stop() {
|
|
this.isRecording = false;
|
|
|
|
if (this.sourceNode && this.workletNode) {
|
|
this.sourceNode.disconnect(this.workletNode);
|
|
}
|
|
|
|
// Concatenate PCM Float32Array chunks into a single AudioBuffer
|
|
const totalSamples = this.pcmChunks.reduce((sum, chunk) => sum + chunk.length, 0);
|
|
if (totalSamples === 0) return null;
|
|
|
|
const audioBuffer = this.audioCtx.createBuffer(1, totalSamples, this.audioCtx.sampleRate);
|
|
const channelData = audioBuffer.getChannelData(0);
|
|
|
|
let offset = 0;
|
|
for (const chunk of this.pcmChunks) {
|
|
channelData.set(chunk, offset);
|
|
offset += chunk.length;
|
|
}
|
|
|
|
return audioBuffer; // Return compiled AudioBuffer for timeline insertion
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 6. UI Components & User Interactions
|
|
|
|
* **Track Header Arming Controls:**
|
|
* **[R] Button (Arm Track):** Highlights red when armed for recording on the target track.
|
|
* **[I] Button (Input Monitor):** Toggles live monitoring for incoming Microphone or Synth audio during performance.
|
|
* **Input Selector Dropdown:** Allows selection of available Microphone devices or USB Hardware MIDI Keyboards.
|
|
|
|
|
|
* **Real-time VU Meter Component:**
|
|
* Displays input signal gain level from $-60\text{ dB}$ to $0\text{ dB}$. Displays red clipping indicators when signal levels exceed $0\text{ dBFS}$.
|
|
|
|
|
|
* **Live Waveform & MIDI Preview Rendering:**
|
|
* **Microphone Recording:** The canvas UI renders incoming waveform signals progressing along the Playhead position in real time.
|
|
* **MIDI Performance:** Rectangular note blocks (green/orange) appear at note-on trigger events and extend until key release (note-off). |