17 KiB
TECHNICAL SPECIFICATION: CLIENT-SIDE REAL-TIME RECORDING ENGINE
Browser-Based Microphone & Hardware MIDI Keyboard Recording Module
1. System Overview
The Client-side recording module enables the DAW to capture live audio signals directly from Microphone/Line-in interfaces (via the Web MediaDevices API) and keypress events from Hardware MIDI Keyboards/Controllers (via the Web MIDI API) in real time. The module operates with low latency and includes hardware latency compensation.
+-----------------------------------------------------------------------------------+
| CLIENT BROWSER |
| |
| +-------------------------+ +-----------------------------+ |
| | Hardware MIDI Keyboard | | Live Microphone / Line-In | |
| +------------+------------+ +--------------+--------------+ |
| | | |
| v (Web MIDI API) v (getUserMedia) |
| +------------+------------+ +--------------+--------------+ |
| | Web MIDI Input Handler | | MediaStreamAudioSourceNode | |
| +------------+------------+ +--------------+--------------+ |
| | | |
| +------------------+ | |
| | | v |
| v v +--------------+--------------+ |
| +------------+-----+ +---------+-----------+ | Track Input Gain Node | |
| | Event Clock / | | WebAudio Virtual | +--------------+--------------+ |
| | Latency Engine | | Synth Engine | | |
| +------------+-----+ +---------+-----------+ +--------+--------+ |
| | | | | |
| v v v v |
| +------------+-----+ (Live Sound) +-------+-------+ +-------+------+ |
| | Recorded MIDI | | AudioWorklet | | Monitoring | |
| | Buffer | | Ring-Buffer | | Switch | |
| +------------+-----+ | Recorder | +-------+------+ |
| | +-------+-------+ | |
| v | v |
| [ Timeline MIDI ] v [ Master Mix ] |
| [ Item Creation ] +-------+-------+ |
| | Float32 PCM | |
| | Audio Buffer | |
| +-------+-------+ |
| | |
| v |
| [ Timeline Audio] |
| [ Item Creation ] |
+-----------------------------------------------------------------------------------+
2. Hardware I/O & API Contracts
2.1 MediaDevices (Microphone Capture)
Permission Request: Uses navigator.mediaDevices.getUserMedia configured to disable automatic browser processing DSP algorithms to capture pure, unprocessed audio signals:
const audioConstraints = {
audio: {
deviceId: selectedDeviceId ? { exact: selectedDeviceId } : undefined,
echoCancellation: false, // Disables echo cancellation to prevent instrument sound distortion
noiseSuppression: false, // Disables automatic noise suppression to preserve full frequency range
autoGainControl: false, // Disables Automatic Gain Control (AGC)
latency: 0 // Requests minimal latency from OS audio driver
}
};
2.2 Web MIDI API Integration
Device Enumeration & Listener Assignment:
- Uses
navigator.requestMIDIAccess({ sysex: false })to scan for USB-connected keyboard devices. - Timestamp Precision: Obtains event timestamps from
MIDIMessageEvent.timeStamp(as aDOMHighResTimeStampin microseconds) and synchronizes them withAudioContext.currentTime.
3. Recording Lifecycle & State Machine
[IDLE] ───► (User Arms Track) ───► [ARMED] ───► (Press Rec + Play) ───► [COUNT-IN / PRE-ROLL]
|
[STOP & COMMIT] ◄─── (Press Stop) ◄─── [RECORDING IN PROGRESS] ◄───────────────+
-
Arming Phase (Record Enable):
-
The user selects an input source and activates the Arm (R) button on the target track.
-
Initializes the input level meter (VU Meter Canvas) to display input volume levels in real time.
-
Pre-Roll / Count-In Phase:
-
Transport triggers the metronome count-in (e.g., 1 Bar = 4 beats). The metronome plays click sounds based on project BPM.
-
The engine does not write data to the Timeline yet, but begins reading the input buffer to prepare memory buffers.
-
Recording Phase:
-
Once the transport passes the Start Bar boundary, incoming MIDI key events or PCM Float32 audio samples are written into the active recording buffer memory.
-
Canvas UI displays real-time visual feedback, rendering waveforms or MIDI note blocks dynamically.
-
Stop & Commit Phase:
-
Pressing Stop halts the recording process.
-
Converts temporary memory buffers into a structured
MIDIItemorAudioItem. -
Inserts the new Item onto the target track within the Main Session or Section tab.
4. Data Structures
4.1 Live MIDI Event Buffer Element Schema
{
"type": "object",
"properties": {
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
"start_beat": { "type": "number", "description": "Start position in beats on the timeline" },
"duration_beats": { "type": "number", "description": "Keypress duration in beats" },
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0 },
"channel": { "type": "integer", "default": 0 }
}
}
4.2 Recording Track Input Configuration State
{
"track_id": "track_midi_01",
"is_armed": true,
"monitoring_enabled": true,
"input_source": {
"device_type": "MIDI_KEYBOARD",
"device_id": "midi_input_usb_keyboard_0",
"channel": 1
},
"input_gain_db": 0.0,
"latency_offset_ms": 12.5
}
5. Core Algorithms & Latency Compensation
5.1 Algorithm 1: Hardware Latency Compensation Formula
When recording, the physical moment a key is pressed or sound enters the microphone is inherently delayed relative to speaker output due to input buffers (L_{\text{input}}) and output buffers (L_{\text{output}}).
Mathematical Formulation
Let:
T_{\text{audio\_ctx}}= Current timestamp in seconds on theAudioContextclock (audioCtx.currentTime).T_{\text{rec\_start}}= Recording start timestamp in seconds.\text{BPM}= Song tempo (Beats Per Minute).\text{TS}_{\text{num}}= Time Signature Numerator (beats per bar).\text{Bar}_{\text{start}}= Target timeline start bar for recording.L_{\text{comp}}= Total hardware latency offset (L_{\text{input}} + L_{\text{output}} + L_{\text{user\_offset}}) in seconds.
Actual Elapsed Audio Time (T_{\text{elapsed}}):
T_{\text{elapsed}} = \max\left(0, T_{\text{audio\_ctx}} - T_{\text{rec\_start}} - L_{\text{comp}}\right)
Audio Time to Beat Conversion (\text{Beat}_{\text{current}}):
\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}
\text{Beat}_{\text{current}} = \frac{T_{\text{elapsed}}}{\text{SecondsPerBeat}} + \left(\text{Bar}_{\text{start}} \times \text{TS}_{\text{num}}\right)
Timeline Placement Mapping:
\text{StartBeat}_{\text{item}} = \text{Beat}_{\text{current}}
5.2 Algorithm 2: AudioWorklet PCM Ring-Buffer Processor
To prevent audio glitches or missing PCM frames when the browser's main thread is processing heavy UI renders, microphone recording runs inside an AudioWorkletProcessor:
// public/processors/pcm-recorder-processor.js
class PCMRecorderProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.bufferSize = 4096;
this.buffer = new Float32Array(this.bufferSize);
this.bufferIndex = 0;
}
process(inputs, outputs, parameters) {
const input = inputs[0];
if (input && input.length > 0) {
const inputChannel = input[0]; // Mono Channel 0
for (let i = 0; i < inputChannel.length; i++) {
this.buffer[this.bufferIndex++] = inputChannel[i];
// When Ring-Buffer fills, send Float32Array to Main Thread
if (this.bufferIndex >= this.bufferSize) {
this.port.postMessage({
type: 'PCM_DATA',
buffer: this.buffer.slice(0, this.bufferSize)
});
this.bufferIndex = 0;
}
}
}
return true; // Keep worklet active
}
}
registerProcessor('pcm-recorder-processor', PCMRecorderProcessor);
5.3 Algorithm 3: Client MIDIRecorder Class Implementation
class ClientMIDIRecorder {
constructor(audioContext, bpm = 120, timeSigNumerator = 4) {
this.audioCtx = audioContext;
this.bpm = bpm;
this.timeSigNum = timeSigNumerator;
this.isRecording = false;
this.activeNotes = new Map(); // Store pitch -> { noteId, startBeat, velocity }
this.recordedNotes = [];
this.recStartAudioTime = 0.0;
this.recStartBar = 0.0;
// Compute round-trip browser latency
this.latencyCompSec = (this.audioCtx.baseLatency || 0) + (this.audioCtx.outputLatency || 0);
}
start(startBar = 0.0) {
this.isRecording = true;
this.recordedNotes = [];
this.activeNotes.clear();
this.recStartBar = startBar;
this.recStartAudioTime = this.audioCtx.currentTime;
this.bindMIDIInputs();
}
bindMIDIInputs() {
if (navigator.requestMIDIAccess) {
navigator.requestMIDIAccess().then(midiAccess => {
for (let input of midiAccess.inputs.values()) {
input.onmidimessage = (event) => this.handleMIDIMessage(event);
}
});
}
}
handleMIDIMessage(event) {
if (!this.isRecording) return;
const [status, pitch, velocity] = event.data;
const command = status >> 4;
// Apply latency compensation formula
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
const secondsPerBeat = 60.0 / this.bpm;
const currentBeat = (currentTimeSec / secondsPerBeat) + (this.recStartBar * this.timeSigNum);
// Command 0x9: Note On
if (command === 0x9 && velocity > 0) {
const noteId = `rec_${Date.now()}_${pitch}`;
this.activeNotes.set(pitch, {
id: noteId,
pitch: pitch,
start_beat: currentBeat,
velocity: velocity / 127.0
});
}
// Command 0x8: Note Off (or Note On with velocity = 0)
else if (command === 0x8 || (command === 0x9 && velocity === 0)) {
if (this.activeNotes.has(pitch)) {
const note = this.activeNotes.get(pitch);
const durationBeats = Math.max(0.125, currentBeat - note.start_beat); // Min 1/32 note
this.recordedNotes.push({
id: note.id,
pitch: note.pitch,
start_beat: note.start_beat,
duration_beats: durationBeats,
velocity: note.velocity,
pan: 0.0
});
this.activeNotes.delete(pitch);
}
}
}
stop() {
this.isRecording = false;
// Flush remaining active keypresses when stop is triggered
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
const currentBeat = (currentTimeSec / (60.0 / this.bpm)) + (this.recStartBar * this.timeSigNum);
for (let [pitch, note] of this.activeNotes.entries()) {
this.recordedNotes.push({
id: note.id,
pitch: note.pitch,
start_beat: note.start_beat,
duration_beats: Math.max(0.25, currentBeat - note.start_beat),
velocity: note.velocity,
pan: 0.0
});
}
this.activeNotes.clear();
return this.recordedNotes;
}
}
5.4 Algorithm 4: Client AudioRecorder & AudioBuffer Splicing Class Implementation
class ClientAudioRecorder {
constructor(audioContext) {
this.audioCtx = audioContext;
this.mediaStream = null;
this.sourceNode = null;
this.workletNode = null;
this.pcmChunks = [];
this.isRecording = false;
}
async initializeInput(deviceId = null) {
const constraints = {
audio: {
deviceId: deviceId ? { exact: deviceId } : undefined,
echoCancellation: false,
noiseSuppression: false,
autoGainControl: false
}
};
this.mediaStream = await navigator.mediaDevices.getUserMedia(constraints);
this.sourceNode = this.audioCtx.createMediaStreamSource(this.mediaStream);
}
async start(destinationTrackGainNode, enableMonitoring = true) {
this.pcmChunks = [];
this.isRecording = true;
// Load Worklet Processor Module
await this.audioCtx.audioWorklet.addModule('/processors/pcm-recorder-processor.js');
this.workletNode = new AudioWorkletNode(this.audioCtx, 'pcm-recorder-processor');
// Receive PCM data streams from AudioWorklet
this.workletNode.port.onmessage = (event) => {
if (this.isRecording && event.data.type === 'PCM_DATA') {
this.pcmChunks.push(new Float32Array(event.data.buffer));
}
};
// Route Audio Nodes
this.sourceNode.connect(this.workletNode);
// Enable Live Input Monitoring if requested
if (enableMonitoring) {
this.sourceNode.connect(destinationTrackGainNode);
}
}
async stop() {
this.isRecording = false;
if (this.sourceNode && this.workletNode) {
this.sourceNode.disconnect(this.workletNode);
}
// Concatenate PCM Float32Array chunks into a single AudioBuffer
const totalSamples = this.pcmChunks.reduce((sum, chunk) => sum + chunk.length, 0);
if (totalSamples === 0) return null;
const audioBuffer = this.audioCtx.createBuffer(1, totalSamples, this.audioCtx.sampleRate);
const channelData = audioBuffer.getChannelData(0);
let offset = 0;
for (const chunk of this.pcmChunks) {
channelData.set(chunk, offset);
offset += chunk.length;
}
return audioBuffer; // Return compiled AudioBuffer for timeline insertion
}
}
6. UI Components & User Interactions
-
Track Header Arming Controls:
-
[R] Button (Arm Track): Highlights red when armed for recording on the target track.
-
[I] Button (Input Monitor): Toggles live monitoring for incoming Microphone or Synth audio during performance.
-
Input Selector Dropdown: Allows selection of available Microphone devices or USB Hardware MIDI Keyboards.
-
Real-time VU Meter Component:
-
Displays input signal gain level from
-60\text{ dB}to0\text{ dB}. Displays red clipping indicators when signal levels exceed0\text{ dBFS}. -
Live Waveform & MIDI Preview Rendering:
-
Microphone Recording: The canvas UI renders incoming waveform signals progressing along the Playhead position in real time.
-
MIDI Performance: Rectangular note blocks (green/orange) appear at note-on trigger events and extend until key release (note-off).