# TECHNICAL SPECIFICATION: CLIENT-SIDE REAL-TIME RECORDING ENGINE ## Browser-Based Microphone & Hardware MIDI Keyboard Recording Module --- ## 1. System Overview The Client-side recording module enables the DAW to capture live audio signals directly from Microphone/Line-in interfaces (via the Web MediaDevices API) and keypress events from Hardware MIDI Keyboards/Controllers (via the Web MIDI API) in real time. The module operates with low latency and includes hardware latency compensation. ```text +-----------------------------------------------------------------------------------+ | CLIENT BROWSER | | | | +-------------------------+ +-----------------------------+ | | | Hardware MIDI Keyboard | | Live Microphone / Line-In | | | +------------+------------+ +--------------+--------------+ | | | | | | v (Web MIDI API) v (getUserMedia) | | +------------+------------+ +--------------+--------------+ | | | Web MIDI Input Handler | | MediaStreamAudioSourceNode | | | +------------+------------+ +--------------+--------------+ | | | | | | +------------------+ | | | | | v | | v v +--------------+--------------+ | | +------------+-----+ +---------+-----------+ | Track Input Gain Node | | | | Event Clock / | | WebAudio Virtual | +--------------+--------------+ | | | Latency Engine | | Synth Engine | | | | +------------+-----+ +---------+-----------+ +--------+--------+ | | | | | | | | v v v v | | +------------+-----+ (Live Sound) +-------+-------+ +-------+------+ | | | Recorded MIDI | | AudioWorklet | | Monitoring | | | | Buffer | | Ring-Buffer | | Switch | | | +------------+-----+ | Recorder | +-------+------+ | | | +-------+-------+ | | | v | v | | [ Timeline MIDI ] v [ Master Mix ] | | [ Item Creation ] +-------+-------+ | | | Float32 PCM | | | | Audio Buffer | | | +-------+-------+ | | | | | v | | [ Timeline Audio] | | [ Item Creation ] | +-----------------------------------------------------------------------------------+ ``` --- ## 2. Hardware I/O & API Contracts ### 2.1 MediaDevices (Microphone Capture) **Permission Request:** Uses `navigator.mediaDevices.getUserMedia` configured to disable automatic browser processing DSP algorithms to capture pure, unprocessed audio signals: ```javascript const audioConstraints = { audio: { deviceId: selectedDeviceId ? { exact: selectedDeviceId } : undefined, echoCancellation: false, // Disables echo cancellation to prevent instrument sound distortion noiseSuppression: false, // Disables automatic noise suppression to preserve full frequency range autoGainControl: false, // Disables Automatic Gain Control (AGC) latency: 0 // Requests minimal latency from OS audio driver } }; ``` ### 2.2 Web MIDI API Integration **Device Enumeration & Listener Assignment:** * Uses `navigator.requestMIDIAccess({ sysex: false })` to scan for USB-connected keyboard devices. * **Timestamp Precision:** Obtains event timestamps from `MIDIMessageEvent.timeStamp` (as a `DOMHighResTimeStamp` in microseconds) and synchronizes them with `AudioContext.currentTime`. --- ## 3. Recording Lifecycle & State Machine ```text [IDLE] ───► (User Arms Track) ───► [ARMED] ───► (Press Rec + Play) ───► [COUNT-IN / PRE-ROLL] | [STOP & COMMIT] ◄─── (Press Stop) ◄─── [RECORDING IN PROGRESS] ◄───────────────+ ``` * **Arming Phase (Record Enable):** * The user selects an input source and activates the Arm (R) button on the target track. * Initializes the input level meter (VU Meter Canvas) to display input volume levels in real time. * **Pre-Roll / Count-In Phase:** * Transport triggers the metronome count-in (e.g., 1 Bar = 4 beats). The metronome plays click sounds based on project BPM. * The engine does not write data to the Timeline yet, but begins reading the input buffer to prepare memory buffers. * **Recording Phase:** * Once the transport passes the Start Bar boundary, incoming MIDI key events or PCM Float32 audio samples are written into the active recording buffer memory. * Canvas UI displays real-time visual feedback, rendering waveforms or MIDI note blocks dynamically. * **Stop & Commit Phase:** * Pressing Stop halts the recording process. * Converts temporary memory buffers into a structured `MIDIItem` or `AudioItem`. * Inserts the new Item onto the target track within the Main Session or Section tab. --- ## 4. Data Structures ### 4.1 Live MIDI Event Buffer Element Schema ```json { "type": "object", "properties": { "pitch": { "type": "integer", "minimum": 0, "maximum": 127 }, "start_beat": { "type": "number", "description": "Start position in beats on the timeline" }, "duration_beats": { "type": "number", "description": "Keypress duration in beats" }, "velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0 }, "channel": { "type": "integer", "default": 0 } } } ``` ### 4.2 Recording Track Input Configuration State ```json { "track_id": "track_midi_01", "is_armed": true, "monitoring_enabled": true, "input_source": { "device_type": "MIDI_KEYBOARD", "device_id": "midi_input_usb_keyboard_0", "channel": 1 }, "input_gain_db": 0.0, "latency_offset_ms": 12.5 } ``` --- ## 5. Core Algorithms & Latency Compensation ### 5.1 Algorithm 1: Hardware Latency Compensation Formula When recording, the physical moment a key is pressed or sound enters the microphone is inherently delayed relative to speaker output due to input buffers ($L_{\text{input}}$) and output buffers ($L_{\text{output}}$). #### Mathematical Formulation Let: * $T_{\text{audio\_ctx}}$ = Current timestamp in seconds on the `AudioContext` clock (`audioCtx.currentTime`). * $T_{\text{rec\_start}}$ = Recording start timestamp in seconds. * $\text{BPM}$ = Song tempo (Beats Per Minute). * $\text{TS}_{\text{num}}$ = Time Signature Numerator (beats per bar). * $\text{Bar}_{\text{start}}$ = Target timeline start bar for recording. * $L_{\text{comp}}$ = Total hardware latency offset ($L_{\text{input}} + L_{\text{output}} + L_{\text{user\_offset}}$) in seconds. **Actual Elapsed Audio Time ($T_{\text{elapsed}}$):** $$T_{\text{elapsed}} = \max\left(0, T_{\text{audio\_ctx}} - T_{\text{rec\_start}} - L_{\text{comp}}\right)$$ **Audio Time to Beat Conversion ($\text{Beat}_{\text{current}}$):** $$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$ $$\text{Beat}_{\text{current}} = \frac{T_{\text{elapsed}}}{\text{SecondsPerBeat}} + \left(\text{Bar}_{\text{start}} \times \text{TS}_{\text{num}}\right)$$ **Timeline Placement Mapping:** $$\text{StartBeat}_{\text{item}} = \text{Beat}_{\text{current}}$$ --- ### 5.2 Algorithm 2: AudioWorklet PCM Ring-Buffer Processor To prevent audio glitches or missing PCM frames when the browser's main thread is processing heavy UI renders, microphone recording runs inside an `AudioWorkletProcessor`: ```javascript // public/processors/pcm-recorder-processor.js class PCMRecorderProcessor extends AudioWorkletProcessor { constructor() { super(); this.bufferSize = 4096; this.buffer = new Float32Array(this.bufferSize); this.bufferIndex = 0; } process(inputs, outputs, parameters) { const input = inputs[0]; if (input && input.length > 0) { const inputChannel = input[0]; // Mono Channel 0 for (let i = 0; i < inputChannel.length; i++) { this.buffer[this.bufferIndex++] = inputChannel[i]; // When Ring-Buffer fills, send Float32Array to Main Thread if (this.bufferIndex >= this.bufferSize) { this.port.postMessage({ type: 'PCM_DATA', buffer: this.buffer.slice(0, this.bufferSize) }); this.bufferIndex = 0; } } } return true; // Keep worklet active } } registerProcessor('pcm-recorder-processor', PCMRecorderProcessor); ``` --- ### 5.3 Algorithm 3: Client MIDIRecorder Class Implementation ```javascript class ClientMIDIRecorder { constructor(audioContext, bpm = 120, timeSigNumerator = 4) { this.audioCtx = audioContext; this.bpm = bpm; this.timeSigNum = timeSigNumerator; this.isRecording = false; this.activeNotes = new Map(); // Store pitch -> { noteId, startBeat, velocity } this.recordedNotes = []; this.recStartAudioTime = 0.0; this.recStartBar = 0.0; // Compute round-trip browser latency this.latencyCompSec = (this.audioCtx.baseLatency || 0) + (this.audioCtx.outputLatency || 0); } start(startBar = 0.0) { this.isRecording = true; this.recordedNotes = []; this.activeNotes.clear(); this.recStartBar = startBar; this.recStartAudioTime = this.audioCtx.currentTime; this.bindMIDIInputs(); } bindMIDIInputs() { if (navigator.requestMIDIAccess) { navigator.requestMIDIAccess().then(midiAccess => { for (let input of midiAccess.inputs.values()) { input.onmidimessage = (event) => this.handleMIDIMessage(event); } }); } } handleMIDIMessage(event) { if (!this.isRecording) return; const [status, pitch, velocity] = event.data; const command = status >> 4; // Apply latency compensation formula const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec); const secondsPerBeat = 60.0 / this.bpm; const currentBeat = (currentTimeSec / secondsPerBeat) + (this.recStartBar * this.timeSigNum); // Command 0x9: Note On if (command === 0x9 && velocity > 0) { const noteId = `rec_${Date.now()}_${pitch}`; this.activeNotes.set(pitch, { id: noteId, pitch: pitch, start_beat: currentBeat, velocity: velocity / 127.0 }); } // Command 0x8: Note Off (or Note On with velocity = 0) else if (command === 0x8 || (command === 0x9 && velocity === 0)) { if (this.activeNotes.has(pitch)) { const note = this.activeNotes.get(pitch); const durationBeats = Math.max(0.125, currentBeat - note.start_beat); // Min 1/32 note this.recordedNotes.push({ id: note.id, pitch: note.pitch, start_beat: note.start_beat, duration_beats: durationBeats, velocity: note.velocity, pan: 0.0 }); this.activeNotes.delete(pitch); } } } stop() { this.isRecording = false; // Flush remaining active keypresses when stop is triggered const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec); const currentBeat = (currentTimeSec / (60.0 / this.bpm)) + (this.recStartBar * this.timeSigNum); for (let [pitch, note] of this.activeNotes.entries()) { this.recordedNotes.push({ id: note.id, pitch: note.pitch, start_beat: note.start_beat, duration_beats: Math.max(0.25, currentBeat - note.start_beat), velocity: note.velocity, pan: 0.0 }); } this.activeNotes.clear(); return this.recordedNotes; } } ``` --- ### 5.4 Algorithm 4: Client AudioRecorder & AudioBuffer Splicing Class Implementation ```javascript class ClientAudioRecorder { constructor(audioContext) { this.audioCtx = audioContext; this.mediaStream = null; this.sourceNode = null; this.workletNode = null; this.pcmChunks = []; this.isRecording = false; } async initializeInput(deviceId = null) { const constraints = { audio: { deviceId: deviceId ? { exact: deviceId } : undefined, echoCancellation: false, noiseSuppression: false, autoGainControl: false } }; this.mediaStream = await navigator.mediaDevices.getUserMedia(constraints); this.sourceNode = this.audioCtx.createMediaStreamSource(this.mediaStream); } async start(destinationTrackGainNode, enableMonitoring = true) { this.pcmChunks = []; this.isRecording = true; // Load Worklet Processor Module await this.audioCtx.audioWorklet.addModule('/processors/pcm-recorder-processor.js'); this.workletNode = new AudioWorkletNode(this.audioCtx, 'pcm-recorder-processor'); // Receive PCM data streams from AudioWorklet this.workletNode.port.onmessage = (event) => { if (this.isRecording && event.data.type === 'PCM_DATA') { this.pcmChunks.push(new Float32Array(event.data.buffer)); } }; // Route Audio Nodes this.sourceNode.connect(this.workletNode); // Enable Live Input Monitoring if requested if (enableMonitoring) { this.sourceNode.connect(destinationTrackGainNode); } } async stop() { this.isRecording = false; if (this.sourceNode && this.workletNode) { this.sourceNode.disconnect(this.workletNode); } // Concatenate PCM Float32Array chunks into a single AudioBuffer const totalSamples = this.pcmChunks.reduce((sum, chunk) => sum + chunk.length, 0); if (totalSamples === 0) return null; const audioBuffer = this.audioCtx.createBuffer(1, totalSamples, this.audioCtx.sampleRate); const channelData = audioBuffer.getChannelData(0); let offset = 0; for (const chunk of this.pcmChunks) { channelData.set(chunk, offset); offset += chunk.length; } return audioBuffer; // Return compiled AudioBuffer for timeline insertion } } ``` --- ## 6. UI Components & User Interactions * **Track Header Arming Controls:** * **[R] Button (Arm Track):** Highlights red when armed for recording on the target track. * **[I] Button (Input Monitor):** Toggles live monitoring for incoming Microphone or Synth audio during performance. * **Input Selector Dropdown:** Allows selection of available Microphone devices or USB Hardware MIDI Keyboards. * **Real-time VU Meter Component:** * Displays input signal gain level from $-60\text{ dB}$ to $0\text{ dB}$. Displays red clipping indicators when signal levels exceed $0\text{ dBFS}$. * **Live Waveform & MIDI Preview Rendering:** * **Microphone Recording:** The canvas UI renders incoming waveform signals progressing along the Playhead position in real time. * **MIDI Performance:** Rectangular note blocks (green/orange) appear at note-on trigger events and extend until key release (note-off).