# TECHNICAL SPECIFICATION: CLIENT-SIDE REAL-TIME RECORDING ENGINE

## Browser-Based Microphone & Hardware MIDI Keyboard Recording Module

---

## 1. System Overview

The Client-side recording module enables the DAW to capture live audio signals directly from Microphone/Line-in interfaces (via the Web MediaDevices API) and keypress events from Hardware MIDI Keyboards/Controllers (via the Web MIDI API) in real time. The module operates with low latency and includes hardware latency compensation.

```text
+-----------------------------------------------------------------------------------+
|                                 CLIENT BROWSER                                    |
|                                                                                   |
|  +-------------------------+                     +-----------------------------+  |
|  | Hardware MIDI Keyboard  |                     | Live Microphone / Line-In   |  |
|  +------------+------------+                     +--------------+--------------+  |
|               |                                                 |                 |
|               v (Web MIDI API)                                  v (getUserMedia)  |
|  +------------+------------+                     +--------------+--------------+  |
|  | Web MIDI Input Handler  |                     |  MediaStreamAudioSourceNode |  |
|  +------------+------------+                     +--------------+--------------+  |
|               |                                                 |                 |
|               +------------------+                              |                 |
|               |                  |                              v                 |
|               v                  v               +--------------+--------------+  |
|  +------------+-----+  +---------+-----------+   |   Track Input Gain Node     |  |
|  | Event Clock /    |  | WebAudio Virtual    |   +--------------+--------------+  |
|  | Latency Engine   |  | Synth Engine        |                  |                 |
|  +------------+-----+  +---------+-----------+         +--------+--------+        |
|               |                  |                     |                 |        |
|               v                  v                     v                 v        |
|  +------------+-----+    (Live Sound)          +-------+-------+ +-------+------+ |
|  | Recorded MIDI    |                          | AudioWorklet  | | Monitoring   | |
|  | Buffer           |                          | Ring-Buffer   | | Switch       | |
|  +------------+-----+                          | Recorder      | +-------+------+ |
|               |                                +-------+-------+         |        |
|               v                                        |                 v        |
|   [ Timeline MIDI   ]                                  v         [ Master Mix ]   |
|   [ Item Creation   ]                          +-------+-------+                  |
|                                                | Float32 PCM   |                  |
|                                                | Audio Buffer  |                  |
|                                                +-------+-------+                  |
|                                                        |                          |
|                                                        v                          |
|                                                [ Timeline Audio]                  |
|                                                [ Item Creation ]                  |
+-----------------------------------------------------------------------------------+

```

---

## 2. Hardware I/O & API Contracts

### 2.1 MediaDevices (Microphone Capture)

**Permission Request:** Uses `navigator.mediaDevices.getUserMedia` configured to disable automatic browser processing DSP algorithms to capture pure, unprocessed audio signals:

```javascript
const audioConstraints = {
    audio: {
        deviceId: selectedDeviceId ? { exact: selectedDeviceId } : undefined,
        echoCancellation: false,   // Disables echo cancellation to prevent instrument sound distortion
        noiseSuppression: false,   // Disables automatic noise suppression to preserve full frequency range
        autoGainControl: false,    // Disables Automatic Gain Control (AGC)
        latency: 0                 // Requests minimal latency from OS audio driver
    }
};

```

### 2.2 Web MIDI API Integration

**Device Enumeration & Listener Assignment:**

* Uses `navigator.requestMIDIAccess({ sysex: false })` to scan for USB-connected keyboard devices.
* **Timestamp Precision:** Obtains event timestamps from `MIDIMessageEvent.timeStamp` (as a `DOMHighResTimeStamp` in microseconds) and synchronizes them with `AudioContext.currentTime`.

---

## 3. Recording Lifecycle & State Machine

```text
 [IDLE] ───► (User Arms Track) ───► [ARMED] ───► (Press Rec + Play) ───► [COUNT-IN / PRE-ROLL]
                                                                                |
 [STOP & COMMIT] ◄─── (Press Stop) ◄─── [RECORDING IN PROGRESS] ◄───────────────+

```

* **Arming Phase (Record Enable):**
* The user selects an input source and activates the Arm (R) button on the target track.
* Initializes the input level meter (VU Meter Canvas) to display input volume levels in real time.


* **Pre-Roll / Count-In Phase:**
* Transport triggers the metronome count-in (e.g., 1 Bar = 4 beats). The metronome plays click sounds based on project BPM.
* The engine does not write data to the Timeline yet, but begins reading the input buffer to prepare memory buffers.


* **Recording Phase:**
* Once the transport passes the Start Bar boundary, incoming MIDI key events or PCM Float32 audio samples are written into the active recording buffer memory.
* Canvas UI displays real-time visual feedback, rendering waveforms or MIDI note blocks dynamically.


* **Stop & Commit Phase:**
* Pressing Stop halts the recording process.
* Converts temporary memory buffers into a structured `MIDIItem` or `AudioItem`.
* Inserts the new Item onto the target track within the Main Session or Section tab.



---

## 4. Data Structures

### 4.1 Live MIDI Event Buffer Element Schema

```json
{
  "type": "object",
  "properties": {
    "pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
    "start_beat": { "type": "number", "description": "Start position in beats on the timeline" },
    "duration_beats": { "type": "number", "description": "Keypress duration in beats" },
    "velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0 },
    "channel": { "type": "integer", "default": 0 }
  }
}

```

### 4.2 Recording Track Input Configuration State

```json
{
  "track_id": "track_midi_01",
  "is_armed": true,
  "monitoring_enabled": true,
  "input_source": {
    "device_type": "MIDI_KEYBOARD",
    "device_id": "midi_input_usb_keyboard_0",
    "channel": 1
  },
  "input_gain_db": 0.0,
  "latency_offset_ms": 12.5
}

```

---

## 5. Core Algorithms & Latency Compensation

### 5.1 Algorithm 1: Hardware Latency Compensation Formula

When recording, the physical moment a key is pressed or sound enters the microphone is inherently delayed relative to speaker output due to input buffers ($L_{\text{input}}$) and output buffers ($L_{\text{output}}$).

#### Mathematical Formulation

Let:

* $T_{\text{audio\_ctx}}$ = Current timestamp in seconds on the `AudioContext` clock (`audioCtx.currentTime`).
* $T_{\text{rec\_start}}$ = Recording start timestamp in seconds.
* $\text{BPM}$ = Song tempo (Beats Per Minute).
* $\text{TS}_{\text{num}}$ = Time Signature Numerator (beats per bar).
* $\text{Bar}_{\text{start}}$ = Target timeline start bar for recording.
* $L_{\text{comp}}$ = Total hardware latency offset ($L_{\text{input}} + L_{\text{output}} + L_{\text{user\_offset}}$) in seconds.

**Actual Elapsed Audio Time ($T_{\text{elapsed}}$):**

$$T_{\text{elapsed}} = \max\left(0, T_{\text{audio\_ctx}} - T_{\text{rec\_start}} - L_{\text{comp}}\right)$$

**Audio Time to Beat Conversion ($\text{Beat}_{\text{current}}$):**

$$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$

$$\text{Beat}_{\text{current}} = \frac{T_{\text{elapsed}}}{\text{SecondsPerBeat}} + \left(\text{Bar}_{\text{start}} \times \text{TS}_{\text{num}}\right)$$

**Timeline Placement Mapping:**

$$\text{StartBeat}_{\text{item}} = \text{Beat}_{\text{current}}$$

---

### 5.2 Algorithm 2: AudioWorklet PCM Ring-Buffer Processor

To prevent audio glitches or missing PCM frames when the browser's main thread is processing heavy UI renders, microphone recording runs inside an `AudioWorkletProcessor`:

```javascript
// public/processors/pcm-recorder-processor.js
class PCMRecorderProcessor extends AudioWorkletProcessor {
    constructor() {
        super();
        this.bufferSize = 4096;
        this.buffer = new Float32Array(this.bufferSize);
        this.bufferIndex = 0;
    }

    process(inputs, outputs, parameters) {
        const input = inputs[0];
        if (input && input.length > 0) {
            const inputChannel = input[0]; // Mono Channel 0
            
            for (let i = 0; i < inputChannel.length; i++) {
                this.buffer[this.bufferIndex++] = inputChannel[i];
                
                // When Ring-Buffer fills, send Float32Array to Main Thread
                if (this.bufferIndex >= this.bufferSize) {
                    this.port.postMessage({
                        type: 'PCM_DATA',
                        buffer: this.buffer.slice(0, this.bufferSize)
                    });
                    this.bufferIndex = 0;
                }
            }
        }
        return true; // Keep worklet active
    }
}

registerProcessor('pcm-recorder-processor', PCMRecorderProcessor);

```

---

### 5.3 Algorithm 3: Client MIDIRecorder Class Implementation

```javascript
class ClientMIDIRecorder {
    constructor(audioContext, bpm = 120, timeSigNumerator = 4) {
        this.audioCtx = audioContext;
        this.bpm = bpm;
        this.timeSigNum = timeSigNumerator;
        this.isRecording = false;
        
        this.activeNotes = new Map(); // Store pitch -> { noteId, startBeat, velocity }
        this.recordedNotes = [];
        this.recStartAudioTime = 0.0;
        this.recStartBar = 0.0;
        
        // Compute round-trip browser latency
        this.latencyCompSec = (this.audioCtx.baseLatency || 0) + (this.audioCtx.outputLatency || 0);
    }

    start(startBar = 0.0) {
        this.isRecording = true;
        this.recordedNotes = [];
        this.activeNotes.clear();
        this.recStartBar = startBar;
        this.recStartAudioTime = this.audioCtx.currentTime;
        
        this.bindMIDIInputs();
    }

    bindMIDIInputs() {
        if (navigator.requestMIDIAccess) {
            navigator.requestMIDIAccess().then(midiAccess => {
                for (let input of midiAccess.inputs.values()) {
                    input.onmidimessage = (event) => this.handleMIDIMessage(event);
                }
            });
        }
    }

    handleMIDIMessage(event) {
        if (!this.isRecording) return;

        const [status, pitch, velocity] = event.data;
        const command = status >> 4;
        
        // Apply latency compensation formula
        const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
        const secondsPerBeat = 60.0 / this.bpm;
        const currentBeat = (currentTimeSec / secondsPerBeat) + (this.recStartBar * this.timeSigNum);

        // Command 0x9: Note On
        if (command === 0x9 && velocity > 0) {
            const noteId = `rec_${Date.now()}_${pitch}`;
            this.activeNotes.set(pitch, {
                id: noteId,
                pitch: pitch,
                start_beat: currentBeat,
                velocity: velocity / 127.0
            });
        } 
        // Command 0x8: Note Off (or Note On with velocity = 0)
        else if (command === 0x8 || (command === 0x9 && velocity === 0)) {
            if (this.activeNotes.has(pitch)) {
                const note = this.activeNotes.get(pitch);
                const durationBeats = Math.max(0.125, currentBeat - note.start_beat); // Min 1/32 note
                
                this.recordedNotes.push({
                    id: note.id,
                    pitch: note.pitch,
                    start_beat: note.start_beat,
                    duration_beats: durationBeats,
                    velocity: note.velocity,
                    pan: 0.0
                });
                
                this.activeNotes.delete(pitch);
            }
        }
    }

    stop() {
        this.isRecording = false;
        
        // Flush remaining active keypresses when stop is triggered
        const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
        const currentBeat = (currentTimeSec / (60.0 / this.bpm)) + (this.recStartBar * this.timeSigNum);

        for (let [pitch, note] of this.activeNotes.entries()) {
            this.recordedNotes.push({
                id: note.id,
                pitch: note.pitch,
                start_beat: note.start_beat,
                duration_beats: Math.max(0.25, currentBeat - note.start_beat),
                velocity: note.velocity,
                pan: 0.0
            });
        }
        this.activeNotes.clear();
        return this.recordedNotes;
    }
}

```

---

### 5.4 Algorithm 4: Client AudioRecorder & AudioBuffer Splicing Class Implementation

```javascript
class ClientAudioRecorder {
    constructor(audioContext) {
        this.audioCtx = audioContext;
        this.mediaStream = null;
        this.sourceNode = null;
        this.workletNode = null;
        this.pcmChunks = [];
        this.isRecording = false;
    }

    async initializeInput(deviceId = null) {
        const constraints = {
            audio: {
                deviceId: deviceId ? { exact: deviceId } : undefined,
                echoCancellation: false,
                noiseSuppression: false,
                autoGainControl: false
            }
        };
        this.mediaStream = await navigator.mediaDevices.getUserMedia(constraints);
        this.sourceNode = this.audioCtx.createMediaStreamSource(this.mediaStream);
    }

    async start(destinationTrackGainNode, enableMonitoring = true) {
        this.pcmChunks = [];
        this.isRecording = true;

        // Load Worklet Processor Module
        await this.audioCtx.audioWorklet.addModule('/processors/pcm-recorder-processor.js');
        this.workletNode = new AudioWorkletNode(this.audioCtx, 'pcm-recorder-processor');

        // Receive PCM data streams from AudioWorklet
        this.workletNode.port.onmessage = (event) => {
            if (this.isRecording && event.data.type === 'PCM_DATA') {
                this.pcmChunks.push(new Float32Array(event.data.buffer));
            }
        };

        // Route Audio Nodes
        this.sourceNode.connect(this.workletNode);

        // Enable Live Input Monitoring if requested
        if (enableMonitoring) {
            this.sourceNode.connect(destinationTrackGainNode);
        }
    }

    async stop() {
        this.isRecording = false;
        
        if (this.sourceNode && this.workletNode) {
            this.sourceNode.disconnect(this.workletNode);
        }

        // Concatenate PCM Float32Array chunks into a single AudioBuffer
        const totalSamples = this.pcmChunks.reduce((sum, chunk) => sum + chunk.length, 0);
        if (totalSamples === 0) return null;

        const audioBuffer = this.audioCtx.createBuffer(1, totalSamples, this.audioCtx.sampleRate);
        const channelData = audioBuffer.getChannelData(0);

        let offset = 0;
        for (const chunk of this.pcmChunks) {
            channelData.set(chunk, offset);
            offset += chunk.length;
        }

        return audioBuffer; // Return compiled AudioBuffer for timeline insertion
    }
}

```

---

## 6. UI Components & User Interactions

* **Track Header Arming Controls:**
* **[R] Button (Arm Track):** Highlights red when armed for recording on the target track.
* **[I] Button (Input Monitor):** Toggles live monitoring for incoming Microphone or Synth audio during performance.
* **Input Selector Dropdown:** Allows selection of available Microphone devices or USB Hardware MIDI Keyboards.


* **Real-time VU Meter Component:**
* Displays input signal gain level from $-60\text{ dB}$ to $0\text{ dB}$. Displays red clipping indicators when signal levels exceed $0\text{ dBFS}$.


* **Live Waveform & MIDI Preview Rendering:**
* **Microphone Recording:** The canvas UI renders incoming waveform signals progressing along the Playhead position in real time.
* **MIDI Performance:** Rectangular note blocks (green/orange) appear at note-on trigger events and extend until key release (note-off).