feat: xử lí track/clip với AI
This commit is contained in:
+214
@@ -0,0 +1,214 @@
|
||||
# Technical Specification: Ultra-Zoom & Sample-Level Waveform Rendering (Sample-Level Waveform Zoom)
|
||||
|
||||
This document defines the technical solution, data flow schema, and graphical optimization algorithms across both the Frontend (HTML5 Canvas) and Backend (Python / Docker) to implement an Ultra-Zoom Waveform feature. This architecture renders discrete sample nodes interconnected by a continuous line vector for absolute Zero-Crossing alignment, referencing the design principles.
|
||||
|
||||
---
|
||||
|
||||
## 1. What is Sample-Level Zoom?
|
||||
|
||||
When displaying an audio waveform at a macro scale (Zoom Out), a single pixel column on the display represents hundreds or thousands of acoustic samples ($N$ samples/pixel). Consequently, the engine deploys a Peak Waveform algorithm that connects the maximum (Max) and minimum (Min) amplitude values within that segment using vertical lines.
|
||||
|
||||
However, when a operator scales the viewport magnification beyond a specific threshold (e.g., a zoom ratio of $Z \ge 100,000\text{ pixels/second}$):
|
||||
|
||||
* A single discrete audio sample occupies a large horizontal footprint on the display (e.g., $5 \rightarrow 15\text{ pixels/sample}$).
|
||||
* The rendering engine must hot-swap its routine from standard vertical peak columns to a **Continuous Polyline with Sample Nodes** loop. Every discrete acoustic sample $x[n]$ is mapped as an independent circle node, with chronologically adjacent nodes joined by a smooth continuous path.
|
||||
|
||||
---
|
||||
|
||||
## 2. Frontend Layout Architecture (HTML5 Canvas & Web Audio API)
|
||||
|
||||
To render thousands of vector coordinate indices fluidly during rapid zooming and scrolling/dragging gestures without locking up the browser thread (Freeze UI), the system integrates the following memory pipeline:
|
||||
|
||||
```text
|
||||
VIEWPORT SLICING ENGINE
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ [ Web Audio Buffer (Full track - Millions of raw sample values) ] │
|
||||
│ │ │
|
||||
│ ▼ (Extract visible boundary region only) │
|
||||
│ [ Visible Sample Array (Restricted to ~200 - 1,000 samples in view) ] │
|
||||
│ │ │
|
||||
│ ▼ (High-speed GPU-accelerated Canvas draw) │
|
||||
│ [ HTML5 Canvas Render: ctx.arc() & ctx.lineTo() ] ──► Screen Viewport │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
### 2.1. Viewport Slicing Technique
|
||||
|
||||
The rendering engine must never iterate through the total sample length of the audio file during a drawing pass. The slice generator isolates only the data segments that correspond directly to the physical visible screen dimensions (visible viewport boundary):
|
||||
|
||||
* **Visible Starting Timestamp:**
|
||||
|
||||
$$T_{\text{start}} = \frac{\text{scrollLeft}}{\text{Zoom}}$$
|
||||
|
||||
|
||||
* **Visible Terminating Timestamp:**
|
||||
|
||||
$$T_{\text{end}} = \frac{\text{scrollLeft} + W_{\text{viewport}}}{\text{Zoom}}$$
|
||||
|
||||
|
||||
* **Starting Array Index Offset:**
|
||||
|
||||
$$n_{\text{start}} = \lfloor T_{\text{start}} \times f_s \rfloor$$
|
||||
|
||||
|
||||
* **Terminating Array Index Offset:**
|
||||
|
||||
$$n_{\text{end}} = \lceil T_{\text{end}} \times f_s \rceil$$
|
||||
|
||||
|
||||
|
||||
### 2.2. Sample Node Graph Canvas Algorithm
|
||||
|
||||
For every absolute sample index $x[i]$ contained within the sliced viewport interval $[n_{\text{start}}, n_{\text{end}}]$, the coordinate translation layer maps the raw data into physical pixel coordinates $(X, Y)$ on the Canvas:
|
||||
|
||||
$$X_i = \left( \frac{i}{f_s} \right) \times \text{Zoom} - \text{scrollLeft}$$
|
||||
|
||||
$$Y_i = \text{mid}_Y + x[i] \cdot \left( \text{height} \times 0.42 \right)$$
|
||||
|
||||
*Where:* $\text{mid}_Y$ maps the horizontal center zero axis (-Inf. dB line), and $x[i] \in [-1.0, 1.0]$ tracks the floating-point sample amplitude value.
|
||||
|
||||
### JavaScript Redraw Core Script (React / JS Context)
|
||||
|
||||
```javascript
|
||||
function drawSampleLevelWaveform(ctx, canvasWidth, canvasHeight, audioBuffer, scrollLeft, zoom) {
|
||||
const data = audioBuffer.getChannelData(0); // Query Left channel data stream
|
||||
const fs = audioBuffer.sampleRate;
|
||||
const midY = canvasHeight / 2;
|
||||
const ampHeight = canvasHeight * 0.42; // Clamps drawing ceiling bounds to 84% of total height
|
||||
|
||||
// 1. Viewport Slicing Matrix Execution
|
||||
const tStart = scrollLeft / zoom;
|
||||
const tEnd = (scrollLeft + canvasWidth) / zoom;
|
||||
const nStart = Math.max(0, Math.floor(tStart * fs));
|
||||
const nEnd = Math.min(data.length, Math.ceil(tEnd * fs));
|
||||
|
||||
ctx.clearRect(0, 0, canvasWidth, canvasHeight);
|
||||
|
||||
// Set up standard studio charcoal theme background canvas
|
||||
ctx.fillStyle = '#1e1e1e';
|
||||
ctx.fillRect(0, 0, canvasWidth, canvasHeight);
|
||||
|
||||
// Overlay symmetrical decibel gridding lines (-6.0 dB, -Inf, -6.0 dB)
|
||||
ctx.strokeStyle = 'rgba(255, 255, 255, 0.08)';
|
||||
ctx.lineWidth = 1;
|
||||
[-0.501, 0, 0.501].forEach(val => {
|
||||
const y = midY + (val * ampHeight);
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(0, y);
|
||||
ctx.lineTo(canvasWidth, y);
|
||||
ctx.stroke();
|
||||
});
|
||||
|
||||
// 2. Continuous Vector Polyline Redraw Configuration
|
||||
ctx.strokeStyle = '#5bc0be'; // Professional sleek light cyan accent theme
|
||||
ctx.lineWidth = 1.5;
|
||||
ctx.beginPath();
|
||||
|
||||
let isFirst = true;
|
||||
for (let i = nStart; i < nEnd; i++) {
|
||||
const xPixel = (i / fs) * zoom - scrollLeft;
|
||||
const yPixel = midY + (data[i] * ampHeight);
|
||||
|
||||
if (isFirst) {
|
||||
ctx.moveTo(xPixel, yPixel);
|
||||
isFirst = false;
|
||||
} else {
|
||||
ctx.lineTo(xPixel, yPixel);
|
||||
}
|
||||
}
|
||||
ctx.stroke();
|
||||
|
||||
// 3. Highlight Discrete Sample Nodes (Luminous node nodes circles)
|
||||
ctx.fillStyle = '#6ee7b7'; // Vivid green emerald node color
|
||||
for (let i = nStart; i < nEnd; i++) {
|
||||
const xPixel = (i / fs) * zoom - scrollLeft;
|
||||
const yPixel = midY + (data[i] * ampHeight);
|
||||
|
||||
// Render point node indicators if the physical pixel delta spacing is >= 4px (Prevents GPU thread thrashing)
|
||||
const nextXPixel = ((i + 1) / fs) * zoom - scrollLeft;
|
||||
if (nextXPixel - xPixel >= 4) {
|
||||
ctx.beginPath();
|
||||
ctx.arc(xPixel, yPixel, 2, 0, 2 * Math.PI);
|
||||
ctx.fill();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Backend Architecture (Python / NumPy / Docker)
|
||||
|
||||
When an operator triggers editing transformations, loop boundary indexing (AI Scan Loops), or an AI Cut on the user interface, precise timestamp scalars (seconds) are pushed to the backend stack. The FastAPI routing layer and Celery task worker process the input metrics via NumPy using sample-accurate precision to eliminate clicking audio defects.
|
||||
|
||||
### 3.1. High-Performance Vectorized Zero-Crossing Analysis via NumPy
|
||||
|
||||
This algorithm targets the exact index offset location where an algebraic sign-inversion occurs (crossing the absolute 0 baseline) closest to the user's cursor selection coordinate:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
def find_exact_zero_crossing_sample(y: np.ndarray, sr: int, target_time: float, search_window_ms: float = 40.0) -> int:
|
||||
"""
|
||||
Scans the signal buffer matrix to extract the exact sample index where amplitude
|
||||
crosses the absolute 0 axis closest to target_time. Mitigates signal phase fracture.
|
||||
"""
|
||||
target_sample = int(target_time * sr)
|
||||
window_samples = int((search_window_ms / 1000.0) * sr)
|
||||
|
||||
# Establish local window limits
|
||||
start_idx = max(0, target_sample - window_samples // 2)
|
||||
end_idx = min(len(y) - 2, target_sample + window_samples // 2)
|
||||
|
||||
y_segment = y[start_idx:end_idx]
|
||||
|
||||
# Vectorized loop matching physical phase boundaries: y[i] * y[i+1] <= 0
|
||||
# This evaluates ultra-fast directly on NumPy's optimized underlying C-layer
|
||||
zero_crossings = np.where(y_segment[:-1] * y_segment[1:] <= 0)[0]
|
||||
|
||||
if len(zero_crossings) == 0:
|
||||
# Fallback: if no phase inversion is detected (extended silence), return the minimum absolute sample value
|
||||
abs_min_idx = np.argmin(np.abs(y_segment))
|
||||
return start_idx + abs_min_idx
|
||||
|
||||
# Translate the localized coordinate index back to global absolute buffer sample indices
|
||||
absolute_crossings = zero_crossings + start_idx
|
||||
|
||||
# Isolate the index that maps closest to the original physical target_sample address
|
||||
distances = np.abs(absolute_crossings - target_sample)
|
||||
best_sample_index = absolute_crossings[np.argmin(distances)]
|
||||
|
||||
return int(best_sample_index)
|
||||
|
||||
```
|
||||
|
||||
### 3.2. Fade-Free Zero-Crossing Splicing Workflow
|
||||
|
||||
Once the exact boundary indices ($N_{\text{start\_zero}}$, $N_{\text{end\_zero}}$) are located using the zero-crossing analyzer:
|
||||
|
||||
1. **Slicing Operation:**
|
||||
```python
|
||||
y_cut = y[N_start_zero : N_end_zero]
|
||||
|
||||
```
|
||||
|
||||
|
||||
2. **Merging & Track Insertion:** The sliced audio block is appended straight into the signal array of the destination track. Because both the initial and terminating boundaries of the cut segment are locked perfectly to a theoretical value of $0\text{V}$, splicing this array into any other silent segment preserves absolute physical phase continuity.
|
||||
3. **Bypassing Fade Modulators:** The physical transient profiles (**Transients**) of percussive assets (Kick Drums, Snares, Claps) remain $100\%$ unwarped. This completely preserves the crisp, punchy acoustic characteristics of the source audio data.
|
||||
|
||||
---
|
||||
|
||||
## 4. Performance Optimization Manual
|
||||
|
||||
* **Double Buffering (Offscreen Canvas Rendering Canvas):** Under extreme magnification scales, client-side horizontal scrolling modifications (`onScroll`) trigger continuous drawing passes. To mitigate visual performance drop, the vector graphs should map onto an un-rendered buffer area (**Offscreen Canvas**) before executing a single block copy to the viewport canvas using the command `ctx.drawImage()`. This eliminates screen tearing or viewport flickering.
|
||||
* **Throttle Rendering Threads:** Wrap interface redraw handlers inside an explicit `requestAnimationFrame()` loop. This throttles the drawing passes to synchronize exactly with the screen hardware refresh rate metrics (typically $60\text{Hz}$ or $120\text{Hz}$), which avoids drawing redundant frames when CPU threads are under heavy loads handling audio decoding.
|
||||
|
||||
---
|
||||
|
||||
Giúp bạn tìm hiểu thêm về cấu trúc này, bạn có muốn khám phá sâu hơn khía cạnh nào không?
|
||||
|
||||
* **Optimizing Audio Codecs:** Cách tối ưu cấu trúc lưu trữ và nén dữ liệu nhị phân khi truyền tải mảng mảng số lớn giữa Docker Server và Web Client.
|
||||
* **PyQt6 High-Frequency Redraw:** Thiết lập vòng lặp vẽ đồ thị `QPainter` đa luồng trên ứng dụng Desktop Python mà không bị treo hàng đợi Event Loop.
|
||||
* **Cubic Spline Interpolation:** Công thức toán học nội suy mượt nâng cao thay thế cho đường thẳng tuyến tính (Linear Polyline) để bo cong sóng âm mịn hơn.
|
||||
Reference in New Issue
Block a user