277 lines
14 KiB
Markdown
277 lines
14 KiB
Markdown
# Technical Specification: AI Loop Scanning System & Fade-Free Zero-Crossing Slicing
|
|
|
|
This document specifies the software architecture, digital signal processing (DSP) algorithms, and API design required to integrate AI-driven automated loop scanning and perfect, fade-free audio slicing (Zero-Crossing Aligned Slicing) without boundary transition effects (Fade-In/Fade-Out).
|
|
|
|
---
|
|
|
|
## 1. Feature 1: AI Loop Scan & Automated Marker Labeling
|
|
|
|
This feature allows users to quickly scan an audio track (driven by backend AI/DSP) to detect segments with the highest rhythmic or musical periodicity (e.g., drum loops, chord progressions, vocal loops) and automatically map both boundaries using the timeline marker system.
|
|
|
|
```text
|
|
AI LOOP SCAN PROCESSING FLOW
|
|
┌───────────────────┐ 1. Send File ID ┌────────────────────────┐
|
|
│ Frontend Client ├──────────────────────►│ Backend FastAPI Server │
|
|
│ (Click "AI Scan") │◄──────────────────────┤ (Celery Task Worker) │
|
|
└───────────────────┘ 4. Return timestamps└───────────┬────────────┘
|
|
▲ [t_start, t_end] │
|
|
│ ▼
|
|
│ 2. Analyze Audio Features
|
|
│ (Self-Similarity Matrix)
|
|
│ │
|
|
│ ▼
|
|
└───────── (Pin Markers automatically) ◄ 3. Snap to Zero-Crossing
|
|
|
|
```
|
|
|
|
### 1.1. Workflow
|
|
|
|
1. The user selects an audio track within the Main Session and clicks the *AI Scan* button on the AI Panel.
|
|
2. The frontend dispatches a request containing the track's `file_id` to the backend gateway.
|
|
3. The backend initiates an asynchronous Celery Task, leveraging the `librosa` acoustic processing library to extract spectral feature matrices (Chromagram/Mel-spectrogram) and search for target loop boundaries exhibiting the highest recurrence correlation.
|
|
4. Once the optimal loop region $[t_{\text{start}}, t_{\text{end}}]$ is calculated, the backend executes a Zero-Crossing Alignment routine to precisely shift both boundaries to the nearest index where the signal amplitude reaches exactly zero.
|
|
5. The processed absolute timestamps $[t'_{\text{start}}, t'_{\text{end}}]$ are returned to the client. The frontend dynamically instantiates and renders timeline markers pinned directly onto that track lane.
|
|
|
|
---
|
|
|
|
## 2. Feature 2: AI Analysis & AI Cut (Fade-Free)
|
|
|
|
When cutting an audio segment at arbitrary time markers, if a slice intersects a high-amplitude point (non-zero), the continuous physical phase of the waveform is abruptly broken (Jump discontinuity). This generates a sharp, vertical step in the amplitude waveform graph, translating mechanically into an audible, harsh popping or ticking artifact ("click" or "pop") through speakers.
|
|
|
|
Standard or basic DAW systems mitigate this issue by adding an ultra-short linear fade envelope (Fade-In/Fade-Out) spanning roughly $5\text{ ms} \rightarrow 10\text{ ms}$. However, this masking method dampens the physical attack phase (transients) of the sound field, which is severely destructive to sharp, high-impact hits such as kick drums or snares.
|
|
|
|
The perfect architecture is a **Fade-Free AI Cut**. It dynamically calculates the closest hardware zero-crossing indices—where the acoustic wave amplitude passes through the central horizontal timeline axis ($0\text{V}$ absolute silence)—and executes the audio slice precisely at those coordinates.
|
|
|
|
```text
|
|
WAVEFORM TIMELINE & FADE-FREE AI CUT PROCESS
|
|
Amplitude
|
|
▲
|
|
+1.0 ┼ / \ / \
|
|
│ / \ / \
|
|
│ User-defined/ \ / \ User-defined
|
|
│ selection marker \ / \ selection marker
|
|
0.0 ┼───────○─────────────○─────○─────────○───────► Time Axis
|
|
│ / \ / \ / \ / \
|
|
│ / \ / \ / \ / \
|
|
-1.0 ┼────/──────\──────/─────○─────\───/─────\─
|
|
▲
|
|
│ [ AI CUTS EXACTLY HERE ]
|
|
│ Amplitude = 0 (Sound is silent)
|
|
│ Absolute zero Click/Pop anomalies!
|
|
|
|
```
|
|
|
|
### 2.1. Workflow
|
|
|
|
1. The user left-clicks and drags a time selection window $[T_{\text{start}}, T_{\text{end}}]$ across the target Waveform Lane.
|
|
2. The user clicks **AI Analysis**: The backend calculates and shifts both bounding coordinates slightly to align with physical zero-crossing sample indices ($T'_{\text{start}}$ and $T'_{\text{end}}$), instantly refreshing the highlighted overlay on the screen viewport.
|
|
3. The user clicks **AI Cut**: The engine slices the raw binary sample stream from index $T'_{\text{start}}$ to $T'_{\text{end}}$ straight inside RAM, generates a new track row directly underneath, and drops the cut clip onto it. The asset remains un-rendered and pure, with absolutely no volume fade multi-stage nodes applied.
|
|
|
|
---
|
|
|
|
## 3. Mathematical Zero-Crossing Optimization Algorithm (DSP Math)
|
|
|
|
Let $x[n]$ represent a single-channel discrete sample array containing mono audio amplitudes ($0$ mapping to the left track lane channel). At the target sample index address $n_{\text{target}}$ derived from the user's raw timeline click event, the engine establishes a symmetrical boundary scanning window of size $W$ (typically set to a $50\text{ ms}$ horizontal time width):
|
|
|
|
$$n_{\text{start}} = n_{\text{target}} - \frac{W \cdot f_s}{2}, \quad n_{\text{end}} = n_{\text{target}} + \frac{W \cdot f_s}{2}$$
|
|
|
|
Where $f_s$ tracks the absolute project Sample Rate hardware clock (e.g., $44100\text{ Hz}$).
|
|
|
|
### 3.1. Physical Phase Inversion Condition (Zero-Crossing Condition)
|
|
|
|
The optimization loop evaluates all internal sample index integers $i \in [n_{\text{start}}, n_{\text{end}}]$ that satisfy the algebraic sign-inversion condition rule:
|
|
|
|
$$x[i] \cdot x[i+1] \le 0$$
|
|
|
|
### 3.2. Optimization Criterion
|
|
|
|
Among all matching coordinate entries captured by the boundary condition filter, the algorithm targets the specific index $i_{\text{best}}$ that minimizes the spatial sample offset relative to the operator's input selection address ($n_{\text{target}}$):
|
|
|
|
$$i_{\text{best}} = \arg\min_{i} \left\vert{} i - n_{\text{target}} \right\vert{}$$
|
|
|
|
At coordinate point $i_{\text{best}}$, the immediate signal amplitude approaches zero ($x[i_{\text{best}}] \approx 0$). Slicing at this address ensures absolute physical phase continuity when the audio stream is partitioned or unlinked.
|
|
|
|
---
|
|
|
|
## 4. Python Backend Implementation Manual (Docker Celery DSP Worker)
|
|
|
|
This prototype Python module (`core/ai_dsp_engine.py`) runs on the backend Celery worker environment to execute automated loop indexing and fade-free zero-crossing slicing:
|
|
|
|
```python
|
|
import numpy as np
|
|
import librosa
|
|
|
|
class AIDSPEngine:
|
|
@staticmethod
|
|
def find_exact_zero_crossing(y: np.ndarray, sr: int, target_time: float, window_ms: float = 50.0) -> float:
|
|
"""
|
|
Locates the absolute nearest physical zero-crossing sample index to target_time (seconds).
|
|
Returns the optimized timeline index position in seconds where amplitude hits 0.
|
|
"""
|
|
target_sample = int(target_time * sr)
|
|
window_samples = int((window_ms / 1000.0) * sr)
|
|
|
|
# Define symmetrical horizontal boundary window
|
|
start_idx = max(0, target_sample - window_samples // 2)
|
|
end_idx = min(len(y) - 2, target_sample + window_samples // 2)
|
|
|
|
y_segment = y[start_idx:end_idx]
|
|
|
|
# DSP Condition logic tracking sign inversion: y[i] * y[i+1] <= 0
|
|
zero_crossings = np.where(y_segment[:-1] * y_segment[1:] <= 0)[0]
|
|
|
|
if len(zero_crossings) == 0:
|
|
# Fallback: if no sign change occurs, return the absolute minimum sample inside the viewport
|
|
abs_min_idx = np.argmin(np.abs(y_segment))
|
|
return float((abs_min_idx + start_idx) / sr)
|
|
|
|
# Translate local segment array address back to absolute buffer coordinates
|
|
absolute_crossings = zero_crossings + start_idx
|
|
|
|
# Isolate the crossing point closest to the raw target_sample baseline
|
|
distances = np.abs(absolute_crossings - target_sample)
|
|
best_sample_idx = absolute_crossings[np.argmin(distances)]
|
|
|
|
return float(best_sample_idx / sr)
|
|
|
|
@classmethod
|
|
def scan_best_loop_regions(cls, y: np.ndarray, sr: int, min_duration: float = 2.0, max_duration: float = 8.0) -> list:
|
|
"""
|
|
Evaluates spectral Self-Similarity Matrices (Recurrence plots) to extract
|
|
the most musically periodic and cohesive loop segments within the track.
|
|
"""
|
|
# 1. Compute harmonic structural properties via Chroma Constant-Q Transform
|
|
chroma = librosa.feature.chroma_cqt(y=y, sr=sr)
|
|
|
|
# 2. Compile the Self-Similarity Matrix (Cosine Recurrence Plot)
|
|
# This maps global structural recurrence profiles across runtime frame vectors
|
|
from sklearn.metrics.pairwise import cosine_similarity
|
|
ssm = cosine_similarity(chroma.T, chroma.T)
|
|
|
|
num_frames = ssm.shape[0]
|
|
hop_length = 512
|
|
frame_duration = hop_length / sr
|
|
|
|
best_score = -1.0
|
|
best_loop = (0.0, 4.0) # Fallback baseline setup to target initial 4 seconds
|
|
|
|
# Scan sub-diagonals to track high-density recurring correlation coefficients
|
|
# Diagonals parallel to the main identity path flag strict periodic cycles
|
|
min_frames = int(min_duration / frame_duration)
|
|
max_frames = int(max_duration / frame_duration)
|
|
|
|
for lag in range(min_frames, min_frames * 4): # Trace delay frames matching typical 1-2 measure blocks
|
|
if lag >= num_frames:
|
|
break
|
|
# Accumulate mean recurrence indices across the active sub-diagonal line
|
|
score = np.mean(np.diagonal(ssm, offset=lag))
|
|
if score > best_score:
|
|
best_score = score
|
|
# Map optimized chronological boundaries
|
|
start_frame = 0
|
|
end_frame = min(num_frames - 1, start_frame + lag)
|
|
|
|
t_start = start_frame * frame_duration
|
|
t_end = end_frame * frame_duration
|
|
|
|
best_loop = (t_start, t_end)
|
|
|
|
# 3. Lock boundaries to precise physical zero-crossings to prevent transient click noise
|
|
t_start_zero = cls.find_exact_zero_crossing(y, sr, best_loop[0])
|
|
t_end_zero = cls.find_exact_zero_crossing(y, sr, best_loop[1])
|
|
|
|
return [{"start_time": t_start_zero, "end_time": t_end_zero, "score": float(best_score)}]
|
|
|
|
@classmethod
|
|
def slice_and_copy_with_zero_crossing(
|
|
cls,
|
|
y: np.ndarray,
|
|
sr: int,
|
|
start_time: float,
|
|
end_time: float
|
|
) -> tuple:
|
|
"""
|
|
Slices an audio data array from start_time to end_time using zero-crossing alignment.
|
|
Strictly bypasses linear or exponential fade configurations.
|
|
"""
|
|
# Align bounding start and termination boundaries directly to zero-amplitude addresses
|
|
t_start_zero = cls.find_exact_zero_crossing(y, sr, start_time)
|
|
t_end_zero = cls.find_exact_zero_crossing(y, sr, end_time)
|
|
|
|
sample_start = int(t_start_zero * sr)
|
|
sample_end = int(t_end_zero * sr)
|
|
|
|
# Squeeze out raw buffer array slice without applying any destructive envelope modifiers
|
|
y_sliced = np.copy(y[sample_start:sample_end])
|
|
|
|
return y_sliced, t_start_zero, t_end_zero
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Serialized API Data Transfer Protocols
|
|
|
|
During data exchange cycles initiated over the AI Panel UI layer, the client application communicates with the FastAPI routing layer via the following structured JSON payloads:
|
|
|
|
### 5.1. API 1: AI Loop Scanning (POST `/api/v1/audio/ai-scan`)
|
|
|
|
* **Request Payload (Client $\rightarrow$ Server):**
|
|
|
|
```json
|
|
{
|
|
"track_id": "1",
|
|
"file_id": "creak_forest_raw.wav",
|
|
"min_loop_duration": 2.0,
|
|
"max_loop_duration": 6.0
|
|
}
|
|
|
|
```
|
|
|
|
* **Response Payload (Server $\rightarrow$ Client):**
|
|
|
|
```json
|
|
{
|
|
"success": true,
|
|
"track_id": "1",
|
|
"suggested_loops": [
|
|
{
|
|
"start_time": 1.4589,
|
|
"end_time": 5.4592,
|
|
"score": 0.892
|
|
}
|
|
]
|
|
}
|
|
|
|
```
|
|
|
|
*(Upon parsing this response, the frontend layout engine executes an automated marker rendering pass, pinning visual handles precisely at `start_time` and `end_time`).*
|
|
|
|
### 5.2. API 2: Fade-Free AI Slicing (POST `/api/v1/audio/ai-cut`)
|
|
|
|
* **Request Payload (Client $\rightarrow$ Server):**
|
|
|
|
```json
|
|
{
|
|
"source_track_id": "1",
|
|
"file_id": "creak_forest_raw.wav",
|
|
"selection_start": 3.120,
|
|
"selection_end": 7.450
|
|
}
|
|
|
|
```
|
|
|
|
* **Response Payload (Server $\rightarrow$ Client):**
|
|
|
|
```json
|
|
{
|
|
"success": true,
|
|
"output_file_id": "ai_cut_creak_forest_3.1s.wav",
|
|
"aligned_start": 3.1192,
|
|
"aligned_end": 7.4504,
|
|
"duration": 4.3312
|
|
}
|
|
|
|
```
|
|
|
|
*(The frontend automatically builds a new track row layout right below the baseline channel, mapping the received `output_file_id` block to mount perfectly at the real-world timeline timestamp indicated by `aligned_start`).* |