# Technical Specification: AI Loop Scanning System & Fade-Free Zero-Crossing Slicing This document specifies the software architecture, digital signal processing (DSP) algorithms, and API design required to integrate AI-driven automated loop scanning and perfect, fade-free audio slicing (Zero-Crossing Aligned Slicing) without boundary transition effects (Fade-In/Fade-Out). --- ## 1. Feature 1: AI Loop Scan & Automated Marker Labeling This feature allows users to quickly scan an audio track (driven by backend AI/DSP) to detect segments with the highest rhythmic or musical periodicity (e.g., drum loops, chord progressions, vocal loops) and automatically map both boundaries using the timeline marker system. ```text AI LOOP SCAN PROCESSING FLOW ┌───────────────────┐ 1. Send File ID ┌────────────────────────┐ │ Frontend Client ├──────────────────────►│ Backend FastAPI Server │ │ (Click "AI Scan") │◄──────────────────────┤ (Celery Task Worker) │ └───────────────────┘ 4. Return timestamps└───────────┬────────────┘ ▲ [t_start, t_end] │ │ ▼ │ 2. Analyze Audio Features │ (Self-Similarity Matrix) │ │ │ ▼ └───────── (Pin Markers automatically) ◄ 3. Snap to Zero-Crossing ``` ### 1.1. Workflow 1. The user selects an audio track within the Main Session and clicks the *AI Scan* button on the AI Panel. 2. The frontend dispatches a request containing the track's `file_id` to the backend gateway. 3. The backend initiates an asynchronous Celery Task, leveraging the `librosa` acoustic processing library to extract spectral feature matrices (Chromagram/Mel-spectrogram) and search for target loop boundaries exhibiting the highest recurrence correlation. 4. Once the optimal loop region $[t_{\text{start}}, t_{\text{end}}]$ is calculated, the backend executes a Zero-Crossing Alignment routine to precisely shift both boundaries to the nearest index where the signal amplitude reaches exactly zero. 5. The processed absolute timestamps $[t'_{\text{start}}, t'_{\text{end}}]$ are returned to the client. The frontend dynamically instantiates and renders timeline markers pinned directly onto that track lane. --- ## 2. Feature 2: AI Analysis & AI Cut (Fade-Free) When cutting an audio segment at arbitrary time markers, if a slice intersects a high-amplitude point (non-zero), the continuous physical phase of the waveform is abruptly broken (Jump discontinuity). This generates a sharp, vertical step in the amplitude waveform graph, translating mechanically into an audible, harsh popping or ticking artifact ("click" or "pop") through speakers. Standard or basic DAW systems mitigate this issue by adding an ultra-short linear fade envelope (Fade-In/Fade-Out) spanning roughly $5\text{ ms} \rightarrow 10\text{ ms}$. However, this masking method dampens the physical attack phase (transients) of the sound field, which is severely destructive to sharp, high-impact hits such as kick drums or snares. The perfect architecture is a **Fade-Free AI Cut**. It dynamically calculates the closest hardware zero-crossing indices—where the acoustic wave amplitude passes through the central horizontal timeline axis ($0\text{V}$ absolute silence)—and executes the audio slice precisely at those coordinates. ```text WAVEFORM TIMELINE & FADE-FREE AI CUT PROCESS Amplitude ▲ +1.0 ┼ / \ / \ │ / \ / \ │ User-defined/ \ / \ User-defined │ selection marker \ / \ selection marker 0.0 ┼───────○─────────────○─────○─────────○───────► Time Axis │ / \ / \ / \ / \ │ / \ / \ / \ / \ -1.0 ┼────/──────\──────/─────○─────\───/─────\─ ▲ │ [ AI CUTS EXACTLY HERE ] │ Amplitude = 0 (Sound is silent) │ Absolute zero Click/Pop anomalies! ``` ### 2.1. Workflow 1. The user left-clicks and drags a time selection window $[T_{\text{start}}, T_{\text{end}}]$ across the target Waveform Lane. 2. The user clicks **AI Analysis**: The backend calculates and shifts both bounding coordinates slightly to align with physical zero-crossing sample indices ($T'_{\text{start}}$ and $T'_{\text{end}}$), instantly refreshing the highlighted overlay on the screen viewport. 3. The user clicks **AI Cut**: The engine slices the raw binary sample stream from index $T'_{\text{start}}$ to $T'_{\text{end}}$ straight inside RAM, generates a new track row directly underneath, and drops the cut clip onto it. The asset remains un-rendered and pure, with absolutely no volume fade multi-stage nodes applied. --- ## 3. Mathematical Zero-Crossing Optimization Algorithm (DSP Math) Let $x[n]$ represent a single-channel discrete sample array containing mono audio amplitudes ($0$ mapping to the left track lane channel). At the target sample index address $n_{\text{target}}$ derived from the user's raw timeline click event, the engine establishes a symmetrical boundary scanning window of size $W$ (typically set to a $50\text{ ms}$ horizontal time width): $$n_{\text{start}} = n_{\text{target}} - \frac{W \cdot f_s}{2}, \quad n_{\text{end}} = n_{\text{target}} + \frac{W \cdot f_s}{2}$$ Where $f_s$ tracks the absolute project Sample Rate hardware clock (e.g., $44100\text{ Hz}$). ### 3.1. Physical Phase Inversion Condition (Zero-Crossing Condition) The optimization loop evaluates all internal sample index integers $i \in [n_{\text{start}}, n_{\text{end}}]$ that satisfy the algebraic sign-inversion condition rule: $$x[i] \cdot x[i+1] \le 0$$ ### 3.2. Optimization Criterion Among all matching coordinate entries captured by the boundary condition filter, the algorithm targets the specific index $i_{\text{best}}$ that minimizes the spatial sample offset relative to the operator's input selection address ($n_{\text{target}}$): $$i_{\text{best}} = \arg\min_{i} \left\vert{} i - n_{\text{target}} \right\vert{}$$ At coordinate point $i_{\text{best}}$, the immediate signal amplitude approaches zero ($x[i_{\text{best}}] \approx 0$). Slicing at this address ensures absolute physical phase continuity when the audio stream is partitioned or unlinked. --- ## 4. Python Backend Implementation Manual (Docker Celery DSP Worker) This prototype Python module (`core/ai_dsp_engine.py`) runs on the backend Celery worker environment to execute automated loop indexing and fade-free zero-crossing slicing: ```python import numpy as np import librosa class AIDSPEngine: @staticmethod def find_exact_zero_crossing(y: np.ndarray, sr: int, target_time: float, window_ms: float = 50.0) -> float: """ Locates the absolute nearest physical zero-crossing sample index to target_time (seconds). Returns the optimized timeline index position in seconds where amplitude hits 0. """ target_sample = int(target_time * sr) window_samples = int((window_ms / 1000.0) * sr) # Define symmetrical horizontal boundary window start_idx = max(0, target_sample - window_samples // 2) end_idx = min(len(y) - 2, target_sample + window_samples // 2) y_segment = y[start_idx:end_idx] # DSP Condition logic tracking sign inversion: y[i] * y[i+1] <= 0 zero_crossings = np.where(y_segment[:-1] * y_segment[1:] <= 0)[0] if len(zero_crossings) == 0: # Fallback: if no sign change occurs, return the absolute minimum sample inside the viewport abs_min_idx = np.argmin(np.abs(y_segment)) return float((abs_min_idx + start_idx) / sr) # Translate local segment array address back to absolute buffer coordinates absolute_crossings = zero_crossings + start_idx # Isolate the crossing point closest to the raw target_sample baseline distances = np.abs(absolute_crossings - target_sample) best_sample_idx = absolute_crossings[np.argmin(distances)] return float(best_sample_idx / sr) @classmethod def scan_best_loop_regions(cls, y: np.ndarray, sr: int, min_duration: float = 2.0, max_duration: float = 8.0) -> list: """ Evaluates spectral Self-Similarity Matrices (Recurrence plots) to extract the most musically periodic and cohesive loop segments within the track. """ # 1. Compute harmonic structural properties via Chroma Constant-Q Transform chroma = librosa.feature.chroma_cqt(y=y, sr=sr) # 2. Compile the Self-Similarity Matrix (Cosine Recurrence Plot) # This maps global structural recurrence profiles across runtime frame vectors from sklearn.metrics.pairwise import cosine_similarity ssm = cosine_similarity(chroma.T, chroma.T) num_frames = ssm.shape[0] hop_length = 512 frame_duration = hop_length / sr best_score = -1.0 best_loop = (0.0, 4.0) # Fallback baseline setup to target initial 4 seconds # Scan sub-diagonals to track high-density recurring correlation coefficients # Diagonals parallel to the main identity path flag strict periodic cycles min_frames = int(min_duration / frame_duration) max_frames = int(max_duration / frame_duration) for lag in range(min_frames, min_frames * 4): # Trace delay frames matching typical 1-2 measure blocks if lag >= num_frames: break # Accumulate mean recurrence indices across the active sub-diagonal line score = np.mean(np.diagonal(ssm, offset=lag)) if score > best_score: best_score = score # Map optimized chronological boundaries start_frame = 0 end_frame = min(num_frames - 1, start_frame + lag) t_start = start_frame * frame_duration t_end = end_frame * frame_duration best_loop = (t_start, t_end) # 3. Lock boundaries to precise physical zero-crossings to prevent transient click noise t_start_zero = cls.find_exact_zero_crossing(y, sr, best_loop[0]) t_end_zero = cls.find_exact_zero_crossing(y, sr, best_loop[1]) return [{"start_time": t_start_zero, "end_time": t_end_zero, "score": float(best_score)}] @classmethod def slice_and_copy_with_zero_crossing( cls, y: np.ndarray, sr: int, start_time: float, end_time: float ) -> tuple: """ Slices an audio data array from start_time to end_time using zero-crossing alignment. Strictly bypasses linear or exponential fade configurations. """ # Align bounding start and termination boundaries directly to zero-amplitude addresses t_start_zero = cls.find_exact_zero_crossing(y, sr, start_time) t_end_zero = cls.find_exact_zero_crossing(y, sr, end_time) sample_start = int(t_start_zero * sr) sample_end = int(t_end_zero * sr) # Squeeze out raw buffer array slice without applying any destructive envelope modifiers y_sliced = np.copy(y[sample_start:sample_end]) return y_sliced, t_start_zero, t_end_zero ``` --- ## 5. Serialized API Data Transfer Protocols During data exchange cycles initiated over the AI Panel UI layer, the client application communicates with the FastAPI routing layer via the following structured JSON payloads: ### 5.1. API 1: AI Loop Scanning (POST `/api/v1/audio/ai-scan`) * **Request Payload (Client $\rightarrow$ Server):** ```json { "track_id": "1", "file_id": "creak_forest_raw.wav", "min_loop_duration": 2.0, "max_loop_duration": 6.0 } ``` * **Response Payload (Server $\rightarrow$ Client):** ```json { "success": true, "track_id": "1", "suggested_loops": [ { "start_time": 1.4589, "end_time": 5.4592, "score": 0.892 } ] } ``` *(Upon parsing this response, the frontend layout engine executes an automated marker rendering pass, pinning visual handles precisely at `start_time` and `end_time`).* ### 5.2. API 2: Fade-Free AI Slicing (POST `/api/v1/audio/ai-cut`) * **Request Payload (Client $\rightarrow$ Server):** ```json { "source_track_id": "1", "file_id": "creak_forest_raw.wav", "selection_start": 3.120, "selection_end": 7.450 } ``` * **Response Payload (Server $\rightarrow$ Client):** ```json { "success": true, "output_file_id": "ai_cut_creak_forest_3.1s.wav", "aligned_start": 3.1192, "aligned_end": 7.4504, "duration": 4.3312 } ``` *(The frontend automatically builds a new track row layout right below the baseline channel, mapping the received `output_file_id` block to mount perfectly at the real-world timeline timestamp indicated by `aligned_start`).*