fix&feat: 12_SUBTAB.md chỉnh sửa audioclip ở subtab
This commit is contained in:
+287
@@ -0,0 +1,287 @@
|
||||
# Technical Specification: Sandbox Isolation & Sub-Tab DSP Editing Algorithms
|
||||
|
||||
This document defines the processing workflow design and digital signal processing (DSP) algorithms dedicated to localized clip editing within a temporary isolated document workspace (Sub-tab).
|
||||
|
||||
---
|
||||
|
||||
## 1. Sandbox Splicing Workflow
|
||||
|
||||
When a user highlights a time region on the Main Tab and triggers "Edit in Sub-tab" or presses the edit keyboard shortcut:
|
||||
|
||||
```text
|
||||
[ MAIN TAB - MULTITRACK ]
|
||||
Track 01: ───[█████ Selected Segment █████]───
|
||||
│
|
||||
▼ (Copy to Clipboard Buffer)
|
||||
[ KHỔI TẠO TAB TẠM THỜI (SUB-TAB) ]
|
||||
- Instantiates a single Track (Height bounds: 48px - 200px via ns-resize)
|
||||
- Timeline Ruler axis resets to t = 0.0s
|
||||
│
|
||||
▼ (Automated Insertion - Auto-Paste)
|
||||
Track 01 (Sub-tab): [█████ Isolated Segment █████] at t = 0s
|
||||
|
||||
```
|
||||
|
||||
* **Extract Buffer:** The underlying engine extracts the binary sample array (`Float32Array`) of the highlighted region from the active track, caching it securely into the application's clipboard buffer memory.
|
||||
* **Sandbox Environment Initialization:**
|
||||
* Appends a temporary document window onto the global Tab Bar (e.g., `Tab: sẤit tiá...n` or `Sub_Edit_1`).
|
||||
* Focuses the viewport down into the sandboxed tab. Here, a single standalone track lane is drawn, mapping the timeline ruler scale to start at $t = 0.0\text{ s}$ up to the absolute duration limit ($T_{\text{clip}}$) of the extracted audio asset.
|
||||
|
||||
|
||||
* **Track Height Resizing:**
|
||||
* Hovering the cursor over the lower layout bounding path of the track lane changes the style configuration to `ns-resize`.
|
||||
* Dragging downward expands the vertical height ceiling (up to an upper bound of $200\text{ px}$), maximizing the waveform amplitude drawing path for precision clip editing. Dragging upward compresses the physical row dimensions (down to a lower constraint of $48\text{ px}$) to protect screen space.
|
||||
|
||||
|
||||
* **Auto-Paste Routine:** The framework automates the insertion sequence, dropping the cached array block onto the root index milestone ($t = 0.0\text{ s}$) inside the isolated single-track layer.
|
||||
|
||||
---
|
||||
|
||||
## 2. Apply & Sync-Back Workflow
|
||||
|
||||
When an editor finishes processing steps inside the sandbox workspace and engages the *Apply* action:
|
||||
|
||||
```text
|
||||
[ SUB-TAB - AUDIO SANDBOX ]
|
||||
y_sub = [█████ Edited Waveform █████]
|
||||
│
|
||||
▼ (Click "Apply" - Trigger Overwrite)
|
||||
[ MAIN TAB - ORIGINAL TRACK ]
|
||||
Track 01: ───[█████ Overwritten Segment █████]─── at t = t_start
|
||||
│
|
||||
(Sub-tab remains open, Undo Stack kept)
|
||||
▼
|
||||
[User presses Undo (Ctrl+Z) inside Sub-tab to iterate]
|
||||
y_sub = [█████ Rollbacked Waveform █████]
|
||||
│
|
||||
(Click "Apply" again)
|
||||
▼
|
||||
Track 01: ───[█████ Corrected Segment █████]──── at t = t_start
|
||||
|
||||
```
|
||||
|
||||
### 2.1. Target Mapping & Metadata Linkage
|
||||
|
||||
Throughout its lifecycle, each sub-tab persistently locks standard metadata records linking back to the origin source elements:
|
||||
|
||||
* `parent_track_id`: Unique identifier referencing the primary source track on the Main Tab.
|
||||
* `parent_clip_id`: Unique identifier tracking the original source audio clip.
|
||||
* `t_start` (seconds): The exact historical start time position of the sliced block on the Main Tab timeline view.
|
||||
* `original_duration` (seconds): The baseline temporal duration of the region prior to modification.
|
||||
|
||||
### 2.2. In-place Overwrite & Splicing
|
||||
|
||||
* **Edited Buffer Extraction:** The system reads the active sample sequence from the sub-tab ($y_{\text{sub}}$) along with its updated duration boundary $T_{\text{sub}}$ (which fluctuates if time-stretching or rate scaling actions have occurred).
|
||||
* **Main Session Integration:**
|
||||
1. The core route mapper checks for the matching `parent_track_id` parameter on the Main Tab.
|
||||
2. Purges the legacy audio segment stretching from $t_{\text{start}}$ through $t_{\text{start}} + T_{\text{original}}$.
|
||||
3. Splices the updated signal array $y_{\text{sub}}$ precisely at the historical insertion index $t_{\text{start}}$.
|
||||
4. **Micro-crossfade:** Executes a ultra-fast crossfade envelope ($10\text{ ms}$) across both the initial and terminating splice boundaries. Blending adjacent files prevents phase cancellation or signal breakage that manifests as transient clicks/pops.
|
||||
|
||||
|
||||
* **Visual Update Tracking:** Commands the canvas engine to redraw the waveform visualization grid for the origin track lane inside the Main Tab view.
|
||||
|
||||
### 2.3. Persistence for Iterative Editing
|
||||
|
||||
* **Tab Lifetime:** Engaging the *Apply* trigger propagates data back to the primary environment but does **not** close down the active sub-tab view.
|
||||
* **Undo Stack Isolation:** The tracking loop containing the localized *Undo/Redo History Stack* inside the sub-tab sandbox remains entirely preserved.
|
||||
* **Iterative Loop Workflow:**
|
||||
1. If monitoring the Main Tab arrangement uncovers an audio anomaly, the user switches focus back to the Sub-tab workspace.
|
||||
2. Pressing `Ctrl + Z` (Undo) rollbacks the localized signal to its earlier state.
|
||||
3. The editor runs separate DSP actions.
|
||||
4. Hitting *Apply* overwrites the updated audio slice over the same target coordinates on the Main Tab.
|
||||
|
||||
|
||||
* **Explicit Destruction Hook:** The sandboxed tab structure is only unmapped when the user clicks the explicit close icon ($\times$) on the horizontal tab bar.
|
||||
|
||||
---
|
||||
|
||||
## 3. Sub-Tab DSP Algorithm Specification
|
||||
|
||||
Editing operations executed inside the sub-tab environment calculate discrete changes over the amplitude sample arrays ($x[n]$). These map to Web Audio API routines on the client layer and standard NumPy/SciPy audio arrays on the Dockerized backend.
|
||||
|
||||
### 3.1. Time-Stretching & Speed Math
|
||||
|
||||
Alters the duration bounds of the audio clip with optional pitch-shifting linking logic:
|
||||
|
||||
* **Pitch-preserving Time-stretching:** Utilizes the Phase Vocoder method to analyze the Short-Time Fourier Transform (STFT) of the signal, shifts spectral frames across the frequency domain, and reconstructs the audio via the Inverse Short-Time Fourier Transform (ISTFT) to align with a new playback velocity ratio $S$:
|
||||
|
||||
$$S = \frac{D}{D'} \times 100\%$$
|
||||
|
||||
|
||||
|
||||
*Where:* $D$ corresponds to the legacy unscaled duration (seconds), and $D'$ maps to the updated value post-resizing (executed by holding down the `Alt` key and dragging the right boundary handle).
|
||||
* **Resampling (Pitch-shifting Speed Scale):** Runs a standard linear interpolation algorithm to resample the core data array size:
|
||||
|
||||
$$x_{\text{new}}[m] = x\left[ \frac{m \cdot D}{D'} \right]$$
|
||||
|
||||
|
||||
|
||||
### 3.2. Peak Normalization
|
||||
|
||||
Amplifies the signal scale uniformly across the active block until the single maximum absolute sample peak reaches a specified ceiling parameter $A_{\text{target}}$ (typically locked at $1.0$ or $0\text{ dBFS}$):
|
||||
|
||||
1. Evaluate the absolute maximum peak within the array bounds:
|
||||
|
||||
$$A_{\text{max}} = \max_{n=0}^{N-1} \vert x[n] \vert$$
|
||||
|
||||
|
||||
2. Compute the static gain multiplier constant $G$:
|
||||
|
||||
$$G = \frac{A_{\text{target}}}{A_{\text{max}}}$$
|
||||
|
||||
|
||||
3. Multiply the entire audio array values by $G$:
|
||||
|
||||
$$x_{\text{norm}}[n] = x[n] \cdot G$$
|
||||
|
||||
|
||||
|
||||
### 3.3. Volume Gain Adjustment (dB Scaling)
|
||||
|
||||
1. Capture the decibel variance target ($\Delta \text{dB}$).
|
||||
2. Translate the logarithmic value into a standard linear scalar multiplier variable $G_{\text{linear}}$:
|
||||
|
||||
$$G_{\text{linear}} = 10^{\frac{\Delta \text{dB}}{20}}$$
|
||||
|
||||
|
||||
3. Apply the gain multiplier directly into the sample values:
|
||||
|
||||
$$x_{\text{gained}}[n] = x[n] \cdot G_{\text{linear}}$$
|
||||
|
||||
|
||||
|
||||
### 3.4. Pitch Shifting
|
||||
|
||||
Shifts the fundamental frequencies of the signal up or down by a specific number of semitones ($n$) while keeping the temporal duration value completely intact.
|
||||
|
||||
* **Frequency Transposition Ratio ($F_{\text{ratio}}$):**
|
||||
|
||||
$$F_{\text{ratio}} = 2^{\frac{n}{12}}$$
|
||||
|
||||
|
||||
* **DSP Processing Pipeline:** Employs either a Pitch Synchronous Overlap and Add (PSOLA) routine or a spectral Phase Vocoder to expand/compress the frequency components, then passes the array into a time-stretching step to return the physical track length to its source metric $T_{\text{clip}}$.
|
||||
|
||||
### 3.5. Linear Fade-In & Fade-Out Curves
|
||||
|
||||
Applies a linear fading envelope over the boundaries of the audio data block.
|
||||
|
||||
* **Linear Fade-In Envelope** (Across a duration bound of $L_{\text{fade}}$ samples):
|
||||
|
||||
$$x_{\text{fade}}[n] = x[n] \cdot \left( \frac{n}{L_{\text{fade}}} \right) \quad \text{for } 0 \le n < L_{\text{fade}}$$
|
||||
|
||||
|
||||
* **Linear Fade-Out Envelope** (Across the final trailing $L_{\text{fade}}$ samples):
|
||||
|
||||
$$x_{\text{fade}}[N - 1 - n] = x[N - 1 - n] \cdot \left( \frac{n}{L_{\text{fade}}} \right) \quad \text{for } 0 \le n < L_{\text{fade}}$$
|
||||
|
||||
|
||||
|
||||
### 3.6. Array Splitting & Merging
|
||||
|
||||
* **Split at Position ($n_{\text{cut}}$):** Unlinks a single sample block $x[n]$ of size $N$ into two separate independent sub-arrays:
|
||||
|
||||
$$x_1[n] = x[n] \quad (0 \le n < n_{\text{cut}})$$
|
||||
|
||||
|
||||
$$x_2[n] = x[n + n_{\text{cut}}] \quad (0 \le n < N - n_{\text{cut}})$$
|
||||
|
||||
|
||||
* **Merge Segments:** Concatenates separate sample sequences end-to-end. The stitching logic runs a $10\text{ ms}$ micro-crossfade overlay envelope at the junction to smooth out phase gaps that prompt click artifacts.
|
||||
|
||||
---
|
||||
|
||||
## 4. Python Backend Implementation Manual
|
||||
|
||||
This prototype Python class (`core/sub_tab_dsp.py`) handles the sandboxed operations and includes the crossfaded structural splicing algorithm designed to run inside the Docker engine:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import scipy.signal as signal
|
||||
import librosa
|
||||
|
||||
class SubTabDSPEngine:
|
||||
@staticmethod
|
||||
def change_speed(y: np.ndarray, sr: int, speed_ratio: float, preserve_pitch: bool = True) -> np.ndarray:
|
||||
"""
|
||||
Alters the playback velocity (Time-Stretching) of a NumPy signal array.
|
||||
"""
|
||||
if speed_ratio == 1.0:
|
||||
return y
|
||||
|
||||
if preserve_pitch:
|
||||
return librosa.effects.time_stretch(y, rate=speed_ratio)
|
||||
else:
|
||||
num_samples_new = int(len(y) / speed_ratio)
|
||||
return signal.resample(y, num_samples_new)
|
||||
|
||||
@staticmethod
|
||||
def normalize(y: np.ndarray, target_db: float = 0.0) -> np.ndarray:
|
||||
"""
|
||||
Performs Peak Normalization on an array to scale it to the target decibel value.
|
||||
"""
|
||||
target_amplitude = 10.0 ** (target_db / 20.0)
|
||||
max_amplitude = np.max(np.abs(y))
|
||||
|
||||
if max_amplitude == 0:
|
||||
return y
|
||||
|
||||
gain = target_amplitude / max_amplitude
|
||||
return y * gain
|
||||
|
||||
@staticmethod
|
||||
def merge_back_to_parent(
|
||||
parent_track_audio: np.ndarray,
|
||||
sr: int,
|
||||
edited_sub_audio: np.ndarray,
|
||||
start_seconds: float,
|
||||
original_duration_seconds: float
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Splices the modified audio segment from the Sub-tab back into the parent track array.
|
||||
Applies a 10ms micro-crossfade at the boundaries to eliminate pop/click noise.
|
||||
"""
|
||||
start_sample = int(start_seconds * sr)
|
||||
original_samples_len = int(original_duration_seconds * sr)
|
||||
edited_samples_len = len(edited_sub_audio)
|
||||
crossfade_samples = int(0.01 * sr) # 10ms crossfade window
|
||||
|
||||
# 1. Allocate the target output array dimension bounds
|
||||
new_total_len = len(parent_track_audio) - original_samples_len + edited_samples_len
|
||||
output_audio = np.zeros(new_total_len, dtype=np.float32)
|
||||
|
||||
# 2. Extract leading unedited block
|
||||
output_audio[:start_sample] = parent_track_audio[:start_sample]
|
||||
|
||||
# 3. Stitch the modified audio payload
|
||||
output_audio[start_sample:start_sample + edited_samples_len] = edited_sub_audio
|
||||
|
||||
# 4. Extract trailing unedited block
|
||||
post_start_original = start_sample + original_samples_len
|
||||
post_start_new = start_sample + edited_samples_len
|
||||
output_audio[post_start_new:] = parent_track_audio[post_start_original:]
|
||||
|
||||
# 5. Execute micro-crossfade across the initial splice junction
|
||||
if start_sample > crossfade_samples:
|
||||
fade_in_ramp = np.linspace(0.0, 1.0, crossfade_samples)
|
||||
fade_out_ramp = np.linspace(1.0, 0.0, crossfade_samples)
|
||||
|
||||
# Smooth 10ms interpolation overlay
|
||||
output_audio[start_sample : start_sample + crossfade_samples] = (
|
||||
edited_sub_audio[:crossfade_samples] * fade_in_ramp +
|
||||
parent_track_audio[start_sample : start_sample + crossfade_samples] * fade_out_ramp
|
||||
)
|
||||
|
||||
# 6. Execute micro-crossfade across the trailing splice junction
|
||||
if post_start_new + crossfade_samples < len(output_audio):
|
||||
fade_in_ramp = np.linspace(0.0, 1.0, crossfade_samples)
|
||||
fade_out_ramp = np.linspace(1.0, 0.0, crossfade_samples)
|
||||
|
||||
output_audio[post_start_new : post_start_new + crossfade_samples] = (
|
||||
parent_track_audio[post_start_original : post_start_original + crossfade_samples] * fade_in_ramp +
|
||||
edited_sub_audio[-crossfade_samples:] * fade_out_ramp
|
||||
)
|
||||
|
||||
return output_audio
|
||||
|
||||
```
|
||||
@@ -0,0 +1,87 @@
|
||||
import numpy as np
|
||||
import scipy.signal as signal
|
||||
import librosa
|
||||
|
||||
class SubTabDSPEngine:
|
||||
@staticmethod
|
||||
def change_speed(y: np.ndarray, sr: int, speed_ratio: float, preserve_pitch: bool = True) -> np.ndarray:
|
||||
"""
|
||||
Alters the playback velocity (Time-Stretching) of a NumPy signal array.
|
||||
"""
|
||||
if speed_ratio == 1.0:
|
||||
return y
|
||||
|
||||
if preserve_pitch:
|
||||
return librosa.effects.time_stretch(y, rate=speed_ratio)
|
||||
else:
|
||||
num_samples_new = int(len(y) / speed_ratio)
|
||||
return signal.resample(y, num_samples_new)
|
||||
|
||||
@staticmethod
|
||||
def normalize(y: np.ndarray, target_db: float = 0.0) -> np.ndarray:
|
||||
"""
|
||||
Performs Peak Normalization on an array to scale it to the target decibel value.
|
||||
"""
|
||||
target_amplitude = 10.0 ** (target_db / 20.0)
|
||||
max_amplitude = np.max(np.abs(y))
|
||||
|
||||
if max_amplitude == 0:
|
||||
return y
|
||||
|
||||
gain = target_amplitude / max_amplitude
|
||||
return y * gain
|
||||
|
||||
@staticmethod
|
||||
def merge_back_to_parent(
|
||||
parent_track_audio: np.ndarray,
|
||||
sr: int,
|
||||
edited_sub_audio: np.ndarray,
|
||||
start_seconds: float,
|
||||
original_duration_seconds: float
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Splices the modified audio segment from the Sub-tab back into the parent track array.
|
||||
Applies a 10ms micro-crossfade at the boundaries to eliminate pop/click noise.
|
||||
"""
|
||||
start_sample = int(start_seconds * sr)
|
||||
original_samples_len = int(original_duration_seconds * sr)
|
||||
edited_samples_len = len(edited_sub_audio)
|
||||
crossfade_samples = int(0.01 * sr) # 10ms crossfade window
|
||||
|
||||
# 1. Allocate the target output array dimension bounds
|
||||
new_total_len = len(parent_track_audio) - original_samples_len + edited_samples_len
|
||||
output_audio = np.zeros(new_total_len, dtype=np.float32)
|
||||
|
||||
# 2. Extract leading unedited block
|
||||
output_audio[:start_sample] = parent_track_audio[:start_sample]
|
||||
|
||||
# 3. Stitch the modified audio payload
|
||||
output_audio[start_sample:start_sample + edited_samples_len] = edited_sub_audio
|
||||
|
||||
# 4. Extract trailing unedited block
|
||||
post_start_original = start_sample + original_samples_len
|
||||
post_start_new = start_sample + edited_samples_len
|
||||
output_audio[post_start_new:] = parent_track_audio[post_start_original:]
|
||||
|
||||
# 5. Execute micro-crossfade across the initial splice junction
|
||||
if start_sample > crossfade_samples:
|
||||
fade_in_ramp = np.linspace(0.0, 1.0, crossfade_samples)
|
||||
fade_out_ramp = np.linspace(1.0, 0.0, crossfade_samples)
|
||||
|
||||
# Smooth 10ms interpolation overlay
|
||||
output_audio[start_sample : start_sample + crossfade_samples] = (
|
||||
edited_sub_audio[:crossfade_samples] * fade_in_ramp +
|
||||
parent_track_audio[start_sample : start_sample + crossfade_samples] * fade_out_ramp
|
||||
)
|
||||
|
||||
# 6. Execute micro-crossfade across the trailing splice junction
|
||||
if post_start_new + crossfade_samples < len(output_audio):
|
||||
fade_in_ramp = np.linspace(0.0, 1.0, crossfade_samples)
|
||||
fade_out_ramp = np.linspace(1.0, 0.0, crossfade_samples)
|
||||
|
||||
output_audio[post_start_new : post_start_new + crossfade_samples] = (
|
||||
parent_track_audio[post_start_original : post_start_original + crossfade_samples] * fade_in_ramp +
|
||||
edited_sub_audio[-crossfade_samples:] * fade_out_ramp
|
||||
)
|
||||
|
||||
return output_audio
|
||||
@@ -1194,8 +1194,9 @@
|
||||
const afterSnap = captureTrackSnapshot(subTab.trackId);
|
||||
pushAction('EDIT_TAB', subTab.trackId, beforeSnap, afterSnap);
|
||||
|
||||
closeSubTab(tabId);
|
||||
showToast('Đã áp dụng chỉnh sửa vào track chính.', 'success');
|
||||
// Tab Lifetime: Engaging the Apply trigger propagates data back to the primary environment but does not close down the active sub-tab view.
|
||||
// closeSubTab(tabId);
|
||||
showToast('Đã áp dụng chỉnh sửa vào track chính.', 'success');
|
||||
};
|
||||
|
||||
const closeSubTab = (tabId) => {
|
||||
|
||||
+3129
-759
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,44 @@
|
||||
import os
|
||||
import sys
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from app.core.sub_tab_dsp import SubTabDSPEngine
|
||||
|
||||
def test_speed_ratio():
|
||||
sr = 44100
|
||||
y = np.sin(2 * np.pi * 440 * np.linspace(0, 1, sr, endpoint=False)).astype(np.float32)
|
||||
# Pitch-preserving speed change
|
||||
y_stretched = SubTabDSPEngine.change_speed(y, sr, 2.0, preserve_pitch=True)
|
||||
assert abs(len(y_stretched) - sr // 2) < 2000 # Librosa might have frame alignment differences
|
||||
|
||||
# Simple resampling speed change
|
||||
y_resampled = SubTabDSPEngine.change_speed(y, sr, 2.0, preserve_pitch=False)
|
||||
assert len(y_resampled) == sr // 2
|
||||
|
||||
def test_normalize():
|
||||
y = np.array([0.1, -0.5, 0.2, 0.4], dtype=np.float32)
|
||||
y_norm = SubTabDSPEngine.normalize(y, target_db=0.0)
|
||||
assert np.max(np.abs(y_norm)) == 1.0
|
||||
|
||||
def test_merge_back_to_parent():
|
||||
sr = 1000
|
||||
parent = np.ones(5000, dtype=np.float32)
|
||||
edited = np.zeros(2000, dtype=np.float32)
|
||||
|
||||
# Merge at t=1.0s (index 1000), original duration 1.5s (1500 samples)
|
||||
res = SubTabDSPEngine.merge_back_to_parent(
|
||||
parent_track_audio=parent,
|
||||
sr=sr,
|
||||
edited_sub_audio=edited,
|
||||
start_seconds=1.0,
|
||||
original_duration_seconds=1.5
|
||||
)
|
||||
|
||||
# Expected length: 5000 - 1500 + 2000 = 5500
|
||||
assert len(res) == 5500
|
||||
# Before 1.0s (1000 samples) should be mostly parent values (1.0)
|
||||
assert np.allclose(res[:900], 1.0)
|
||||
# Inside the edited range should be zero (except crossfades)
|
||||
assert np.allclose(res[1100:2900], 0.0)
|
||||
Reference in New Issue
Block a user