SonicForgeStudio/md/36_USE_VST3.md

OPERATION GUIDE & AUDIO PLAYBACK WORKFLOW FOR MIDI TRACKS (SOUNDFONT / VST3)

This document describes in detail the user interface interaction workflow when using the Synth button and explains the underlying technical architecture required for MIDI Notes on a Track to output audio via a selected SoundFont or VST3 Plugin.

---

1. User Interface Workflow Description

Activating the Instrument Selection Menu

• On the Track Control Panel (the left-side pane of Track 01), the user clicks the 🎵 Synth: BAN-DI button (or the orange Synth button below it).
• A dropdown selection menu appears directly underneath the button.

Instrument Menu Layout

• None (Default Synth): Uses the application's default synthesizer (a simple Oscillator Synth).
• SOUNDFONTS: Displays a list of SoundFont (.sf2) soundbanks loaded into the system (e.g., SoundFont_DSK_Asia, SoundFont_SGM_v2, weedsgm3).
• VST INSTRUMENTS: Displays a list of native 64-bit Linux VST3 Plugins (e.g., Vital, DecentSampler, libSurge XT).

Selection & State Update Operations

• The user clicks to select an instrument (e.g., selecting SoundFont_DSK_Asia or Vital).
• The menu closes, and the button label updates to reflect the chosen instrument (e.g., 🎵 DSK_Asia or 🎵 Vital).
• The instrument configuration payload is directly assigned to the Track State object (session.tracks[0].synth_engine).

---

2. Technical Execution Flow for MIDI Note Audio Output (SoundFont / VST3)

To ensure that the purple MIDI note bars on the Timeline or Piano Roll play back audio accurately using the chosen instrument, the system processes tasks across two primary workflows:

                                  +------------------------------------+
                                  | User selects SoundFont / VST3      |
                                  +-----------------+------------------+
                                                    |
                         +--------------------------+--------------------------+
                         |                                                     |
                         v                                                     v
          [ 1. Real-time Client Preview ]                        [ 2. Server-side Offline Export ]
          (Audio Playback in Browser)                             (High-Quality WAV Rendering)
                         |                                                     |
    +--------------------+--------------------+                   +------------+------------+
    |                                         |                   |                         |
    v                                         v                   v                         v
(If SoundFont)                          (If VST3)           (If SoundFont)            (If VST3)
FluidSynth Wasm /                       Load Wasm Module /  PyFluidSynth C-API        Python Pedalboard
SoundfontPlayer.js                      AudioWorklet Synth  Dispatches Bank/Program   Loads .vst3 binary
Dispatches programChange                Preview Synth       Renders Audio Buffer      Renders PCM Buffer
          |                                   |                   |                         |
          +-----------------+-----------------+                   +------------+------------+
                            |                                                  |
                            v                                                  v
                 AudioContext Destination                            Audio Export Output File
                 (User Speakers)                                     (Downloaded WAV File)

A. Real-time Client Playback (Browser Audio Preview)

When the user clicks the Play button or clicks a key on the Piano Roll:

1. Audio Routing Update:

• The client reads the instrument parameters from track.synth_engine.
• If SoundFont (.sf2) is selected: The client dispatches controllerChange(channel, 0, bank) and programChange(channel, program) configuration calls to the soundfontPlayer.js module (running FluidSynth WebAssembly).
• If VST3 Plugin (Vital, DecentSampler, etc.) is selected: Because browsers cannot natively run Linux .vst3/.so binary executables directly, the client uses an equivalent WebAssembly Synth or Preview Synth to output real-time audio with 0\text{ ms} latency.

2. Note Scheduling:

• The Transport driver (PrecisionAudioScheduler) scans for MIDI notes located within the moving Playhead range.

• Each MIDI note includes: pitch (0–127), start_beat (start position), duration_beats (length), and velocity (keypress intensity 0.0–1.0).

• The scheduler converts beat timing to absolute time in seconds (exactAudioTime) and dispatches audio events:

• noteOn(pitch, velocity, exactAudioTime)

• noteOff(pitch, exactAudioTime + durationSec)

• Audio signals generated by the WebAssembly Engine travel through Track Gain Node \rightarrow Track Pan Node \rightarrow Master Bus \rightarrow AudioContext.destination (User Speakers).

B. Server-side Offline Render (High-Quality WAV Export)

When the user exports a track (Bounce Track / Export WAV), the Python Backend on the server receives the project's JSON payload:

1. Reading Track Instrument Metadata:

{
  "track_id": "track_01",
  "synth_engine": {
    "type": "VST3",
    "plugin_id": "Vital",
    "soundfont_bank": 0,
    "soundfont_program": 0
  }
}

2. Rendering SoundFont (.sf2) Instruments:

• render_engine.py initializes a FluidSynth instance.
• Calls fl.program_select(channel, sf_id, bank, program).
• Feeds the list of MIDI notes directly to FluidSynth to render an Audio Buffer.

3. Rendering VST3 (.vst3) Instruments:

• vst_engine.py invokes pedalboard.VST3Plugin("/opt/daw_engine/vst3/Vital.vst3").

• If DecentSampler is selected, it loads the corresponding Pianobook sample preset file (.dspreset).

• Converts all MIDI Notes into an array of pedalboard.Message events:

• Inserts control_change (Bank Select) and program_change events at timestamp 0.0\text{ s}.

• Inserts note_on and note_off events matching the pitch and duration parameters of each note.

• Feeds the MIDI message stream into the VST3 instance to generate a high-fidelity Float32 PCM audio stream.

• Mixes down the Track PCM Audio Buffers into the Master Mix and creates the final .wav output file.

---

3. Instrument Selection Checklist

To ensure that selecting an instrument via the Synth button produces audio output successfully:

• Track is Unmuted: Verify that the Mute button [M] is not active (orange/red) and that the Solo button [S] on other tracks is not muting the current track.
• MIDI Notes in Valid Key Range: Some instruments (such as Bass or Horns) operate within constrained pitch boundaries (e.g., C1 to C5). Ensure the notes drawn on the Piano Roll fall within the playable range of the selected SoundFont or VST3 instrument.
• VST3 / SoundFont Files Ready on Server: Confirm that the .vst3 binary files are placed inside /opt/daw_engine/vst3/ and .sf2 files are present in /opt/daw_engine/soundfonts/.
• Appropriate Volume / Gain Settings: Verify that the Track 01 Volume slider is configured to 0\text{ dB} to avoid signal clipping or silent playback.