From fde9a91c9bcbad2e9d425d0baa38390b96c617ed Mon Sep 17 00:00:00 2001 From: 3dtours Date: Sat, 25 Jul 2026 20:46:53 +0700 Subject: [PATCH] =?UTF-8?q?fix:=20=C4=91=C3=A3=20s=E1=BB=ADa=20l=E1=BB=97i?= =?UTF-8?q?=20MIDI=20note=20b=E1=BB=8B=20flicker?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- app/storage/sonicforge.db | Bin 118784 -> 118784 bytes md/34_VST_LINUX.md | 344 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 344 insertions(+) diff --git a/app/storage/sonicforge.db b/app/storage/sonicforge.db index 98f37b78882fe318bde21028203272663bc2593d..a3e82e72b319194186335ecf910ce59544983890 100644 GIT binary patch literal 118784 zcmeI5ON=AOd4OkjWvzB)d7PKWb}WRUAXtmcw!2>a7$b;B+L1HyYFFL|62n#wi=`d?K=-TC^BZW1*PhvR;jG{nZIH#RpnzS3xHY+U%ajg5_$;qhsB zT!F`Pvj_Z0ug&b2^MAr;Sf`I(+WMc3D_36M_|#{&{(b8QTTedui>Y5g-B|O#<)!#pO%eufDqZ?Y~UIRxgTA;w0(^hrRGYjDNrK z#_j#RyZep1d#~TvZ{SxNuU%=(F8!|HI67(E-T$k*jkj+fytQ}x8;!5+e`9BUODBrk zC*PgFK>SL zb18+i+FHSajEUVDry;^VaSCgRkDyciz~$zCRtFFTL>m_8TvsH#`~EfMvg3 ze)n6SxU~JsE1TchPlqLK^wDIPgfaT{`NH6!7tr{m&t^kVshDvx9ET60px5nplc3)! zu~Igo)eoNr$vA8umzgqWoygSl&u<^Ravn0&*?^%xE`9s2H!p2(Z*P8gdk!u|7GEgA ziV>!Tg!JdC^54*joKA(HfQ5YY=Bp!uv{A4(u&7HFLkB7b4%*ZUk z7vm%xbi#4xi#g=}Bu;|%Q8<9nn}6a06Ve_>l_n(rM3_`zJLx`-O4Eu>75VCO&u{&?^|`ATmfxC=a(oed$qU+_yabQUt-}rYM;8$w0z`la5CI}U z1c(3;AOb{y2oM1x@V*GVyS;hk-u9Q@eQEphrOS3lvasc30X|f4%~>yKWoS=O08?oJ>01q2FS@a2>a!TCR{x$+l{T?bhL;=kY^F zI37FXUaPI7+p?J>?9O4QW3y2Bhli@o+4Qsbe*V4xxcYa0`@ec&e{<_+8}N@VB0vO) z01+SpM1Tko0U|&IhyW2F0z}|rM&OGV@6{?aO!NO=ZEXGOW7a_umk1C6B0vO)01+Sp zM1Tko0U|&IhyW2-M&N~u+n3Yr1=IZhmm6EZT=o`vod^&CB0vO)01+SpM1Tko0U|&I zhyW4zNE7({#V=2@{qkXg>G}VkZfyPZBi#cMjtCF|B0vO)01+SpM1Tko0U|&IhyW2N z68MwN=k8tDtn)1cuWfvV&i@zT(O*P>2oM1xKm>>Y5g-CYfCvx)B0vN_P6X>Y5g-B|Ac1QagUuxBkAgaz`Y-HU3@)d4w`E5P zWwUEhsN^;22>2fja;JT*>Y5g-CY zfCvx)BJiOT(Chz{|9|MiKmrf}B0vO)01+SpM1Tko0U|&IhyW4zKnbMZ|G)8H8~WdG z@b&J0o@`zT?)}+!e*EI)3zur|j;OPP;`)34^3RQb{r;ely!Yep_Z#h_pZq9n^nUWA z$ve%_c=#}CCqcK<+--Wca68_iy2d24^==IRn#UaXnreA|r`=+05&6xXW$I0^=nel!k~;kdagOD}e!!LZ+j zJ7+J)VSm(%0=OSOT7oS08MGR92SFUgad$Y-EvFLeC!yIv*w;e5wRdo{acBR|or7C9 z;n{8+jEBRdxtok9(N41$4IU&%K`R``AfvO>3@6DjXb=0NNfHIqSF?MQao9eF+rBw# zhvkjj)h{({{Y-K)N}D~negJM9&$T^Yayys&Es(Qu z0QVO~4-W3`zXealNjOe4o>cD61n>jujF;uRo`bKQX1muN1=Dv84-exgeKW`cy^e>I zaT{1-^#<(N?sNh@+*yOZ*`RP7 z?)tXlyN(O$3Lm_t=0&%C{GfC6FyMLf4l|pV9#_g^!dF6io^aRJynEP5?8i^zK%nM@ z*}U>~qulm|%>_uZw&ouv4<9++>C-@>=HcUuqwO-q7~;RS<{#SQ!}#RsX`oQ^U^a@) zdm;x94EWlbmm(IE{-a*tpypw0i_QCTrgM%h9R;efCe1q?JsJ1I(NW-{){U9QEgM|K zOzCvr*Qa@-@re?#ngkwd-k5jXV%}8aGTmQW?<4heG8}}%K)buyw0E^xo0`bf&TxzS zKzy!x>+8IH(mCn&)G%Pk%(=$2=ayszqXTYrO=Itj#?ez33^qen&jrmcc3*L1@?c)9 zsr7z*qE7q%alnw#a}BG*El1{x2Zm?G=EqSx^Dy8aU^ok&#ptqaaYwNEWjoc#T|0M^ z@H-( zdTh>YP|^7X_h5;H!?}%AxsFKKj~O$ISv~|$`Bg0v(=p)u5{5|g2_;#zB4G zr%1>J>L!bjgzsc+Nh$mqB6)xTOC(&|c0E_xOt}&g(BiN`zW;?Z|9|eC4fsbF5g-CY z;MYaq?=M`q3?;dL{G+TQSLbw7oqOt5e6$XXQ8u0Dg zX`|DMZ&|prv-|UUm=5@7qRy(G`k7hPbMM~ugIn7PlEsT0s)FQ0;JD2i4e_ zgqRRS?S-wV2X*4NyWwEixIOIkb{iE%(YFuwZr%!R-@0)__g0sCr?AqJZ#aGch0ba5 zIjgm%Z3x>=)AbNICt8|}a9XiWv&{59n@KIpd9}{;9!GKa z9k&}{aXIRoS2@$BAba#Ubg_e)GIAJ*JlE5dBb}4hDbor+>y=~VCJ=>JWJk~1CC_s@ zgF*KMwPPe4YE5187_qr6vU{tW8-{uRQvm6#koNf$oOnNFbcW;-J7CC*ir%q4CM zn|a|$S1+(+3T`N#UpY$C8CKoY{amuxF)Xsd4e#hD=0<7im$F5*@+)&UaA)qZ=8J2yrLEtt)wOg|&Mf}SLsmEv{ZuVq z4pX_lFHg#gY4HBcpWOfWIdJIwR9Bm?*oUs;M9s6avllP#{rr1B`{Q4{@ZZ0#V4vC{ z0z`la5CI}U1c(3;AOb{y2oM1xKm>>Y5g-CYfCvx)B0vO)01+SpM1Tko0U|&IK5zn; zp@q-D;}77m4UgZ0$M3@9ci^!Fk59ql_u-*m)A#A$v&R?Uce;oG5g-CYfCvx)B0vO) z01+SpM1Tko0U|&IhyW2F0z`la5CI}U1c(3;AOb{y2oM1xKm>>Y5g-CYfCvx)B0vO) z01+SpM1Tko0U|&IhyW2F0z`la5CI}U1c(3;AOb{y2oM1xKm>>Y5g-CYfCvx)B0vO) z01+SpM1Tko0U|&IhyW2F0z`la5CI}U1c(3;AOb{y2oM1xKm>>Y5g-CYfCvx)B0vO) z01+SpM1Tko0U|&IhyW2F0z`la5CI}U1c(3;AOb{y2z=}bFu24{^KsN0w!6tmbJw;! z?zk>x5Tj>BN6?auFNFEZ28zOPbf!^g;l}L{A{cOw_lkgD*iB`}gA4#NEkqFcmT`(|MBA*H2 zskw&NVdU6IB9oB-#YF(1wX$l%6tU6u{uwz8Nn|qma3tbLB9oD{nCqjtw$SGHPwqVkCY=`?k zfL`k19BjPJkpr2OTIS&d^HeT5H$0pnQ5}EcVB>9yGvD}!4O zK|+4%vB48<&Sx6AL9dotBNrQO)3g*<^i{N&d@EwGpZ3~`#~gEB6nMfNx(dx$G5f_aic7$wlX23 zT@uN+AU;J5Dt#OCg`5u@GfmSfHgb{G_ASqcR4}`=+p18eg?1n}l(oIBhs?G~N!XTj z=VJoypxQJQrkn%!;3?-xt{5bM)2OnBa(mc(!-ru_`TzyGpVMycx)$g|domtJgCt10 z{iwO?Timx%wgZy2hrQvrxjUHjdS~$VIBKVfcAN~y@P>EJ&IaGw`}se7>2E&unU|q> z@mXfs94y%UGHesX5I%+lCVr_Q;AqyF#Q@g=Z34TSU6tM!8Z8djG{BzD`!s>8BL$jh zrWBsV1&*|H%6UPe7#xW;Onr{$nUVi{mdCV7E6=T>9A=O2rqXc;hzj$-k{Q~Qx{EyT zhX$52LXnZl$hV|&kjIB9VU8MussK(v%xt!p5-FIq%OzS67%^S5WBIVbpqdidR>5uP zl(jg;a9mxpo+WTxWDFaK1R=;0k-)OBJR?#@_*jgw?K3vvu}r+w2n3*6K(3Z2E}X?7*)Fb zqFE}LN0eMoN!1YwTkJ9@*AvL=gVibPI2Qt?QA|g^nA8e^h`=@*lmVM20QLT5nwM*p zZ$s=>$2iFa1{`GaZ0umGwR|zBV%65J<5VOz;U*_;F$dT8Divx;$!1<@pC**GI4v;Z zj9bz{>20N3DLOiH1z?rXNMb8)gpnYUG)QDH@*R(K+@4F~&Lm~jHM+e5y$lF?#`R+(^% zEV#53gZOKHRg1KEndI4v+Y%XY$A~4FvLL!ePNmWY2<3@9i~~bv3|ziQQn#EWk^L?? z6~}_?8P)hJb!qX0%gRVuv|C5Jk@?nPRzaB@UA3(oE>*~?bg`nK)5XXHJkiqVO<|L5 z&N9gB98}h-PO*!H2s7)SE%{A-2-HaHh=eS535^F98&|c;6CVN>sDaM4-gS-WZG{CJ zh3vJNv2h4sIfhi02-c8&v|J+qGafal2WSU7FCf=JgsrtP$+SSWjTE}Fd{i_=W;8h~ zi)Q*_2>q0x^|DOwc0YjSL!hFD!gj?xcfZCl4i`+VoDSLoYx(@Tjm=S|v z5@lNkAhZNS(2q~lY2QB%7#ek*Ef!mMq!Lgc6)uOj*_D&EM)Tzj3wS;nUA8Umph(l! zs#wu!7lq{TyJ0?8sePkiPzDHH^1_8dS#u;-iBZsaDAQC~!p4rptqhw=0|J{!=H&pL zdDc;MH1$f>sg@Mj*}$SH3^KL`dSeTgsg6i!>XqD^Xx3l*jrUuCgm!J4}T z=>$r3^?G`Ft`~~}W*#6VUgOehcZTsRTOSlC>80tS$>W7(H?1TU5-m>PRuBmVO1jCS zNWyorwMkeR^=pV^#!UI%t!WZo0}?|vS~D+-q@2PUK&59eQcBYVtf1TlR|!#8Q6?8J zyNOD-!3)c7#%wdIOLZV2YhCu;z6+bw@~d*XErw*2|MhHKs@)9PZXFh4NRUw$(|~GS zDsJB}25z?HE}M0A$`cIPZWnBQQ6$i5K3@Q_^}>32#ud{gGh52nlBQb}>rg_r+cjpK zuwZ2o#g@AGSs_wk`v&ZfEUHtsHF3tN7A9=G!H{803M!Szc57uTi&S#`kcB?$9w~-X zTf;hn5H8wi!=O_YLP*$dHoss36uGRBnJ{dy4O3K4B)n$JA(}Nr()-6UzYqHp>!=Sl z*dq)=&AR!1WY2i34M3#J8@$uQUeaj7lh zD5y4FtpgbxUL~%aWQCXw2F@5coKb4KR7WIavQ57RB0+qdFJ)C*$HmS++mD#(Qyr0z z#V(7a;1jd;+H}WEdCCkv_j2|Ba2-gIEgnx%W8vN*h;21Qx;SyOSdTHzBf&(+|0w=&}>t#02TCw?%Q>b+R-3k7&U~ATqIh#t zA*ZdEGfK4ea(dYD{@L06xdGSKdJyUzm(9E5uzlRH^)t!INI$c8@A|L3v!HvY*`FkE(_z?)BM7HpbLRYy?jTOa@DA9`sgG6a9h*B%^yFwfd@v6C6|YQ1 znY|&2`f!5soAt&acRSrI$g_P|veT0H!dBD+(ckWdgJI+Lu-Dsd zymfH>pm6}7miO%KgT0%#g4?%l+|bYIj;E3~by5Wus>jg@X3MU!mQgFF>(QO|;=o~!^a6n*G7MUesCz|q@@Mo)GYir&;>?HQ% zr*VKYKx6RbQp94?f7A;WxNm0pGFyDLVapPz#+q6`9X%QM!_iTI z>uttN;})!JW=f~~UXewv*!j`;M2T2U0_2s=+5>DH^Wc;<(*3pdK2lF7!$CLkCR%#Qs!6o;Qb(1ry*9jmjinblb_u5ns$!xg=ZaQ1aZSfA&-v0Wfhr!3 zp~7_Ra=IQWa9zYKOwh5*86;djF(J{h%lvEd1S++ZYK*`BW0#Qs|KV%|aS_Ha1#3Cx zNJlTjB>kSlB}Y{0=;ajC;*(&IGe7gUQqa-MrRfnJy@cBTa!SaE)6vUwzY@ZgW}YH{j@{-kxsk*Ei1Y*E++C18?J`1%2)Etg@~a*lU19 zU%Q;nH)O5N?bf{37eN=k210+YBHjr>U%RZo;~a9W{Fo6O9xpOlVg5U~o^7OUyxvzF eqn-V5O4j_Msuj-4(AO^Ob^N7K)6MW` z_sQPT?sb^LJNcjc8TwpyKg;9zT|h*?bBGdR4sj4Mk2r)lj3^?GAj*hJFJJ5D!b^X= z6h74}WTdj(SQ{SiJs(msAxtW$Y}8MlS!;DxR^M&DwcP5g*B2R420KYYj22RiS;if9?|lp9 zqz(_^dS!|l<~*E%YT(j+Xr(=AgG){e&+b4eAlfMAlCWXx<~*^6N}2>8>sB+#jNse~ zsio3m-O7QE(}GAV{EN4tcEE5OetD~uQ$nEv)KHWcQ~3Oq6nW)OddjyooV#ffV(rla$xEa!28KQcw6vPWQ;j}7#~ga#@4 zTRTvlvPua5!VVPu8#iEKc$Dvr!Vwvb;v=XL&FPS-wPO~NPE*aDnG5A>pF$bQ8DtmdQ%bYZjQ=+e6FH(Z zBM4=CuTn5RU%zoXZ3yePwxcIa5G0XOh57AGsQZOY$oo&u!(o5%J17jPyKwavyd{aG zO2=ly5&F&F;ZPRmIt=A^e*?P^TaeRwqkej&v-Ea-QPD^)dVl D^@6Vy diff --git a/md/34_VST_LINUX.md b/md/34_VST_LINUX.md index e69de29..7d8d7bf 100644 --- a/md/34_VST_LINUX.md +++ b/md/34_VST_LINUX.md @@ -0,0 +1,344 @@ +# TECHNICAL PLAN: DECENT SAMPLER + PIANOBOOK INSTALLATION AND SOUNDFONT MAPPING EXTRACTION FOR AI AGENT + +--- + +## 1. Executive Overview + +The system needs to fulfill two core requirements: + +* **Integrate DecentSampler & Pianobook on Linux Server:** +* Install `DecentSampler.vst3` (Linux 64-bit) into the Docker Server environment. +* Structure the Pianobook sample library directories (`.dspreset` + `.wav` files). +* Integrate Preset Loading into Python `pedalboard` for offline rendering steps. + + +* **Resolve the AI Agent's "Instrument Information Blindness" regarding SoundFont (`.sf2`):** +* *Current State:* The AI Agent and the system only load raw `.sf2` files without knowing what instruments are contained within (which Bank, which Program/Patch number, or what the instrument names are). +* *Solution:* +* Build a **SoundFont Inspection Engine** using Python (`sf2utils`) to scan `.sf2` files upon upload/scan and extract the instrument catalog table (Bank, Program/Preset ID, Instrument Name). +* Export a Catalog table (`soundfont_catalog.json`) and pass this context to the AI Agent. +* Update the AI Tool Schema so the AI accurately passes `soundfont_id`, `bank`, and `program` (MIDI Program Change) when creating a Track. + + + + + +--- + +## 2. DecentSampler + Pianobook Installation Plan (Server Backend) + +### 2.1 Installing DecentSampler Linux Native VST3 in Docker + +1. **Download DecentSampler Linux VST3:** +* Download the official Linux 64-bit build from the DecentSampler website (`DecentSampler_Linux_x64.tar.gz` or `.vst3` file). + + +2. **Server Directory Structure:** +```text +/opt/daw_engine/ +├── vst3/ +│ └── DecentSampler.vst3/ <-- Native Linux VST3 Binary +├── soundfonts/ +│ ├── GeneralUser_GS.sf2 +│ └── SGM-V2.01.sf2 +└── samples/ + └── pianobook/ <-- Pianobook Sample Libraries + ├── salamander_grand_piano/ + │ ├── salamander_piano.dspreset + │ └── samples/ (*.wav) + └── acoustic_guitar/ + ├── guitar.dspreset + └── samples/ (*.wav) + +``` + + +3. **Additions to `Dockerfile`:** +```dockerfile +# Install audio system dependencies +RUN apt-get update && apt-get install -y \ + libgl1-mesa-glx \ + libfreetype6 \ + libcurl4 \ + && rm -rf /var/lib/apt/lists/* + +# Copy DecentSampler VST3 to Server +COPY ./vst_plugins/DecentSampler.vst3 /opt/daw_engine/vst3/DecentSampler.vst3 + +``` + + + +### 2.2 Integrating DecentSampler into the Python Engine (`app/core/vst_engine.py`) + +The `pedalboard` library supports loading VST3 plugins and preset files for DecentSampler: + +```python +import os +from pedalboard import VST3Plugin + +class DecentSamplerManager: + def __init__(self, vst_path="/opt/daw_engine/vst3/DecentSampler.vst3"): + self.vst_path = vst_path + + def create_decent_sampler_instance(self, dspreset_path: str) -> VST3Plugin: + """ + Instantiates VST3 DecentSampler and loads the Pianobook sample preset (.dspreset) file. + """ + if not os.path.exists(self.vst_path): + raise FileNotFoundError(f"DecentSampler VST3 not found at {self.vst_path}") + + plugin = VST3Plugin(self.vst_path) + + # Load the Pianobook preset file into DecentSampler VST3 + if os.path.exists(dspreset_path): + plugin.load_preset(dspreset_path) + + return plugin + +``` + +--- + +## 3. Designing the SoundFont Inspection Engine (Bank/Program Extraction) + +Every SoundFont (`.sf2`) file is a collection of Presets (or Programs). For the AI Agent to know what sound presets exist inside the `.sf2` file, the Backend must scan and parse the `.sf2` file. + +### 3.1 Adding Python SoundFont Inspection Libraries + +Add to `requirements.txt`: + +```text +sf2utils>=0.9.0 +mido>=1.3.0 + +``` + +### 3.2 Building the SoundFont Metadata Inspection Service (`app/core/soundfont_inspector.py`) + +```python +import os +import json +from sf2utils.sf2parse import Sf2File + +class SoundFontInspector: + def __init__(self, sf_dir="/opt/daw_engine/soundfonts"): + self.sf_dir = sf_dir + + def inspect_sf2_file(self, filepath: str) -> dict: + """ + Parses an .sf2 file and returns a complete instrument catalog (Bank, Program, Instrument Name). + """ + if not os.path.exists(filepath): + return {} + + sf_name = os.path.basename(filepath) + sf_id = os.path.splitext(sf_name)[0].lower() + + instruments = [] + with open(filepath, 'rb') as f: + sf2 = Sf2File(f) + for preset in sf2.presets: + # Ignore EOP (End of Header) preset + if preset.name.strip() == "EOP" or (preset.bank == 128 and preset.preset == 127): + continue + + instruments.append({ + "bank": preset.bank, # Bank number (0 = General MIDI Standard, 128 = Percussion/Drums) + "program": preset.preset, # Program/Patch number (0-127) + "name": preset.name.strip(), # Instrument name (e.g. "Stereo Grand", "Violin", "Brass Section") + "is_percussion": (preset.bank == 128) + }) + + return { + "soundfont_id": sf_id, + "filename": sf_name, + "total_instruments": len(instruments), + "instruments": instruments + } + + def generate_full_catalog(self, output_json_path="/opt/daw_engine/soundfont_catalog.json"): + """ + Scans all .sf2 files in the directory and builds a Catalog JSON for the AI Agent. + """ + catalog = {} + for root, dirs, files in os.walk(self.sf_dir): + for file in files: + if file.endswith(('.sf2', '.SF2')): + full_path = os.path.join(root, file) + sf_info = self.inspect_sf2_file(full_path) + catalog[sf_info["soundfont_id"]] = sf_info + + with open(output_json_path, 'w', encoding='utf-8') as f: + json.dump(catalog, f, ensure_ascii=False, indent=2) + + return catalog + +``` + +### 3.3 Catalog File Output Structure (`soundfont_catalog.json`) + +This JSON file serves as an Instrument Dictionary for the AI Agent: + +```json +{ + "generaluser_gs": { + "soundfont_id": "generaluser_gs", + "filename": "GeneralUser_GS.sf2", + "total_instruments": 128, + "instruments": [ + { "bank": 0, "program": 0, "name": "Stereo Grand Piano", "is_percussion": false }, + { "bank": 0, "program": 19, "name": "Church Organ", "is_percussion": false }, + { "bank": 0, "program": 40, "name": "Violin Ensemble", "is_percussion": false }, + { "bank": 0, "program": 56, "name": "Trumpet", "is_percussion": false }, + { "bank": 128, "program": 0, "name": "Standard Drum Kit", "is_percussion": true } + ] + } +} + +``` + +--- + +## 4. Guiding the AI Agent in Instrument Selection & Accurate Note Loading + +When the user types: *"Create a Piano track and a Strings section track for 8 bars"*, the AI Agent needs to know precisely which soundfont, bank, and program numbers to assign to the Tracks. + +### 4.1 Updating the AI Tool Function Spec (`tools_spec.json`) + +Add `soundfont_bank` and `soundfont_program` fields to the Tool Schema sent to the AI: + +```json +{ + "type": "function", + "function": { + "name": "generate_multitrack_midi", + "description": "Generates multi-track MIDI data along with appropriate SoundFont Program configurations.", + "parameters": { + "type": "object", + "properties": { + "tracks": { + "type": "array", + "items": { + "type": "object", + "properties": { + "track_name": { "type": "string" }, + "instrument_type": { "type": "string", "enum": ["PIANO", "STRINGS", "BRASS", "SYNTH", "DRUMS"] }, + "soundfont_id": { + "type": "string", + "description": "ID of the SoundFont to use (e.g. 'generaluser_gs')" + }, + "soundfont_bank": { + "type": "integer", + "default": 0, + "description": "MIDI Bank code (0 for melodic instruments, 128 for Drums)" + }, + "soundfont_program": { + "type": "integer", + "description": "MIDI Program Number (0-127) corresponding to the instrument name in the Catalog" + }, + "notes": { "type": "array", "items": { "type": "object" } } + }, + "required": ["track_name", "soundfont_id", "soundfont_bank", "soundfont_program", "notes"] + } + } + } + } + } +} + +``` + +### 4.2 Injecting the Catalog into the AI Agent's Context Prompt (Prompt Template) + +Before sending the user query to the LLM, the system reads `soundfont_catalog.json` and injects a condensed catalog table into the System Instruction: + +```python +# System Context Prompt Injector +def build_ai_system_instruction(catalog_data: dict) -> str: + sf_summary = [] + for sf_id, sf_info in catalog_data.items(): + sf_summary.append(f"SoundFont ID: '{sf_id}' (File: {sf_info['filename']}):") + for inst in sf_info['instruments'][:20]: # Inject primary instrument lists + sf_summary.append( + f" - [{inst['name']}]: bank={inst['bank']}, program={inst['program']}" + ) + + catalog_context = "\n".join(sf_summary) + + system_instruction = f""" +You are an AI Copilot for a DAW. Below is the Catalog of available SoundFonts on the system: + +{catalog_context} + +MANDATORY RULES WHEN CREATING TRACKS: +1. When creating any track, you MUST look up the catalog above and fill in the correct `soundfont_id`, `soundfont_bank`, and `soundfont_program`. +2. Example: If the user requests "Piano", select soundfont_id="generaluser_gs", soundfont_bank=0, soundfont_program=0 ("Stereo Grand Piano"). +3. If the user requests "Violin/Strings", select soundfont_bank=0, soundfont_program=40 ("Violin Ensemble"). +4. If the user requests "Drums", select soundfont_bank=128, soundfont_program=0 ("Standard Drum Kit"). +""" + return system_instruction + +``` + +### 4.3 Applying Program Change on Client & Server Render Layers + +#### A. Client Browser Side (FluidSynth Wasm / SoundFont Player) + +Upon receiving JSON from the AI, the Frontend invokes the Bank and Program selection function to trigger the correct sound: + +```javascript +// Client-side Javascript (spessasynth / fluidsynth.wasm) +function applyAITrackInstrument(trackId, soundfontBank, soundfontProgram) { + const channel = getTrackMIDIChannel(trackId); + + // Send MIDI Bank Select (CC 0) + synthInstance.controllerChange(channel, 0, soundfontBank); + + // Send MIDI Program Change + synthInstance.programChange(channel, soundfontProgram); +} + +``` + +#### B. Server Offline Render Side (`app/core/render_engine.py`) + +When rendering to a WAV file, Python inserts a MIDI Program Change event ahead of the Track's note sequence: + +```python +import mido + +def create_midi_track_with_program(notes_data, bank=0, program=0): + midi_track = mido.MidiTrack() + + # 1. Insert Bank Select (Control Change 0) + midi_track.append(mido.Message('control_change', channel=0, control=0, value=bank, time=0)) + + # 2. Insert Program Change (Instrument Sound Selection) + midi_track.append(mido.Message('program_change', channel=0, program=program, time=0)) + + # 3. Insert MIDI notes generated by AI + for note in notes_data: + start_tick = int(note['start_beat'] * 480) # 480 ticks per beat + dur_tick = int(note['duration_beats'] * 480) + pitch = int(note['pitch']) + vel = int(note['velocity'] * 127) + + midi_track.append(mido.Message('note_on', note=pitch, velocity=vel, time=start_tick)) + midi_track.append(mido.Message('note_off', note=pitch, velocity=0, time=dur_tick)) + + return midi_track + +``` + +--- + +## 5. Action Checklist + +* [ ] **Step 1:** Download Linux 64-bit `DecentSampler.vst3` and copy it into `/opt/daw_engine/vst3/`. +* [ ] **Step 2:** Download Pianobook sound libraries (e.g. Salamander Grand Piano) and extract them to `/opt/daw_engine/samples/pianobook/`. +* [ ] **Step 3:** Add `sf2utils` to `requirements.txt` and install it in Docker. +* [ ] **Step 4:** Create `app/core/soundfont_inspector.py` to automatically scan all `.sf2` files in the project and generate `soundfont_catalog.json`. +* [ ] **Step 5:** Build API Endpoint `GET /api/v1/plugins/soundfonts/catalog` returning the extracted instrument catalog. +* [ ] **Step 6:** Update the Prompt Template and AI Tool Schema to support `soundfont_bank` & `soundfont_program`. +* [ ] **Step 7:** End-to-End Verification: Type Prompt *"Generate 8 bars of Brass horn music"* $\rightarrow$ AI reads Catalog and selects Program `56` $\rightarrow$ Web Audio & Server Render produce the correct Brass horn instrument sound. \ No newline at end of file