Compare commits
660 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e99e54773f | |||
| 0ba31c57bf | |||
| 454dd91f96 | |||
| e9f29e09ca | |||
| 289746f187 | |||
| b3ced7a7b3 | |||
| 4233c1eeda | |||
| 35f7c3822f | |||
| d37f4e7557 | |||
| 8a9d6b3c27 | |||
| 55d3464b1e | |||
| cfc114b9d7 | |||
| 856a8183b6 | |||
| 0272912cff | |||
| c047934fc4 | |||
| fc663921ff | |||
| fe5e8cb58e | |||
| 01fcf51fea | |||
| 1af2119444 | |||
| e5b4321a55 | |||
| ec5fedec33 | |||
| 8c1a8ead56 | |||
| ed91e4534c | |||
| 8fc1c2641b | |||
| a9da813cb1 | |||
| 6f55d36085 | |||
| 7325fbfc45 | |||
| 94b2d2ef41 | |||
| f616edce49 | |||
| 6018263044 | |||
| 36edc9daca | |||
| 71278f2aba | |||
| 022ba38a5e | |||
| 71c3bafdb5 | |||
| 34ad02dd6b | |||
| 47b1633bd3 | |||
| 0182abf7ea | |||
| 9b6de7f857 | |||
| 84ab4ae823 | |||
| 25471e6ea5 | |||
| 0dc95386f2 | |||
| d8227904b6 | |||
| 4d10b9485b | |||
| adccf6430a | |||
| 73025fc387 | |||
| 9f762c2a78 | |||
| a0b6110fd1 | |||
| e88ce2e3ea | |||
| 827693dfc4 | |||
| d04631c7d3 | |||
| 224fd54a56 | |||
| befc0d35fd | |||
| 5e3a283bd9 | |||
| 4a04bc04ef | |||
| dc384b3102 | |||
| 0b9079632a | |||
| 9be6991222 | |||
| 0853a13f62 | |||
| c09f994ede | |||
| 249e2afea2 | |||
| b65188fcc8 | |||
| ba5883951d | |||
| 7000529d8d | |||
| 1de6bd6b57 | |||
| c435d00897 | |||
| b075a5eca7 | |||
| 36ee2bdf90 | |||
| 5423686c73 | |||
| 4db777eb5d | |||
| 1989bfb105 | |||
| d1995307f5 | |||
| f647291523 | |||
| fdaabe3a08 | |||
| 548ab1258f | |||
| 2c2ee59d7d | |||
| 7e5f406226 | |||
| ad5cda5a40 | |||
| 05e6a467d8 | |||
| 4b43f027ce | |||
| 175bce6743 | |||
| 3380b0e201 | |||
| 619a2484ce | |||
| 24ac703d4d | |||
| c47eb5d718 | |||
| 3a345c3f4e | |||
| 86b304e173 | |||
| 01d02a9d71 | |||
| 2bf9f8bcb8 | |||
| 99a445b789 | |||
| ee39dda5b9 | |||
| be8430fbd9 | |||
| f1b00eb060 | |||
| d81afba93d | |||
| 223254eb07 | |||
| 698fa818e9 | |||
| 60d05e2b4d | |||
| 0ea994fb2f | |||
| eb370319cb | |||
| 9b25766fca | |||
| dfbeb24a33 | |||
| 3c4f106e14 | |||
| 4f5aee6830 | |||
| 42d64f93d8 | |||
| 6c11daccf8 | |||
| 88007dce34 | |||
| 1790f743d6 | |||
| ef6370f92e | |||
| 3cac617aff | |||
| e1550c8ffe | |||
| c60803ee2a | |||
| 9c5507f7b2 | |||
| 47e87c8363 | |||
| 652fb98d2c | |||
| 845d1594bb | |||
| 59aa9a32ff | |||
| bd296c63f1 | |||
| 468a1f11a4 | |||
| 0fdb8318f0 | |||
| 437980ea9f | |||
| bc36e40967 | |||
| 6ce77c4f20 | |||
| bc951e75af | |||
| 5e8664af1c | |||
| b1aec24392 | |||
| be3205645f | |||
| 0894f97386 | |||
| ca34a496eb | |||
| e76a6df86b | |||
| 64b9fbadd5 | |||
| 323be6a6f4 | |||
| 5ed6acd4f5 | |||
| 6c227fe6af | |||
| 7b8e10419f | |||
| 872ecabf85 | |||
| 4005088e89 | |||
| 9c487614f1 | |||
| 58a294d845 | |||
| 313d1be5af | |||
| 3386a2b4c2 | |||
| 64f43cd5ad | |||
| 2fb70e2e11 | |||
| 546ad3dd23 | |||
| 402cac56a3 | |||
| 1de69b8f4e | |||
| a9be38b76f | |||
| 938c8d7f87 | |||
| 15193b13a7 | |||
| 4db3e3484a | |||
| f0479dc3d8 | |||
| 7836cfcc76 | |||
| 19f02a5a49 | |||
| dc7ec3af4c | |||
| c073ca7a49 | |||
| 9f6f9763c1 | |||
| bbd06f23d5 | |||
| b67c0264fb | |||
| 74597cfe8b | |||
| 0c1a27ef2c | |||
| eb61015cff | |||
| 6e1f116658 | |||
| 2c55db372a | |||
| e64070a83e | |||
| 66eef405a7 | |||
| 971dad84e7 | |||
| 4cead1010e | |||
| 7b5d4bf27a | |||
| 47af85c072 | |||
| d4ad3bc1f7 | |||
| f721b8242f | |||
| a1767e338a | |||
| 8051cf3edb | |||
| 35e767795a | |||
| 7bfdd72850 | |||
| 30ce0eb306 | |||
| 41b5a70a60 | |||
| fa7917cf54 | |||
| a3e681afc0 | |||
| 8775c960e3 | |||
| 33e35a1af1 | |||
| d4262acb9f | |||
| e40fdf15ad | |||
| 0d763c5a0b | |||
| ad6fa1feb9 | |||
| 9abb0e6a50 | |||
| 6c6a326b7d | |||
| 8e8fc58082 | |||
| ec355da58d | |||
| 9c3a435f26 | |||
| 12d2639455 | |||
| e99bc8624f | |||
| 9869566d3e | |||
| 0f99788d21 | |||
| e544d5e694 | |||
| 614d83b971 | |||
| 895ad0b7d9 | |||
| 98ca21bd9a | |||
| c3d87562e9 | |||
| 5ed943b3d6 | |||
| fa7815bb53 | |||
| caa95177ef | |||
| b00f0cff95 | |||
| 2187440681 | |||
| 33cf804c88 | |||
| a68bebac2a | |||
| 35fb29201d | |||
| 393dd7190f | |||
| fc81f19b1c | |||
| 61f4b90af0 | |||
| a56c616817 | |||
| 9cbf5940f4 | |||
| 4c3d649506 | |||
| 0dc3f10406 | |||
| 6a46b1ffe4 | |||
| 71edf7ffb3 | |||
| c09ad7f48d | |||
| ad29df46d2 | |||
| 2a9748e12e | |||
| bb49035bee | |||
| 6c8eb79aac | |||
| 54c1ee9e94 | |||
| a41284ccf7 | |||
| 6604f85e0d | |||
| cff3950670 | |||
| 3f7abe0c95 | |||
| 2eb327056d | |||
| 421535ca0e | |||
| 77c7486d4e | |||
| 10c0a743c6 | |||
| ae71acef0f | |||
| f0b62d5a1c | |||
| f84431c9a6 | |||
| b5f3f987b7 | |||
| 5ec329933a | |||
| aafc750274 | |||
| 62a41d4502 | |||
| fa7ffe88bf | |||
| 385d62cfaf | |||
| f4a364df51 | |||
| 32b61553f9 | |||
| 67551075be | |||
| fd31e8051d | |||
| 43691a127e | |||
| 6fa6578dd1 | |||
| 2dec94aa7e | |||
| ea40f0c540 | |||
| 5d31563df6 | |||
| 2f2f84066c | |||
| 52ec9379d6 | |||
| 9c2fa52e1b | |||
| 815a6faaca | |||
| 9befc20851 | |||
| 18e177fa2d | |||
| f2ac70eee0 | |||
| ed51cab765 | |||
| 447504ea67 | |||
| e4d6405b37 | |||
| 2baf034852 | |||
| d40ac47d65 | |||
| 14e3a561a3 | |||
| d0def4bf1e | |||
| e68fe27d7f | |||
| 1dbece1fc7 | |||
| 1ca2ec6d01 | |||
| 292c787a31 | |||
| b84507bd64 | |||
| 0992879b49 | |||
| 74fdc2b303 | |||
| c99527e687 | |||
| e055e8b5c8 | |||
| 85ad22ee4f | |||
| b8dfc9b211 | |||
| fd24c2dcd8 | |||
| 80a5352645 | |||
| 4b732844c9 | |||
| 430e87445e | |||
| 74992f7b55 | |||
| 8beef7cb4d | |||
| 08cd4d72cf | |||
| 081a17d537 | |||
| 6dec171af7 | |||
| e38391ac7d | |||
| b42190e6bc | |||
| 40502eda86 | |||
| f429f6b1d3 | |||
| 912fb2d354 | |||
| 59cf1d67db | |||
| 0420eccb82 | |||
| b5344e41f7 | |||
| 8c5d71c2f0 | |||
| 4a9cbcdef1 | |||
| e14554a75a | |||
| a60607c673 | |||
| c9176a27fa | |||
| db4bdd2471 | |||
| 60141cef3f | |||
| 9c0909c3fa | |||
| 6ff2f72c58 | |||
| ab4b91a158 | |||
| 1628a34ea0 | |||
| db45efe53c | |||
| fd23b8c4b0 | |||
| 1d3c7a7955 | |||
| 3ed62b3343 | |||
| 3a775fdda7 | |||
| 509e171c20 | |||
| 0500d16bc7 | |||
| a98dd58744 | |||
| 1ba9e8379f | |||
| b9113f8a06 | |||
| 91ddc3cb6f | |||
| fb0cd248ad | |||
| d21091b69b | |||
| 53a27b5af4 | |||
| 035d75e504 | |||
| c4a320a1c1 | |||
| 095a0680e8 | |||
| 4ed0b1e5be | |||
| 29206d62a3 | |||
| 57235e68d2 | |||
| 47d1a14f50 | |||
| 2b04b32ea8 | |||
| 15ec35d8d9 | |||
| 8965643450 | |||
| 16e33b2e6a | |||
| 838e173dd7 | |||
| ce7d23c351 | |||
| 987ecffc6c | |||
| 7a2ac3a851 | |||
| cb23d36bc8 | |||
| 638d56ce63 | |||
| 3d7bae94f2 | |||
| b947237c3c | |||
| 00566e8ea6 | |||
| d9ad60768e | |||
| 5c7f922eb0 | |||
| ae6abc3ea5 | |||
| cdfe88ba76 | |||
| 05113e17f3 | |||
| 4d2da5a272 | |||
| 4ca6e7c6ad | |||
| 2c012e3348 | |||
| a887db587f | |||
| 6fe0cd8d61 | |||
| e801023beb | |||
| ef1238b87d | |||
| 81dc9a0354 | |||
| 18d2503cb9 | |||
| 187686aa82 | |||
| fa4a38bf3e | |||
| 4def86168b | |||
| 9357e4fcc8 | |||
| f7a46e0baa | |||
| 1829b61472 | |||
| 8aeac14e4b | |||
| 48e4047a84 | |||
| 0e051297b0 | |||
| 62ac2e2d88 | |||
| e785b88ea7 | |||
| e866ad1c0d | |||
| 90ff65f31d | |||
| 0246255792 | |||
| d5ef016d49 | |||
| e6439c2d9d | |||
| 5a49d2012b | |||
| 7eb26a004f | |||
| 5c47b68a9d | |||
| cf61b8431d | |||
| e5bd442b7c | |||
| 55a27b5957 | |||
| 981a184176 | |||
| 138d05698a | |||
| 3a6b44e195 | |||
| fba3664a1c | |||
| 2ad29226c4 | |||
| d18c0085b6 | |||
| 7544ee2e5f | |||
| d04442ad45 | |||
| dff2c81848 | |||
| ef891e9e04 | |||
| 720d80dbf9 | |||
| 06480516eb | |||
| bf0469558a | |||
| af6c77b9ee | |||
| 3a84323dd8 | |||
| 3c12ce2a22 | |||
| e739eb54bf | |||
| f59cc304d4 | |||
| 6a409b954d | |||
| cde29386d2 | |||
| ea5043c981 | |||
| 5915049260 | |||
| d97f82e93f | |||
| f98ade5d58 | |||
| 993610ca37 | |||
| 509da0c724 | |||
| 1764fc29b5 | |||
| e4dadb3158 | |||
| 45a86f09bd | |||
| 0f5f494173 | |||
| 3b6f1ad05a | |||
| 7bbe1f3be0 | |||
| 9038cfab86 | |||
| 8155b4f2fa | |||
| cdd5f8c96b | |||
| 1daf355dd4 | |||
| bdaa5594bd | |||
| 5ab93953e7 | |||
| f96064cbab | |||
| 473a0afe0a | |||
| 87655f4275 | |||
| 28be934a19 | |||
| 40c531087c | |||
| 2f2cb3a066 | |||
| 63496f27f4 | |||
| de70d4a837 | |||
| 400856c0a3 | |||
| d117a363da | |||
| cca970c348 | |||
| 5eb0c26ce4 | |||
| 6242eba3c8 | |||
| 3c7a002b2c | |||
| 8f4a4cd0eb | |||
| f41422da03 | |||
| 4064b90874 | |||
| 9cad9f6c95 | |||
| 91e228ad6a | |||
| 09231d9b21 | |||
| 6d9060a52b | |||
| 6a2522d039 | |||
| 263c4be409 | |||
| 50847fc440 | |||
| 4d42b8407c | |||
| c884f3c533 | |||
| 5096c618b4 | |||
| e7a0a85eca | |||
| 72186168d2 | |||
| e843f55483 | |||
| 7af0e61fb0 | |||
| 09def59cac | |||
| 0d1ef0f895 | |||
| 0271be4484 | |||
| 23601007bf | |||
| 29c4c45791 | |||
| 3d2930dca5 | |||
| e51b7fd355 | |||
| 44e0a6d736 | |||
| bfd7146bfc | |||
| bfb636f0bc | |||
| 443e09b592 | |||
| ef6fe0e718 | |||
| 4ff2ce0b9c | |||
| c2bfeb7163 | |||
| 96aa76f194 | |||
| 013465f27a | |||
| 1059f67979 | |||
| 280c8f598f | |||
| d48262d468 | |||
| 2ab989132b | |||
| 3e805179c2 | |||
| f25a574126 | |||
| 89c7237379 | |||
| f16467eba1 | |||
| b1bbf567ab | |||
| 8204bc016d | |||
| b0d7f35a1e | |||
| 653c0747cb | |||
| 4e880850f5 | |||
| 1ffdec68c6 | |||
| 523a754949 | |||
| 79999065a7 | |||
| 49c9f019db | |||
| e18b261943 | |||
| d3c28d2fa7 | |||
| 7601cc72c9 | |||
| f7b499cda9 | |||
| 5e096b5594 | |||
| 724b42d506 | |||
| d3a3adca8a | |||
| 7ede9bbf72 | |||
| b6b1faf083 | |||
| bf4f2a5f11 | |||
| 90ca8f484b | |||
| 6926c79357 | |||
| 9717470bb6 | |||
| e57f6d96d9 | |||
| 5b500ebdf3 | |||
| 9438f44052 | |||
| 3901af75b2 | |||
| 27d9556a9d | |||
| bc0c2d3279 | |||
| 034455598d | |||
| 8450bd6de4 | |||
| d2993eee69 | |||
| fe379c45e2 | |||
| 3ae48f2f2f | |||
| d20fafb75e | |||
| 0b7dd82407 | |||
| 7107a0cc14 | |||
| 8ec4b05e21 | |||
| 9237d01f35 | |||
| a7417b9bfa | |||
| a765998455 | |||
| bd55d035de | |||
| 8516961d80 | |||
| c917db200e | |||
| e5196e79c8 | |||
| ce30289f46 | |||
| 5bd92ee186 | |||
| 89f2fed1d9 | |||
| 4306ff302a | |||
| 790a5823f6 | |||
| fccdb15ad7 | |||
| a181f7a24d | |||
| 1a5c1eff43 | |||
| 9bfcd1870c | |||
| 39680cb95c | |||
| 2ce731e6a9 | |||
| 025aba3265 | |||
| 57fd110fe3 | |||
| c1a3d3a6dd | |||
| fde9a91c9b | |||
| f2d02a358f | |||
| ce319573ab | |||
| 29b682e3ba | |||
| f4ec769650 | |||
| 966e673fae | |||
| a9f56d0fcd | |||
| 8d83a46e08 | |||
| 8fdf93f5d1 | |||
| 8849f33d07 | |||
| 97855f47d1 | |||
| d858628a8f | |||
| f3ad111376 | |||
| 7f681f23c1 | |||
| 6cc3ae6ab8 | |||
| 432dec635a | |||
| e6dd7ff13c | |||
| da760dbb86 | |||
| c3edda5c81 | |||
| 68eb6d5261 | |||
| 513a6b082c | |||
| 1151f1afa0 | |||
| 5af8536e53 | |||
| dffb8562df | |||
| 9498a0d4b0 | |||
| 2c91c0156a | |||
| cdff404437 | |||
| 1b711ef014 | |||
| 1a97a4b33b | |||
| a59a36ec90 | |||
| 7dd6300500 | |||
| 5628802e08 | |||
| 2fed3eee0b | |||
| eaedfebdf0 | |||
| f5bc0a8810 | |||
| 221561d58a | |||
| 0a17cc91ea | |||
| 7174773cb3 | |||
| 0beb957185 | |||
| 963a5519af | |||
| bf22679506 | |||
| 66cb05f26c | |||
| 5497bd1256 | |||
| 225c44d98a | |||
| 51787e055e | |||
| ac78d0aa7a | |||
| e90c929885 | |||
| e0531be65b | |||
| a956ae407c | |||
| 569ae9ef3e | |||
| 0039f85913 | |||
| 7530738444 | |||
| a6a12b36a7 | |||
| c397545383 | |||
| 5bd61516aa | |||
| 01cd3d1867 | |||
| cb3e8117b2 | |||
| 8be4ce99e1 | |||
| e14006ffcc | |||
| b410de8855 | |||
| f299b93372 | |||
| 70d1abaa7a | |||
| db759fc4fc | |||
| 8372ee1b98 | |||
| ff44fd4b8d | |||
| 6822f5ed33 | |||
| 52ecc38e13 | |||
| fbbe2c933c | |||
| e56ee5cb0a | |||
| b746f9659b | |||
| 373c018025 | |||
| 5a152e4951 | |||
| 7c99fca168 | |||
| 8dc57d4501 | |||
| 0554934ebe | |||
| dc8b1f59fa | |||
| 186c5353d4 | |||
| e5a563d343 | |||
| 6aed5e2ec8 | |||
| f1057ab7f2 | |||
| aa63a310fc | |||
| c5c3ec19cc | |||
| 893f42c36c | |||
| 8dee5c8cdb | |||
| eb805c04a4 | |||
| a90a3cfad1 | |||
| 46530c8896 | |||
| b04a76ebfb | |||
| 1c0d35f69b | |||
| d262f2e778 | |||
| 9a61a1f616 | |||
| b9eb840b11 | |||
| 17a284ad17 | |||
| 6a17e7036c | |||
| 1a28ebabee | |||
| f0210dd019 | |||
| dac661e6c6 | |||
| a21266736c | |||
| 055f351a17 | |||
| b46a2c45f2 | |||
| 225f23516f | |||
| 0b2382573f | |||
| b460d40824 | |||
| 92877d02b7 | |||
| 0bffe2d0d9 | |||
| af5d20d2e5 | |||
| 0e44e44ecb | |||
| 6545e1746e | |||
| 97a53fe59a | |||
| 5a80657b2b | |||
| 9843a33a9b | |||
| 78b9358539 | |||
| f3d48e8837 | |||
| 0155abd0da | |||
| c4302da931 | |||
| 9bf9f38864 | |||
| 6f0fac9f2d | |||
| ec178b42f3 | |||
| 7136cbe904 | |||
| d39b3f74ff | |||
| a8b484bd15 | |||
| 6b7872c636 | |||
| b9c524230d | |||
| 610c384bca | |||
| b4ec7981a6 | |||
| d6fe1326c5 | |||
| 8a85dd2dfc | |||
| b78193dfad | |||
| 58089f40f9 | |||
| 6fd9db54cf | |||
| 108943ae81 | |||
| 4791bb22d9 | |||
| 022fbb3351 | |||
| e0b849fdf2 | |||
| 09bb431a2b | |||
| 90ab2c1824 | |||
| 04403b0af7 | |||
| bbc42c630e | |||
| 9de9ae965f | |||
| d5143b440a |
+7
-1
@@ -17,9 +17,15 @@ app/storage/uploads/*
|
||||
app/storage/processed/*
|
||||
!app/storage/uploads/.gitkeep
|
||||
!app/storage/processed/.gitkeep
|
||||
app/storage/*.db
|
||||
app/storage/sf_scan_state.json
|
||||
.DS_Store
|
||||
|
||||
# VST3 and sample library directories (proprietary binaries)
|
||||
vst_plugins/
|
||||
samples/
|
||||
.idea/
|
||||
.vscode/
|
||||
*.log
|
||||
celerybeat-schedule
|
||||
node_modules/
|
||||
node_modules
|
||||
@@ -0,0 +1,12 @@
|
||||
# Kilo Post-Task Automation Rule
|
||||
|
||||
Mỗi khi bạn (Kilo) hoàn thành một tác vụ viết code, sửa lỗi, hoặc refactor code theo yêu cầu của tôi, bạn BẮT BUỘC phải thực hiện bước cuối cùng sau:
|
||||
|
||||
1. Kiểm tra sự tồn tại của file `wiki.md` ở thư mục gốc (nếu chưa có, hãy tạo mới).
|
||||
2. Tự động chèn (append) một đoạn báo cáo ngắn về task vừa hoàn thành vào CUỐI file `wiki.md` theo cấu trúc sau:
|
||||
|
||||
### [YYYY-MM-DD HH:mm] Task: <Tên ngắn gọn của công việc>
|
||||
- **Tóm tắt thay đổi:** <Mô tả 1-2 câu về nội dung đã thực hiện>
|
||||
- **Các file ảnh hưởng:** `<file_1>`, `<file_2>`
|
||||
- **Ghi chú/Test (nếu có):** <Các lưu ý hoặc lệnh chạy test/build nếu có>
|
||||
---
|
||||
@@ -0,0 +1,17 @@
|
||||
# Kilo Post-Task Automation Rule
|
||||
|
||||
Mỗi khi bạn (Kilo) hoàn thành một tác vụ viết code, sửa lỗi, hoặc refactor code theo yêu cầu của tôi, bạn BẮT BUỘC phải thực hiện bước cuối cùng sau:
|
||||
|
||||
**BẮT BUỘC**
|
||||
|
||||
1. Kiểm tra sự tồn tại của file `wiki.md` ở thư mục gốc (nếu chưa có, hãy tạo mới).
|
||||
2. Tự động chèn (append) một đoạn báo cáo ngắn về task vừa hoàn thành vào CUỐI file `wiki.md` theo cấu trúc sau:
|
||||
|
||||
### [YYYY-MM-DD HH:mm] Task: <Tên ngắn gọn của công việc>
|
||||
- **Tóm tắt thay đổi:** <Mô tả 1-2 câu về nội dung đã thực hiện>
|
||||
- **Các file ảnh hưởng:** `<file_1>`, `<file_2>`
|
||||
- **Ghi chú/Test (nếu có):** <Các lưu ý hoặc lệnh chạy test/build nếu có>
|
||||
---
|
||||
|
||||
3. Kiểm tra `git status`, add các file code đã sửa (chỉ add file code, KHÔNG add file database/soundfonts), và commit với message ngắn gọn.
|
||||
4. Nếu có file `.gitignore` bị conflict, resolve bằng `git checkout --theirs .gitignore && git add .gitignore` trước khi commit.
|
||||
@@ -0,0 +1,205 @@
|
||||
# Plan: TCP Resizable Width + Instrument Search Dropdown + Section Save Fix
|
||||
|
||||
## Task 1: User-Resizable TCP Width
|
||||
|
||||
**Files:** `app/static/js/app.jsx`
|
||||
|
||||
### Root Cause
|
||||
TCP containers are hardcoded `w-[320px]` (lines 14253, 14809). Components like Synth button, FX button, volume/pan sliders, input select overflow when content is wide.
|
||||
|
||||
### Implementation Steps
|
||||
|
||||
**1a — Add TCP width state**
|
||||
Add near line 6394 (near existing `rightSidebarWidth` state):
|
||||
```javascript
|
||||
const [tcpWidth, setTcpWidth] = useState(320);
|
||||
```
|
||||
|
||||
**1b — Add TCP resize handler**
|
||||
Add near line 6365 (near `startColResize`):
|
||||
```javascript
|
||||
const startTcpResize = e => {
|
||||
e.preventDefault();
|
||||
const startX = e.clientX;
|
||||
const startW = tcpWidth;
|
||||
const onMove = ev => {
|
||||
const deltaX = ev.clientX - startX;
|
||||
const newWidth = Math.max(280, Math.min(600, startW + deltaX));
|
||||
setTcpWidth(newWidth);
|
||||
};
|
||||
const onUp = () => {
|
||||
document.removeEventListener('mousemove', onMove);
|
||||
document.removeEventListener('mouseup', onUp);
|
||||
};
|
||||
document.addEventListener('mousemove', onMove);
|
||||
document.addEventListener('mouseup', onUp);
|
||||
};
|
||||
```
|
||||
|
||||
**1c — Replace `w-[320px]` with dynamic width in main TCP container (line 14253)**
|
||||
Change `className: "w-[320px] shrink-0 ..."` to `style: { width: tcpWidth + 'px', ... }`.
|
||||
|
||||
**1d — Replace `w-[320px]` with dynamic width in sub-tab TCP container (line 14809)**
|
||||
Same pattern as 1c.
|
||||
|
||||
**1e — Add resize handle (right edge of TCP)**
|
||||
Add a vertical resize handle bar on the right edge of both TCP containers. Pattern:
|
||||
```jsx
|
||||
React.createElement("div", {
|
||||
onMouseDown: startTcpResize,
|
||||
className: "absolute right-0 top-0 bottom-0 w-1 cursor-col-resize z-40 hover:bg-cyan-500/50 transition-colors",
|
||||
style: { right: 0 }
|
||||
})
|
||||
```
|
||||
|
||||
**1f — Ensure the main layout accommodates variable TCP width**
|
||||
The main timeline area should use `flex-1` so it fills remaining space. Verify existing layout handles this.
|
||||
|
||||
### Verification
|
||||
- Drag TCP right edge → width changes between 280px and 600px
|
||||
- Components fit properly at various widths
|
||||
- Timeline area fills remaining space
|
||||
- Works in section-tab view too
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Instrument Search Dropdown in TCP
|
||||
|
||||
**Files:** `app/static/js/app.jsx`
|
||||
|
||||
### Root Cause
|
||||
Current instrument selector is a modal overlay (lines 15661-15726) with no search/filter. Requires clicking Synth button → modal → scroll to find instrument.
|
||||
|
||||
### Implementation Steps
|
||||
|
||||
**2a — Add per-track dropdown open/close state**
|
||||
Add state:
|
||||
```javascript
|
||||
const [instrumentDropdownTrackId, setInstrumentDropdownTrackId] = useState(null);
|
||||
```
|
||||
This tracks which track's dropdown is open (null = all closed).
|
||||
|
||||
**2b — Add search query state**
|
||||
```javascript
|
||||
const [instrumentSearchQuery, setInstrumentSearchQuery] = useState('');
|
||||
```
|
||||
|
||||
**2c — Replace Synth button (top toolbar, line 14444-14448) with dropdown toggle**
|
||||
Convert the icon-only `<button>` into a container that:
|
||||
1. Shows current instrument name (truncated) + chevron-down icon when assigned
|
||||
2. Shows "Synth" + chevron-down icon when no instrument
|
||||
3. Click toggles `instrumentDropdownTrackId` for this track
|
||||
|
||||
**2d — Render the dropdown panel (conditional, below the button)**
|
||||
When `instrumentDropdownTrackId === track.id`, render a dropdown panel:
|
||||
```jsx
|
||||
React.createElement("div", {
|
||||
className: "absolute left-0 top-full mt-0.5 z-50 bg-[#1e1e1e] border border-zinc-700 rounded shadow-xl w-[220px] max-h-[300px] flex flex-col",
|
||||
onClick: e => e.stopPropagation()
|
||||
},
|
||||
// Search input
|
||||
React.createElement("input", {
|
||||
type: "text",
|
||||
placeholder: "Tìm nhạc cụ...",
|
||||
value: instrumentSearchQuery,
|
||||
onChange: e => setInstrumentSearchQuery(e.target.value),
|
||||
className: "w-full bg-black border-b border-zinc-700 px-2 py-1.5 text-xs text-zinc-200 outline-none"
|
||||
}),
|
||||
// Filtered list
|
||||
React.createElement("div", {
|
||||
className: "flex-1 overflow-y-auto"
|
||||
},
|
||||
// Filtered items from instrumentSelectorData
|
||||
// "None (Default Synth)" always shown first
|
||||
// Then filtered soundfonts
|
||||
// Then filtered VST instruments
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**2e — Filter logic**
|
||||
```javascript
|
||||
const filteredInstruments = useMemo(() => {
|
||||
if (!instrumentSelectorData) return { soundfonts: [], vst: [] };
|
||||
const q = instrumentSearchQuery.toLowerCase();
|
||||
return {
|
||||
soundfonts: (instrumentSelectorData.soundfonts || []).filter(sf =>
|
||||
(sf.display || sf.name || sf.id).toLowerCase().includes(q)
|
||||
),
|
||||
vst: (instrumentSelectorData.vst_instruments || []).filter(v =>
|
||||
(v.name || v.id).toLowerCase().includes(q)
|
||||
)
|
||||
};
|
||||
}, [instrumentSearchQuery, instrumentSelectorData]);
|
||||
```
|
||||
|
||||
**2f — Click outside to close**
|
||||
Add a global click handler that closes the dropdown when clicking outside.
|
||||
|
||||
**2g — Preload `instrumentSelectorData` on first TCP mount**
|
||||
Instead of only loading on `openInstrumentSelector`, preload `listPlugins()` when the first track renders (or on app mount).
|
||||
|
||||
**2h — Apply selection**
|
||||
On click of a dropdown item, call existing `setTrackInstrumentWithProgram` or `setTrackInstrument`. Close dropdown.
|
||||
|
||||
### Verification
|
||||
- Click Synth button → dropdown opens with search input focused
|
||||
- Type instrument name → list filters in real-time
|
||||
- Click instrument → dropdown closes, track assigned, Synth button shows name
|
||||
- Click outside → dropdown closes
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Section-Tab Save Fix — Replace Instead of Draw On Top
|
||||
|
||||
**Files:** `app/static/js/app.jsx`
|
||||
|
||||
### Root Cause
|
||||
`handleEditSectionInTab` (line 7178) clones ALL main-session tracks (empty) into the session-tab when `section.tracks` doesn't exist. `handleSaveSectionTab` (line 7156) saves ALL those empty tracks + edited ones into `s.tracks`. The rendering code (lines 696-791) draws ALL stored tracks inside the section box, creating a cluttered preview with empty/minimal tracks.
|
||||
|
||||
### Implementation Steps
|
||||
|
||||
**3a — Fix `handleEditSectionInTab` (line 7169) to only initialize relevant track**
|
||||
Change the fallback cloning (line 7178) from cloning ALL main tracks to creating a minimal set of tracks based on the section's parent track:
|
||||
```javascript
|
||||
const clonedTracks = section.tracks ? section.tracks : [{
|
||||
...tracks.find(tr => tr.id === trackId),
|
||||
clips: [],
|
||||
sections: [],
|
||||
midiItems: [],
|
||||
markers: [],
|
||||
isArmed: false,
|
||||
monitoringEnabled: true,
|
||||
instrumentId: null,
|
||||
instrumentProgram: undefined,
|
||||
instrumentName: null
|
||||
}];
|
||||
```
|
||||
This only clones the track that owns the section, not ALL main tracks.
|
||||
|
||||
**3b — Fix `handleSaveSectionTab` (line 7138) to filter non-empty tracks**
|
||||
After building the updated section, filter `tab.tracks` to only include tracks that have actual content:
|
||||
```javascript
|
||||
const contentTracks = tab.tracks.filter(t =>
|
||||
(t.clips && t.clips.length > 0) ||
|
||||
(t.midiItems && t.midiItems.length > 0)
|
||||
);
|
||||
```
|
||||
Store `tracks: contentTracks` instead of `tracks: tab.tracks`.
|
||||
|
||||
**3c — Improve section-item rendering (lines 694-791)**
|
||||
The rendering already draws waveform from clips and MIDI notes from midiItems. Ensure:
|
||||
- Waveform rendering for clips with `clip.buffer` is correct (already done at lines 720-739)
|
||||
- MIDI note colors are per-track-index (already done at line 783: `noteColors[trackIdx % noteColors.length]`)
|
||||
- Add a subtle track label inside each sub-track row so users can identify which track is which
|
||||
|
||||
**3d — Ensure waveform preview is properly sized**
|
||||
The section preview currently allocates `subTrackHeight = (height - 24) / maxSubTracks` for each sub-track (line 698). Verify this is sufficient for waveform + MIDI note rendering when there are 1-2 tracks (typical case).
|
||||
|
||||
### Verification
|
||||
- Open a section for editing → session-tab shows only the relevant track(s), not all main tracks
|
||||
- Add MIDI items, sound clips, soundfonts, FX to tracks
|
||||
- Save section → section-item shows waveform preview + MIDI note preview (replacing previous content, not appending)
|
||||
- Open section again → previous edits are loaded correctly
|
||||
- Multiple save cycles → no doubling of content
|
||||
- Waveform rendered as background, MIDI notes in distinct colors per track
|
||||
@@ -0,0 +1,212 @@
|
||||
# Piano Roll: 4 tính năng
|
||||
|
||||
## 1. Auto-scroll brush khi drag gần cạnh
|
||||
|
||||
**File**: `app/static/js/app.jsx`
|
||||
|
||||
**Vị trí**: Trong `handleGridMouseMove`, cuối block `draggedNote.mode === 'draw'` (trước `return;` ở dòng ~5237).
|
||||
|
||||
**Code thêm** (sau visitedPitches/brushIds logic, trước `return;`):
|
||||
```js
|
||||
const container = gridScrollRef.current;
|
||||
if (container) {
|
||||
const cr = container.getBoundingClientRect();
|
||||
const edgeThreshold = 30;
|
||||
const scrollStep = 6;
|
||||
if (e.clientY < cr.top + edgeThreshold) {
|
||||
container.scrollTop = Math.max(0, container.scrollTop - scrollStep);
|
||||
} else if (e.clientY > cr.bottom - edgeThreshold) {
|
||||
container.scrollTop = Math.min(container.scrollHeight - container.clientHeight, container.scrollTop + scrollStep);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**⚠ Edge case**: Nếu chuột dừng tại mép, `mousemove` ngưng → cuộn dừng. Để cuộn liên tục, dùng `setInterval` khi vào threshold. Tạm thời chấp nhập giới hạn này.
|
||||
|
||||
---
|
||||
|
||||
## 2. Ctrl+drag velocity với selected notes
|
||||
|
||||
**File**: `app/static/js/app.jsx`
|
||||
|
||||
### 2a. `handleCCMouseDown` (dòng ~5360)
|
||||
Thay block `if (e.ctrlKey)` hiện tại:
|
||||
|
||||
```js
|
||||
if (e.ctrlKey) {
|
||||
if (selectedNoteIds.length > 0) {
|
||||
selectedNoteIds.forEach(id => {
|
||||
const idx = notes.findIndex(n => n.id === id);
|
||||
if (idx !== -1) paintNote(idx, val);
|
||||
});
|
||||
ccDragRef.current = { active: true, lastBeat: beat, selectedMode: true, lastPainted: selectedNoteIds.map(id => notes.findIndex(n => n.id === id)).filter(i => i !== -1) };
|
||||
} else {
|
||||
if (noteIdx !== -1) paintNote(noteIdx, val);
|
||||
ccDragRef.current = { active: true, lastBeat: beat, lastPainted: noteIdx !== -1 ? [noteIdx] : [] };
|
||||
}
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
### 2b. `handleCCMouseMove` (sau dòng ~5381)
|
||||
Thêm block đầu `handleCCMouseMove` (SAU khi lấy `drag`, `painted`, TRƯỚC `candidateIdx`):
|
||||
|
||||
```js
|
||||
if (drag.selectedMode && selectedNoteIds.length > 0) {
|
||||
selectedNoteIds.forEach(id => {
|
||||
const idx = notes.findIndex(n => n.id === id);
|
||||
if (idx !== -1 && !painted.includes(idx)) {
|
||||
setNotes(prev => prev.map((n, i) => {
|
||||
if (i !== idx) return n;
|
||||
if (ccMode === 'pan') return { ...n, pan: (val - 0.5) * 2.0 };
|
||||
return { ...n, velocity: val };
|
||||
}));
|
||||
}
|
||||
});
|
||||
// Cập nhật lastPainted TRỰC TIẾP trên ref (không qua setNotes callback)
|
||||
const newPainted = selectedNoteIds
|
||||
.map(id => notes.findIndex(n => n.id === id))
|
||||
.filter(i => i !== -1 && !painted.includes(i));
|
||||
ccDragRef.current.lastPainted = [...painted, ...newPainted];
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
⚠ **Không gán `drag.lastPainted` bên trong `setNotes` callback** — `drag` là `ccDragRef.current`, gán trực tiếp vào ref ngoài callback để tránh stale closure.
|
||||
|
||||
### 2b2. Cleanup `selectedMode` khi mouseup (dòng ~5409)
|
||||
Trong `handleCCMouseUp` (và `onMouseLeave`), thêm reset:
|
||||
```js
|
||||
if (ccDragRef.current) ccDragRef.current.selectedMode = false;
|
||||
```
|
||||
|
||||
### 2c. CC canvas rendering (dòng ~4911-4932)
|
||||
Thêm `isSelected` vào loop notes; đổi màu xanh dương `#3b82f6` khi selected:
|
||||
|
||||
```js
|
||||
const isSelected = selectedNoteIds.includes(note.id);
|
||||
// ...
|
||||
ctx.strokeStyle = ccMode === 'pan' ? (isSelected ? '#60a5fa' : '#a78bfa') : (isSelected ? '#3b82f6' : '#fbbf24');
|
||||
ctx.fillStyle = ccMode === 'pan' ? (isSelected ? '#3b82f6' : '#c084fc') : (isSelected ? '#3b82f6' : '#fbbf24');
|
||||
```
|
||||
|
||||
Thêm `selectedNoteIds` vào dependency array của effect.
|
||||
|
||||
---
|
||||
|
||||
## 3. SNAP trong MIDI tab
|
||||
|
||||
**Ghi chú**: `getSnapBeat(beat, mode)` đã xử lý `mode === 'free'` bằng cách return `beat` không đổi. Không cần check `snapVal !== 'free'` riêng.
|
||||
|
||||
**File**: `app/static/js/app.jsx`
|
||||
|
||||
### 3a. Selection marquee — create (dòng ~5003-5008)
|
||||
Snap `startBeat` khi tạo marquee:
|
||||
|
||||
```js
|
||||
const snapStart = getSnapBeat(beat, snapVal);
|
||||
setSelectionMarquee({
|
||||
startBeat: snapStart, startPitch: pitch,
|
||||
currentBeat: snapStart, currentPitch: pitch
|
||||
});
|
||||
```
|
||||
|
||||
### 3a2. Selection marquee — drag update (dòng ~5140-5144)
|
||||
Snap `currentBeat` khi kéo marquee:
|
||||
|
||||
```js
|
||||
const snappedBeat = getSnapBeat(beat, snapVal);
|
||||
const marquee = {
|
||||
...selectionMarquee,
|
||||
currentBeat: snappedBeat,
|
||||
currentPitch: pitch
|
||||
};
|
||||
```
|
||||
|
||||
### 3b. Ruler drag loop range (dòng ~5742-5763)
|
||||
Snap `clickBeat` khởi tạo, snap `beat` trong onMove:
|
||||
|
||||
```js
|
||||
const snappedStartBeat = getSnapBeat(clickBeat, snapVal);
|
||||
const startData = { startX: e.clientX, startBeat: snappedStartBeat, scrollLeft: e.currentTarget.scrollLeft };
|
||||
// ...
|
||||
const rawBeat = Math.max(0, bx / pixelsPerBeat);
|
||||
const beat = getSnapBeat(rawBeat, snapVal);
|
||||
```
|
||||
|
||||
### 3c. Shift+Click ruler loop (dòng ~5732-5739)
|
||||
Đổi `Math.round(clickBeat / 4) * 4` thành `getSnapBeat(clickBeat, snapVal)`:
|
||||
|
||||
```js
|
||||
const beatSnap = getSnapBeat(clickBeat, snapVal);
|
||||
```
|
||||
|
||||
### 3d. Ruler loop handles (dòng ~5791, ~5808, ~5829)
|
||||
Đổi `Math.round(bx / pixelsPerBeat / 4) * 4` thành `getSnapBeat(bx / pixelsPerBeat, snapVal)` ở cả 3 handle (left resize, right resize, grab body).
|
||||
|
||||
---
|
||||
|
||||
## 4. Synth button trong MIDI tab toolbar
|
||||
|
||||
**File**: `app/static/js/app.jsx`
|
||||
|
||||
### 4a. Prop `onInstrumentSelect` (dòng ~4546)
|
||||
Thêm `onInstrumentSelect` vào props destructuring.
|
||||
|
||||
### 4b. Button synth trong toolbar (dòng ~5662-5700)
|
||||
Chèn button sau MIDI input select, trước transport buttons:
|
||||
|
||||
```jsx
|
||||
React.createElement("button", {
|
||||
onClick: () => onInstrumentSelect && onInstrumentSelect(st.trackId),
|
||||
title: st.instrumentName || "Synth",
|
||||
className: `px-1.5 py-0.5 text-[10px] rounded font-mono font-bold border transition flex items-center gap-0.5 max-w-[50px] ${st.instrumentName ? 'bg-violet-900 text-violet-300 border-violet-700' : 'bg-zinc-800 text-zinc-500 border-transparent hover:text-zinc-300'}`
|
||||
}, React.createElement("i", { "data-lucide": "music", className: "w-3 h-3 shrink-0" }), React.createElement("span", { className: "truncate text-[9px]" }, st.instrumentName || 'Synth'))
|
||||
```
|
||||
|
||||
### 4c. Pass callback từ App (dòng ~15631-15652)
|
||||
Thêm `onInstrumentSelect: (trackId) => { openInstrumentSelector(trackId); }`.
|
||||
|
||||
### 4d. Đồng bộ subTab instrument (dòng ~6343-6353)
|
||||
Trong `setTrackInstrumentWithProgram`, thêm cập nhật `subTabs`:
|
||||
|
||||
```js
|
||||
setSubTabs(prev => prev.map(s => {
|
||||
if (s.trackId !== trackId) return s;
|
||||
return { ...s, instrumentProgram: programNumber !== undefined ? programNumber : undefined, instrumentName: displayName, instrumentId };
|
||||
}));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Layout fix: thêm `h-full` (QUAN TRỌNG)
|
||||
|
||||
**File**: `app/static/js/app.jsx`, dòng ~5623
|
||||
|
||||
Đổi `className` của outer div từ:
|
||||
```
|
||||
"flex-1 flex overflow-hidden min-h-0 bg-[#1e1e1e] flex-col"
|
||||
```
|
||||
thành:
|
||||
```
|
||||
"flex-1 flex overflow-hidden min-h-0 bg-[#1e1e1e] flex-col h-full"
|
||||
```
|
||||
|
||||
**Lý do**: `h-full` cung cấp height tham chiếu cho flex chain, tránh content area cao 0px.
|
||||
|
||||
---
|
||||
|
||||
## Thứ tự thực hiện
|
||||
|
||||
1. Sửa layout: thêm `h-full`
|
||||
2. Feature 3: SNAP (4 edits nhỏ — dễ verify)
|
||||
3. Feature 2: Velocity selected notes
|
||||
4. Feature 1: Auto-scroll brush
|
||||
5. Feature 4: Synth button (liên quan nhiều component nhất)
|
||||
|
||||
## Kiểm tra
|
||||
|
||||
```bash
|
||||
cd /home/locpham/SonicForgeStudio && npm run build
|
||||
```
|
||||
Build phải pass. Nếu lỗi paren, kiểm tra đóng `()` tại cuối return statement.
|
||||
@@ -0,0 +1,193 @@
|
||||
# Plan: Apply & Install `md/34_VST_LINUX.md` (Revised per `md/35_VST_PLAN.md` + `md/35.1_VST_FIX.md`)
|
||||
|
||||
## Context Summary
|
||||
|
||||
- **Dockerfile** already has `libgl1`, `libasound2`, `libjack-jackd2-0`, `libfreetype6`, Xvfb, fluidsynth. Missing `libcurl4`.
|
||||
- `requirements.txt` has `mido` but missing `sf2utils`.
|
||||
- `app/core/vst_engine.py` has `PluginManager` with FluidSynth C-API-based `list_soundfont_instruments()`.
|
||||
- `app/core/render_engine.py` FluidSynth path hardcodes `program_select(0, fid, 0, 0)`.
|
||||
- `app/static/js/services/aiGateway.js` — `generate_multitrack_midi` tool has NO `soundfont_id`/`soundfont_bank`/`soundfont_program`.
|
||||
- `soundfontPlayer.js` — oscillator-based, uses `program` only for ADSR/osc-type selection; no `controllerChange()` or `programChange()` channel-state methods.
|
||||
- No `soundfont_catalog.json` generation or catalog API endpoint.
|
||||
- `docker-compose.yml` mounts `.` to `/app`.
|
||||
|
||||
---
|
||||
|
||||
## 5 Mandatory Refinements (from 35_VST_PLAN.md)
|
||||
|
||||
1. **Condensed Catalog for AI**: `get_condensed_catalog_summary()` → max 40–50 instruments categorized by GM groups (Piano, Organ, Guitar, Bass, Strings, Ensemble, Brass, Reed, Pipe, Synth Lead, Synth Pad, Drum Kit). Avoids token overflow.
|
||||
2. **Non-Blocking Catalog Generation**: `generate_full_catalog()` runs on first API access, not on startup. Cache in memory; refresh on SF2 upload via background task.
|
||||
3. **Dual Directory Scanning**: system `/opt/daw_engine/soundfonts/` AND user uploads `app/storage/uploads/soundfonts/`.
|
||||
4. **DecentSampler CWD Fix**: `os.chdir()` to `.dspreset` parent directory before `load_preset()`, so relative `samples/*.wav` paths resolve.
|
||||
5. **Robust Error Handling**: per-file `try/except` in `SoundFontInspector` — skip corrupted SF2 files with warning instead of crashing.
|
||||
|
||||
---
|
||||
|
||||
## Task A: SoundFont Inspection Engine (sf2utils)
|
||||
|
||||
### A1 — Add dependency
|
||||
- Add `sf2utils>=0.9.0` to `requirements.txt`.
|
||||
|
||||
### A2 — Create `app/core/soundfont_inspector.py`
|
||||
- `inspect_sf2_file(filepath)` — wrapped in `try/except` per Refinement 5. Returns `{soundfont_id, filename, total_instruments, instruments: [{bank, program, name, is_percussion}]}`.
|
||||
- `generate_full_catalog(output_json_path)` — scans system dir `/opt/daw_engine/soundfonts/` + user upload dir (Refinement 3). Writes `soundfont_catalog.json`.
|
||||
- `get_condensed_catalog_summary()` — returns categorized dict with ≤50 entries per Refinement 1. Groups instruments by GM category (Piano=0-7, Chromatic Perc=8-15, Organ=16-23, Guitar=24-31, Bass=32-39, Strings=40-47, Ensemble=48-55, Brass=56-63, Reed=64-71, Pipe=72-79, Synth Lead=80-89, Synth Pad=90-103, Drum Kit=128).
|
||||
- `invalidate_catalog_cache()` — resets in-memory cache; called after SF2 upload.
|
||||
|
||||
### A3 — API Endpoint `GET /api/v1/plugins/soundfonts/catalog`
|
||||
- In `app/api/v1/plugins.py`:
|
||||
- Response shape: `{ full_catalog: {...}, condensed_catalog: {...} }`.
|
||||
- Lazily generate on first call, cache in memory (Refinement 2).
|
||||
- `POST /upload-soundfont` success handler: calls `invalidate_catalog_cache()` + triggers a background task (FastAPI `BackgroundTasks`) to re-scan. Does NOT block HTTP response.
|
||||
|
||||
### A4 — JS API wrapper
|
||||
- In `app/static/js/services/api.js`, add `SonicAPI.getSoundfontCatalog()` → `GET /api/v1/plugins/soundfonts/catalog`.
|
||||
|
||||
---
|
||||
|
||||
## Task B: DecentSampler + Pianobook Support
|
||||
|
||||
### B1 — Dockerfile updates
|
||||
- Add `libcurl4` to `apt-get install`.
|
||||
- Pre-create `/opt/daw_engine/vst3/` and `/opt/daw_engine/samples/pianobook/` with `mkdir -p`.
|
||||
|
||||
### B2 — Host dirs
|
||||
- Create `vst_plugins/` and `samples/pianobook/` at repo root. Add both to `.gitignore`.
|
||||
|
||||
### B3 — DecentSamplerManager in `app/core/vst_engine.py`
|
||||
- `create_decent_sampler_instance(dspreset_path)`:
|
||||
- Resolve to absolute path with `os.path.abspath()`.
|
||||
- Save original CWD with `os.getcwd()`, then `os.chdir()` to `.dspreset` parent dir before `load_preset()` (Refinement 4).
|
||||
- Restore original CWD in `finally` block.
|
||||
- Return `VST3Plugin` instance ready for rendering.
|
||||
|
||||
### B4 — Wire into `app/core/render_engine.py`
|
||||
- If track selects a Pianobook instrument (e.g. `instrument_source: "pianobook"`), route through `DecentSamplerManager` instead of FluidSynth or synth fallback.
|
||||
- The Pianobook path uses `pedalboard.Pedalboard([vst])` with MIDI messages, same as other VST3 paths.
|
||||
|
||||
---
|
||||
|
||||
## Task C: AI Tool Schema & Prompt Injection
|
||||
|
||||
### C1 — Update `generate_multitrack_midi` tool in `aiGateway.js`
|
||||
- Add to `parameters.properties.tracks.items.properties`:
|
||||
- `soundfont_id`: `{ type: "string", description: "ID of the SoundFont file (e.g. 'generaluser_gs')" }`
|
||||
- `soundfont_bank`: `{ type: "integer", default: 0, description: "MIDI Bank. 0 = melodic, 128 = drums/percussion" }`
|
||||
- `soundfont_program`: `{ type: "integer", description: "MIDI Program number 0-127 from instrument catalog" }`
|
||||
- Add all 3 to `required` array.
|
||||
|
||||
### C2 — Inject condensed catalog into system instruction
|
||||
- Modify `buildUserMessage()` in `aiGateway.js`:
|
||||
- When `systemInstruction` is empty and global `window.__soundfontCatalog` exists, prepend a `system` message block containing the condensed catalog text.
|
||||
- Format: one line per GM category with bank/program examples.
|
||||
- **Enforce bank rule** (per 35.1_VST_FIX.md §3.B): Add explicit instruction — _`soundfont_bank: 0` for all melodic instruments, `soundfont_bank: 128` for Drum Kits._
|
||||
|
||||
### C3 — Fetch catalog on frontend startup
|
||||
- In `app/static/js/app.jsx`, after auth check, call `SonicAPI.getSoundfontCatalog()`.
|
||||
- Store result in `window.__soundfontCatalog = { condensed_catalog, full_catalog }`.
|
||||
- Re-fetch after any SF2 upload succeeds.
|
||||
|
||||
---
|
||||
|
||||
## Task D: Server Render — Program Change & Channel Mapping
|
||||
|
||||
### D1 — Read bank/program from track metadata
|
||||
- In `render_engine.py` MIDI rendering block, extract `soundfont_bank` and `soundfont_program` from track dict.
|
||||
- Default: bank=0, program=0.
|
||||
|
||||
### D2 — MIDI channel routing + FluidSynth update
|
||||
- **Channel rules** (per 35.1_VST_FIX.md §5.B):
|
||||
- `bank == 128` or track has `is_percussion: true` → `midi_channel = 9` (GM channel 10, percussion).
|
||||
- Otherwise → assign channels sequentially from 0–8, one per unique percussion-group track.
|
||||
- Replace hardcoded `fl.program_select(0, fid, 0, 0)` with:
|
||||
```python
|
||||
midi_channel = 9 if (bank == 128 or track.get("is_percussion")) else channel_counter
|
||||
fl.program_select(midi_channel, fid, bank, prog)
|
||||
```
|
||||
- All note_on/note_off events for that track must use the same `midi_channel`.
|
||||
|
||||
### D3 — VST3/Pedalboard path: CC + PC insertion
|
||||
- Extend `midi_events_to_messages()` or add a wrapper that inserts two MIDI messages at sample_offset=0 before note messages:
|
||||
- `MidiMessage(control_change=0, value=bank, sample_offset=0)` — CONTROL_CHANGE CC 0 (Bank Select MSB)
|
||||
- `MidiMessage(program_change=program, sample_offset=0)` — PROGRAM_CHANGE
|
||||
- These are prepended to the message list before `Pedalboard([vst])` processes the buffer.
|
||||
|
||||
---
|
||||
|
||||
## Task E: Client SoundFont Player — Program Change & Channel Allocation
|
||||
|
||||
### E1 — Add channel-state tracking to `soundfontPlayer.js`
|
||||
- Add internal `_channels` array (size 16), each entry: `{ bank: 0, program: 0 }`.
|
||||
- `controllerChange(channel, controller, value)`:
|
||||
- If `controller === 0` (Bank Select MSB), store `bank` for that channel.
|
||||
- `programChange(channel, program)`:
|
||||
- Store `program` for that channel.
|
||||
- Modify `playNote()` to accept an optional `channel` parameter and use the stored bank/program for ADSR/osc-type selection.
|
||||
|
||||
### E2 — Add `applyAITrackInstrument(trackId, bank, program)`
|
||||
- New function in `soundfontPlayer.js`:
|
||||
- Determine MIDI channel: `bank === 128 || isPercussion ? 9 : track_index % 9`.
|
||||
- Call `controllerChange(channel, 0, bank)`.
|
||||
- Call `programChange(channel, program)`.
|
||||
- Called from `app.jsx` after AI returns `generate_multitrack_midi` with track instrument data.
|
||||
|
||||
### E3 — Wire into post-AI pipeline in `app.jsx`
|
||||
- In the DAW command dispatch loop (around line 12853), after processing `generate_multitrack_midi` function call:
|
||||
- For each returned track with `soundfont_bank`/`soundfont_program`, call `applyAITrackInstrument()`.
|
||||
- Log the action to `aiActionLog`.
|
||||
|
||||
---
|
||||
|
||||
## Task F: Background Cache Invalidation on Upload
|
||||
|
||||
### F1 — Update `POST /upload-soundfont` in `plugins.py`
|
||||
- After saving the uploaded SF2 file:
|
||||
1. Call `SoundFontInspector.invalidate_catalog_cache()`.
|
||||
2. Use FastAPI `BackgroundTasks` to queue a re-scan: `background_tasks.add_task(generate_full_catalog)`.
|
||||
3. Return HTTP 200 immediately (not block on scan).
|
||||
|
||||
### F2 — Frontend catalog re-fetch after upload
|
||||
- In `app.jsx` upload handler, after `SonicAPI.uploadSoundFont()` succeeds, call `SonicAPI.getSoundfontCatalog()` and update `window.__soundfontCatalog`.
|
||||
|
||||
---
|
||||
|
||||
## Task G: Validation
|
||||
|
||||
### G1 — Catalog API
|
||||
- `GET /api/v1/plugins/soundfonts/catalog` → valid JSON with `{ full_catalog: {...}, condensed_catalog: {...} }`.
|
||||
- Condensed catalog contains ≤50 entries, grouped by GM category.
|
||||
|
||||
### G2 — AI generation
|
||||
- Input: _"Compose 8 bars of Brass horns and a drum kit"_
|
||||
- Verify AI returns `generate_multitrack_midi` call with:
|
||||
- Brass track: `program: 56`, `bank: 0`, `soundfont_id: "generaluser_gs"`.
|
||||
- Drums track: `program: 0`, `bank: 128`, `soundfont_id: "generaluser_gs"`.
|
||||
|
||||
### G3 — Client instrument switching
|
||||
- After AI response, verify `applyAITrackInstrument` is called with correct bank/program per track.
|
||||
- Verify MIDI channel allocation: melodic → ch0-8, drums → ch9.
|
||||
- Verify `controllerChange(CC0)` + `programChange()` dispatched per channel.
|
||||
|
||||
### G4 — Server render
|
||||
- Export WAV, verify correct Brass horn and Drum sound.
|
||||
- For FluidSynth path: confirm `program_select` uses correct channel, bank, program.
|
||||
- For VST3 path: confirm CC0 + PC inserted before notes.
|
||||
|
||||
### G5 — Upload cache invalidation
|
||||
- Upload a new `.sf2` file → verify `catalog` endpoint updates without manual restart.
|
||||
- Upload a corrupted `.sf2` file → verify it is skipped gracefully (Refinement 5).
|
||||
|
||||
### G6 — Regression
|
||||
- `pytest tests/` passes with no regressions.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **A1–A4** (sf2utils + soundfont_inspector + catalog API + JS wrapper) — foundational.
|
||||
2. **C1–C3** (AI tool schema + condensed prompt injection + startup fetch) — depends on A3/A4.
|
||||
3. **D1–D3** (server render program change + channel mapping + CC/PC insertion) — depends on C1 for field names.
|
||||
4. **F1–F2** (background cache invalidation on upload) — depends on A3.
|
||||
5. **E1–E3** (client program change + channel allocation + post-AI wiring) — independent of D, but shares channel routing logic.
|
||||
6. **B1–B4** (DecentSampler) — last, requires manual VST3 binary download.
|
||||
7. **G1–G6** (validation).
|
||||
@@ -0,0 +1,206 @@
|
||||
# Technical Analysis & Implementation Plan: SF3 + SpessaSynth Client (md/37_SF_CONVERT.md)
|
||||
|
||||
## 1. Assessment Summary
|
||||
|
||||
The spec proposes 2 steps:
|
||||
1. **Server**: Convert `.sf2` → `.sf3` (Ogg Vorbis compressed) via `mscore` CLI
|
||||
2. **Client**: Replace oscillator emulation with SpessaSynth library for authentic SoundFont playback
|
||||
|
||||
### Current State vs Spec
|
||||
|
||||
| Requirement | Status | Impact |
|
||||
|---|---|---|
|
||||
| `mscore` in Dockerfile | ❌ Not installed | 200MB+ dependency |
|
||||
| `soundfont_converter.py` | ❌ Does not exist | Needs creation |
|
||||
| `GET /soundfonts/download/{sf_id}` endpoint | ❌ Missing | Blocks client download |
|
||||
| `spessasynth_lib` CDN import in `index.html` | ❌ Not present | Blocks client upgrade |
|
||||
| `soundfontStorage.js` (IndexedDB) | ❌ Not created | Needed for caching |
|
||||
| `SonicSF.init(audioCtx)` | ❌ Not called anywhere | New integration point |
|
||||
| `playNote` signature compat | ⚠️ Spec uses 4 params, codebase uses 6–8 | Must bridge |
|
||||
| `applyAITrackInstrument` signature | ⚠️ Spec has `channel` first, code passes it last | Must bridge |
|
||||
|
||||
### Key Risk: `mscore` Dependency
|
||||
|
||||
`mscore` (MuseScore) pulls in Qt, fontconfig, audio drivers — easily 200-400MB in the container. **Alternative approaches:**
|
||||
|
||||
1. **`fluidsynth` built-in conversion**: `fluidsynth` already installed (`libfluidsynth3`, `pyfluidsynth`). Can convert SF2→SF3 via `--convert` flag or using `fluid_synth_sffd_*` APIs, but the low-level Python bindings don't expose this.
|
||||
|
||||
2. **`sf2convert`/`sf2pack`**: Smaller tools, but less commonly packaged.
|
||||
|
||||
3. **Python + `libsndfile`/`pydub`/`ogg`**: Parse SF2, extract WAV samples, compress to Ogg, rebuild SF3 structure. Complex — would need a SF2 parser and Ogg encoder.
|
||||
|
||||
4. **Python `acousticbrainz-sf2convert`**: Lightweight Python library specifically for SF2↔SF3.
|
||||
|
||||
**Recommendation**: Install `mscore` via `apt-get install -y mscore --no-install-recommends` to minimize deps. If the image grows too much (~1.5GB+), fall back to `fluidsynth` command-line conversion (`fluidsynth --convert`).
|
||||
|
||||
### Key Risk: `playNote` API Compatibility
|
||||
|
||||
The spec's proposed API:
|
||||
```javascript
|
||||
playNote(pitch, velocity=0.8, durationSec=1.0, channel=0)
|
||||
```
|
||||
|
||||
Current callers use:
|
||||
```javascript
|
||||
playNote(pitch, velocity, durationMs, startTime, program, destinationNode, channel?, synthEngine?)
|
||||
```
|
||||
|
||||
**Strategy**: Don't replace. Instead, wrap SpessaSynth inside the existing `SonicSF` object. Map:
|
||||
- `velocity` (0–1) → MIDI velocity (1–127)
|
||||
- `startTime` → if in future, use `setTimeout` for note scheduling (not perfect but adequate for preview)
|
||||
- `destinationNode` → SpessaSynth routes to its own internal destination, but can add a gain node stage
|
||||
- `program` → internal `_channels[channel].program` state (as now), but SpessaSynth also gets `programChange(ch, program)`
|
||||
- `synthEngine` → load the right SF3 and set bank/program on SpessaSynth
|
||||
|
||||
**Backward compat**: Keep ALL existing methods. Add SpessaSynth as an optional enhanced engine. If SpessaSynth isn't loaded (CDN fails), fall back to oscillator emulation.
|
||||
|
||||
---
|
||||
|
||||
## 2. Implementation Plan
|
||||
|
||||
### Task 1 — Server: SF2→SF3 Conversion Pipeline
|
||||
|
||||
#### 1.1 Docker Dependency
|
||||
- `Dockerfile`: Add `mscore` with `--no-install-recommends`
|
||||
```dockerfile
|
||||
RUN apt-get install -y --no-install-recommends mscore && rm -rf /var/lib/apt/lists/*
|
||||
```
|
||||
- Test: `docker compose build` — measure image size delta
|
||||
|
||||
#### 1.2 `app/core/soundfont_converter.py`
|
||||
- `SoundFontConverter` class with:
|
||||
- `convert_sf2_to_sf3(sf2_path)` → runs `mscore -o output.sf3 input.sf2`
|
||||
- `batch_convert_all()` → walks `target_dirs`, converts missing `.sf3`
|
||||
- `target_dirs` = `["/opt/daw_engine/soundfonts", "app/storage/uploads/soundfonts"]`
|
||||
- Progress logging, error handling, cache validation (skip if `.sf3` newer than `.sf2`)
|
||||
- Integrate with server startup:
|
||||
- In `app/main.py` startup event, call `SoundFontConverter().batch_convert_all()` as background task (non-blocking, don't delay boot)
|
||||
|
||||
#### 1.3 API Download Endpoint
|
||||
- `app/api/v1/plugins.py`: Add:
|
||||
```python
|
||||
@router.get("/soundfonts/download/{sf_id}")
|
||||
async def download_soundfont_asset(sf_id: str):
|
||||
# Search system_dir, upload_dir for .sf3 or .sf2
|
||||
# Return FileResponse
|
||||
# Fallback: .sf3 → .sf2 → 404
|
||||
```
|
||||
- Use `_find_sf2_path` from `render_engine.py` logic (case-insensitive, multi-dir)
|
||||
- Set `media_type="application/octet-stream"` and proper `Content-Disposition`
|
||||
|
||||
#### 1.4 Update `SoundFontInspector` to Also Scan `.sf3`
|
||||
- `soundfont_inspector.py:80`: Change `.sf2` filter to `('.sf2', '.sf3')`
|
||||
- Ensure the catalog includes `.sf3` files as available instruments
|
||||
|
||||
---
|
||||
|
||||
### Task 2 — Client: SpessaSynth Integration
|
||||
|
||||
#### 2.1 `index.html` — Add SpessaSynth CDN
|
||||
```html
|
||||
<script type="module">
|
||||
import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
|
||||
window.SpessaSynthClass = Synthesizer;
|
||||
</script>
|
||||
```
|
||||
- Add BEFORE `soundfontPlayer.js` so the class is available when the player initializes
|
||||
|
||||
#### 2.2 `app/static/js/services/soundfontStorage.js` — IndexedDB Cache
|
||||
- `SoundFontStorage` class with `openDB()`, `getBuffer(sfId)`, `saveBuffer(sfId, arrayBuffer)`
|
||||
- Uses `indexedDB` with DB name `"DAW_SoundFont_Cache"`, store name `"sf3_buffers"`
|
||||
- Export singleton `sfStorage`
|
||||
|
||||
#### 2.3 `app/static/js/services/soundfontPlayer.js` — Dual-Mode Rewrite
|
||||
|
||||
**Architecture**: Keep the existing `window.SonicSF` as the public API. Internally use SpessaSynth when available, fall back to oscillator when not.
|
||||
|
||||
**Changes to existing methods:**
|
||||
|
||||
| Method | Change |
|
||||
|---|---|
|
||||
| `init(audioCtx)` | NEW — creates SpessaSynth instance, triggers default SF load |
|
||||
| `loadSoundFont(sfId)` | REWORK — try IndexedDB → fetch `/api/v1/plugins/soundfonts/download/{sfId}` → load into SpessaSynth |
|
||||
| `playNote(...)` | ADD SpessaSynth path: if initialized, delegate to `synthInstance.noteOn/noteOff`; else use oscillator fallback |
|
||||
| `applyAITrackInstrument(bank, program, synthEngine?)` | ADD SpessaSynth path: call `controllerChange` + `programChange` on SpessaSynth |
|
||||
| `stopAll()` | ADD SpessaSynth: `allNotesOff(channel)` or `programReset()` |
|
||||
| `controllerChange(ch, cc, val)` | ADD SpessaSynth: delegate if initialized |
|
||||
| `programChange(ch, prog)` | ADD SpessaSynth: delegate if initialized |
|
||||
|
||||
**Backward compat guarantee:**
|
||||
- All existing callers continue to work unchanged
|
||||
- 6-arg `playNote(pitch, vel, durMs, startTime, program, destNode)` → SpessaSynth ignores `startTime` (schedules immediately) and `destNode` (uses internal routing)
|
||||
- 8-arg `playNote(..., channel, synthEngine)` → SpessaSynth uses `synthEngine.soundfont_id` for SF loading, `synthEngine.soundfont_program` for program selection
|
||||
- If SpessaSynth not loaded/CDN fails → transparent fallback to existing oscillator code
|
||||
|
||||
#### 2.4 `app.jsx` — Integration Points
|
||||
|
||||
- **Startup** (in a useEffect or the existing audio context initialization):
|
||||
```javascript
|
||||
if (window.SonicSF && window.SonicSF.init) {
|
||||
SonicSF.init(getAudioContext());
|
||||
}
|
||||
```
|
||||
- **Synth selection**: After `setTrackInstrumentWithProgram` / `setTrackInstrument`, trigger SF load:
|
||||
```javascript
|
||||
if (synthEngine && synthEngine.type === 'soundfont') {
|
||||
SonicSF.loadSoundFont(synthEngine.soundfont_id);
|
||||
}
|
||||
```
|
||||
- **AI track creation**: Already calls `applyAITrackInstrument` — SpessaSynth path handles it
|
||||
|
||||
---
|
||||
|
||||
### Task 3 — Backward Compatibility & Migration
|
||||
|
||||
#### 3.1 Fallback Behavior
|
||||
- If `window.SpessaSynthClass` is undefined (CDN blocked, offline): fall back to existing oscillator code
|
||||
- If `.sf3` download fails: fall back to oscillator
|
||||
- If SpessaSynth throws: catch error, log warning, fall back to oscillator
|
||||
|
||||
#### 3.2 Testing Matrix
|
||||
|
||||
| Scenario | Expected |
|
||||
|---|---|
|
||||
| SpessaSynth loaded + SF3 cached | Authentic playback, 0ms load |
|
||||
| SpessaSynth loaded + SF3 needs download | Authentic playback after 1-2s load |
|
||||
| CDN blocked (offline) | Transparent oscillator fallback |
|
||||
| SF3 not available on server | Transparent oscillator fallback |
|
||||
| MIDI keyboard + armed track | Authentic or oscillator based on availability |
|
||||
|
||||
---
|
||||
|
||||
## 3. Files Affected
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `Dockerfile` | Add `mscore --no-install-recommends` |
|
||||
| `app/core/soundfont_converter.py` | **NEW** — SF2→SF3 convert + batch scan |
|
||||
| `app/api/v1/plugins.py` | Add `GET /soundfonts/download/{sf_id}` |
|
||||
| `app/core/soundfont_inspector.py` | Accept `.sf3` in scan filter |
|
||||
| `app/main.py` | Add startup background conversion |
|
||||
| `app/templates/index.html` | Add SpessaSynth CDN `<script type="module">` |
|
||||
| `app/static/js/services/soundfontStorage.js` | **NEW** — IndexedDB cache |
|
||||
| `app/static/js/services/soundfontPlayer.js` | Dual-mode rewrite (SpessaSynth + oscillator fallback) |
|
||||
| `app/static/js/app.jsx` | Add init + SF load calls |
|
||||
|
||||
## 4. Validation
|
||||
|
||||
```bash
|
||||
# Server-side
|
||||
docker compose build # verify mscore installs
|
||||
docker compose exec web python -c "from app.core.soundfont_converter import SoundFontConverter; print('OK')"
|
||||
curl -s http://localhost:8000/api/v1/plugins/soundfonts/download/sgm_v2.01 | head -c 4 | file -
|
||||
docker compose exec web ls -la /opt/daw_engine/soundfonts/*.sf3 # verify conversion
|
||||
|
||||
# Client-side
|
||||
# Open browser → DevTools → check SpessaSynth loaded (window.SpessaSynthClass)
|
||||
# Select SoundFont → check network tab for .sf3 download
|
||||
# Play MIDI notes → hear authentic instrument (not oscillator)
|
||||
```
|
||||
|
||||
## 5. Open Questions
|
||||
|
||||
1. **`mscore` image size**: Measure actual delta. If >300MB, consider `fluidsynth --convert` alternative.
|
||||
2. **CDN reliability**: SpessaSynth loaded from jsdelivr CDN — consider bundling or NPM install as fallback.
|
||||
3. **Startup delay**: `batch_convert_all()` could take minutes for large SF2s. Run as background Celery task, not inline startup.
|
||||
4. **SGM_v2.01.sf2 (529MB)**: Conversion time for this file. Need to handle gracefully (streaming, timeout).
|
||||
@@ -0,0 +1,398 @@
|
||||
# Technical Analysis & Implementation Plan: SpessaSynth → FluidSynth WASM Migration
|
||||
|
||||
## 1. Key Findings from Codebase Investigation
|
||||
|
||||
### 1.1 Current State (SpessaSynth)
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| Engine | `spessasynth_lib@4.3.1` (CDN via jsDelivr + importmap) |
|
||||
| File | `app/static/js/services/soundfontPlayer.js` — IIFE, `window.SonicSF` singleton |
|
||||
| API surface | 10 methods: `init`, `playNote`, `stopNote`, `stopAll`, `selectInstrument`, `controllerChange`, `programChange`, `pitchBend`, `sustainActive`, `applyAITrackInstrument`, `allocateChannel`, `getChannelState`, `loadSoundFont` (internal) |
|
||||
| Call sites | 45 in `app.jsx` across piano roll, MIDI keyboard, timeline playback, external MIDI input, AI track generation |
|
||||
| storage | `app/static/js/services/soundfontStorage.js` — IndexedDB cache, `window.SonicSFStorage` |
|
||||
| Server render | `app/core/render_engine.py` uses `pyfluidsynth` (C++ FluidSynth) |
|
||||
| Server SF3 | `app/core/soundfont_converter.py` converts SF2→SF3 via `fluidsynth`/`mscore` CLI |
|
||||
| API endpoint | `GET /api/v1/plugins/soundfonts/download/{sf_id}` — serves both `.sf2` and `.sf3` |
|
||||
| Docker | `libfluidsynth3`, `fluidsynth` CLI installed in container |
|
||||
|
||||
### 1.2 Target State (FluidSynth WASM) — Corrected Spec
|
||||
|
||||
**CDN package**: `fluidsynth-wasm` does NOT exist on npm/CDN.
|
||||
**Real package**: `@enikey87/fluidsynth-emscripten@0.1.1`
|
||||
|
||||
| File | Size | Purpose |
|
||||
|---|---|---|
|
||||
| `libfluidsynth-2.3.0-sf3.js` | 130 KB | JS loader, SF3 support, separate WASM |
|
||||
| `libfluidsynth-2.3.0-sf3.wasm` | 1.7 MB | WASM binary with Ogg/SF3 support |
|
||||
| `libfluidsynth-2.3.0-sf3-all-in-one.js` | 2.39 MB | JS+WASM embedded (no separate .wasm fetch) |
|
||||
|
||||
**CDN URL** (chosen: separate .wasm for caching):
|
||||
```
|
||||
https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js
|
||||
https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.wasm
|
||||
```
|
||||
|
||||
### 1.3 CRITICAL Architecture Flaw in Spec
|
||||
|
||||
The spec's code (md/39_SF3_NEW.md) passes `wasmModule` and `synthPtr` via `postMessage` to AudioWorklet. **This cannot work** because:
|
||||
1. AudioWorklet runs in a separate audio thread with an isolated global scope
|
||||
2. `synthPtr` (a C pointer / integer) refers to WASM linear memory in the main thread's `WebAssembly.Module` instance
|
||||
3. Loading `libfluidsynth-2.3.0-sf3.js` inside the worklet creates a **separate WASM instance** with its own memory
|
||||
4. The spec also uses `_fluid_synth_write_float` inside `process()` — this blocks the audio thread if audio buffer underruns occur
|
||||
|
||||
**Correct architecture (per README):**
|
||||
> "To use libfluidsynth-X.X.X.js in AudioWorklet, load it into AudioWorklet before your worklet JS file."
|
||||
|
||||
The FluidSynth instance must be created **inside** the AudioWorklet. MIDI events + ArrayBuffer data are sent from main thread to worklet via `port.postMessage`.
|
||||
|
||||
### 1.4 `playNote` Signature Complexity
|
||||
|
||||
Current `playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine)` has 8 parameters and 16 call sites. The `synthEngine` object carries `{ soundfont_id, soundfont_bank, soundfont_program }` which must be converted to FluidSynth's `bank_select` + `program_change` before `noteon`.
|
||||
|
||||
### 1.5 Additional `window.SonicSF` Methods Not in Spec
|
||||
|
||||
Must preserve in new implementation:
|
||||
- `stopNote(channel, pitch)` — MIDI note-off with extra CC 64/120 cleanup
|
||||
- `controllerChange(channel, controller, value)` — MIDI CC forwarding
|
||||
- `programChange(channel, program)` — pure JS state tracking
|
||||
- `pitchBend(channel, value)` — 14-bit bend value
|
||||
- `sustainActive(channel)` — getter for sustain pedal state
|
||||
- `allocateChannel(bank)` — channel allocation (percussion = ch 9)
|
||||
- `getChannelState(channel)` — JS-side { bank, program, isPercussion } state
|
||||
- `saveToIndexedDB / loadFromIndexedDB` — legacy storage
|
||||
|
||||
---
|
||||
|
||||
## 2. Implementation Plan
|
||||
|
||||
### Phase 1: Preparation (Infrastructure)
|
||||
|
||||
**Task 1.1 — Create worklet directory**
|
||||
```
|
||||
mkdir -p app/static/js/worklets/
|
||||
touch app/static/js/worklets/.gitkeep
|
||||
```
|
||||
|
||||
**Task 1.2 — Update `index.html`**
|
||||
- Remove SpessaSynth importmap + module script (lines 13-25)
|
||||
- Add FluidSynth WASM CDN script:
|
||||
```html
|
||||
<script src="https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js"></script>
|
||||
```
|
||||
|
||||
### Phase 2: AudioWorklet Processor
|
||||
|
||||
**Task 2.1 — Create `app/static/js/worklets/fluidsynth-worklet.js`**
|
||||
|
||||
This file runs inside AudioWorkletGlobalScope. It:
|
||||
- Receives the FluidSynth WASM module (loaded via `addModule()`)
|
||||
- Maintains its own `_fluid_synth` instance
|
||||
- Receives commands from main thread via `port.onmessage`:
|
||||
- `INIT_SYNTH` — create settings + synth, store `synthPtr`
|
||||
- `LOAD_SF` — receive ArrayBuffer, write to MEMFS, call `_fluid_synth_sfload`
|
||||
- `NOTE_ON` — `_fluid_synth_noteon(synthPtr, channel, pitch, velocity)`
|
||||
- `NOTE_OFF` — `_fluid_synth_noteoff(synthPtr, channel, pitch)`
|
||||
- `CC` — `_fluid_synth_cc(synthPtr, channel, controller, value)`
|
||||
- `PROGRAM_CHANGE` — `_fluid_synth_program_change / bank_select`
|
||||
- `PITCH_BEND` — `_fluid_synth_pitch_bend(synthPtr, channel, value)`
|
||||
- `ALL_NOTES_OFF` / `ALL_SOUNDS_OFF`
|
||||
- `SET_GAIN` — `_fluid_synth_set_gain`
|
||||
- In `process(inputs, outputs)`:
|
||||
- Get `synthPtr` from closure
|
||||
- Call `_fluid_synth_write_float(synthPtr, bufferSize, leftPtr, 0, 1, rightPtr, 0, 1)`
|
||||
- Return `true` to keep processor alive
|
||||
|
||||
Key constraint: The FluidSynth WASM's `_fluid_synth_write_float` needs access to the output channel Float32Array's **byteOffset** relative to the WASM heap. AudioWorklet `output` arrays are not backed by WASM memory. Two solutions:
|
||||
- **A**: After rendering, copy from WASM heap Float32Array to output channels (less efficient but safe)
|
||||
- **B**: Load FluidSynth WASM inside worklet, allocate output buffers inside WASM heap (complex)
|
||||
|
||||
**Recommendation: Solution A** — simpler and avoids memory management issues.
|
||||
|
||||
### Phase 3: Rewrite `soundfontPlayer.js`
|
||||
|
||||
**Task 3.1 — Rewrite as IIFE-compatible `window.SonicSF`**
|
||||
|
||||
Keep the IIFE pattern (`window.SonicSF = SonicSF`) to minimize diff in `app.jsx`. Internal implementation uses `FluidSynthWasmWorkletBridge` class.
|
||||
|
||||
```javascript
|
||||
// Internal state
|
||||
const _channels = Array.from({ length: 16 }, () => ({ bank: 0, program: 0, isPercussion: false }));
|
||||
const _sustainStates = new Array(16).fill(false);
|
||||
let _workletNode = null;
|
||||
let _audioCtx = null;
|
||||
let _initialized = false;
|
||||
let _initPromise = null;
|
||||
let _currentSfId = null;
|
||||
let _sfHandleMap = new Map(); // sfId -> sfHandle (integer, tracked on JS side)
|
||||
let _fluidModule = null; // guarded global, used before worklet takes over
|
||||
```
|
||||
|
||||
**Key architectural decision:**
|
||||
|
||||
The FluidSynth instance lives in the AudioWorklet. The main thread `soundfontPlayer.js`:
|
||||
1. Loads the WASM module only for FS operations (MEMFS file writing) - needed because worklet can't do `fetch()`
|
||||
2. Creates an `AudioWorkletNode`, registers the worklet
|
||||
3. Sends ArrayBuffer data + MIDI commands to worklet via `port.postMessage`
|
||||
4. Tracks JS-side state (channel, bank, program) for methods like `getChannelState`, `allocateChannel`
|
||||
|
||||
**Alternative (simpler, recommended for v1):**
|
||||
Run FluidSynth entirely on **main thread**, render PCM buffers in a rAF/setInterval loop, feed to a `ScriptProcessorNode` or a simple AudioWorklet that just outputs pre-rendered buffers. This avoids the AudioWorklet WASM complexity.
|
||||
|
||||
**Decision:** Use main-thread FluidSynth + simple AudioWorklet output. Rationale:
|
||||
- Much simpler implementation
|
||||
- FluidSynth `_fluid_synth_write_float` is fast enough for real-time rendering
|
||||
- No need to manage two WASM instances
|
||||
- Can reuse the spec's API surface directly
|
||||
- MIDI events can be processed synchronously on main thread
|
||||
|
||||
**Architecture:**
|
||||
```
|
||||
Main thread:
|
||||
FluidSynth Module (loaded via CDN <script> tag)
|
||||
└─ _new_fluid_settings, _new_fluid_synth
|
||||
└─ MEMFS: FS.writeFile('/soundfonts/...', data)
|
||||
└─ _fluid_synth_sfload, _fluid_synth_noteon, etc.
|
||||
└─ _fluid_synth_write_float(bufSize, ...) -> Float32Array
|
||||
|
||||
AudioWorklet (simple passthrough):
|
||||
└─ Receives Float32Array PCM buffers via postMessage
|
||||
└─ Outputs to audio destination in process()
|
||||
```
|
||||
|
||||
### 3.2 Method Implementation Details
|
||||
|
||||
**`init(audioCtx)`:**
|
||||
```javascript
|
||||
async init(audioContext) {
|
||||
this.audioCtx = audioContext;
|
||||
// resolve createFluidSynthModule() from CDN
|
||||
this.fluidModule = await new Promise(resolve => {
|
||||
// libfluidsynth-2.3.0-sf3.js exposes createFluidSynthModule()
|
||||
resolve(window.createFluidSynthModule());
|
||||
});
|
||||
// create settings, synth
|
||||
const settingsPtr = this.fluidModule._new_fluid_settings();
|
||||
this.fluidModule._fluid_settings_setnum(settingsPtr, "synth.sample-rate", this.audioCtx.sampleRate);
|
||||
this.fluidModule._fluid_settings_setnum(settingsPtr, "synth.gain", 1.0);
|
||||
this.synthPtr = this.fluidModule._new_fluid_synth(settingsPtr);
|
||||
// create MEMFS dir
|
||||
this.fluidModule.FS.mkdir('/soundfonts');
|
||||
// create AudioWorklet node
|
||||
this.workletNode = new AudioWorkletNode(this.audioCtx, 'fluidsynth-bridge');
|
||||
this.workletNode.connect(this.audioCtx.destination);
|
||||
// start render loop
|
||||
this._startRenderLoop();
|
||||
}
|
||||
```
|
||||
|
||||
**`_startRenderLoop()`:**
|
||||
Uses `requestAnimationFrame` or scheduler to periodically:
|
||||
1. Check if worklet Node needs more data
|
||||
2. Call `_fluid_synth_write_float` into a Float32Array
|
||||
3. Send via `postMessage` to worklet
|
||||
|
||||
**`loadSoundFont(sfId)`:**
|
||||
```javascript
|
||||
async loadSoundFont(sfId) {
|
||||
let buffer = await sfStorage.getBuffer(sfId);
|
||||
if (!buffer) {
|
||||
const resp = await fetch(`/api/v1/plugins/soundfonts/download/${sfId}`);
|
||||
buffer = await resp.arrayBuffer();
|
||||
await sfStorage.saveBuffer(sfId, buffer);
|
||||
}
|
||||
this.fluidModule.FS.writeFile(`/soundfonts/${sfId}.sf3`, new Uint8Array(buffer));
|
||||
const sfHandle = this.fluidModule._fluid_synth_sfload(this.synthPtr, `/soundfonts/${sfId}.sf3`, 1);
|
||||
// store handle
|
||||
return sfHandle !== -1;
|
||||
}
|
||||
```
|
||||
|
||||
**`playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine)`:**
|
||||
```javascript
|
||||
function playNote(note, velocity, durationMs, startTime, program, destNode, channel, synthEngine) {
|
||||
// Handle synthEngine (bank/program setup)
|
||||
let ch = channel ?? 0;
|
||||
let bank = 0, prog = 0;
|
||||
if (synthEngine) {
|
||||
bank = synthEngine.soundfont_bank ?? 0;
|
||||
prog = synthEngine.soundfont_program ?? 0;
|
||||
this.fluidModule._fluid_synth_bank_select(this.synthPtr, ch, bank);
|
||||
this.fluidModule._fluid_synth_program_change(this.synthPtr, ch, prog);
|
||||
} else if (program !== undefined) {
|
||||
prog = program;
|
||||
this.fluidModule._fluid_synth_program_change(this.synthPtr, ch, prog);
|
||||
}
|
||||
// Note on
|
||||
this.fluidModule._fluid_synth_noteon(this.synthPtr, ch, midiPitch, midiVel);
|
||||
// Schedule note off
|
||||
if (durationMs > 0 && durationMs < 60000) { // skip held notes
|
||||
setTimeout(() => {
|
||||
this.fluidModule._fluid_synth_noteoff(this.synthPtr, ch, midiPitch);
|
||||
}, durationMs);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`stopNote(channel, pitch)`:**
|
||||
→ `this.fluidModule._fluid_synth_noteoff(synthPtr, channel, pitch)`
|
||||
Also set sustain off + all notes off (preserve current behavior).
|
||||
|
||||
**`stopAll()`:**
|
||||
→ `_fluid_synth_all_notes_off` + `_fluid_synth_all_sounds_off` for all 16 channels.
|
||||
|
||||
**`controllerChange(channel, controller, value)`:**
|
||||
→ `this.fluidModule._fluid_synth_cc(synthPtr, channel, controller, value)`
|
||||
+ track JS-side state for `_channels[channel].bank`, `_sustainStates[channel]`.
|
||||
|
||||
**`pitchBend(channel, value)`:**
|
||||
→ `this.fluidModule._fluid_synth_pitch_bend(synthPtr, channel, value)`.
|
||||
|
||||
**`selectInstrument(channel, bank, program, sfId)`:**
|
||||
→ Load SF if needed, then `_fluid_synth_bank_select` + `_fluid_synth_program_change`.
|
||||
|
||||
**`sustainActive(channel)`:**
|
||||
→ Return `_sustainStates[channel]` (tracked by `controllerChange` for CC 64).
|
||||
|
||||
**`applyAITrackInstrument(bank, program, synthEngine)`:**
|
||||
→ Same as current: allocate channel, call `selectInstrument` with `synthEngine.soundfont_id`.
|
||||
|
||||
### Phase 4: Bridge AudioWorklet
|
||||
|
||||
**Task 4.1 — Create `app/static/js/worklets/fluidsynth-bridge.js`**
|
||||
|
||||
Simple AudioWorkletProcessor that receives pre-rendered PCM buffers:
|
||||
```javascript
|
||||
class FluidSynthBridge extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.audioQueue = [[], []]; // L/R channel queues
|
||||
this.port.onmessage = (e) => {
|
||||
if (e.data.type === 'PCM_FRAME') {
|
||||
this.audioQueue[0].push(...e.data.left);
|
||||
this.audioQueue[1].push(...e.data.right);
|
||||
}
|
||||
};
|
||||
}
|
||||
process(inputs, outputs) {
|
||||
const out = outputs[0];
|
||||
if (!out) return true;
|
||||
const len = out[0].length;
|
||||
const left = this.audioQueue[0].splice(0, len);
|
||||
const right = this.audioQueue[1].splice(0, len);
|
||||
// Fill output; zero-fill if buffer underrun
|
||||
for (let i = 0; i < len; i++) {
|
||||
out[0][i] = i < left.length ? left[i] : 0;
|
||||
out[1][i] = i < right.length ? right[i] : 0;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('fluidsynth-bridge', FluidSynthBridge);
|
||||
```
|
||||
|
||||
### Phase 5: Optimize Render Loop
|
||||
|
||||
**Task 5.1 — Implement ring-buffer approach**
|
||||
|
||||
The render loop must balance latency vs. buffer underruns:
|
||||
- Render ~512 samples per frame (≈11.6ms at 44.1kHz)
|
||||
- Queue 3 frames ahead (≈35ms buffer → safe against GC pauses)
|
||||
- Use `AudioWorkletNode.port.postMessage` with `transferable: true` for zero-copy
|
||||
|
||||
Alternative: Use SharedArrayBuffer for lock-free ring buffer (requires COOP/COEP headers).
|
||||
|
||||
### Phase 6: Remove SpessaSynth from index.html
|
||||
|
||||
**Task 6.1 — Clean up CDN imports**
|
||||
- Remove: `<script type="importmap">` block (lines 13-19)
|
||||
- Remove: SpessaSynth module script (lines 20-25)
|
||||
- Remove: `spessasynth_core` importmap entry
|
||||
|
||||
### Phase 7: API Endpoint for WASM asset (optional)
|
||||
|
||||
If self-hosting is preferred over CDN, serve `libfluidsynth-2.3.0-sf3.js` and `.wasm` from static files directory.
|
||||
|
||||
---
|
||||
|
||||
## 3. Files to Modify/Create
|
||||
|
||||
| Action | File | Description |
|
||||
|---|---|---|
|
||||
| **CREATE** | `app/static/js/worklets/fluidsynth-bridge.js` | Bridge AudioWorkletProcessor (PCM passthrough) |
|
||||
| **MODIFY** | `app/static/js/services/soundfontPlayer.js` | Rewrite engine from SpessaSynth to FluidSynth WASM; preserve `window.SonicSF` API surface |
|
||||
| **MODIFY** | `app/templates/index.html` | Replace SpessaSynth CDN imports with `@enikey87/fluidsynth-emscripten` CDN |
|
||||
| **MODIFY** | `app/static/js/services/soundfontStorage.js` | Minor: expose `sfStorage` as named export + `window.SonicSFStorage` (likely no change needed) |
|
||||
| **NO CHANGE** | `app/static/js/app.jsx` | Zero changes — `window.SonicSF` API surface preserved |
|
||||
| **NO CHANGE** | `app/api/v1/plugins.py` | Download endpoint already serves `.sf2`/`.sf3` correctly |
|
||||
| **NO CHANGE** | `app/core/render_engine.py` | Already uses pyfluidsynth (server-side) |
|
||||
| **NO CHANGE** | `app/core/soundfont_converter.py` | Already converts SF2→SF3 |
|
||||
| **NO CHANGE** | Dockerfile | Already has `libfluidsynth3`, `fluidsynth` CLI |
|
||||
|
||||
---
|
||||
|
||||
## 4. API Signature Mapping
|
||||
|
||||
| Old (SpessaSynth) | New (FluidSynth WASM) | Notes |
|
||||
|---|---|---|
|
||||
| `_synthInstance.noteOn(ch, pitch, vel)` | `_fluid_synth_noteon(ptr, ch, pitch, vel)` | Direct C function |
|
||||
| `_synthInstance.noteOff(ch, pitch)` | `_fluid_synth_noteoff(ptr, ch, pitch)` | Direct C function |
|
||||
| `_synthInstance.controllerChange(ch, cc, val)` | `_fluid_synth_cc(ptr, ch, cc, val)` | Direct C function |
|
||||
| `_synthInstance.programChange(ch, prog)` | `_fluid_synth_program_change(ptr, ch, prog)` | Direct C function |
|
||||
| `_synthInstance.pitchBend(ch, val)` | `_fluid_synth_pitch_bend(ptr, ch, val)` | Direct C function |
|
||||
| `soundBankManager.addSoundBank(buf, id)` | `FS.writeFile(path, uint8arr)` + `_fluid_synth_sfload(ptr, path, 1)` | MEMFS-based |
|
||||
| `soundBankManager.soundBankList` | JS-side `_sfHandleMap` (Map) | Track manually |
|
||||
| `controllerChange(ch, 120, 0)` (all sound off) | `_fluid_synth_all_sounds_off(ptr, ch)` | Direct C function |
|
||||
| `controllerChange(ch, 123, 0)` (all notes off) | `_fluid_synth_all_notes_off(ptr, ch)` | Direct C function |
|
||||
|
||||
---
|
||||
|
||||
## 5. Risk Assessment
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|---|---|---|
|
||||
| FluidSynth WASM CPU usage causes audio glitches | Medium | Use triple-buffered rendering at 512-sample blocks; monitor with `performance.now()` |
|
||||
| `_fluid_synth_write_float` blocks main thread | Medium | Move rendering to a separate rAF loop, not synchronized to audio callback |
|
||||
| SF3 loading latency on first load | Low | IndexedDB caching already in place; loading ≈2-5MB via MEMFS is sub-100ms |
|
||||
| CDN availability for `@enikey87/fluidsynth-emscripten` | Low | Package is 17.9MB unpacked; fallback host via jsDelivr + self-host option |
|
||||
| Backward compatibility with existing MIDI sessions | Medium | All `window.SonicSF` methods preserved with same signatures; `synthEngine` object handled correctly |
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation
|
||||
|
||||
```bash
|
||||
# 1. Build frontend
|
||||
cd /home/locpham/SonicForgeStudio
|
||||
cd app/static/js && npx babel app.jsx --config-file ../../babel.config.json -o app.precompiled.js
|
||||
|
||||
# 2. Start server
|
||||
cd /home/locpham/SonicForgeStudio
|
||||
docker compose up -d
|
||||
|
||||
# 3. Browser tests
|
||||
# - Open DevTools → check `window.createFluidSynthModule` exists
|
||||
# - Load app → check `SonicSF.init(audioCtx)` creates FluidSynth synth
|
||||
# - Select SoundFont instrument → check MEMFS write + sfload succeeds
|
||||
# - Play piano roll notes → hear authentic FluidSynth output
|
||||
# - Play timeline → notes play with correct timing
|
||||
# - External MIDI controller → CC, pitch bend, note on/off work
|
||||
# - Transport stop → all notes stop immediately
|
||||
# - Sustain pedal → notes sustain on CC 64 >= 64
|
||||
# - Stress: play 50+ simultaneous notes → no stuck notes
|
||||
```
|
||||
|
||||
**Server-side parity validation:**
|
||||
- Render a project with server (`/api/v1/plugins/render`)
|
||||
- Play same project in client browser
|
||||
- Compare WAV spectrograms: should be identical (same C++ FluidSynth core)
|
||||
|
||||
---
|
||||
|
||||
## 7. Open Questions for User
|
||||
|
||||
1. **Self-host WASM vs CDN?** — `@enikey87/fluidsynth-emscripten` from jsDelivr (5.9MB) or serve from `app/static/` folder. CDN reduces server load but requires internet. If user wants offline-capable, suggest self-host.
|
||||
|
||||
2. **Separate WASM file vs all-in-one?** — `libfluidsynth-2.3.0-sf3.js` (130KB) + `libfluidsynth-2.3.0-sf3.wasm` (1.7MB) allow browser to cache WASM separately. Or `libfluidsynth-2.3.0-sf3-all-in-one.js` (2.39MB) single file but no caching benefit. Recommend separate files.
|
||||
|
||||
3. **Main-thread render vs worklet-own-synth?** — The spec's worklet-own-synth approach is theoretically ideal but complex (MIDI events + file data must cross thread boundary). Recommend v1 as main-thread render + simple bridge worklet. Can optimize later.
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
# Plan: MIDI Ghost Notes + Dropdown Item Switcher + Session Sync Mode
|
||||
|
||||
## Overview
|
||||
Three features built on each other:
|
||||
1. **Dropdown** at tab title position listing ALL MIDI items across all tracks
|
||||
2. **Item switching** — selected item becomes editable, all others become ghost notes
|
||||
3. **Session sync mode toggle** — viewport aligns with session bars (ghost visible) or resets to bar 0 (isolated, no ghost)
|
||||
|
||||
---
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### A. Ghost note scope = all items except the selected one
|
||||
Not just "other tracks" — ALL MIDI items in `activeTracks` except the one matching `targetItemId` contribute ghost notes if overlapping.
|
||||
|
||||
### B. Two viewport modes, togglable
|
||||
| Mode | Viewport origin | Ghost notes | Bar labels |
|
||||
|------|----------------|-------------|------------|
|
||||
| **Session Sync** (default) | `item.startTime / secondsPerBar` | Visible | `Bar N` (session-absolute) |
|
||||
| **Isolated** | bar 0 | Hidden | `Bar N` (0-based) |
|
||||
|
||||
### C. Ghost notes computed (not persisted)
|
||||
No schema changes. Extraction runs in `useMemo` inside `PianoRollTabEditor`.
|
||||
|
||||
### D. Hit-testing exclusion is automatic
|
||||
Ghost notes are in separate `ghostLayers` state; mouse handlers only iterate `notes`.
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| **NEW** `app/static/js/services/ghostNoteExtractor.js` | Extraction logic |
|
||||
| `app/static/js/app.jsx` (~16113) | Pass `activeTracks` prop to `PianoRollTabEditor` |
|
||||
| `app/static/js/app.jsx` (~4563-6070) | All PianoRollTabEditor changes below |
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step Implementation
|
||||
|
||||
### Step 1: Create `ghostNoteExtractor.js`
|
||||
|
||||
`app/static/js/services/ghostNoteExtractor.js`
|
||||
|
||||
```js
|
||||
export function extractGhostLayers(activeTracks, targetTrackId, targetItemId, bpm)
|
||||
```
|
||||
|
||||
Algorithm:
|
||||
1. `secondsPerBeat = 60 / bpm`
|
||||
2. Find `targetItem` across all tracks → `windowStartBeat = targetItem.startTime / secondsPerBeat`, `windowEndBeat = (targetItem.startTime + targetItem.duration) / secondsPerBeat`
|
||||
3. Iterate ALL tracks, ALL MIDI items:
|
||||
- Skip non-MIDI tracks (`!t.midiItems || !t.midiItems.length`)
|
||||
- Skip muted tracks (`t.muted`)
|
||||
- Skip item matching `targetItemId` (the active item)
|
||||
4. For each candidate item:
|
||||
- `itemStartBeat = item.startTime / secondsPerBeat`
|
||||
- `itemEndBeat = (item.startTime + item.duration) / secondsPerBeat`
|
||||
- Overlap test: `itemStartBeat < windowEndBeat && itemEndBeat > windowStartBeat`
|
||||
- For each overlapping note:
|
||||
- `noteAbsStart = itemStartBeat + note.start_beat`
|
||||
- `noteAbsEnd = noteAbsStart + note.duration_beats`
|
||||
- Clip: keep if `noteAbsStart < windowEndBeat && noteAbsEnd > windowStartBeat`
|
||||
- `clampedDur = Math.min(noteAbsEnd, windowEndBeat) - Math.max(noteAbsStart, windowStartBeat)`
|
||||
- Push: `{ id: ghost_${note.id}, pitch, relative_start_beat: noteAbsStart - windowStartBeat, duration_beats: clampedDur, velocity, original_track_name: t.name, original_track_color: t.color || '#888' }`
|
||||
5. Group by track → `ghostLayers: [{ track_id, track_name, track_color, notes }]`
|
||||
6. Return `ghostLayers`
|
||||
|
||||
### Step 2: Pass `activeTracks` to PianoRollTabEditor
|
||||
|
||||
At render site (~line 16113), add:
|
||||
```js
|
||||
activeTracks: activeTracks,
|
||||
```
|
||||
|
||||
Add `activeTracks` to destructured props in `PianoRollTabEditor` function signature (~line 4563).
|
||||
|
||||
### Step 3: New state & derived data
|
||||
|
||||
Inside `PianoRollTabEditor` (~line 4570), after existing `React.useState` declarations:
|
||||
|
||||
```js
|
||||
const [showGhostNotes, setShowGhostNotes] = React.useState(true);
|
||||
const [sessionSyncMode, setSessionSyncMode] = React.useState(true);
|
||||
|
||||
// Compute all MIDI items for dropdown
|
||||
const allMidiItems = React.useMemo(() => {
|
||||
const result = [];
|
||||
(activeTracks || []).forEach(t => {
|
||||
if (!t.midiItems || !t.midiItems.length) return;
|
||||
t.midiItems.forEach(m => {
|
||||
result.push({ ...m, _trackId: t.id, _trackName: t.name });
|
||||
});
|
||||
});
|
||||
return result;
|
||||
}, [activeTracks]);
|
||||
|
||||
// Compute ghost layers
|
||||
const ghostLayers = React.useMemo(() => {
|
||||
if (!activeTracks || !st || !st.target_id) return [];
|
||||
return extractGhostLayers(activeTracks, st.trackId, st.target_id, parseInt(bpm) || 120);
|
||||
}, [activeTracks, st.trackId, st.target_id, bpm]);
|
||||
|
||||
// Compute session offset for bar labels
|
||||
const secondsPerBar = (60.0 / (parseInt(bpm) || 120)) * 4;
|
||||
const targetTrack = React.useMemo(
|
||||
() => (activeTracks || []).find(t => t.id === st.trackId),
|
||||
[activeTracks, st.trackId]
|
||||
);
|
||||
const activeTargetItem = React.useMemo(
|
||||
() => targetTrack ? (targetTrack.midiItems || []).find(m => m.id === st.target_id) : null,
|
||||
[targetTrack, st.target_id]
|
||||
);
|
||||
const sessionStartBar = sessionSyncMode && activeTargetItem
|
||||
? (activeTargetItem.startTime / secondsPerBar) : 0;
|
||||
```
|
||||
|
||||
### Step 4: Dropdown at tab title position
|
||||
|
||||
Replace the static title (line 5748-5752) with a dropdown:
|
||||
|
||||
```js
|
||||
/* 1a. TAB TITLE DROPDOWN */
|
||||
React.createElement("div", { className: "relative inline-block text-xs" },
|
||||
React.createElement("select", {
|
||||
value: st.target_id,
|
||||
onChange: e => handleSwitchMidiItem(e.target.value),
|
||||
className: "bg-zinc-800 border border-zinc-700 text-yellow-500 font-bold rounded px-1.5 py-0.5 outline-none focus:border-yellow-500 max-w-[200px]"
|
||||
}, allMidiItems.map(m =>
|
||||
React.createElement("option", {
|
||||
key: m.id,
|
||||
value: m.id
|
||||
}, `${m._trackName} - ${m.name || 'MIDI'}`)
|
||||
))
|
||||
)
|
||||
```
|
||||
|
||||
### Step 5: Switch handler function
|
||||
|
||||
Add before the return statement:
|
||||
|
||||
```js
|
||||
const handleSwitchMidiItem = (itemId) => {
|
||||
if (itemId === st.target_id) return;
|
||||
|
||||
// Save current notes first
|
||||
onSaveNotes(st.id, st.trackId, st.target_id, notes);
|
||||
|
||||
// Find selected item
|
||||
const match = allMidiItems.find(m => m.id === itemId);
|
||||
if (!match) return;
|
||||
|
||||
// Update subTab state (triggers ghost re-compute via useMemo)
|
||||
setSubTabs(prev => prev.map(s => s.id === st.id ? {
|
||||
...s,
|
||||
trackId: match._trackId,
|
||||
target_id: match.id,
|
||||
label: `Piano Roll: ${match.name || 'MIDI'}`,
|
||||
notes: match.notes || [],
|
||||
duration: match.duration || 4,
|
||||
instrumentProgram: activeTracks.find(t => t.id === match._trackId)?.instrumentProgram,
|
||||
instrumentName: activeTracks.find(t => t.id === match._trackId)?.instrumentName,
|
||||
note_selection: [],
|
||||
currentTime: 0,
|
||||
} : s));
|
||||
|
||||
// Reset local state
|
||||
setSelectedNoteIds([]);
|
||||
setLoopStartBeat(null);
|
||||
setLoopEndBeat(null);
|
||||
};
|
||||
```
|
||||
|
||||
### Step 6: Session sync toggle button
|
||||
|
||||
In toolbar (~line 5798, near CC toggle), add:
|
||||
|
||||
```js
|
||||
/* Session sync mode toggle */
|
||||
React.createElement("button", {
|
||||
onClick: () => setSessionSyncMode(!sessionSyncMode),
|
||||
className: `px-2 py-1 rounded text-xs ${sessionSyncMode ? 'bg-cyan-900/60 text-cyan-300 border border-cyan-700' : 'text-zinc-500 hover:text-zinc-300'}`,
|
||||
title: sessionSyncMode ? "Session-synced mode (ghost visible)" : "Isolated mode (bar 0, no ghost)"
|
||||
}, sessionSyncMode ? "🌐 Session" : "📋 Isolated")
|
||||
```
|
||||
|
||||
And the Ghost toggle:
|
||||
|
||||
```js
|
||||
React.createElement("button", {
|
||||
onClick: () => setShowGhostNotes(!showGhostNotes),
|
||||
disabled: !sessionSyncMode,
|
||||
className: `px-2 py-1 rounded text-xs ${!sessionSyncMode ? 'opacity-30 cursor-not-allowed' : showGhostNotes ? 'bg-purple-900/60 text-purple-300 border border-purple-700' : 'text-zinc-500 hover:text-zinc-300'}`,
|
||||
title: "Toggle ghost notes visibility"
|
||||
}, "👻 Ghost")
|
||||
```
|
||||
|
||||
Ghost toggle disabled in isolated mode (no ghost notes to show).
|
||||
|
||||
### Step 7: Bar labels with session offset
|
||||
|
||||
Modify `renderBarLabels()` (~line 5711-5734):
|
||||
|
||||
Replace `Bar ${bar}` with:
|
||||
```js
|
||||
const displayBar = bar + Math.floor(sessionStartBar);
|
||||
`Bar ${displayBar}`
|
||||
```
|
||||
|
||||
And the seek click handler:
|
||||
```js
|
||||
const barTime = (bar + Math.floor(sessionStartBar)) * 4 * beatSec;
|
||||
```
|
||||
|
||||
### Step 8: Auto-scroll to session position
|
||||
|
||||
Add `useEffect`:
|
||||
|
||||
```js
|
||||
React.useEffect(() => {
|
||||
if (sessionSyncMode && gridScrollRef.current && activeTargetItem) {
|
||||
const scrollTargetBeats = sessionStartBar * 4;
|
||||
gridScrollRef.current.scrollLeft = scrollTargetBeats * pixelsPerBeat;
|
||||
}
|
||||
}, [sessionSyncMode, sessionStartBar, st.target_id, pixelsPerBeat]);
|
||||
```
|
||||
|
||||
### Step 9: Ghost note canvas layer
|
||||
|
||||
In the note-drawing `useLayoutEffect` (~line 4834), insert **before** active note rendering:
|
||||
|
||||
```js
|
||||
/* Layer 2: Ghost Notes */
|
||||
if (showGhostNotes && sessionSyncMode && ghostLayers.length > 0) {
|
||||
ghostLayers.forEach(layer => {
|
||||
ctx.save();
|
||||
ctx.globalAlpha = 0.25;
|
||||
ctx.fillStyle = layer.track_color || '#888';
|
||||
ctx.strokeStyle = layer.track_color || '#888';
|
||||
layer.notes.forEach(note => {
|
||||
const x = note.relative_start_beat * pixelsPerBeat;
|
||||
const y = (127 - note.pitch) * NoteHeight;
|
||||
const w = note.duration_beats * pixelsPerBeat;
|
||||
const h = NoteHeight - 1;
|
||||
ctx.fillRect(x, y, w, h);
|
||||
ctx.setLineDash([2, 2]);
|
||||
ctx.lineWidth = 1;
|
||||
ctx.strokeRect(x, y, w, h);
|
||||
ctx.setLineDash([]);
|
||||
});
|
||||
ctx.restore();
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Add `showGhostNotes`, `sessionSyncMode`, `ghostLayers` to dependency array.
|
||||
|
||||
### Step 10: Ghost notes in CC Lane
|
||||
|
||||
Skip ghost notes in CC lane (only active notes). CC lane already only iterates `notes`, not `ghostLayers`. No changes needed.
|
||||
|
||||
---
|
||||
|
||||
## Edge Cases
|
||||
|
||||
- **Dropdown with single MIDI item**: Only one option, no ghost notes (nothing to ghost).
|
||||
- **Item deleted while piano roll is open**: `handleSwitchMidiItem` fails gracefully (item not found → no-op). `ghostLayers` `useMemo` returns `[]`.
|
||||
- **BPM change mid-edit**: All beat computations update via `useMemo`/React reactivity.
|
||||
- **Session sync → Isolated switch**: Scroll resets to 0, bar labels change to 0-based, ghost notes disappear.
|
||||
- **Isolated → Session sync switch**: Scroll jumps to session position, ghost notes reappear.
|
||||
- **Color fallback**: Use track's `color` prop; if `#888` as default.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
1. Open MIDI item → dropdown shows all MIDI items across all tracks
|
||||
2. Select different item from dropdown → active notes switch, ghost notes re-compute
|
||||
3. Ghost notes from ALL non-selected items appear (same track + other tracks)
|
||||
4. Ghost toggle hides/shows ghost notes (disabled in isolated mode)
|
||||
5. Session sync mode shows correct bar labels (e.g. `Bar 2` if item starts at bar 2)
|
||||
6. Isolated mode shows `Bar 0, 1, 2...` regardless of item's session position
|
||||
7. Switching items in isolated mode: notes change, viewport stays at bar 0
|
||||
8. Ghost notes cannot be clicked/dragged (excluded from hit-testing)
|
||||
9. Muted tracks' MIDI items are excluded from ghost notes
|
||||
10. Items outside target window are excluded from ghost notes
|
||||
@@ -0,0 +1,197 @@
|
||||
# Implementation Plan: Ozone Mastering Module
|
||||
|
||||
## Nguồn DUY NHẤT để tạo giao diện
|
||||
|
||||
**Toàn bộ giao diện của Mastering Modal được lấy TRỰC TIẾP từ HTML trong `md/45_MASTERING_MODULE.md` dòng 74–885.**
|
||||
|
||||
Không thiết kế giao diện mới. Không suy luận UI. Lấy chính xác:
|
||||
- **Cấu trúc HTML**: copy verbatim từ spec, chuyển sang JSX (className thay class, onChange thay onchange, v.v.)
|
||||
- **CSS styles**: copy verbatim từ spec (oz-panel, oz-card, oz-card-active, knob-dial, knob-pointer, gradient backgrounds)
|
||||
- **JavaScript logic**: port verbatim từ spec vào React hooks (useState thay ozState, useRef thay DOM refs, useEffect thay event listeners)
|
||||
- **Web Audio DSP graph**: copy y hệt node graph + connection từ spec (BiquadFilterNode, GainNode, DynamicsCompressorNode, AnalyserNode)
|
||||
- **Canvas renderers**: copy y hệt drawing logic từ spec (frequency grid, FFT fill, EQ curve, polar vectorscope, VU meter gradient)
|
||||
|
||||
## Goal
|
||||
Dịch spec HTML `md/45_MASTERING_MODULE.md` (lines 74–885) thành `MasteringModal` React component trong `app.jsx`.
|
||||
|
||||
## Ánh xạ HTML → React (chi tiết từ spec)
|
||||
|
||||
### HTML lines 74–118: Dependencies & Styles
|
||||
|
||||
| Spec | Action |
|
||||
|------|--------|
|
||||
| `<script src="cdn.tailwindcss.com">` | Đã có trong `index.html:9` |
|
||||
| `<link FA 6.4.0>` | Đã có trong `index.html:10` (dùng lucide thay FA, giữ nguyên) |
|
||||
| `<link Google Fonts Inter>` | Đã có font-family trong `index.html` style |
|
||||
| CSS `.oz-panel` `oz-card` `oz-card-active` `knob-*` scrollbar | Thêm vào `index.html` `<style>` block |
|
||||
| CSS `.font-mono { JetBrains Mono }` | Thêm vào `index.html` `<style>` |
|
||||
|
||||
**→ Sửa `app/templates/index.html`**: append ~30 dòng CSS class (oz-panel, oz-card, knob-dial, scrollbar tối)
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 121–161: Header Bar
|
||||
|
||||
| Spec Element | React JSX Mapping |
|
||||
|---|---|
|
||||
| Logo `.fa-atom` + "OZONE ADVANCED" | `<div>` với cùng class & lucide icon `wand-2` thay FA |
|
||||
| Play/Stop buttons + preset select | JSX nguyên bản, class giữ nguyên |
|
||||
| Target LUFS text | JSX nguyên bản |
|
||||
|
||||
**→ Trong `MasteringModal` return JSX**, copy verbatim HTML, chỉ thay `.fa-*` → lucide `i` tương ứng.
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 163–213: Module Chain Strip
|
||||
|
||||
| Spec Element | React |
|
||||
|---|---|
|
||||
| `#cardDynamicEq` + power toggle + icon | JSX với `onClick` → set activeModule('eq') |
|
||||
| `#cardImager` + power toggle + icon | JSX với `onClick` → set activeModule('imager') |
|
||||
| `#cardMaximizer` + power toggle + icon | JSX với `onClick` → set activeModule('maximizer') |
|
||||
| Empty add-module slot `.border-dashed` | JSX verbatim |
|
||||
| `oz-card-active` class toggle | `className={\`oz-card ${ozState.activeModule==='eq' ? 'oz-card-active' : ''}\`}` |
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 216–325: Dynamic EQ View
|
||||
|
||||
| Spec Element | React |
|
||||
|---|---|
|
||||
| `#viewDynamicEq` container | `className={ozState.activeModule === 'eq' ? '' : 'hidden'}` |
|
||||
| `#eqCanvas` | `<canvas ref={el => canvasRefs.current.eq = el}>` |
|
||||
| 4-band knobs (Low/Mid1/Mid2/High) | Mỗi band là `<div>` với `.knob-container`, `data-param`, pointer events |
|
||||
| Knob dial HTML `.knob-dial > .knob-pointer` | JSX verbatim + inline `transform: rotate(Xdeg)` |
|
||||
| Band labels & freq text | JSX verbatim |
|
||||
|
||||
**Knob interaction (spec lines 794–843)**: port pointerdown/move/up logic vào `useEffect` với `.knob-container` querySelectorAll, update `ozState[param]` + gọi `updateAudioGraphValues`.
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 327–363: Stereo Imager View
|
||||
|
||||
| Spec Element | React |
|
||||
|---|---|
|
||||
| `#viewImager` | `className={ozState.activeModule === 'imager' ? '' : 'hidden'}` |
|
||||
| `#imagerCanvas` (polar vectorscope) | `<canvas ref={...}>` |
|
||||
| 4-band width sliders `#w1Slider`–`#w4Slider` | `<input type="range" value={ozState.w1} onChange={...}>` |
|
||||
| Value spans `#w1Val`–`#w4Val` | `<span>{ozState.w1}%</span>` |
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 366–421: Maximizer View
|
||||
|
||||
| Spec Element | React |
|
||||
|---|---|
|
||||
| `#viewMaximizer` | `className={ozState.activeModule === 'maximizer' ? '' : 'hidden'}` |
|
||||
| Maximizer gain knob (`.knob-dial w-24 h-24`) | JSX verbatim, pointer events |
|
||||
| Ceiling level text | `<span>{ozState.ceiling.toFixed(2)} dB</span>` |
|
||||
| Upward compress knob | JSX verbatim + pointer events |
|
||||
| Soft clipper knob | JSX verbatim + pointer events |
|
||||
| Transient emphasis knob | JSX verbatim + pointer events |
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 425–480: Right Sidebar (I/O Meters + Utilities)
|
||||
|
||||
| Spec Element | React |
|
||||
|---|---|
|
||||
| `#inPeakText` / `#outPeakText` | `<span>{inPeak.toFixed(1)} dB</span>` etc. |
|
||||
| `#inMeterCanvas` / `#outMeterCanvas` | `<canvas ref={...}>` |
|
||||
| Bypass / Gain Match / Codec / Dither buttons | JSX verbatim |
|
||||
| Module active state checkboxes (power toggles) | Thêm `onClick` set `ozState.eqActive` etc. + gọi `updateAudioGraphValues` |
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 484–585: Audio Engine (JavaScript → React)
|
||||
|
||||
| Spec Function | React Mapping |
|
||||
|---|---|
|
||||
| `ozState` object | `const [ozState, setOzState] = useState({...})` — values copy verbatim |
|
||||
| `initOzoneAudioEngine()` | `useEffect` on mount: tạo AudioContext (dùng `window.SonicAudio?.getAudioContext()` nếu có), tạo AnalyserNode (fftSize=2048), BiquadFilter nodes (lowshelf 100Hz, peaking 822Hz Q0.7, peaking 3.2kHz Q1.2, highshelf 10kHz), GainNode boost, DynamicsCompressorNode (threshold -0.1, knee 0, ratio 20, attack 0.001, release 0.05), outputAnalyser |
|
||||
| Node graph connection | `inputAnalyser → eqLow → eqMid1 → eqMid2 → eqHigh → boostGain → compressor → outputAnalyser → destination` |
|
||||
| `updateAudioGraphValues()` | Copy verbatim logic: `eqLowFilter.gain.setTargetAtTime(ozState.eqActive ? ozState.eqLowGain : 0, now, 0.01)` etc. |
|
||||
| `startAudioDemo()` | Copy verbatim: create 4-second stereo buffer, kick/snare/synth oscillator, loop playback |
|
||||
| `stopAudioDemo()` | Copy verbatim: `sourceNode.stop(); sourceNode.disconnect();` |
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 586–791: Canvas Renderers (JavaScript → React)
|
||||
|
||||
| Spec Canvas | React Mapping |
|
||||
|---|---|
|
||||
| `eqCanvas` (lines 680–738) | `useEffect` với `requestAnimationFrame` loop: draw freq grid lines, FFT spectrum bar fill từ `outputAnalyser.getByteFrequencyData`, draw composite EQ curve |
|
||||
| `imagerCanvas` (lines 742–765) | `requestAnimationFrame`: draw polar circle guides + random scatter dots with `ozState.w3` width |
|
||||
| `inMeterCanvas`/`outMeterCanvas` (lines 769–787) | `requestAnimationFrame`: `renderMeterBar()` with linear gradient green→yellow→red, dBFS numeric readout |
|
||||
| `resizeAll()` | `useEffect` + `window.addEventListener('resize', resizeAll)` |
|
||||
|
||||
---
|
||||
|
||||
### HTML lines 793–882: Interactions & Init (JavaScript → React)
|
||||
|
||||
| Spec | React |
|
||||
|---|---|
|
||||
| `setupKnobs()` (lines 794–843) | `useEffect` on mount: query `.knob-container`, attach pointerdown/move/up, update UI rotation + `ozState` + `updateAudioGraphValues` |
|
||||
| `setupModuleNavigation()` (lines 846–868) | Inline JSX click handlers thay vì event listeners |
|
||||
| `window.addEventListener('load', ...)` (lines 871–882) | `useEffect` on mount: init audio engine, setup event handlers, start canvas renderers |
|
||||
|
||||
---
|
||||
|
||||
## Task List
|
||||
|
||||
### Task 1: Append CSS classes vào app/templates/index.html
|
||||
Thêm ~30 dòng CSS vào `<style>` block (`index.html:28-245`):
|
||||
- `.oz-panel` `oz-card` `oz-card-active`
|
||||
- `.knob-container` `.knob-dial` `.knob-pointer`
|
||||
- Scrollbar dark theme
|
||||
- `.font-mono` JetBrains Mono override
|
||||
- Gradient panel backgrounds
|
||||
|
||||
### Task 2: Thêm state + menu item trong app.jsx
|
||||
- `showMasteringModal` state (dòng 7708)
|
||||
- Menu item "Mastering Suite" (dòng 15585)
|
||||
- Render `MasteringModal` (dòng 18664)
|
||||
|
||||
### Task 3: Viết MasteringModal component trong app.jsx (trước dòng 6694)
|
||||
~500 dòng gồm:
|
||||
|
||||
**3a. State**: `ozState` (activeModule, gains, widths, maximizer params), `isPlaying`, `canvasRefs`, `audioRefs`
|
||||
|
||||
**3b. useEffect (mount)**: `initOzoneAudioEngine()` — copy verbatim từ spec lines 529-584, dùng `window.SonicAudio?.getAudioContext()` fallback
|
||||
|
||||
**3c. useEffect (mount)**: `setupKnobs()` — port pointer events từ spec lines 794-843, gọi `updateAudioGraphValues()` khi giá trị thay đổi
|
||||
|
||||
**3d. useEffect (mount + deps)**: Canvas render loop — port `startCanvasRenderers()` từ spec lines 650-791, sử dụng `ozState` thay vì global
|
||||
|
||||
**3e. useEffect (cleanup)**: Stop audio + disconnect nodes khi unmount
|
||||
|
||||
**3f. JSX return**: ~250 dòng JSX, dịch sát spec HTML lines 121-480:
|
||||
- Overlay modal container `fixed inset-0 z-[100] bg-black/60 backdrop-blur-sm`
|
||||
- Bên trong: `flex flex-col` container với `h-[90vh] w-[90vw] max-w-[1400px]`
|
||||
- Header bar (logo, transport, preset select, LUFS)
|
||||
- Module chain strip (3 module cards + add slot, `oz-card-active` toggle)
|
||||
- Main workspace:
|
||||
- Sub toolbar (Delta listen, IRC mode select, Learn Input Gain)
|
||||
- `viewDynamicEq` (EQ canvas + 4-band knob panel, show khi `activeModule==='eq'`)
|
||||
- `viewImager` (vectorscope canvas + 4-band sliders, show khi `activeModule==='imager'`)
|
||||
- `viewMaximizer` (gain knob, upward comp, soft clip, transient knobs, show khi `activeModule==='maximizer'`)
|
||||
- Right sidebar I/O meters (in/out peak text, meter canvases, bypass/gain match/codec/dither buttons)
|
||||
- Close button (góc trên phải)
|
||||
|
||||
### Task 4: Build & Verify
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
Kiểm tra: mở app → menu → "Mastering Suite" → modal hiển thị, Play Reference Loop phát âm thanh, canvas vẽ, knobs kéo được.
|
||||
|
||||
## Files Affected
|
||||
| File | Lines | Change |
|
||||
|------|-------|--------|
|
||||
| `app/templates/index.html` | +30 | Thêm CSS classes (oz-panel, oz-card, knob-dial, scrollbar) |
|
||||
| `app/static/js/app.jsx` | ~dòng 6690 + 7708 + 15585 + 18664 | Thêm MasteringModal component + state + menu item + render |
|
||||
|
||||
## Skipped (PONYTALL)
|
||||
- Real M/S processing: phase 1 chỉ UI + vectorscope viz. Thêm actual M/S khi user upload file stereo thật.
|
||||
- Delta Listen / Gain Match / Codec / Dither: UI button renders, DSP stubbed. Thêm sau.
|
||||
- Drag-drop module reorder: spec define UX. Phase 1: fixed chain.
|
||||
- Preset save/load: UI dropdown + defaults. Backend integration sau.
|
||||
+10
@@ -18,18 +18,28 @@ RUN apt-get update && apt-get install -y \
|
||||
xvfb \
|
||||
ffmpeg \
|
||||
libsndfile1 \
|
||||
libfluidsynth3 \
|
||||
libcurl4 \
|
||||
build-essential \
|
||||
fluidsynth \
|
||||
vorbis-tools \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Thiết lập biến môi trường hiển thị cho X11 ảo
|
||||
ENV DISPLAY=:99
|
||||
|
||||
# Create VST3 and sample directories
|
||||
RUN mkdir -p /opt/daw_engine/vst3 /opt/daw_engine/soundfonts /opt/daw_engine/samples/pianobook
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
# Copy VST3 plugins if present
|
||||
COPY ./vst_plugins/ /opt/daw_engine/vst3/
|
||||
|
||||
# Tạo thư mục chứa file nhạc và cấp quyền ghi
|
||||
RUN mkdir -p /app/app/storage/uploads /app/app/storage/processed && chmod -R 777 /app/app/storage
|
||||
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
|
||||
SonicForge Studio là một hệ thống xử lý âm thanh chuyên nghiệp kết hợp giao diện Web Audio API phía client với công cụ DSP/AI mạnh mẽ trên server (Python/Celery).
|
||||
|
||||

|
||||
|
||||
## 🎯 Tính Năng Chính
|
||||
|
||||
### Client-side (Web Audio API)
|
||||
|
||||
+163
@@ -124,6 +124,169 @@ celery -A app.tasks.worker.celery_app worker --loglevel=info
|
||||
4. File sẽ download tự động
|
||||
5. Mở file bằng audio player để kiểm tra
|
||||
|
||||
## 🎹 FluidSynth WASM Migration — Manual Test Plan
|
||||
|
||||
### Môi trường
|
||||
- Mở DevTools Console (F12) → Tab Console (bật `Verbose` để thấy `[SonicSF]` logs)
|
||||
- Tab Network: filter `fluidsynth`, `sf3`, `.wasm`
|
||||
|
||||
### Test A: FluidSynth WASM Load & Init
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| A1 | Mở http://localhost:8000 | Console: `[FluidSynth] Loaded from: https://cdn.jsdelivr.net/...` |
|
||||
| A2 | Kiểm tra Network tab | `.wasm` file tải thành công (status 200) |
|
||||
| A3 | Check window.__FluidSynthModuleFactory | `typeof window.__FluidSynthModuleFactory === 'function'` |
|
||||
| A4 | Tương tác với app (click vào DAW) | Console: `[SonicSF] FluidSynth WASM Engine initialized.` |
|
||||
| A5 | Kiểm tra AudioWorklet | Console: `Worklet reg success` hoặc check `audioWorklet` trong Application tab |
|
||||
|
||||
### Test B: SoundFont Loading
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| B1 | Mở Plugin Manager → tab SoundFont | Danh sách SoundFont hiển thị |
|
||||
| B2 | Chọn 1 SoundFont instrument (vd: Piano) | Console: `[SonicSF] SoundFont loaded: <sfId>` |
|
||||
| B3 | Kiểm tra Network tab | Request `download/<sfId>` status 200 |
|
||||
| B4 | Chuyển đổi instrument khác (vd: Violin) | Console: `Bank Select + Program Change` (nếu cùng SF, không tải lại) |
|
||||
| B5 | Load SoundFont có loop samples (vd: Tremolo Strings, Pad, Synth) | loadSoundFont success, không lỗi |
|
||||
|
||||
### Test C: Piano Roll Playback
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| C1 | Mở Piano Roll tab | Grid hiển thị notes |
|
||||
| C2 | Click vào 1 note trên grid | Note phát ra → âm thanh giống nhạc cụ thật (không phải oscillator beep) |
|
||||
| C3 | Click và kéo thả chuột trên grid → draw note mới | Âm thanh phát ngay lập tức |
|
||||
| C4 | Scroll wheel trên piano roll notes | Các note scroll qua phát âm thanh preview |
|
||||
| C5 | Vẽ note dài (full measure) | Note kéo dài đúng độ dài, không bị tắt giữa chừng |
|
||||
|
||||
### Test D: MIDI Keyboard (Virtual & Hardware)
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| D1 | Click vào phím đàn virtual (piano keybed) | Note phát ra = âm thanh instrument đúng |
|
||||
| D2 | Kéo chuột ngang trên keybed | Các note phát liên tục, glide không bị stuck |
|
||||
| D3 | Kết nối MIDI keyboard qua WebMIDI | Console: `MIDI access granted` |
|
||||
| D4 | Nhấn phím trên MIDI keyboard | Note phát ra ngay, không delay |
|
||||
| D5 | Nhả phím MIDI | Note tắt ngay (không stuck, không sustain dài) |
|
||||
| D6 | Sustain pedal (CC 64) | Nhấn pedal → notes sustain; nhả → notes release |
|
||||
| D7 | Pitch bend wheel | Cao độ thay đổi real-time |
|
||||
| D8 | Modulation wheel (CC 1) | Âm thanh thay đổi (nếu instrument hỗ trợ) |
|
||||
|
||||
### Test E: Timeline Playback (MIDI Tracks)
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| E1 | Tạo track MIDI mới | Track được tạo |
|
||||
| E2 | Gán SoundFont instrument cho track | Console log bank/program change |
|
||||
| E3 | Thêm MIDI notes vào track, click Play | Notes phát đúng pitch, đúng thời điểm, đúng instrument |
|
||||
| E4 | Click Pause → Play | Nhạc tiếp tục từ vị trí pause |
|
||||
| E5 | Click Stop | Tất cả notes tắt ngay lập tức |
|
||||
| E6 | Seek playhead → Play | Play từ vị trí mới, notes cũ tắt |
|
||||
| E7 | Set loop region → Play | Loop playback hoạt động |
|
||||
| E8 | Chuyển track instrument khác → Play | Âm thanh thay đổi theo instrument mới |
|
||||
|
||||
### Test F: Tremolo/Sustain/Loop Instrument Stress Test
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| F1 | Chọn Tremolo Strings (GM#44) | Load SF thành công |
|
||||
| F2 | Play note → nhanh chóng NoteOff | **QUAN TRỌNG**: Note tắt ngay, không bị stuck loop |
|
||||
| F3 | Play nhiều note liên tiếp (staccato) | Mỗi note tắt hẳn trước khi note kế phát |
|
||||
| F4 | Chọn Saxophone (GM#65-67) | Load SF thành công |
|
||||
| F5 | Play note giữ 3s → NoteOff | Saxophone release envelope chạy đúng, không stuck |
|
||||
| F6 | Chọn Pad/Synth (GM#88-95) | Các instrument loop dài không bị stuck |
|
||||
| F7 | Play 10+ notes cùng lúc → Stop All | Tất cả notes tắt ngay |
|
||||
| F8 | **So sánh**: Test F1-F7 cũ: SpessaSynth bị stuck notes cần CC120+noteOn+post. FluidSynth WASM: chỉ cần noteOff thường. | FluidSynth handle loop Gen 54 đúng spec |
|
||||
|
||||
### Test G: Multi-SoundFont Switching
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| G1 | Load SoundFont A (vd: GeneralUser) | Handle ID log |
|
||||
| G2 | Chuyển track sang instrument từ SF A | SF A active |
|
||||
| G3 | Tạo track 2, load SoundFont B (vd: SGM) | SF B load vào MEMFS |
|
||||
| G4 | Play track 1 (SF A) + track 2 (SF B) | Cả 2 soundfont phát đồng thời, mỗi track instrument đúng |
|
||||
| G5 | Unload SF A, load SF C | SF A đã unload, SF C active |
|
||||
|
||||
### Test H: Transport Controls
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| H1 | Đang play → click Stop | Console: `FluidSynth: All notes stopped.` |
|
||||
| H2 | Play với nhiều notes đang vang → Stop | Âm thanh tắt ngay lập tức (CC 120 all sound off) |
|
||||
| H3 | Play → Pause → Seek → Play | Seek không bị stuck notes |
|
||||
| H4 | Play → Reload trang | Audio context mới, FluidSynth init lại |
|
||||
|
||||
### Test I: Fallback Behavior
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| I1 | Chặn CDN request (DevTools → Network → Offline) | `fluidsynthLoader.js` detect localhost → vẫn dùng CDN? Set `window.__FLUIDSYNTH_CDN` = null |
|
||||
| I2 | Nếu FluidSynth init fail | Console: `FluidSynth init failed`. Fallback oscillator hoạt động (âm beep) |
|
||||
| I3 | Nếu loadSoundFont fail (network down) | Console: `SoundFont not found`. Fallback oscillator cho note preview |
|
||||
|
||||
### Test J: Memory & Performance
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| J1 | Load SF lần đầu | Network download + MEMFS write + sfload |
|
||||
| J2 | Load lại SF lần 2 (đã cache IndexedDB) | `_sfHandleMap.has(sfId)` → true, skip download |
|
||||
| J3 | Check Performance tab (DevTools) | `_fluid_synth_write_float` không block main thread > 5ms |
|
||||
| J4 | Play liên tục 5 phút | Không memory leak, không audio glitch |
|
||||
| J5 | Load SF 3-4MB (SGM v2.01) | MEMFS write + sfload < 500ms |
|
||||
|
||||
### Test K: Audio Parity (Client vs Server)
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| K1 | Tạo project với MIDI notes + SoundFont | Client preview âm thanh |
|
||||
| K2 | Export WAV server-side (Render) | Server dùng pyfluidsynth (C++ core) |
|
||||
| K3 | **So sánh** WAV export vs Client preview | Giống nhau 100% (cùng FluidSynth engine) |
|
||||
| K4 | Test với SF3 files | Cả client (FluidSynth WASM) và server (pyfluidsynth) đều xử lý SF3 |
|
||||
|
||||
### Test L: Regression — Tính năng không thay đổi
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|----------------|
|
||||
| L1 | AI track generation | `applyAITrackInstrument` works → bank/program change |
|
||||
| L2 | Audio file playback | Không ảnh hưởng (vẫn dùng AudioEngine cũ) |
|
||||
| L3 | VST instrument tracks | Không ảnh hưởng (dùng VST engine riêng) |
|
||||
| L4 | Upload/download file | Không thay đổi |
|
||||
| L5 | Multi-track mix | Không thay đổi |
|
||||
|
||||
### Test Environment Setup
|
||||
|
||||
```bash
|
||||
# 1. Start server
|
||||
cd /home/locpham/SonicForgeStudio
|
||||
docker compose up -d --build
|
||||
|
||||
# 2. Clear browser cache trước khi test lần đầu (cache-bust version đã update)
|
||||
# Chrome: DevTools → Network → Disable cache (khi DevTools mở)
|
||||
|
||||
# 3. Kiểm tra console logs
|
||||
# Mở DevTools Console, filter: [SonicSF] [FluidSynth]
|
||||
|
||||
# 4. Force re-download SF (xóa IndexedDB cache nếu cần)
|
||||
# Application → IndexedDB → DAW_SoundFont_Cache → Clear
|
||||
```
|
||||
|
||||
### Checklist
|
||||
|
||||
- [ ] A1-A5: FluidSynth WASM load + init
|
||||
- [ ] B1-B5: SoundFont load + switch (nhiều SF)
|
||||
- [ ] C1-C5: Piano roll note play
|
||||
- [ ] D1-D8: MIDI keyboard (virtual + hardware)
|
||||
- [ ] E1-E8: Timeline playback
|
||||
- [ ] F1-F8: **Tremolo/Sustain loop stress** — key test
|
||||
- [ ] G1-G5: Multi-SoundFont switching
|
||||
- [ ] H1-H4: Transport controls (stop, seek)
|
||||
- [ ] I1-I3: Fallback oscillator
|
||||
- [ ] J1-J5: Memory & performance
|
||||
- [ ] K1-K4: Audio parity client vs server
|
||||
- [ ] L1-L5: Regression (features không thay đổi)
|
||||
|
||||
## 🐛 Known Issues
|
||||
|
||||
### Docker Environment
|
||||
|
||||
+30
-1
@@ -20,6 +20,12 @@ class UpdateUserRoleRequest(BaseModel):
|
||||
role: str # 'admin', 'standard', 'premium'
|
||||
is_active: Optional[bool] = True
|
||||
|
||||
class CreateUserRequest(BaseModel):
|
||||
username: str
|
||||
email: str
|
||||
password: str
|
||||
role: Optional[str] = 'standard'
|
||||
|
||||
@router.get("/users")
|
||||
async def list_users(admin: dict = Depends(require_admin)):
|
||||
conn = get_db_connection()
|
||||
@@ -52,6 +58,29 @@ async def list_users(admin: dict = Depends(require_admin)):
|
||||
})
|
||||
return users
|
||||
|
||||
@router.post("/users")
|
||||
async def create_user(req: CreateUserRequest, admin: dict = Depends(require_admin)):
|
||||
import uuid, time
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT id FROM users WHERE username = ? OR email = ?", (req.username, req.email))
|
||||
if cursor.fetchone():
|
||||
conn.close()
|
||||
raise HTTPException(status_code=409, detail="Tên đăng nhập hoặc email đã tồn tại")
|
||||
user_id = str(uuid.uuid4())
|
||||
hashed = hash_password(req.password)
|
||||
now = time.time()
|
||||
cursor.execute("""
|
||||
INSERT INTO users (id, username, email, hashed_password, role, must_change_password, created_at, is_active)
|
||||
VALUES (?, ?, ?, ?, ?, 1, ?, 1)
|
||||
""", (user_id, req.username, req.email, hashed, req.role, now))
|
||||
cursor.execute("""
|
||||
INSERT INTO user_quotas (user_id, storage_limit_mb, max_tracks) VALUES (?, 500, 16)
|
||||
""", (user_id,))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"message": f"Đã tạo người dùng '{req.username}' thành công", "user_id": user_id}
|
||||
|
||||
@router.put("/users/{user_id}/role")
|
||||
async def update_user_role(user_id: str, req: UpdateUserRoleRequest, admin: dict = Depends(require_admin)):
|
||||
conn = get_db_connection()
|
||||
@@ -81,7 +110,7 @@ async def delete_user(user_id: str, admin: dict = Depends(require_admin)):
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM users WHERE id = ?", (user_id,))
|
||||
cursor.execute("DELETE FROM user_quotas WHERE id = ?", (user_id,))
|
||||
cursor.execute("DELETE FROM user_quotas WHERE user_id = ?", (user_id,))
|
||||
cursor.execute("DELETE FROM projects WHERE user_id = ?", (user_id,))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
import json, os, time
|
||||
from fastapi import APIRouter, HTTPException, Header
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, List
|
||||
from app.core.auth import decode_token
|
||||
from app.config import settings
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
DATA_FILE = os.path.join(settings.PROCESSED_DIR, "ai_presets.json")
|
||||
|
||||
DEFAULT_PRESETS = [
|
||||
{
|
||||
"id": "preset_epic_orchestra_intro",
|
||||
"name": "Epic Orchestra Intro (8 Bars)",
|
||||
"keywords": ["epic orchestra", "epic orchestral", "hoành tráng", "nhạc phim epic"],
|
||||
"category": "Orchestral / Film Score",
|
||||
"default_bars": 8,
|
||||
"default_bpm": 130,
|
||||
"default_scale": "C Minor",
|
||||
"system_instruction_template": "You are a professional film composer. Create a powerful, dramatic 8-bar orchestral intro. Keep the note density low (e.g. use mostly whole notes, half notes, or quarter notes) and do NOT generate dense 16th notes or complex drum rolls. This is critical to avoid output token limit timeouts. The required structure to return via the `generate_multitrack_midi` tool consists of 3 tracks: 1. Strings: plays smooth legato chord changes (one chord per 1 or 2 bars). 2. Brass Theme: plays a swelling simple melodic line in the C3-C5 range. 3. Epic Percussion: hits heavily on beats 1 and 3. Ensure the duration is precisely 8 bars (32 beats).",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": False,
|
||||
"created_at": "2026-07-23T16:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "preset_pop_piano_chords",
|
||||
"name": "Pop Piano Chords (4 Bars)",
|
||||
"keywords": ["pop piano", "piano chords", "ballad piano", "hợp âm piano"],
|
||||
"category": "Pop / Ballad",
|
||||
"default_bars": 4,
|
||||
"default_bpm": 90,
|
||||
"default_scale": "C Major",
|
||||
"system_instruction_template": "You are a professional Pop Piano player. Generate a beautiful 4-bar piano chord progression (e.g. C - G - Am - F) with pleasant chord voicing and simple accompaniment. Return the MIDI notes via `generate_multitrack_midi` function on a track named 'Pop Piano'. Keep notes simple, using mostly whole/half/quarter notes. Ensure the duration of the track is precisely 4 bars (16 beats).",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": False,
|
||||
"created_at": "2026-07-23T16:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "preset_cyberpunk_synth",
|
||||
"name": "Cyberpunk Synthwave (8 Bars)",
|
||||
"keywords": ["cyberpunk synth", "synthwave", "cyberpunk", "futuristic synth"],
|
||||
"category": "Electronic / Synthwave",
|
||||
"default_bars": 8,
|
||||
"default_bpm": 120,
|
||||
"default_scale": "A Minor",
|
||||
"system_instruction_template": "You are a Synthwave producer. Generate a driving 8-bar cyberpunk synth theme. Return MIDI notes via `generate_multitrack_midi` containing: 1. Synth Bass: eighth notes on pitch A1, C2, G1. 2. Synth Lead: simple melodic line in high register C4-E5. Keep notes clean and concise to ensure fast generation.",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": False,
|
||||
"created_at": "2026-07-23T16:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "preset_rearrange_jazz_swing",
|
||||
"name": "Jazz Swing Variation",
|
||||
"keywords": ["jazz", "swing", "rearrange to jazz", "jazz improv"],
|
||||
"category": "Rearrange / Variation",
|
||||
"default_bars": 4,
|
||||
"default_bpm": 120,
|
||||
"default_scale": "C Major",
|
||||
"system_instruction_template": "Rearrange the original melody into a rhythmic Jazz Swing style. Apply syncopation, 7th/9th chord extensions, and off-beat rhythmic feel.",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": True,
|
||||
"created_at": "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "preset_rearrange_synth_arpeggio",
|
||||
"name": "80s Synthwave Arpeggiator",
|
||||
"keywords": ["synthwave", "arpeggio", "arp", "16th variation", "80s synth"],
|
||||
"category": "Rearrange / Variation",
|
||||
"default_bars": 4,
|
||||
"default_bpm": 120,
|
||||
"default_scale": "C Minor",
|
||||
"system_instruction_template": "Transform this sequence into a driving, shimmering 16th-note Arpeggio string in classic 80s Synthwave/Trance style.",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": False,
|
||||
"created_at": "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "preset_rearrange_harmonize",
|
||||
"name": "Chords Harmonization (4-Note)",
|
||||
"keywords": ["harmonize", "harmony", "add chord notes", "rich melody"],
|
||||
"category": "Rearrange / Variation",
|
||||
"default_bars": 4,
|
||||
"default_bpm": 120,
|
||||
"default_scale": "C Major",
|
||||
"system_instruction_template": "Add harmonizing notes to create lush 4-note chord voicing extensions beneath the lead melody.",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": False,
|
||||
"created_at": "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "preset_rearrange_cinematic_strings",
|
||||
"name": "Cinematic Strings Staccato",
|
||||
"keywords": ["cinematic", "strings", "staccato", "staccato strings"],
|
||||
"category": "Rearrange / Variation",
|
||||
"default_bars": 4,
|
||||
"default_bpm": 120,
|
||||
"default_scale": "C Minor",
|
||||
"system_instruction_template": "Rearrange the original line into a dramatic, driving Staccato rhythm for a Strings Ensemble.",
|
||||
"is_user_defined": False,
|
||||
"is_favorite": True,
|
||||
"created_at": "2026-07-28T11:00:00Z"
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
class AIPresetSchema(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
keywords: List[str]
|
||||
category: str = "General"
|
||||
default_bars: int = 8
|
||||
default_bpm: int = 120
|
||||
default_scale: str = "C Major"
|
||||
system_instruction_template: str
|
||||
is_user_defined: bool = True
|
||||
is_favorite: bool = False
|
||||
created_at: Optional[str] = None
|
||||
|
||||
|
||||
def _load_data():
|
||||
if not os.path.exists(DATA_FILE):
|
||||
return {"user_presets": {}}
|
||||
try:
|
||||
with open(DATA_FILE, "r") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return {"user_presets": {}}
|
||||
|
||||
|
||||
def _save_data(user_presets):
|
||||
os.makedirs(os.path.dirname(DATA_FILE), exist_ok=True)
|
||||
with open(DATA_FILE, "w") as f:
|
||||
json.dump({"user_presets": user_presets}, f, indent=2)
|
||||
|
||||
|
||||
def _get_user_id(authorization):
|
||||
if not authorization or not authorization.startswith("Bearer "):
|
||||
return "anonymous"
|
||||
token = authorization.split(" ")[1]
|
||||
payload = decode_token(token)
|
||||
if not payload:
|
||||
return "anonymous"
|
||||
return payload.get("user_id", "anonymous")
|
||||
|
||||
|
||||
@router.get("/presets")
|
||||
async def list_presets(authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
data = _load_data()
|
||||
user_presets = data.get("user_presets", {}).get(uid, [])
|
||||
merged = DEFAULT_PRESETS + user_presets
|
||||
return {"success": True, "presets": merged}
|
||||
|
||||
|
||||
@router.post("/presets")
|
||||
async def save_preset(req: AIPresetSchema, authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
if uid == "anonymous":
|
||||
raise HTTPException(status_code=401, detail="Authentication required to save presets")
|
||||
|
||||
data = _load_data()
|
||||
user_presets = data.get("user_presets", {}).get(uid, [])
|
||||
|
||||
if not req.created_at:
|
||||
req.created_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
|
||||
existing_idx = next((i for i, p in enumerate(user_presets) if p["id"] == req.id), None)
|
||||
preset_dict = req.model_dump()
|
||||
|
||||
if existing_idx is not None:
|
||||
user_presets[existing_idx] = preset_dict
|
||||
else:
|
||||
user_presets.append(preset_dict)
|
||||
|
||||
if "user_presets" not in data:
|
||||
data["user_presets"] = {}
|
||||
data["user_presets"][uid] = user_presets
|
||||
_save_data(data["user_presets"])
|
||||
return {"success": True, "preset": preset_dict}
|
||||
|
||||
|
||||
@router.delete("/presets/{preset_id}")
|
||||
async def delete_preset(preset_id: str, authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
if uid == "anonymous":
|
||||
raise HTTPException(status_code=401, detail="Authentication required to delete presets")
|
||||
|
||||
data = _load_data()
|
||||
user_presets = data.get("user_presets", {}).get(uid, [])
|
||||
filtered = [p for p in user_presets if p["id"] != preset_id]
|
||||
|
||||
if len(filtered) == len(user_presets):
|
||||
raise HTTPException(status_code=404, detail="Preset not found")
|
||||
|
||||
if "user_presets" not in data:
|
||||
data["user_presets"] = {}
|
||||
data["user_presets"][uid] = filtered
|
||||
_save_data(data["user_presets"])
|
||||
return {"success": True, "message": "Preset deleted"}
|
||||
@@ -0,0 +1,142 @@
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import json
|
||||
import socket
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Any, Dict
|
||||
|
||||
from app.api.v1.auth import get_current_user
|
||||
from app.api.v1.user_config import _load_ai_configs, _get_default_providers
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
class ProxyRequest(BaseModel):
|
||||
url: str
|
||||
headers: Dict[str, str] = {}
|
||||
body: Dict[str, Any] = {}
|
||||
|
||||
|
||||
# Ranges that are never legitimate AI endpoints: cloud metadata + this host.
|
||||
_BLOCKED_NETWORKS = [
|
||||
ipaddress.ip_network("169.254.0.0/16"), # link-local / cloud metadata
|
||||
ipaddress.ip_network("0.0.0.0/8"),
|
||||
]
|
||||
# Private ranges: only reachable when the target host is one the user has
|
||||
# explicitly configured as an AI provider (e.g. local Ollama/LM Studio).
|
||||
_PRIVATE_NETWORKS = [
|
||||
ipaddress.ip_network("10.0.0.0/8"),
|
||||
ipaddress.ip_network("172.16.0.0/12"),
|
||||
ipaddress.ip_network("192.168.0.0/16"),
|
||||
ipaddress.ip_network("127.0.0.0/8"),
|
||||
ipaddress.ip_network("::1/128"),
|
||||
ipaddress.ip_network("fc00::/7"), # ULA
|
||||
]
|
||||
|
||||
_LOOPBACK_HOSTS = {"localhost", "127.0.0.1", "::1", "0.0.0.0"}
|
||||
|
||||
|
||||
def _configured_ai_hosts(user_id: str) -> set:
|
||||
"""Hosts the user has configured as AI providers (from saved config + defaults)."""
|
||||
hosts = set()
|
||||
configs = _load_ai_configs()
|
||||
providers = configs.get(user_id) or _get_default_providers()
|
||||
for p in providers:
|
||||
base = (p.get("api_base_url") or "").strip()
|
||||
if not base:
|
||||
continue
|
||||
try:
|
||||
host = urlparse(base).hostname
|
||||
if host:
|
||||
hosts.add(host.lower())
|
||||
except Exception:
|
||||
continue
|
||||
return hosts
|
||||
|
||||
|
||||
async def _resolve_host_ips(hostname: str):
|
||||
"""Resolve hostname to IPs (non-blocking). Returns list of ipaddress objects."""
|
||||
loop = asyncio.get_event_loop()
|
||||
try:
|
||||
infos = await loop.run_in_executor(None, socket.getaddrinfo, hostname, None)
|
||||
ips = []
|
||||
for info in infos:
|
||||
try:
|
||||
ips.append(ipaddress.ip_address(info[4][0]))
|
||||
except ValueError:
|
||||
continue
|
||||
return ips
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
async def _validate_target_url(url: str, user_id: str):
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
raise HTTPException(status_code=400, detail="URL chỉ hỗ trợ giao thức http/https")
|
||||
if parsed.username or parsed.password:
|
||||
raise HTTPException(status_code=400, detail="URL không được chứa thông tin đăng nhập")
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
if not hostname:
|
||||
raise HTTPException(status_code=400, detail="URL không hợp lệ")
|
||||
|
||||
allowed_hosts = _configured_ai_hosts(user_id)
|
||||
|
||||
# Hostname-level fast path for loopback hosts
|
||||
if hostname in _LOOPBACK_HOSTS:
|
||||
if hostname in allowed_hosts:
|
||||
return
|
||||
raise HTTPException(status_code=403, detail="Target nội bộ không nằm trong danh sách AI provider đã cấu hình")
|
||||
|
||||
# Try direct IP parse (hostname may itself be an IP)
|
||||
try:
|
||||
ip = ipaddress.ip_address(hostname)
|
||||
ips = [ip]
|
||||
except ValueError:
|
||||
ips = await _resolve_host_ips(hostname)
|
||||
|
||||
if not ips:
|
||||
raise HTTPException(status_code=502, detail="Không phân giải được hostname")
|
||||
|
||||
for ip in ips:
|
||||
if any(ip in net for net in _BLOCKED_NETWORKS):
|
||||
raise HTTPException(status_code=403, detail="Target bị chặn (metadata/link-local không được phép)")
|
||||
if any(ip in net for net in _PRIVATE_NETWORKS):
|
||||
if hostname in allowed_hosts:
|
||||
continue
|
||||
raise HTTPException(status_code=403, detail="Target IP nội bộ không nằm trong danh sách AI provider đã cấu hình")
|
||||
|
||||
|
||||
@router.post("/proxy")
|
||||
async def proxy_llm(req: ProxyRequest, current_user: dict = Depends(get_current_user)):
|
||||
await _validate_target_url(req.url, current_user["user_id"])
|
||||
# Never forward the app's own auth token upstream.
|
||||
headers = {
|
||||
k: v for k, v in req.headers.items()
|
||||
if k.lower() not in ("host", "origin", "referer", "x-auth-token")
|
||||
}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=180.0, follow_redirects=False) as client:
|
||||
resp = await client.post(req.url, headers=headers, json=req.body)
|
||||
raw = resp.text
|
||||
try:
|
||||
return resp.json()
|
||||
except json.JSONDecodeError:
|
||||
try:
|
||||
return json.loads(raw[:raw.find('\n')])
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return {"content": raw}
|
||||
except httpx.TimeoutException:
|
||||
raise HTTPException(status_code=504, detail="AI provider timeout")
|
||||
except httpx.ConnectError as e:
|
||||
msg = f"Cannot connect to AI provider: {e}"
|
||||
if 'localhost' in req.url or '127.0.0.1' in req.url:
|
||||
msg += "\nNếu app chạy trong Docker, localhost trỏ vào container, không ra host.\nHãy thay localhost bằng host.docker.internal hoặc IP bridge Docker (172.17.0.1)."
|
||||
raise HTTPException(status_code=502, detail=msg)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
+193
-81
@@ -1,14 +1,39 @@
|
||||
import os
|
||||
import uuid
|
||||
import asyncio
|
||||
from fastapi import APIRouter, UploadFile, File, HTTPException, Query
|
||||
import json
|
||||
from fastapi import APIRouter, UploadFile, File, HTTPException, Query, Depends
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional
|
||||
from typing import Optional, List
|
||||
import json
|
||||
from app.config import settings
|
||||
from app.api.v1.auth import get_current_user, enforce_password_changed
|
||||
from app.api.v1.projects import get_optional_user
|
||||
from app.models.user import get_db_connection
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
MAX_AUDIO_UPLOAD_BYTES = 1024 * 1024 * 1024 # 1 GB
|
||||
|
||||
def _safe_file_id(file_id: str) -> str:
|
||||
"""Strip any path components from a client-supplied file id."""
|
||||
if not file_id:
|
||||
return ""
|
||||
return os.path.basename(file_id.replace("\\", "/"))
|
||||
|
||||
def _resolve_storage_path(file_id: str) -> str:
|
||||
"""Return the existing file path (processed first, then uploads) for a
|
||||
sanitized file id, or '' when not found."""
|
||||
fid = _safe_file_id(file_id)
|
||||
if not fid:
|
||||
return ""
|
||||
for d in (settings.PROCESSED_DIR, settings.UPLOADS_DIR):
|
||||
p = os.path.join(d, fid)
|
||||
if os.path.isfile(p):
|
||||
return p
|
||||
return ""
|
||||
|
||||
class EditRequest(BaseModel):
|
||||
file_id: str
|
||||
cut_start_ms: Optional[float] = None
|
||||
@@ -52,17 +77,43 @@ class PythonToolRequest(BaseModel):
|
||||
wave_type: Optional[str] = "sine"
|
||||
|
||||
@router.post("/upload")
|
||||
async def upload_audio(file: UploadFile = File(...)):
|
||||
ext = os.path.splitext(file.filename)[1]
|
||||
async def upload_audio(file: UploadFile = File(...), current_user: Optional[dict] = Depends(get_optional_user)):
|
||||
if current_user:
|
||||
enforce_password_changed(current_user)
|
||||
user_id = current_user["user_id"] if current_user else "anonymous"
|
||||
ext = os.path.splitext(file.filename or "")[1]
|
||||
if not ext:
|
||||
ext = ".wav"
|
||||
file_id = f"{uuid.uuid4()}{ext}"
|
||||
file_id = f"user_{user_id}_{uuid.uuid4()}{ext}"
|
||||
file_path = os.path.join(settings.UPLOADS_DIR, file_id)
|
||||
|
||||
# Stream upload in chunks with a hard size cap (avoids loading a multi-GB
|
||||
# WAV into RAM and bounds disk usage).
|
||||
with open(file_path, "wb") as f:
|
||||
content = await file.read()
|
||||
f.write(content)
|
||||
|
||||
size = 0
|
||||
while True:
|
||||
chunk = await file.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
size += len(chunk)
|
||||
if size > MAX_AUDIO_UPLOAD_BYTES:
|
||||
f.close()
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise HTTPException(status_code=413, detail="File âm thanh quá lớn (giới hạn 1GB)")
|
||||
f.write(chunk)
|
||||
|
||||
# Save original filename as sidecar metadata
|
||||
import json
|
||||
meta_path = os.path.join(settings.UPLOADS_DIR, file_id + ".meta")
|
||||
try:
|
||||
with open(meta_path, "w") as mf:
|
||||
json.dump({"original_name": file.filename}, mf)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Trigger celery task
|
||||
from app.tasks.worker import analyze_audio_task
|
||||
task = analyze_audio_task.delay(file_id)
|
||||
@@ -75,15 +126,12 @@ async def upload_audio(file: UploadFile = File(...)):
|
||||
|
||||
@router.post("/edit")
|
||||
async def edit_audio(req: EditRequest):
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, req.file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, req.file_id)
|
||||
|
||||
# Use uploaded file if it exists, or look in processed if it was already edited
|
||||
if not os.path.exists(upload_path) and not os.path.exists(processed_path):
|
||||
if not _resolve_storage_path(req.file_id):
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
from app.tasks.worker import edit_audio_task
|
||||
task = edit_audio_task.delay(req.dict())
|
||||
task = edit_audio_task.delay(req.model_dump())
|
||||
|
||||
return {
|
||||
"task_id": task.id
|
||||
@@ -91,15 +139,10 @@ async def edit_audio(req: EditRequest):
|
||||
|
||||
@router.get("/download/{file_id}")
|
||||
async def download_audio(file_id: str):
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, file_id)
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, file_id)
|
||||
|
||||
if os.path.exists(processed_path):
|
||||
return FileResponse(processed_path, media_type="audio/wav", filename=file_id)
|
||||
elif os.path.exists(upload_path):
|
||||
return FileResponse(upload_path, media_type="audio/wav", filename=file_id)
|
||||
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
path = _resolve_storage_path(file_id)
|
||||
if not path:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
return FileResponse(path, media_type="audio/wav", filename=os.path.basename(path))
|
||||
|
||||
@router.get("/waveform/{file_id}")
|
||||
async def get_waveform(file_id: str, num_peaks: int = Query(default=800, ge=50, le=4000)):
|
||||
@@ -107,14 +150,8 @@ async def get_waveform(file_id: str, num_peaks: int = Query(default=800, ge=50,
|
||||
API endpoint vẽ Peak Waveform đồng bộ (Week 2).
|
||||
Trả về dữ liệu peak waveform cho hiển thị đồ thị sóng âm trên Frontend.
|
||||
"""
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, file_id)
|
||||
|
||||
if os.path.exists(processed_path):
|
||||
file_path = processed_path
|
||||
elif os.path.exists(upload_path):
|
||||
file_path = upload_path
|
||||
else:
|
||||
file_path = _resolve_storage_path(file_id)
|
||||
if not file_path:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
from app.core.dsp_utils import generate_peak_waveform
|
||||
@@ -125,14 +162,8 @@ async def get_waveform_rms(file_id: str, num_points: int = Query(default=800, ge
|
||||
"""
|
||||
API endpoint vẽ RMS Waveform (mượt hơn peak).
|
||||
"""
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, file_id)
|
||||
|
||||
if os.path.exists(processed_path):
|
||||
file_path = processed_path
|
||||
elif os.path.exists(upload_path):
|
||||
file_path = upload_path
|
||||
else:
|
||||
file_path = _resolve_storage_path(file_id)
|
||||
if not file_path:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
from app.core.dsp_utils import generate_rms_waveform
|
||||
@@ -144,26 +175,20 @@ async def analyze_audio_with_ai(req: AIAnalysisRequest):
|
||||
API endpoint phân tích cấu trúc khuôn nhạc bằng AI (Week 4).
|
||||
Gọi OpenAI Compatible API (DeepSeek/Ollama) để phân đoạn bố cục.
|
||||
"""
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, req.file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, req.file_id)
|
||||
|
||||
if os.path.exists(processed_path):
|
||||
file_path = processed_path
|
||||
elif os.path.exists(upload_path):
|
||||
file_path = upload_path
|
||||
else:
|
||||
file_path = _resolve_storage_path(req.file_id)
|
||||
if not file_path:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
from app.tasks.worker import analyze_ai_task
|
||||
task = analyze_ai_task.delay(
|
||||
file_id=req.file_id,
|
||||
file_id=_safe_file_id(req.file_id),
|
||||
api_base_url=req.api_base_url,
|
||||
model=req.model
|
||||
)
|
||||
|
||||
return {
|
||||
"task_id": task.id,
|
||||
"file_id": req.file_id
|
||||
"file_id": _safe_file_id(req.file_id)
|
||||
}
|
||||
|
||||
@router.post("/export")
|
||||
@@ -171,19 +196,13 @@ async def export_audio(req: ExportRequest):
|
||||
"""
|
||||
API endpoint xuất tệp âm thanh sang nhiều định dạng (WAV/MP3/OGG).
|
||||
"""
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, req.file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, req.file_id)
|
||||
|
||||
if os.path.exists(processed_path):
|
||||
source_path = processed_path
|
||||
elif os.path.exists(upload_path):
|
||||
source_path = upload_path
|
||||
else:
|
||||
source_path = _resolve_storage_path(req.file_id)
|
||||
if not source_path:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
from app.tasks.worker import export_audio_task
|
||||
task = export_audio_task.delay(
|
||||
file_id=req.file_id,
|
||||
file_id=_safe_file_id(req.file_id),
|
||||
format=req.format,
|
||||
sample_rate=req.sample_rate,
|
||||
bit_depth=req.bit_depth
|
||||
@@ -191,29 +210,24 @@ async def export_audio(req: ExportRequest):
|
||||
|
||||
return {
|
||||
"task_id": task.id,
|
||||
"file_id": req.file_id
|
||||
"file_id": _safe_file_id(req.file_id)
|
||||
}
|
||||
|
||||
@router.post("/ai-scan")
|
||||
async def ai_scan_audio(req: AIScanRequest):
|
||||
async def ai_scan_audio(req: AIScanRequest, current_user: Optional[dict] = Depends(get_optional_user)):
|
||||
"""
|
||||
17_AI_SCAN.md Feature 1: AI Loop Scan & Automated Marker Labeling.
|
||||
Uses AIDSPEngine to find optimal recurring loop region with zero-crossing alignment.
|
||||
"""
|
||||
if current_user:
|
||||
enforce_password_changed(current_user)
|
||||
from app.core.ai_dsp_engine import AIDSPEngine
|
||||
import soundfile as sf
|
||||
import numpy as np
|
||||
|
||||
file_path = None
|
||||
if req.file_id:
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, req.file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, req.file_id)
|
||||
if os.path.exists(processed_path):
|
||||
file_path = processed_path
|
||||
elif os.path.exists(upload_path):
|
||||
file_path = upload_path
|
||||
file_path = _resolve_storage_path(req.file_id) if req.file_id else ""
|
||||
|
||||
if file_path and os.path.exists(file_path):
|
||||
if file_path:
|
||||
data, sr = sf.read(file_path)
|
||||
if data.ndim > 1:
|
||||
data = data.T
|
||||
@@ -231,28 +245,24 @@ async def ai_scan_audio(req: AIScanRequest):
|
||||
}
|
||||
|
||||
@router.post("/ai-cut")
|
||||
async def ai_cut_audio(req: AICutRequest):
|
||||
async def ai_cut_audio(req: AICutRequest, current_user: Optional[dict] = Depends(get_optional_user)):
|
||||
"""
|
||||
17_AI_SCAN.md Feature 2: Fade-Free AI Cut (Zero-Crossing Aligned Slicing).
|
||||
Executes raw binary sample slice at exact zero-crossing coordinates.
|
||||
"""
|
||||
user_id = current_user["user_id"] if current_user else "anonymous"
|
||||
if current_user:
|
||||
enforce_password_changed(current_user)
|
||||
from app.core.ai_dsp_engine import AIDSPEngine
|
||||
import soundfile as sf
|
||||
import numpy as np
|
||||
|
||||
output_file_id = f"ai_cut_{uuid.uuid4().hex[:8]}.wav"
|
||||
output_file_id = f"user_{user_id}_ai_cut_{uuid.uuid4().hex[:8]}.wav"
|
||||
out_path = os.path.join(settings.PROCESSED_DIR, output_file_id)
|
||||
|
||||
file_path = None
|
||||
if req.file_id:
|
||||
upload_path = os.path.join(settings.UPLOADS_DIR, req.file_id)
|
||||
processed_path = os.path.join(settings.PROCESSED_DIR, req.file_id)
|
||||
if os.path.exists(processed_path):
|
||||
file_path = processed_path
|
||||
elif os.path.exists(upload_path):
|
||||
file_path = upload_path
|
||||
file_path = _resolve_storage_path(req.file_id) if req.file_id else ""
|
||||
|
||||
if file_path and os.path.exists(file_path):
|
||||
if file_path:
|
||||
data, sr = sf.read(file_path)
|
||||
if data.ndim > 1:
|
||||
data = data.T
|
||||
@@ -273,11 +283,14 @@ async def ai_cut_audio(req: AICutRequest):
|
||||
}
|
||||
|
||||
@router.post("/python-tool")
|
||||
async def run_python_dsp_tool(req: PythonToolRequest):
|
||||
async def run_python_dsp_tool(req: PythonToolRequest, current_user: Optional[dict] = Depends(get_optional_user)):
|
||||
"""
|
||||
Non-AI Python DSP Tools endpoint.
|
||||
Handles normalize peak, invert phase, swap channels, zero-crossing align, and synth wave generation.
|
||||
"""
|
||||
user_id = current_user["user_id"] if current_user else "anonymous"
|
||||
if current_user:
|
||||
enforce_password_changed(current_user)
|
||||
from app.core.python_tools_engine import PythonToolsEngine
|
||||
from app.core.ai_dsp_engine import AIDSPEngine
|
||||
import soundfile as sf
|
||||
@@ -285,7 +298,7 @@ async def run_python_dsp_tool(req: PythonToolRequest):
|
||||
|
||||
if req.tool_type == "synth_wave":
|
||||
wave = PythonToolsEngine.generate_synth_wave(req.wave_type or "sine", req.freq or 440.0, req.duration or 2.0)
|
||||
output_file_id = f"synth_{req.wave_type}_{uuid.uuid4().hex[:6]}.wav"
|
||||
output_file_id = f"user_{user_id}_synth_{req.wave_type}_{uuid.uuid4().hex[:6]}.wav"
|
||||
out_path = os.path.join(settings.PROCESSED_DIR, output_file_id)
|
||||
sf.write(out_path, wave, 44100)
|
||||
return {
|
||||
@@ -306,3 +319,102 @@ async def run_python_dsp_tool(req: PythonToolRequest):
|
||||
"success": True,
|
||||
"message": f"Python Tool '{req.tool_type}' executed successfully for track {req.track_id}"
|
||||
}
|
||||
|
||||
class MyFilesRequest(BaseModel):
|
||||
active_file_ids: List[str] = []
|
||||
|
||||
@router.post("/my-files")
|
||||
async def list_user_files(req: MyFilesRequest, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
prefix = f"user_{user_id}_"
|
||||
|
||||
# Scan all user's projects to find referenced files
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT data_json FROM projects WHERE user_id = ?", (user_id,))
|
||||
rows = cursor.fetchall()
|
||||
conn.close()
|
||||
|
||||
referenced_in_db = set()
|
||||
for row in rows:
|
||||
try:
|
||||
proj = json.loads(row["data_json"])
|
||||
for track in proj.get("tracks", []):
|
||||
fid = track.get("serverFileId")
|
||||
if fid:
|
||||
referenced_in_db.add(fid)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
active_set = set(req.active_file_ids) | referenced_in_db
|
||||
|
||||
files_map = {}
|
||||
|
||||
def scan_dir(directory, type_label):
|
||||
if not os.path.exists(directory):
|
||||
return
|
||||
for filename in os.listdir(directory):
|
||||
if filename.startswith(prefix):
|
||||
filepath = os.path.join(directory, filename)
|
||||
if os.path.isfile(filepath):
|
||||
stat = os.stat(filepath)
|
||||
is_in_use = filename in active_set
|
||||
|
||||
if filename in files_map:
|
||||
files_map[filename]["size_mb"] = round(files_map[filename]["size_mb"] + stat.st_size / (1024 * 1024), 2)
|
||||
else:
|
||||
original_name = filename
|
||||
meta_path = os.path.join(directory, filename + ".meta")
|
||||
if os.path.isfile(meta_path):
|
||||
try:
|
||||
with open(meta_path, "r") as mf:
|
||||
meta = json.load(mf)
|
||||
original_name = meta.get("original_name", filename)
|
||||
except Exception:
|
||||
pass
|
||||
files_map[filename] = {
|
||||
"file_id": filename,
|
||||
"original_name": original_name,
|
||||
"size_mb": round(stat.st_size / (1024 * 1024), 2),
|
||||
"created_at": stat.st_mtime,
|
||||
"type": type_label,
|
||||
"is_in_use": is_in_use
|
||||
}
|
||||
|
||||
scan_dir(settings.UPLOADS_DIR, "Upload")
|
||||
scan_dir(settings.PROCESSED_DIR, "Processed")
|
||||
|
||||
user_files = list(files_map.values())
|
||||
user_files.sort(key=lambda x: x["created_at"], reverse=True)
|
||||
return user_files
|
||||
|
||||
@router.delete("/my-files/{file_id}")
|
||||
async def delete_user_file(file_id: str, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
prefix = f"user_{user_id}_"
|
||||
|
||||
# Guard: only own files can be deleted
|
||||
if not file_id.startswith(prefix):
|
||||
raise HTTPException(status_code=403, detail="Bạn không có quyền xóa tệp này")
|
||||
|
||||
deleted = False
|
||||
for directory in [settings.UPLOADS_DIR, settings.PROCESSED_DIR]:
|
||||
filepath = os.path.join(directory, file_id)
|
||||
if os.path.exists(filepath):
|
||||
try:
|
||||
os.remove(filepath)
|
||||
deleted = True
|
||||
except Exception:
|
||||
pass
|
||||
# Clean up sidecar metadata file
|
||||
meta_path = os.path.join(directory, file_id + ".meta")
|
||||
if os.path.isfile(meta_path):
|
||||
try:
|
||||
os.remove(meta_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy tệp trên server")
|
||||
|
||||
return {"success": True, "message": "Đã xóa tệp thành công"}
|
||||
|
||||
+79
-14
@@ -1,10 +1,15 @@
|
||||
import uuid
|
||||
import time
|
||||
from fastapi import APIRouter, HTTPException, Header, Depends
|
||||
import threading
|
||||
from fastapi import APIRouter, HTTPException, Header, Depends, Request, Response
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel, EmailStr
|
||||
from typing import Optional
|
||||
from app.models.user import get_db_connection
|
||||
from app.core.auth import hash_password, verify_password, create_token, decode_token, seed_admin
|
||||
from app.core.auth import (
|
||||
hash_password, verify_password, create_token, decode_token, seed_admin,
|
||||
COOKIE_NAME, X_AUTH_HEADER,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -21,10 +26,49 @@ class ChangePasswordRequest(BaseModel):
|
||||
old_password: str
|
||||
new_password: str
|
||||
|
||||
def get_current_user(authorization: Optional[str] = Header(None)):
|
||||
if not authorization or not authorization.startswith("Bearer "):
|
||||
# ── Brute-force guard: in-memory per-IP failed-login limiter ──
|
||||
_LOGIN_FAILURES = {} # ip -> [timestamps]
|
||||
_LOGIN_LOCK = threading.Lock()
|
||||
MAX_LOGIN_ATTEMPTS = 10
|
||||
LOGIN_WINDOW_SEC = 900 # 15 min
|
||||
LOGIN_BLOCK_SEC = 900
|
||||
|
||||
def _check_login_ratelimit(ip: str):
|
||||
now = time.time()
|
||||
with _LOGIN_LOCK:
|
||||
stamps = [t for t in _LOGIN_FAILURES.get(ip, []) if now - t < LOGIN_WINDOW_SEC]
|
||||
if len(stamps) >= MAX_LOGIN_ATTEMPTS:
|
||||
raise HTTPException(status_code=429, detail="Quá nhiều lần đăng nhập thất bại. Vui lòng thử lại sau 15 phút.")
|
||||
_LOGIN_FAILURES[ip] = stamps
|
||||
|
||||
def _record_login_failure(ip: str):
|
||||
now = time.time()
|
||||
with _LOGIN_LOCK:
|
||||
stamps = _LOGIN_FAILURES.setdefault(ip, [])
|
||||
stamps.append(now)
|
||||
_LOGIN_FAILURES[ip] = [t for t in stamps if now - t < LOGIN_WINDOW_SEC]
|
||||
|
||||
def _record_login_success(ip: str):
|
||||
with _LOGIN_LOCK:
|
||||
_LOGIN_FAILURES.pop(ip, None)
|
||||
|
||||
def _set_auth_cookie(response: Response, token: str):
|
||||
response.set_cookie(
|
||||
COOKIE_NAME, token,
|
||||
max_age=7 * 24 * 3600, httponly=True, samesite="lax",
|
||||
# path="/" (default); secure flag set by proxy when behind TLS
|
||||
)
|
||||
|
||||
def get_current_user(request: Request, authorization: Optional[str] = Header(None), x_auth_token: Optional[str] = Header(None)):
|
||||
token = None
|
||||
if authorization and authorization.startswith("Bearer "):
|
||||
token = authorization.split(" ")[1]
|
||||
elif x_auth_token:
|
||||
token = x_auth_token
|
||||
elif request.cookies.get(COOKIE_NAME):
|
||||
token = request.cookies.get(COOKIE_NAME)
|
||||
if not token:
|
||||
raise HTTPException(status_code=401, detail="Thiếu Token xác thực hoặc Token không hợp lệ")
|
||||
token = authorization.split(" ")[1]
|
||||
payload = decode_token(token)
|
||||
if not payload:
|
||||
raise HTTPException(status_code=401, detail="Token đã hết hạn hoặc không hợp lệ")
|
||||
@@ -39,7 +83,10 @@ def enforce_password_changed(user: dict):
|
||||
)
|
||||
|
||||
@router.post("/login")
|
||||
async def login(req: LoginRequest):
|
||||
async def login(req: LoginRequest, request: Request):
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
_check_login_ratelimit(client_ip)
|
||||
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
@@ -65,14 +112,17 @@ async def login(req: LoginRequest):
|
||||
conn.close()
|
||||
|
||||
if not user or not user["is_active"]:
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(status_code=400, detail="Tài khoản hoặc mật khẩu không chính xác")
|
||||
|
||||
if not verify_password(password, user["hashed_password"]):
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(status_code=400, detail="Tài khoản hoặc mật khẩu không chính xác")
|
||||
|
||||
token = create_token(user["id"], user["username"], user["role"], user["must_change_password"])
|
||||
|
||||
return {
|
||||
_record_login_success(client_ip)
|
||||
|
||||
resp = JSONResponse({
|
||||
"access_token": token,
|
||||
"user": {
|
||||
"id": user["id"],
|
||||
@@ -81,13 +131,24 @@ async def login(req: LoginRequest):
|
||||
"role": user["role"],
|
||||
"must_change_password": bool(user["must_change_password"])
|
||||
}
|
||||
}
|
||||
})
|
||||
_set_auth_cookie(resp, token)
|
||||
return resp
|
||||
|
||||
def _validate_password_strength(password: str):
|
||||
"""Minimal strength policy: >= 8 chars and not trivially common."""
|
||||
if len(password) < 8:
|
||||
raise HTTPException(status_code=400, detail="Mật khẩu phải có ít nhất 8 ký tự")
|
||||
lowered = password.lower()
|
||||
if lowered in ("admin123", "password", "12345678", "123456789", "qwerty123"):
|
||||
raise HTTPException(status_code=400, detail="Mật khẩu quá dễ đoán, vui lòng chọn mật khẩu khác")
|
||||
|
||||
@router.post("/register")
|
||||
async def register(req: RegisterRequest):
|
||||
async def register(req: RegisterRequest, request: Request):
|
||||
username = req.username.strip()
|
||||
email = req.email.strip()
|
||||
password = req.password.strip()
|
||||
_validate_password_strength(password)
|
||||
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
@@ -115,7 +176,7 @@ async def register(req: RegisterRequest):
|
||||
conn.close()
|
||||
|
||||
token = create_token(user_id, username, "standard", False)
|
||||
return {
|
||||
resp = JSONResponse({
|
||||
"access_token": token,
|
||||
"user": {
|
||||
"id": user_id,
|
||||
@@ -124,7 +185,9 @@ async def register(req: RegisterRequest):
|
||||
"role": "standard",
|
||||
"must_change_password": False
|
||||
}
|
||||
}
|
||||
})
|
||||
_set_auth_cookie(resp, token)
|
||||
return resp
|
||||
|
||||
@router.post("/change-password")
|
||||
async def change_password(req: ChangePasswordRequest, current_user: dict = Depends(get_current_user)):
|
||||
@@ -153,10 +216,12 @@ async def change_password(req: ChangePasswordRequest, current_user: dict = Depen
|
||||
conn.close()
|
||||
|
||||
new_token = create_token(updated_user["id"], updated_user["username"], updated_user["role"], False)
|
||||
return {
|
||||
resp = JSONResponse({
|
||||
"message": "Đổi mật khẩu thành công!",
|
||||
"access_token": new_token
|
||||
}
|
||||
})
|
||||
_set_auth_cookie(resp, new_token)
|
||||
return resp
|
||||
|
||||
@router.get("/profile")
|
||||
async def get_profile(current_user: dict = Depends(get_current_user)):
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
import os
|
||||
import platform
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from app.api.v1.auth import get_current_user
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
MEDIA_EXTS = {
|
||||
".wav", ".mp3", ".ogg", ".flac", ".aiff", ".aif", ".m4a", ".aac", ".opus",
|
||||
".mid", ".midi"
|
||||
}
|
||||
|
||||
AUDIO_EXTS = {".wav", ".mp3", ".ogg", ".flac", ".aiff", ".aif", ".m4a", ".aac", ".opus"}
|
||||
MIDI_EXTS = {".mid", ".midi"}
|
||||
|
||||
|
||||
def _safe_path(path: str) -> str:
|
||||
if not path:
|
||||
raise HTTPException(status_code=400, detail="Thiếu path")
|
||||
if "\x00" in path:
|
||||
raise HTTPException(status_code=400, detail="Path không hợp lệ")
|
||||
return os.path.normpath(path)
|
||||
|
||||
|
||||
REAL_FS_TYPES = {
|
||||
"ext2", "ext3", "ext4", "xfs", "btrfs", "jfs", "reiserfs",
|
||||
"ntfs", "ntfs3", "vfat", "exfat", "fat", "hfs", "hfsplus", "apfs",
|
||||
"zfs", "f2fs", "iso9660", "udf", "nfs", "nfs4", "cifs", "smb3", "fuseblk",
|
||||
}
|
||||
|
||||
PSEUDO_FS_TYPES = {
|
||||
"proc", "sysfs", "devpts", "tmpfs", "devtmpfs", "overlay", "squashfs",
|
||||
"cgroup", "cgroup2", "pstore", "securityfs", "debugfs", "tracefs",
|
||||
"configfs", "fusectl", "hugetlbfs", "mqueue", "binfmt_misc", "nsfs",
|
||||
"autofs", "ramfs", "efivarfs", "rpc_pipefs", "fuse", "fusefs",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/computer")
|
||||
async def list_computer_roots(current_user: dict = Depends(get_current_user)):
|
||||
"""Liệt kê các ổ đĩa / mount point thật của máy (My Computer)."""
|
||||
system = platform.system()
|
||||
roots = []
|
||||
if system == "Windows":
|
||||
import string
|
||||
for drive in string.ascii_uppercase:
|
||||
root = drive + ":\\"
|
||||
try:
|
||||
if os.path.exists(root):
|
||||
roots.append({"path": root, "name": drive + ":", "is_dir": True})
|
||||
except OSError:
|
||||
continue
|
||||
else:
|
||||
# Unix/Linux/macOS: chỉ liệt kê filesystem thật, bỏ pseudo/docker/systemd mounts
|
||||
seen = set()
|
||||
try:
|
||||
with open("/proc/mounts", "r") as f:
|
||||
for line in f:
|
||||
parts = line.split()
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
device, mount, fstype = parts[0], parts[1], parts[2]
|
||||
if fstype in PSEUDO_FS_TYPES:
|
||||
continue
|
||||
if fstype not in REAL_FS_TYPES:
|
||||
# giữ mount point root "/" nếu không thuộc pseudo
|
||||
if mount != "/":
|
||||
continue
|
||||
if mount in seen:
|
||||
continue
|
||||
seen.add(mount)
|
||||
# lọc mount point rác kiểu /run/credentials/...
|
||||
if mount.startswith("/run/") or mount.startswith("/var/lib/docker"):
|
||||
continue
|
||||
try:
|
||||
if os.path.isdir(mount):
|
||||
label = mount if mount != "/" else "Root (/)"
|
||||
roots.append({"path": mount, "name": label, "is_dir": True})
|
||||
except OSError:
|
||||
pass
|
||||
except OSError:
|
||||
pass
|
||||
# macOS fallback: liệt kê /Volumes
|
||||
if system == "Darwin":
|
||||
try:
|
||||
for name in sorted(os.listdir("/Volumes")):
|
||||
full = os.path.join("/Volumes", name)
|
||||
if os.path.isdir(full):
|
||||
roots.append({"path": full, "name": name, "is_dir": True})
|
||||
except OSError:
|
||||
pass
|
||||
if not roots:
|
||||
roots = [{"path": "/", "name": "Root (/)", "is_dir": True}]
|
||||
return {"system": system, "roots": roots}
|
||||
|
||||
|
||||
@router.get("/browse")
|
||||
async def browse_directory(path: str = Query(...), current_user: dict = Depends(get_current_user)):
|
||||
"""Liệt kê nội dung một thư mục trên máy: thư mục con + file audio/MIDI."""
|
||||
resolved = _safe_path(path)
|
||||
if not os.path.isdir(resolved):
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy thư mục")
|
||||
|
||||
dirs, files = [], []
|
||||
try:
|
||||
entries = os.listdir(resolved)
|
||||
except OSError as e:
|
||||
raise HTTPException(status_code=403, detail=f"Không thể đọc thư mục: {e}")
|
||||
|
||||
for name in entries:
|
||||
if name.startswith("."):
|
||||
continue
|
||||
full = os.path.join(resolved, name)
|
||||
try:
|
||||
if os.path.isdir(full):
|
||||
dirs.append({"name": name, "path": full, "is_dir": True})
|
||||
else:
|
||||
ext = os.path.splitext(name)[1].lower()
|
||||
try:
|
||||
size = os.path.getsize(full)
|
||||
except OSError:
|
||||
size = 0
|
||||
kind = "midi" if ext in MIDI_EXTS else ("audio" if ext in AUDIO_EXTS else "other")
|
||||
files.append({
|
||||
"name": name,
|
||||
"path": full,
|
||||
"is_dir": False,
|
||||
"size_mb": round(size / (1024 * 1024), 2),
|
||||
"ext": ext,
|
||||
"kind": kind
|
||||
})
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
dirs.sort(key=lambda d: d["name"].lower())
|
||||
files.sort(key=lambda f: f["name"].lower())
|
||||
parent = os.path.dirname(resolved)
|
||||
return {
|
||||
"path": resolved,
|
||||
"parent": parent if parent != resolved else None,
|
||||
"dirs": dirs,
|
||||
"files": files
|
||||
}
|
||||
|
||||
|
||||
@router.get("/file")
|
||||
async def serve_local_file(path: str = Query(...), current_user: dict = Depends(get_current_user)):
|
||||
"""Phục vụ file audio/MIDI cục bộ để preview."""
|
||||
resolved = _safe_path(path)
|
||||
if not os.path.isfile(resolved):
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy file")
|
||||
ext = os.path.splitext(resolved)[1].lower()
|
||||
if ext not in MEDIA_EXTS:
|
||||
raise HTTPException(status_code=403, detail="Loại file không được hỗ trợ preview")
|
||||
media_type = "audio/wav" if ext in AUDIO_EXTS else "audio/midi"
|
||||
return FileResponse(resolved, media_type=media_type, filename=os.path.basename(resolved))
|
||||
@@ -54,7 +54,7 @@ async def mix_multitrack_session(req: MultitrackSessionRequest):
|
||||
|
||||
# Gửi task xuống Celery Worker
|
||||
from app.tasks.worker import mix_multitrack_task
|
||||
task = mix_multitrack_task.delay(req.dict())
|
||||
task = mix_multitrack_task.delay(req.model_dump())
|
||||
|
||||
return {
|
||||
"task_id": task.id,
|
||||
@@ -69,7 +69,7 @@ async def process_session(req: MultitrackSessionRequest):
|
||||
Xử lý từng clip, sau đó hòa âm tất cả tracks lại với nhau.
|
||||
"""
|
||||
from app.tasks.worker import process_multitrack_session_task
|
||||
task = process_multitrack_session_task.delay(req.dict())
|
||||
task = process_multitrack_session_task.delay(req.model_dump())
|
||||
|
||||
return {
|
||||
"task_id": task.id,
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
import os, uuid, json, tempfile
|
||||
from fastapi import APIRouter, HTTPException, Depends, UploadFile, File, BackgroundTasks
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, Any
|
||||
from app.config import settings
|
||||
from app.core.vst_engine import PluginManager, HAS_PEDALBOARD, HAS_PYFLUIDSYNTH
|
||||
from app.core.render_engine import PythonRenderEngine
|
||||
from app.core.soundfont_inspector import SoundFontInspector
|
||||
from app.core.soundfont_converter import SoundFontConverter
|
||||
from app.core.soundfont_scanner import SoundFontAutoScanner
|
||||
from app.api.v1.auth import get_current_user, enforce_password_changed
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
UPLOAD_SF_DIR = os.path.join(settings.STORAGE_DIR, "soundfonts")
|
||||
os.makedirs(UPLOAD_SF_DIR, exist_ok=True)
|
||||
|
||||
SYSTEM_SF_DIR = "/opt/daw_engine/soundfonts"
|
||||
|
||||
_inspector = None
|
||||
_scanner = None
|
||||
|
||||
def get_inspector():
|
||||
global _inspector
|
||||
if _inspector is None:
|
||||
_inspector = SoundFontInspector(system_sf_dir=SYSTEM_SF_DIR, upload_sf_dir=UPLOAD_SF_DIR)
|
||||
return _inspector
|
||||
|
||||
def get_scanner():
|
||||
global _scanner
|
||||
if _scanner is None:
|
||||
_scanner = SoundFontAutoScanner(system_sf_dir=SYSTEM_SF_DIR, upload_sf_dir=UPLOAD_SF_DIR)
|
||||
_scanner.scan_once()
|
||||
return _scanner
|
||||
|
||||
|
||||
@router.get("/available")
|
||||
async def list_plugins(current_user: dict = Depends(get_current_user)):
|
||||
pm = PluginManager(upload_sf_dir=UPLOAD_SF_DIR)
|
||||
return pm.list_available()
|
||||
|
||||
|
||||
@router.get("/default-soundfonts")
|
||||
async def list_default_soundfonts():
|
||||
static_sf_dir = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), "static", "soundfonts")
|
||||
result = []
|
||||
if os.path.isdir(static_sf_dir):
|
||||
for f in os.listdir(static_sf_dir):
|
||||
if f.endswith(".sf2") or f.endswith(".sf3"):
|
||||
result.append({
|
||||
"id": os.path.splitext(f)[0],
|
||||
"name": f,
|
||||
"file": f,
|
||||
"url": f"/soundfonts/{f}"
|
||||
})
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/soundfonts/catalog")
|
||||
async def soundfont_catalog(current_user: dict = Depends(get_current_user)):
|
||||
scanner = get_scanner()
|
||||
full_catalog = scanner.get_catalog()
|
||||
inspector = get_inspector()
|
||||
condensed_catalog = inspector.get_condensed_catalog_summary(full_catalog)
|
||||
return {"full_catalog": full_catalog, "condensed_catalog": condensed_catalog}
|
||||
|
||||
|
||||
@router.get("/soundfont-instruments/{sf_id}")
|
||||
async def list_soundfont_instruments(sf_id: str, current_user: dict = Depends(get_current_user)):
|
||||
pm = PluginManager(upload_sf_dir=UPLOAD_SF_DIR)
|
||||
presets = pm.list_soundfont_instruments(sf_id)
|
||||
return {"presets": presets, "count": len(presets)}
|
||||
|
||||
|
||||
@router.post("/upload-soundfont")
|
||||
async def upload_soundfont(
|
||||
file: UploadFile = File(...),
|
||||
background_tasks: BackgroundTasks = None,
|
||||
current_user: dict = Depends(get_current_user)
|
||||
):
|
||||
enforce_password_changed(current_user)
|
||||
if not (file.filename and (file.filename.endswith(".sf2") or file.filename.endswith(".sf3"))):
|
||||
raise HTTPException(status_code=400, detail="Only .sf2 / .sf3 files are allowed")
|
||||
|
||||
# Stream upload in chunks with a hard size cap (SGM-class fonts can exceed
|
||||
# 500MB; reading the whole body into RAM would OOM the server).
|
||||
MAX_SF_UPLOAD_BYTES = 2 * 1024 * 1024 * 1024 # 2 GB
|
||||
contents = bytearray()
|
||||
while True:
|
||||
chunk = await file.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
contents.extend(chunk)
|
||||
if len(contents) > MAX_SF_UPLOAD_BYTES:
|
||||
raise HTTPException(status_code=413, detail="SoundFont quá lớn (giới hạn 2GB)")
|
||||
|
||||
if not PluginManager.validate_sf2_header(bytes(contents[:4096])):
|
||||
raise HTTPException(status_code=400, detail="Invalid SoundFont file: missing RIFF/sfbk header")
|
||||
|
||||
file_ext = os.path.splitext(file.filename)[1]
|
||||
file_uuid = str(uuid.uuid4())
|
||||
# Store original name in a sidecar file
|
||||
base_name = os.path.splitext(file.filename)[0].replace('/', '_').replace('\\', '_')
|
||||
file_id = file_uuid + file_ext
|
||||
dest_path = os.path.join(UPLOAD_SF_DIR, file_id)
|
||||
with open(dest_path, "wb") as f:
|
||||
f.write(contents)
|
||||
|
||||
# Save metadata with original name
|
||||
meta_path = os.path.join(UPLOAD_SF_DIR, file_uuid + ".meta")
|
||||
with open(meta_path, "w", encoding="utf-8") as f:
|
||||
import json
|
||||
json.dump({"original_name": file.filename, "uuid": file_uuid, "file": file_id}, f)
|
||||
|
||||
scanner = get_scanner()
|
||||
if background_tasks:
|
||||
background_tasks.add_task(scanner.scan_once)
|
||||
else:
|
||||
scanner.scan_once()
|
||||
return {"id": file_id, "name": file.filename, "path": dest_path, "size_bytes": len(contents)}
|
||||
|
||||
|
||||
@router.delete("/soundfont/{sf_id}")
|
||||
async def delete_soundfont(sf_id: str, current_user: dict = Depends(get_current_user)):
|
||||
base_id = sf_id.replace("sf_", "")
|
||||
deleted = False
|
||||
for d in [UPLOAD_SF_DIR, os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), "static", "soundfonts"), SYSTEM_SF_DIR]:
|
||||
if not os.path.isdir(d):
|
||||
continue
|
||||
for f in os.listdir(d):
|
||||
if os.path.splitext(f)[0] == base_id:
|
||||
# Skip system dir — only allow deleting uploads
|
||||
if d == SYSTEM_SF_DIR:
|
||||
raise HTTPException(status_code=403, detail="System soundfonts cannot be deleted via this endpoint")
|
||||
path = os.path.join(d, f)
|
||||
os.remove(path)
|
||||
meta_path = os.path.join(d, os.path.splitext(f)[0] + ".meta")
|
||||
if os.path.isfile(meta_path):
|
||||
os.remove(meta_path)
|
||||
deleted = True
|
||||
break
|
||||
if deleted:
|
||||
break
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="SoundFont not found")
|
||||
get_scanner().scan_once()
|
||||
return {"deleted": True, "sf_id": sf_id}
|
||||
|
||||
|
||||
@router.get("/soundfonts/download/{sf_id}")
|
||||
async def download_soundfont_asset(sf_id: str):
|
||||
clean_id = sf_id.replace("sf_", "") if sf_id.startswith("sf_") else sf_id
|
||||
# Cũng tìm trong static/soundfonts (font bundled theo deployment) — trước
|
||||
# đây chỉ UPLOAD + SYSTEM → font bundled 404 → incognito (IndexedDB rỗng)
|
||||
# không tải được font → instrument CÂM (browser thường dùng cache nên OK).
|
||||
static_sf_dir = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), "static", "soundfonts")
|
||||
for base_dir in [UPLOAD_SF_DIR, SYSTEM_SF_DIR, static_sf_dir]:
|
||||
if not os.path.isdir(base_dir):
|
||||
continue
|
||||
# Prefer SF2: the client FluidSynth WASM cannot decode SF3 (Ogg Vorbis)
|
||||
# samples, so any SF3 would play silence in the browser.
|
||||
for fname in os.listdir(base_dir):
|
||||
fbase, fext = os.path.splitext(fname)
|
||||
if fext.lower() == ".sf2" and fbase.lower() == clean_id.lower():
|
||||
full = os.path.join(base_dir, fname)
|
||||
return FileResponse(full, media_type="application/octet-stream", filename="soundfont.sf2")
|
||||
# Only an SF3 exists -> decompress it to a playable SF2 on demand (cached)
|
||||
for fname in os.listdir(base_dir):
|
||||
fbase, fext = os.path.splitext(fname)
|
||||
if fext.lower() == ".sf3" and fbase.lower() == clean_id.lower():
|
||||
full = os.path.join(base_dir, fname)
|
||||
try:
|
||||
from app.core.soundfont_converter import SoundFontConverter
|
||||
sf2_path = os.path.join(UPLOAD_SF_DIR, clean_id + ".sf2")
|
||||
if os.path.exists(sf2_path) and os.path.getmtime(sf2_path) >= os.path.getmtime(full):
|
||||
return FileResponse(sf2_path, media_type="application/octet-stream", filename="soundfont.sf2")
|
||||
result = SoundFontConverter().sf3_to_sf2(full, sf2_path)
|
||||
if result != full and os.path.exists(result):
|
||||
return FileResponse(result, media_type="application/octet-stream", filename="soundfont.sf2")
|
||||
except Exception as e:
|
||||
print(f"[soundfont-download] SF3->SF2 conversion failed for {full}: {e}")
|
||||
return FileResponse(full, media_type="application/octet-stream", filename="soundfont.sf3")
|
||||
raise HTTPException(status_code=404, detail="SoundFont asset not found")
|
||||
|
||||
|
||||
class RenderRequest(BaseModel):
|
||||
project_json: dict
|
||||
output_filename: Optional[str] = "render_output.wav"
|
||||
|
||||
|
||||
@router.post("/render")
|
||||
async def render_project(
|
||||
req: RenderRequest,
|
||||
current_user: dict = Depends(get_current_user)
|
||||
):
|
||||
enforce_password_changed(current_user)
|
||||
engine = PythonRenderEngine()
|
||||
# Prevent path traversal: strip any directory components and force .wav.
|
||||
safe_name = os.path.basename((req.output_filename or "render_output.wav").replace("\\", "/"))
|
||||
if not safe_name.lower().endswith(".wav"):
|
||||
safe_name += ".wav"
|
||||
output_path = os.path.join(settings.PROCESSED_DIR, safe_name)
|
||||
try:
|
||||
result_path = engine.render_project(req.project_json, output_path)
|
||||
return {"url": f"/static/audio/processed/{os.path.basename(result_path)}", "path": result_path}
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Render failed: {str(e)}")
|
||||
+348
-10
@@ -1,14 +1,135 @@
|
||||
import time
|
||||
import json
|
||||
import uuid
|
||||
from fastapi import APIRouter, HTTPException, Depends, Header
|
||||
import os
|
||||
from fastapi import APIRouter, HTTPException, Depends, Header, Query
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, Any, Dict
|
||||
from typing import Optional, Any, Dict, List
|
||||
from jsonschema import validate, ValidationError
|
||||
from app.models.user import get_db_connection
|
||||
from app.api.v1.auth import get_current_user, decode_token
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
SCHEMA_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), "models", "project_schema.json")
|
||||
|
||||
def upgrade_project_json_if_needed(project_data: dict) -> dict:
|
||||
if "main_session" in project_data:
|
||||
return project_data
|
||||
|
||||
tracks = project_data.get("tracks", [])
|
||||
# Legacy format stores item start times in SECONDS; convert using the real
|
||||
# seconds-per-bar (old code hardcoded /4.0 which shifted every item's
|
||||
# position for any tempo other than the one where 1 bar = 4s).
|
||||
bpm_val = float(project_data.get("bpm", 120.0) or 120.0)
|
||||
seconds_per_bar = (60.0 / bpm_val) * 4
|
||||
upgraded_tracks = []
|
||||
for t in tracks:
|
||||
track_id = str(t.get("id", ""))
|
||||
track_name = t.get("name", "Track")
|
||||
vol = t.get("volumeDb", 0.0)
|
||||
pan = t.get("pan", 0.0)
|
||||
muted = t.get("muted", False)
|
||||
solo = t.get("solo", False)
|
||||
|
||||
items = []
|
||||
for c in t.get("clips", []):
|
||||
items.append({
|
||||
"id": c.get("id"),
|
||||
"name": c.get("name", "Audio Clip"),
|
||||
"type": "AUDIO_ITEM",
|
||||
"start_bar": round(c.get("startTime", 0.0) / seconds_per_bar, 6),
|
||||
"duration_bars": round((c.get("duration", 4.0) if c.get("duration") else 4.0) / seconds_per_bar, 6),
|
||||
"clip_start_offset_bars": 0.0,
|
||||
"source_data": {
|
||||
"audio_file_url": f"/static/audio/uploads/{t.get('serverFileId')}" if t.get("serverFileId") else "",
|
||||
"server_file_id": t.get("serverFileId") or None,
|
||||
"sample_rate": 44100,
|
||||
"channels": 2,
|
||||
"gain": 1.0
|
||||
}
|
||||
})
|
||||
for m in t.get("midiItems", []):
|
||||
items.append({
|
||||
"id": m.get("id"),
|
||||
"name": m.get("name", "MIDI Item"),
|
||||
"type": "MIDI_ITEM",
|
||||
"start_bar": round(m.get("startTime", 0.0) / seconds_per_bar, 6),
|
||||
"duration_bars": round((m.get("duration", 4.0) or 4.0) / seconds_per_bar, 6),
|
||||
"clip_start_offset_bars": 0.0,
|
||||
"source_data": {
|
||||
"total_buffer_bars": round((m.get("duration", 8.0) or 8.0) / seconds_per_bar, 6),
|
||||
"notes": m.get("notes", [])
|
||||
}
|
||||
})
|
||||
|
||||
upgraded_tracks.append({
|
||||
"id": track_id,
|
||||
"name": track_name,
|
||||
"type": "MIDI" if t.get("midiItems") else "AUDIO",
|
||||
"volume_db": vol,
|
||||
"pan": pan,
|
||||
"mute": muted,
|
||||
"solo": solo,
|
||||
"fx_chain": [],
|
||||
"synth_engine": {
|
||||
"plugin_id": "synth",
|
||||
"preset_id": "default",
|
||||
"parameters": {}
|
||||
},
|
||||
"items": items
|
||||
})
|
||||
|
||||
return {
|
||||
"project_id": project_data.get("id", "temp_project"),
|
||||
"metadata": {
|
||||
"title": project_data.get("name", "Dự án mới"),
|
||||
"bpm": 120.0,
|
||||
"time_signature_numerator": 4,
|
||||
"time_signature_denominator": 4,
|
||||
"sample_rate": 44100
|
||||
},
|
||||
"main_session": {
|
||||
"id": "main",
|
||||
"name": "MAIN SESSION",
|
||||
"is_root": True,
|
||||
"length_bars": 16.0,
|
||||
"auto_compute_length": True,
|
||||
"tracks": upgraded_tracks
|
||||
},
|
||||
"section_store": {}
|
||||
}
|
||||
|
||||
def validate_project_data(data_json: str) -> str:
|
||||
try:
|
||||
data = json.loads(data_json)
|
||||
if "project_id" not in data:
|
||||
data["project_id"] = "temp_legacy_" + str(int(time.time()))
|
||||
# Strip null synth_engine from tracks (breaks schema validation)
|
||||
for session_key in ["main_session"] + [k for k in data.get("section_store", {})]:
|
||||
session = data.get(session_key)
|
||||
if not session:
|
||||
continue
|
||||
for track in session.get("tracks", []):
|
||||
if "synth_engine" in track and track["synth_engine"] is None:
|
||||
del track["synth_engine"]
|
||||
if "main_session" not in data:
|
||||
data = upgrade_project_json_if_needed(data)
|
||||
data_json = json.dumps(data)
|
||||
|
||||
if os.path.exists(SCHEMA_PATH):
|
||||
with open(SCHEMA_PATH, "r") as f:
|
||||
schema = json.load(f)
|
||||
validate(instance=data, schema=schema)
|
||||
return data_json
|
||||
except json.JSONDecodeError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Cấu trúc JSON không hợp lệ: {str(e)}")
|
||||
except ValidationError as e:
|
||||
path = " -> ".join(str(p) for p in e.path)
|
||||
raise HTTPException(status_code=400, detail=f"Lỗi xác thực project schema tại [{path}]: {e.message}")
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=400, detail=f"Lỗi xác thực dự án: {str(e)}")
|
||||
|
||||
class SaveProjectRequest(BaseModel):
|
||||
name: str
|
||||
data_json: str
|
||||
@@ -24,11 +145,12 @@ def get_optional_user(authorization: Optional[str] = Header(None)) -> Optional[d
|
||||
|
||||
@router.post("/temp")
|
||||
async def save_temp_project(req: SaveTempProjectRequest, current_user: Optional[dict] = Depends(get_optional_user)):
|
||||
validated_data_json = validate_project_data(req.data_json)
|
||||
user_id = current_user["user_id"] if current_user else "anonymous"
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
size_bytes = len(req.data_json.encode("utf-8"))
|
||||
size_bytes = len(validated_data_json.encode("utf-8"))
|
||||
now = time.time()
|
||||
temp_id = f"temp_{user_id}"
|
||||
|
||||
@@ -36,7 +158,7 @@ async def save_temp_project(req: SaveTempProjectRequest, current_user: Optional[
|
||||
INSERT INTO projects (id, user_id, name, data_json, is_temp, size_bytes, updated_at)
|
||||
VALUES (?, ?, 'Dự án tạm chưa lưu', ?, 1, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET data_json = excluded.data_json, size_bytes = excluded.size_bytes, updated_at = excluded.updated_at
|
||||
""", (temp_id, user_id, req.data_json, size_bytes, now))
|
||||
""", (temp_id, user_id, validated_data_json, size_bytes, now))
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
@@ -64,6 +186,7 @@ async def get_temp_project(current_user: Optional[dict] = Depends(get_optional_u
|
||||
|
||||
@router.post("/cloud")
|
||||
async def save_cloud_project(req: SaveProjectRequest, current_user: dict = Depends(get_current_user)):
|
||||
validated_data_json = validate_project_data(req.data_json)
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
@@ -76,7 +199,7 @@ async def save_cloud_project(req: SaveProjectRequest, current_user: dict = Depen
|
||||
used_row = cursor.fetchone()
|
||||
used_bytes = used_row["total_used"] if used_row and used_row["total_used"] else 0
|
||||
|
||||
new_size_bytes = len(req.data_json.encode("utf-8"))
|
||||
new_size_bytes = len(validated_data_json.encode("utf-8"))
|
||||
max_bytes = storage_limit_mb * 1024 * 1024
|
||||
|
||||
if used_bytes + new_size_bytes > max_bytes:
|
||||
@@ -92,7 +215,7 @@ async def save_cloud_project(req: SaveProjectRequest, current_user: dict = Depen
|
||||
cursor.execute("""
|
||||
INSERT INTO projects (id, user_id, name, data_json, is_temp, size_bytes, updated_at)
|
||||
VALUES (?, ?, ?, ?, 0, ?, ?)
|
||||
""", (project_id, user_id, req.name, req.data_json, new_size_bytes, now))
|
||||
""", (project_id, user_id, req.name, validated_data_json, new_size_bytes, now))
|
||||
|
||||
temp_id = f"temp_{user_id}"
|
||||
cursor.execute("DELETE FROM projects WHERE id = ? AND is_temp = 1", (temp_id,))
|
||||
@@ -111,9 +234,11 @@ async def list_cloud_projects(current_user: dict = Depends(get_current_user)):
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT id, name, size_bytes, updated_at FROM projects
|
||||
WHERE user_id = ? AND is_temp = 0
|
||||
ORDER BY updated_at DESC
|
||||
SELECT p.id, p.name, p.size_bytes, p.updated_at,
|
||||
(SELECT COUNT(*) FROM project_backups pb WHERE pb.project_id = p.id AND pb.user_id = p.user_id) as backup_count
|
||||
FROM projects p
|
||||
WHERE p.user_id = ? AND p.is_temp = 0
|
||||
ORDER BY p.updated_at DESC
|
||||
""", (user_id,))
|
||||
rows = cursor.fetchall()
|
||||
conn.close()
|
||||
@@ -123,6 +248,219 @@ async def list_cloud_projects(current_user: dict = Depends(get_current_user)):
|
||||
"id": r["id"],
|
||||
"name": r["name"],
|
||||
"size_mb": round(r["size_bytes"] / (1024 * 1024), 2),
|
||||
"updated_at": r["updated_at"]
|
||||
"updated_at": r["updated_at"],
|
||||
"backup_count": r["backup_count"]
|
||||
} for r in rows
|
||||
]
|
||||
|
||||
@router.get("/cloud/{project_id}")
|
||||
async def get_cloud_project(project_id: str, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT name, data_json FROM projects WHERE id = ? AND user_id = ? AND is_temp = 0", (project_id, user_id))
|
||||
row = cursor.fetchone()
|
||||
conn.close()
|
||||
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy dự án")
|
||||
|
||||
return {
|
||||
"id": project_id,
|
||||
"name": row["name"],
|
||||
"data_json": row["data_json"]
|
||||
}
|
||||
|
||||
@router.delete("/cloud/{project_id}")
|
||||
async def delete_cloud_project(project_id: str, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM projects WHERE id = ? AND user_id = ? AND is_temp = 0", (project_id, user_id))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"success": True, "message": "Đã xóa dự án thành công"}
|
||||
|
||||
@router.put("/cloud/{project_id}")
|
||||
async def update_cloud_project(project_id: str, req: SaveProjectRequest, current_user: dict = Depends(get_current_user)):
|
||||
validated_data_json = validate_project_data(req.data_json)
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("SELECT id, size_bytes FROM projects WHERE id = ? AND user_id = ? AND is_temp = 0", (project_id, user_id))
|
||||
existing = cursor.fetchone()
|
||||
if not existing:
|
||||
conn.close()
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy dự án để cập nhật")
|
||||
|
||||
new_size_bytes = len(validated_data_json.encode("utf-8"))
|
||||
|
||||
# Enforce storage quota (same rule as save_cloud_project — previously
|
||||
# update bypassed the quota entirely).
|
||||
cursor.execute("SELECT storage_limit_mb FROM user_quotas WHERE user_id = ?", (user_id,))
|
||||
quota_row = cursor.fetchone()
|
||||
storage_limit_mb = quota_row["storage_limit_mb"] if quota_row else 500
|
||||
cursor.execute("SELECT SUM(size_bytes) as total_used FROM projects WHERE user_id = ? AND is_temp = 0", (user_id,))
|
||||
used_row = cursor.fetchone()
|
||||
used_bytes = (used_row["total_used"] if used_row and used_row["total_used"] else 0) - (existing["size_bytes"] or 0)
|
||||
max_bytes = storage_limit_mb * 1024 * 1024
|
||||
if used_bytes + new_size_bytes > max_bytes:
|
||||
conn.close()
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Dung lượng dự án vượt quá hạn mức Quota ({storage_limit_mb}MB). Vui lòng dọn dẹp hoặc nâng cấp tài khoản."
|
||||
)
|
||||
|
||||
now = time.time()
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE projects
|
||||
SET name = ?, data_json = ?, size_bytes = ?, updated_at = ?
|
||||
WHERE id = ? AND user_id = ?
|
||||
""", (req.name, validated_data_json, new_size_bytes, now, project_id, user_id))
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"success": True, "message": "Đã cập nhật dự án thành công"}
|
||||
|
||||
class BackupConfigRequest(BaseModel):
|
||||
max_count: int = 10
|
||||
|
||||
@router.post("/cloud/{project_id}/backup")
|
||||
async def create_project_backup(project_id: str, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT name, data_json FROM projects WHERE id = ? AND user_id = ? AND is_temp = 0", (project_id, user_id))
|
||||
row = cursor.fetchone()
|
||||
if not row:
|
||||
conn.close()
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy dự án")
|
||||
|
||||
backup_id = f"backup_{uuid.uuid4().hex[:12]}"
|
||||
now = time.time()
|
||||
size_bytes = len(row["data_json"].encode("utf-8"))
|
||||
backup_name = f"[Backup] {row['name']} ({time.strftime('%Y-%m-%d %H:%M', time.localtime(now))})"
|
||||
|
||||
cursor.execute("""
|
||||
INSERT INTO project_backups (id, user_id, project_id, name, data_json, size_bytes, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
""", (backup_id, user_id, project_id, backup_name, row["data_json"], size_bytes, now))
|
||||
|
||||
# Enforce retention limit: lấy max_count từ query param mặc định 10
|
||||
cursor.execute("SELECT COUNT(*) as cnt FROM project_backups WHERE project_id = ? AND user_id = ?", (project_id, user_id))
|
||||
count = cursor.fetchone()["cnt"]
|
||||
max_backup = 10 # default
|
||||
if count > max_backup:
|
||||
excess = count - max_backup
|
||||
cursor.execute("""
|
||||
DELETE FROM project_backups WHERE id IN (
|
||||
SELECT id FROM project_backups WHERE project_id = ? AND user_id = ?
|
||||
ORDER BY created_at ASC LIMIT ?
|
||||
)
|
||||
""", (project_id, user_id, excess))
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {
|
||||
"backup_id": backup_id,
|
||||
"name": backup_name,
|
||||
"created_at": now,
|
||||
"max_backups": max_backup
|
||||
}
|
||||
|
||||
@router.get("/cloud/{project_id}/backups")
|
||||
async def list_project_backups(project_id: str, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT id, name, size_bytes, created_at FROM project_backups
|
||||
WHERE project_id = ? AND user_id = ?
|
||||
ORDER BY created_at DESC
|
||||
""", (project_id, user_id))
|
||||
rows = cursor.fetchall()
|
||||
conn.close()
|
||||
return [
|
||||
{
|
||||
"id": r["id"],
|
||||
"name": r["name"],
|
||||
"size_mb": round(r["size_bytes"] / (1024 * 1024), 2),
|
||||
"created_at": r["created_at"]
|
||||
} for r in rows
|
||||
]
|
||||
|
||||
@router.delete("/cloud/backups/{backup_id}")
|
||||
async def delete_project_backup(backup_id: str, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM project_backups WHERE id = ? AND user_id = ?", (backup_id, user_id))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"success": True, "message": "Đã xóa bản backup"}
|
||||
|
||||
class CleanupBackupsRequest(BaseModel):
|
||||
keep: int = 10
|
||||
|
||||
@router.post("/cloud/backups/cleanup")
|
||||
async def cleanup_all_backups(req: CleanupBackupsRequest, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
keep = max(5, min(20, req.keep))
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Lấy tất cả project_id của user
|
||||
cursor.execute("SELECT DISTINCT project_id FROM project_backups WHERE user_id = ?", (user_id,))
|
||||
projects = cursor.fetchall()
|
||||
total_deleted = 0
|
||||
for p in projects:
|
||||
pid = p["project_id"]
|
||||
cursor.execute("SELECT COUNT(*) as cnt FROM project_backups WHERE project_id = ? AND user_id = ?", (pid, user_id))
|
||||
cnt = cursor.fetchone()["cnt"]
|
||||
if cnt > keep:
|
||||
excess = cnt - keep
|
||||
cursor.execute("""
|
||||
DELETE FROM project_backups WHERE id IN (
|
||||
SELECT id FROM project_backups WHERE project_id = ? AND user_id = ?
|
||||
ORDER BY created_at ASC LIMIT ?
|
||||
)
|
||||
""", (pid, user_id, excess))
|
||||
total_deleted += excess
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"success": True, "deleted": total_deleted, "keep": keep}
|
||||
|
||||
class RenderProjectRequest(BaseModel):
|
||||
sample_rate: Optional[int] = 44100
|
||||
|
||||
@router.post("/cloud/{project_id}/render")
|
||||
async def render_project_endpoint(project_id: str, req: RenderProjectRequest, current_user: dict = Depends(get_current_user)):
|
||||
user_id = current_user["user_id"]
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT name, data_json FROM projects WHERE id = ? AND user_id = ? AND is_temp = 0", (project_id, user_id))
|
||||
row = cursor.fetchone()
|
||||
conn.close()
|
||||
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Không tìm thấy dự án để kết xuất")
|
||||
|
||||
# Validate schema
|
||||
validate_project_data(row["data_json"])
|
||||
|
||||
# Trigger Celery task
|
||||
from app.tasks.worker import render_project_task
|
||||
task = render_project_task.delay(
|
||||
project_id=project_id,
|
||||
project_name=row["name"],
|
||||
project_json_str=row["data_json"],
|
||||
sample_rate=req.sample_rate or 44100
|
||||
)
|
||||
|
||||
return {
|
||||
"task_id": task.id,
|
||||
"status": "processing"
|
||||
}
|
||||
|
||||
+75
-53
@@ -1,12 +1,55 @@
|
||||
from fastapi import APIRouter, HTTPException, Depends
|
||||
import json, os
|
||||
from fastapi import APIRouter, HTTPException, Header, Depends
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, List
|
||||
from typing import Optional, List, Dict, Any
|
||||
import time
|
||||
from app.core.auth import decode_token
|
||||
from app.config import settings
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
# In-memory / per-user AI provider configurations storage dictionary
|
||||
USER_AI_CONFIGS = {}
|
||||
DATA_FILE = os.path.join(settings.PROCESSED_DIR, "user_configs.json")
|
||||
|
||||
def _load_all():
|
||||
if not os.path.exists(DATA_FILE):
|
||||
return {"ai_configs": {}, "preferences": {}}
|
||||
try:
|
||||
with open(DATA_FILE, "r") as f:
|
||||
return json.load(f)
|
||||
except: return {"ai_configs": {}, "preferences": {}}
|
||||
|
||||
def _save_all(ai_configs=None, preferences=None):
|
||||
data = _load_all()
|
||||
if ai_configs is not None: data["ai_configs"] = ai_configs
|
||||
if preferences is not None: data["preferences"] = preferences
|
||||
os.makedirs(os.path.dirname(DATA_FILE), exist_ok=True)
|
||||
with open(DATA_FILE, "w") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
def _get_user_id(authorization):
|
||||
if not authorization or not authorization.startswith("Bearer "):
|
||||
return "anonymous"
|
||||
token = authorization.split(" ")[1]
|
||||
payload = decode_token(token)
|
||||
if not payload:
|
||||
return "anonymous"
|
||||
return payload.get("user_id", "anonymous")
|
||||
|
||||
def _load_ai_configs():
|
||||
data = _load_all()
|
||||
return data.get("ai_configs", {})
|
||||
|
||||
def _load_preferences():
|
||||
data = _load_all()
|
||||
return data.get("preferences", {})
|
||||
|
||||
def _get_default_providers():
|
||||
return [
|
||||
{"id": "openai_default", "name": "OpenAI Official", "provider_type": "openai", "api_base_url": "https://api.openai.com/v1", "api_key": "", "model_name": "gpt-4o", "temperature": 0.7, "is_active": True},
|
||||
{"id": "openai_compat_default", "name": "OpenAI Compatible (Ollama/LocalAI/DeepSeek)", "provider_type": "openai_compatible", "api_base_url": "http://localhost:11434/v1", "api_key": "ollama", "model_name": "deepseek-r1", "temperature": 0.7, "is_active": False},
|
||||
{"id": "anthropic_default", "name": "Anthropic Claude", "provider_type": "anthropic", "api_base_url": "https://api.anthropic.com/v1", "api_key": "", "model_name": "claude-3-5-sonnet", "temperature": 0.7, "is_active": False},
|
||||
{"id": "gemini_default", "name": "Google Gemini", "provider_type": "gemini", "api_base_url": "https://generativelanguage.googleapis.com", "api_key": "", "model_name": "gemini-1.5-pro", "temperature": 0.7, "is_active": False}
|
||||
]
|
||||
|
||||
class AIProviderSetting(BaseModel):
|
||||
id: str
|
||||
@@ -21,61 +64,40 @@ class AIProviderSetting(BaseModel):
|
||||
class SaveAIConfigRequest(BaseModel):
|
||||
providers: List[AIProviderSetting]
|
||||
|
||||
class SavePreferencesRequest(BaseModel):
|
||||
preferences: Dict[str, Any]
|
||||
|
||||
@router.get("/preferences")
|
||||
async def get_user_preferences(authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
prefs = _load_preferences()
|
||||
return {"success": True, "preferences": prefs.get(uid, {})}
|
||||
|
||||
@router.post("/preferences")
|
||||
async def save_user_preferences(req: SavePreferencesRequest, authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
prefs = _load_preferences()
|
||||
prefs[uid] = req.preferences
|
||||
_save_all(preferences=prefs)
|
||||
return {"success": True, "message": "Đã lưu cấu hình người dùng."}
|
||||
|
||||
@router.get("/config/ai")
|
||||
async def get_user_ai_config():
|
||||
"""Fetch user's AI provider configurations."""
|
||||
if "default_user" not in USER_AI_CONFIGS:
|
||||
USER_AI_CONFIGS["default_user"] = [
|
||||
{
|
||||
"id": "openai_default",
|
||||
"name": "OpenAI Official",
|
||||
"provider_type": "openai",
|
||||
"api_base_url": "https://api.openai.com/v1",
|
||||
"api_key": "",
|
||||
"model_name": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"is_active": True
|
||||
},
|
||||
{
|
||||
"id": "openai_compat_default",
|
||||
"name": "OpenAI Compatible (Ollama/LocalAI/DeepSeek)",
|
||||
"provider_type": "openai_compatible",
|
||||
"api_base_url": "http://localhost:11434/v1",
|
||||
"api_key": "ollama",
|
||||
"model_name": "deepseek-r1",
|
||||
"temperature": 0.7,
|
||||
"is_active": False
|
||||
},
|
||||
{
|
||||
"id": "anthropic_default",
|
||||
"name": "Anthropic Claude",
|
||||
"provider_type": "anthropic",
|
||||
"api_base_url": "https://api.anthropic.com/v1",
|
||||
"api_key": "",
|
||||
"model_name": "claude-3-5-sonnet",
|
||||
"temperature": 0.7,
|
||||
"is_active": False
|
||||
},
|
||||
{
|
||||
"id": "gemini_default",
|
||||
"name": "Google Gemini",
|
||||
"provider_type": "gemini",
|
||||
"api_base_url": "https://generativelanguage.googleapis.com",
|
||||
"api_key": "",
|
||||
"model_name": "gemini-1.5-pro",
|
||||
"temperature": 0.7,
|
||||
"is_active": False
|
||||
}
|
||||
]
|
||||
async def get_user_ai_config(authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
configs = _load_ai_configs()
|
||||
if uid not in configs:
|
||||
configs[uid] = _get_default_providers()
|
||||
return {
|
||||
"success": True,
|
||||
"providers": USER_AI_CONFIGS["default_user"]
|
||||
"providers": configs[uid]
|
||||
}
|
||||
|
||||
@router.post("/config/ai")
|
||||
async def save_user_ai_config(req: SaveAIConfigRequest):
|
||||
"""Save user's AI provider configurations."""
|
||||
USER_AI_CONFIGS["default_user"] = [p.dict() for p in req.providers]
|
||||
async def save_user_ai_config(req: SaveAIConfigRequest, authorization: Optional[str] = Header(None)):
|
||||
uid = _get_user_id(authorization)
|
||||
configs = _load_ai_configs()
|
||||
configs[uid] = [p.model_dump() for p in req.providers]
|
||||
_save_all(ai_configs=configs)
|
||||
return {
|
||||
"success": True,
|
||||
"message": "Đã lưu cấu hình AI Providers thành công!"
|
||||
|
||||
+57
-19
@@ -5,25 +5,69 @@ import time
|
||||
import base64
|
||||
import uuid
|
||||
import os
|
||||
import secrets
|
||||
from typing import Optional, Dict, Any
|
||||
from app.models.user import get_db_connection
|
||||
from app.config import settings
|
||||
|
||||
SECRET_KEY = os.getenv("SECRET_KEY", "sonicforge_secret_key_super_secure_2026")
|
||||
COOKIE_NAME = "sf_token"
|
||||
X_AUTH_HEADER = "X-Auth-Token"
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
def _load_or_create_secret_key() -> str:
|
||||
"""Persistent random SECRET_KEY.
|
||||
|
||||
Priority: env SECRET_KEY > {STORAGE_DIR}/.secret_key (auto-generated on
|
||||
first run). Never falls back to a hardcoded value: a known secret lets
|
||||
anyone forge admin tokens.
|
||||
"""
|
||||
Hash password using PBKDF2 HMAC SHA-256 with salt.
|
||||
env_key = os.getenv("SECRET_KEY", "").strip()
|
||||
if env_key:
|
||||
return env_key
|
||||
key_file = os.path.join(settings.STORAGE_DIR, ".secret_key")
|
||||
try:
|
||||
os.makedirs(settings.STORAGE_DIR, exist_ok=True)
|
||||
if os.path.exists(key_file):
|
||||
with open(key_file, "r") as f:
|
||||
key = f.read().strip()
|
||||
if len(key) >= 32:
|
||||
return key
|
||||
key = secrets.token_hex(32)
|
||||
with open(key_file, "w") as f:
|
||||
f.write(key)
|
||||
return key
|
||||
except Exception:
|
||||
# Last resort: ephemeral random key (all tokens invalid on restart).
|
||||
return secrets.token_hex(32)
|
||||
|
||||
SECRET_KEY = _load_or_create_secret_key()
|
||||
|
||||
def hash_password(password: str, salt: Optional[str] = None) -> str:
|
||||
"""
|
||||
Hash password using PBKDF2 HMAC SHA-256 with a per-user random salt.
|
||||
Guarantees raw passwords are NEVER stored or exposed in plaintext.
|
||||
Returns format: 'salt_hex:hashed_key_hex' (colon-delimited, stores both values).
|
||||
If salt is provided, uses that salt (for verification).
|
||||
If salt is None, generates a new 32-byte random salt.
|
||||
"""
|
||||
salt = b"sonicforge_crypto_salt_2026_secure_"
|
||||
key = hashlib.pbkdf2_hmac('sha256', password.encode('utf-8'), salt, 100000)
|
||||
return key.hex()
|
||||
if salt is None:
|
||||
salt = secrets.token_hex(32)
|
||||
key = hashlib.pbkdf2_hmac('sha256', password.encode('utf-8'), salt.encode('utf-8'), 600000)
|
||||
return f"{salt}:{key.hex()}"
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify plain password against PBKDF2 hashed password using constant-time comparison."""
|
||||
computed_hash = hash_password(plain_password)
|
||||
return hmac.compare_digest(computed_hash, hashed_password)
|
||||
def verify_password(plain_password: str, stored_value: str) -> bool:
|
||||
"""Verify plain password against stored 'salt:hash' using constant-time comparison.
|
||||
Also supports legacy-format hash (hex only, without salt) for backward compatibility."""
|
||||
if ':' in stored_value:
|
||||
parts = stored_value.split(':', 1)
|
||||
salt = parts[0]
|
||||
expected_hash = parts[1]
|
||||
computed = hash_password(plain_password, salt)
|
||||
computed_hash = computed.split(':', 1)[1]
|
||||
return hmac.compare_digest(computed_hash, expected_hash)
|
||||
else:
|
||||
expected_hash = stored_value
|
||||
computed_hash = hashlib.pbkdf2_hmac('sha256', plain_password.encode('utf-8'), b"sonicforge_crypto_salt_2026_secure_", 600000).hex()
|
||||
return hmac.compare_digest(computed_hash, expected_hash)
|
||||
|
||||
def create_token(user_id: str, username: str, role: str, must_change_password: bool) -> str:
|
||||
payload = {
|
||||
@@ -56,7 +100,7 @@ def decode_token(token: str) -> Optional[Dict[str, Any]]:
|
||||
return None
|
||||
|
||||
def seed_admin():
|
||||
"""Seed default admin account on initial launch if not exists or update password hash if outdated."""
|
||||
"""Seed default admin account on initial launch only — never resets existing password."""
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
@@ -64,9 +108,8 @@ def seed_admin():
|
||||
hashed_pwd = hash_password(default_pwd)
|
||||
now = time.time()
|
||||
|
||||
cursor.execute("SELECT id, hashed_password, must_change_password FROM users WHERE username = ?", ("admin",))
|
||||
row = cursor.fetchone()
|
||||
if not row:
|
||||
cursor.execute("SELECT id FROM users WHERE username = ?", ("admin",))
|
||||
if not cursor.fetchone():
|
||||
admin_id = str(uuid.uuid4())
|
||||
cursor.execute("""
|
||||
INSERT INTO users (id, username, email, hashed_password, role, must_change_password, created_at, is_active)
|
||||
@@ -78,11 +121,6 @@ def seed_admin():
|
||||
VALUES (?, 10240, 64)
|
||||
""", (admin_id,))
|
||||
conn.commit()
|
||||
else:
|
||||
# Kiểm tra và sửa password admin mặc định nếu cần
|
||||
if not verify_password(default_pwd, row["hashed_password"]):
|
||||
cursor.execute("UPDATE users SET hashed_password = ?, must_change_password = 1 WHERE id = ?", (hashed_pwd, row["id"]))
|
||||
conn.commit()
|
||||
|
||||
conn.close()
|
||||
|
||||
|
||||
@@ -0,0 +1,445 @@
|
||||
import os, logging, math
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import scipy.signal as signal
|
||||
from app.config import settings
|
||||
from app.core.vst_engine import (
|
||||
render_midi_events_to_audio,
|
||||
PluginManager,
|
||||
DecentSamplerManager,
|
||||
HAS_PEDALBOARD,
|
||||
HAS_PYFLUIDSYNTH,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
UPLOAD_SF_DIR = os.path.join(settings.STORAGE_DIR, "soundfonts")
|
||||
SYSTEM_SF_DIR = "/opt/daw_engine/soundfonts"
|
||||
|
||||
SYS_SOUNDFONTS = [
|
||||
("GeneralUser_GS.sf2", "GeneralUser GS"),
|
||||
("SGM_v2.01.sf2", "SGM v2.01"),
|
||||
("SGM-V2.01.sf2", "SGM v2.01"),
|
||||
]
|
||||
|
||||
def _find_sf2_path(sf_id: str) -> str:
|
||||
clean_id = sf_id.replace("sf_", "") if sf_id.startswith("sf_") else sf_id
|
||||
clean_lower = clean_id.lower()
|
||||
sf_lower = sf_id.lower()
|
||||
for base_dir in [UPLOAD_SF_DIR, SYSTEM_SF_DIR]:
|
||||
if not os.path.isdir(base_dir):
|
||||
continue
|
||||
for fname in os.listdir(base_dir):
|
||||
fbase, fext = os.path.splitext(fname)
|
||||
if fext.lower() in (".sf2", ".sf3") and (fbase.lower() == clean_lower or fbase.lower() == sf_lower):
|
||||
return os.path.join(base_dir, fname)
|
||||
static_dir = os.path.join(settings.APP_DIR, "static", "soundfonts")
|
||||
if os.path.isdir(static_dir):
|
||||
for fname in os.listdir(static_dir):
|
||||
fbase, fext = os.path.splitext(fname)
|
||||
if fext.lower() in (".sf2", ".sf3") and (fbase.lower() == clean_lower or fbase.lower() == sf_lower):
|
||||
return os.path.join(static_dir, fname)
|
||||
return ""
|
||||
|
||||
def _find_default_sf2() -> str:
|
||||
for sf_name, _ in SYS_SOUNDFONTS:
|
||||
for base_dir in [SYSTEM_SF_DIR, UPLOAD_SF_DIR]:
|
||||
p = os.path.join(base_dir, sf_name)
|
||||
if os.path.exists(p):
|
||||
return p
|
||||
return ""
|
||||
|
||||
if HAS_PEDALBOARD:
|
||||
try:
|
||||
from pedalboard import Pedalboard, Gain, Chorus, Reverb
|
||||
except Exception:
|
||||
HAS_PEDALBOARD = False
|
||||
|
||||
|
||||
|
||||
class PythonRenderEngine:
|
||||
def __init__(self, sample_rate=44100):
|
||||
self.sample_rate = sample_rate
|
||||
|
||||
def bars_to_samples(self, bars: float, bpm: float, time_sig_num: int) -> int:
|
||||
seconds_per_beat = 60.0 / max(20.0, bpm)
|
||||
seconds_per_bar = seconds_per_beat * time_sig_num
|
||||
return int(bars * seconds_per_bar * self.sample_rate)
|
||||
|
||||
def resolve_file_path(self, url_or_id: str) -> str:
|
||||
if not url_or_id:
|
||||
return ""
|
||||
base = os.path.basename(url_or_id)
|
||||
# Check uploads directory
|
||||
p_uploads = os.path.join(settings.UPLOADS_DIR, base)
|
||||
if os.path.exists(p_uploads):
|
||||
return p_uploads
|
||||
# Check processed directory
|
||||
p_processed = os.path.join(settings.PROCESSED_DIR, base)
|
||||
if os.path.exists(p_processed):
|
||||
return p_processed
|
||||
# Check general storage directory
|
||||
p_storage = os.path.join(settings.STORAGE_DIR, base)
|
||||
if os.path.exists(p_storage):
|
||||
return p_storage
|
||||
# Direct check
|
||||
if os.path.exists(url_or_id):
|
||||
return url_or_id
|
||||
return url_or_id
|
||||
|
||||
def render_session_container(self, session: dict, section_store: dict, bpm: float, time_sig_num: int, total_samples: int, _cache: dict = None) -> np.ndarray:
|
||||
session_buffer = np.zeros((2, total_samples), dtype=np.float32)
|
||||
|
||||
# Solo semantics: when any track is soloed, only soloed tracks sound.
|
||||
tracks = session.get("tracks", [])
|
||||
solo_ids = {t.get("id") for t in tracks if t.get("solo")}
|
||||
|
||||
_channel_counter = 0
|
||||
|
||||
for track in tracks:
|
||||
if solo_ids and track.get("id") not in solo_ids:
|
||||
continue
|
||||
track_type = track.get("type", "AUDIO")
|
||||
track_buffer = np.zeros((2, total_samples), dtype=np.float32)
|
||||
|
||||
# Parse synth_engine struct (Task C) — fall back to flat fields
|
||||
se = track.get("synth_engine", {}) or {}
|
||||
instrument_id = se.get("plugin_id") or track.get("instrument_id", "") or track.get("instrument", "")
|
||||
instrument_source = se.get("type") or track.get("instrument_source", "soundfont")
|
||||
soundfont_bank = se.get("soundfont_bank") if se.get("soundfont_bank") is not None else track.get("soundfont_bank", 0)
|
||||
soundfont_program = se.get("soundfont_program") if se.get("soundfont_program") is not None else track.get("soundfont_program", 0)
|
||||
soundfont_id = se.get("soundfont_id") or track.get("soundfont_id", "")
|
||||
is_percussion = track.get("is_percussion", False) or (soundfont_bank == 128)
|
||||
midi_channel = 9 if is_percussion else (_channel_counter % 9)
|
||||
if not is_percussion:
|
||||
_channel_counter += 1
|
||||
|
||||
for item in track.get("items", []):
|
||||
start_sample = self.bars_to_samples(item["start_bar"], bpm, time_sig_num)
|
||||
dur_samples = self.bars_to_samples(item["duration_bars"], bpm, time_sig_num)
|
||||
offset_sample = self.bars_to_samples(item["clip_start_offset_bars"], bpm, time_sig_num)
|
||||
|
||||
item_type = item.get("type")
|
||||
if item_type == "AUDIO_ITEM":
|
||||
source_data = item.get("source_data", {})
|
||||
audio_url = source_data.get("audio_file_url", "")
|
||||
resolved_path = self.resolve_file_path(audio_url)
|
||||
|
||||
if resolved_path and os.path.exists(resolved_path):
|
||||
try:
|
||||
audio_data, sr = sf.read(resolved_path, dtype='float32')
|
||||
if sr != self.sample_rate:
|
||||
# Proper resampling: previously a silent no-op that
|
||||
# played 48kHz audio at the wrong speed/pitch.
|
||||
from scipy.signal import resample_poly
|
||||
g = math.gcd(sr, self.sample_rate)
|
||||
audio_data = resample_poly(
|
||||
audio_data,
|
||||
up=self.sample_rate // g,
|
||||
down=sr // g,
|
||||
axis=-1,
|
||||
)
|
||||
sr = self.sample_rate
|
||||
|
||||
# Handle channel mapping (Mono/Stereo)
|
||||
if len(audio_data.shape) == 1:
|
||||
audio_data = np.vstack([audio_data, audio_data])
|
||||
else:
|
||||
audio_data = audio_data.T # Shape: (channels, samples)
|
||||
|
||||
# Trim source offset & duration
|
||||
src_len = audio_data.shape[1]
|
||||
if offset_sample < src_len:
|
||||
actual_dur = min(dur_samples, src_len - offset_sample)
|
||||
sliced_audio = audio_data[:, offset_sample : offset_sample + actual_dur]
|
||||
|
||||
# Apply gain
|
||||
gain_val = source_data.get("gain", 1.0)
|
||||
sliced_audio = sliced_audio * gain_val
|
||||
|
||||
# Write to track buffer with boundaries
|
||||
write_end = min(start_sample + sliced_audio.shape[1], total_samples)
|
||||
actual_len = write_end - start_sample
|
||||
if actual_len > 0:
|
||||
track_buffer[:, start_sample:write_end] += sliced_audio[:, :actual_len]
|
||||
except Exception as e:
|
||||
logger.warning("[RenderEngine] Error reading audio file %s: %s", resolved_path, e)
|
||||
|
||||
elif item_type == "MIDI_ITEM":
|
||||
source_data = item.get("source_data", {})
|
||||
notes = source_data.get("notes", [])
|
||||
|
||||
# Convert to midi events required by vst_engine
|
||||
midi_events = []
|
||||
for note in notes:
|
||||
note_start_bar = note["start_beat"] / time_sig_num
|
||||
# Filter notes within the non-destructive visible window
|
||||
offset_bar = item["clip_start_offset_bars"]
|
||||
dur_bar = item["duration_bars"]
|
||||
if note_start_bar >= offset_bar and note_start_bar < (offset_bar + dur_bar):
|
||||
rel_bar_in_item = note_start_bar - offset_bar
|
||||
target_global_bar = item["start_bar"] + rel_bar_in_item
|
||||
midi_events.append({
|
||||
"note": note["pitch"],
|
||||
"start_beat": target_global_bar * time_sig_num,
|
||||
"duration_beats": note["duration_beats"],
|
||||
"velocity": int(note.get("velocity", 0.8) * 127)
|
||||
})
|
||||
|
||||
if midi_events:
|
||||
try:
|
||||
plugin_mgr = PluginManager()
|
||||
vst = plugin_mgr.load_vst(instrument_id) if instrument_id else None
|
||||
|
||||
if instrument_source == "pianobook":
|
||||
dspreset_path = track.get("dspreset_path", "")
|
||||
if dspreset_path and os.path.exists(dspreset_path) and HAS_PEDALBOARD:
|
||||
from pedalboard import Pedalboard
|
||||
ds_manager = DecentSamplerManager()
|
||||
try:
|
||||
vst = ds_manager.create_decent_sampler_instance(dspreset_path)
|
||||
midi_messages = PluginManager.midi_events_to_messages(
|
||||
midi_events, bpm, self.sample_rate
|
||||
)
|
||||
total_needed = 0
|
||||
for ev in midi_events:
|
||||
end_sec = (ev.get("start_beat", 0) + ev.get("duration_beats", 1)) * (60.0 / bpm)
|
||||
dur_samples = int(end_sec * self.sample_rate)
|
||||
if dur_samples > total_needed:
|
||||
total_needed = dur_samples
|
||||
total_needed = max(total_needed, 1024)
|
||||
silent = np.zeros((2, total_needed), dtype=np.float32)
|
||||
board = Pedalboard([vst])
|
||||
synth_buffer = board(silent, sample_rate=self.sample_rate, midi_messages=midi_messages)
|
||||
except Exception as e:
|
||||
logger.warning(f"[RenderEngine] DecentSampler/Pianobook error: {e}")
|
||||
synth_buffer = render_midi_events_to_audio(
|
||||
midi_events=midi_events, sr=self.sample_rate, bpm=bpm, instrument='synth'
|
||||
)
|
||||
else:
|
||||
synth_buffer = render_midi_events_to_audio(
|
||||
midi_events=midi_events, sr=self.sample_rate, bpm=bpm, instrument='synth'
|
||||
)
|
||||
elif vst and HAS_PEDALBOARD:
|
||||
from pedalboard import Pedalboard
|
||||
midi_messages = PluginManager.midi_events_to_messages(
|
||||
midi_events, bpm, self.sample_rate,
|
||||
bank=soundfont_bank, program=soundfont_program
|
||||
)
|
||||
total_needed = 0
|
||||
for ev in midi_events:
|
||||
end_sec = (ev.get("start_beat", 0) + ev.get("duration_beats", 1)) * (60.0 / bpm)
|
||||
dur_samples = int(end_sec * self.sample_rate)
|
||||
if dur_samples > total_needed:
|
||||
total_needed = dur_samples
|
||||
total_needed = max(total_needed, 1024)
|
||||
silent = np.zeros((2, total_needed), dtype=np.float32)
|
||||
board = Pedalboard([vst])
|
||||
synth_buffer = board(silent, sample_rate=self.sample_rate, midi_messages=midi_messages)
|
||||
elif instrument_id and (instrument_id.startswith("sf_") or soundfont_id):
|
||||
sf_path = _find_sf2_path(soundfont_id or instrument_id)
|
||||
# 3-level fallback: selected SF → default SF → oscillator synth
|
||||
if not sf_path or not os.path.exists(sf_path) or not HAS_PYFLUIDSYNTH:
|
||||
if not sf_path or not os.path.exists(sf_path):
|
||||
logger.warning(f"[RenderEngine] SoundFont not found for {soundfont_id or instrument_id}, trying default")
|
||||
sf_path = _find_default_sf2() if HAS_PYFLUIDSYNTH else ""
|
||||
|
||||
if sf_path and os.path.exists(sf_path) and HAS_PYFLUIDSYNTH:
|
||||
import fluidsynth as _fs
|
||||
_settings = _fs.new_fluid_settings()
|
||||
_fs.fluid_settings_setnum(_settings, b'synth.sample-rate', float(self.sample_rate))
|
||||
_fl = _fs.new_fluid_synth(_settings)
|
||||
_fid = _fs.fluid_synth_sfload(_fl, sf_path.encode("utf-8"), 1)
|
||||
_fs.fluid_synth_program_select(_fl, midi_channel, _fid, soundfont_bank, soundfont_program)
|
||||
beat_sec = 60.0 / bpm
|
||||
total_sec = 0
|
||||
for ev in midi_events:
|
||||
end_sec = (ev.get("start_beat", 0) + ev.get("duration_beats", 1)) * beat_sec
|
||||
if end_sec > total_sec:
|
||||
total_sec = end_sec
|
||||
sf_total_samples = int((total_sec + 1.0) * self.sample_rate)
|
||||
midi_data = np.zeros((2, sf_total_samples), dtype=np.float32)
|
||||
_cursor = 0
|
||||
for ev in sorted(midi_events, key=lambda e: e.get("start_beat", 0)):
|
||||
note = ev.get("note", 60)
|
||||
velocity = ev.get("velocity", 100)
|
||||
start_beat = ev.get("start_beat", 0.0)
|
||||
dur_beats = ev.get("duration_beats", 1.0)
|
||||
start_sec = start_beat * beat_sec
|
||||
dur_sec = dur_beats * beat_sec
|
||||
start_s = int(start_sec * self.sample_rate)
|
||||
dur_s = int(dur_sec * self.sample_rate)
|
||||
# Advance synth time by rendering silence
|
||||
if start_s > _cursor:
|
||||
gap = start_s - _cursor
|
||||
_fs.fluid_synth_write_s16_stereo(_fl, gap)
|
||||
_cursor = start_s
|
||||
# Start note
|
||||
_fs.fluid_synth_noteon(_fl, midi_channel, note, min(velocity, 127))
|
||||
block_s16 = _fs.fluid_synth_write_s16_stereo(_fl, dur_s)
|
||||
_fs.fluid_synth_noteoff(_fl, midi_channel, note)
|
||||
block = block_s16.astype(np.float32).reshape(-1, 2).T / 32768.0
|
||||
end_s = min(_cursor + block.shape[1], sf_total_samples)
|
||||
actual = end_s - _cursor
|
||||
if actual > 0 and block.shape[1] > 0:
|
||||
midi_data[:, _cursor:end_s] += block[:, :actual]
|
||||
_cursor = end_s
|
||||
synth_buffer = midi_data
|
||||
_fs.delete_fluid_synth(_fl)
|
||||
else:
|
||||
if not HAS_PYFLUIDSYNTH:
|
||||
logger.warning("[RenderEngine] pyfluidsynth not available, falling back to oscillator synth")
|
||||
synth_buffer = render_midi_events_to_audio(
|
||||
midi_events=midi_events, sr=self.sample_rate, bpm=bpm, instrument='synth'
|
||||
)
|
||||
else:
|
||||
synth_buffer = render_midi_events_to_audio(
|
||||
midi_events=midi_events, sr=self.sample_rate, bpm=bpm, instrument='synth'
|
||||
)
|
||||
actual_len = min(synth_buffer.shape[1], total_samples)
|
||||
track_buffer[:, :actual_len] += synth_buffer[:, :actual_len]
|
||||
except Exception as e:
|
||||
logger.warning("[RenderEngine] Error rendering MIDI: %s", e)
|
||||
|
||||
elif item_type == "SECTION_ITEM":
|
||||
source_data = item.get("source_data", {})
|
||||
sec_id = source_data.get("referenced_section_id", "")
|
||||
if sec_id and sec_id in section_store:
|
||||
# Render nested section recursively, cached per section id
|
||||
# so repeated section instances don't re-render every time.
|
||||
cache = _cache if _cache is not None else {}
|
||||
if sec_id in cache:
|
||||
sec_buffer = cache[sec_id]
|
||||
else:
|
||||
sec_buffer = self.render_session_container(
|
||||
session=section_store[sec_id],
|
||||
section_store=section_store,
|
||||
bpm=bpm,
|
||||
time_sig_num=time_sig_num,
|
||||
total_samples=total_samples,
|
||||
_cache=cache,
|
||||
)
|
||||
cache[sec_id] = sec_buffer
|
||||
|
||||
# Apply non-destructive crop/slicing on section buffer
|
||||
if offset_sample < total_samples:
|
||||
actual_dur = min(dur_samples, total_samples - offset_sample)
|
||||
sliced_sec = sec_buffer[:, offset_sample : offset_sample + actual_dur]
|
||||
|
||||
# Write to track buffer
|
||||
write_end = min(start_sample + sliced_sec.shape[1], total_samples)
|
||||
actual_len = write_end - start_sample
|
||||
if actual_len > 0:
|
||||
track_buffer[:, start_sample:write_end] += sliced_sec[:, :actual_len]
|
||||
|
||||
# Apply Track Gain (via Pedalboard or fallback)
|
||||
vol_db = track.get("volume_db", 0.0)
|
||||
pan = track.get("pan", 0.0)
|
||||
mute = track.get("mute", False)
|
||||
|
||||
if mute:
|
||||
continue
|
||||
|
||||
# Apply Track FX (Chorus or Reverb)
|
||||
fx_type = track.get("fx_type")
|
||||
if fx_type == "chorus":
|
||||
if HAS_PEDALBOARD:
|
||||
try:
|
||||
board = Pedalboard([Chorus(rate_hz=1.5, depth=0.25)])
|
||||
track_buffer = board(track_buffer, sample_rate=self.sample_rate)
|
||||
except Exception as e:
|
||||
logger.warning("[RenderEngine] Pedalboard Chorus failed: %s", e)
|
||||
else:
|
||||
# Fallback chorus using simple LFO delay modulation in scipy/numpy
|
||||
try:
|
||||
# 1.5 Hz sine LFO, modulating delay time between 15ms and 25ms (average 20ms)
|
||||
lfo = 0.020 + 0.005 * np.sin(2 * np.pi * 1.5 * np.arange(total_samples) / self.sample_rate)
|
||||
dry = track_buffer * 0.6
|
||||
wet = np.zeros_like(track_buffer)
|
||||
for ch in range(2):
|
||||
indices = np.arange(total_samples) - (lfo * self.sample_rate)
|
||||
indices = np.clip(indices, 0, total_samples - 1).astype(np.int32)
|
||||
wet[ch, :] = track_buffer[ch, indices]
|
||||
track_buffer = dry + wet * 0.5
|
||||
except Exception as e:
|
||||
logger.warning("[RenderEngine] Fallback Chorus failed: %s", e)
|
||||
elif fx_type == "reverb":
|
||||
if HAS_PEDALBOARD:
|
||||
try:
|
||||
board = Pedalboard([Reverb(room_size=0.5, wet_level=0.4, dry_level=0.6)])
|
||||
track_buffer = board(track_buffer, sample_rate=self.sample_rate)
|
||||
except Exception as e:
|
||||
logger.warning("[RenderEngine] Pedalboard Reverb failed: %s", e)
|
||||
else:
|
||||
# Fallback reverb using exponentially decaying noise room impulse response
|
||||
try:
|
||||
# Generate impulse response (decaying noise)
|
||||
len_ir = int(self.sample_rate * 2.0)
|
||||
t_ir = np.arange(len_ir) / self.sample_rate
|
||||
decay = np.exp(-t_ir / 0.5)
|
||||
ir_l = (np.random.rand(len_ir) * 2 - 1) * decay
|
||||
ir_r = (np.random.rand(len_ir) * 2 - 1) * decay
|
||||
|
||||
dry = track_buffer * 0.6
|
||||
wet = np.zeros_like(track_buffer)
|
||||
for ch in range(2):
|
||||
ir = ir_l if ch == 0 else ir_r
|
||||
# Convolve
|
||||
conv = signal.convolve(track_buffer[ch, :], ir, mode='full')[:total_samples]
|
||||
wet[ch, :] = conv
|
||||
track_buffer = dry + wet * 0.4
|
||||
except Exception as e:
|
||||
logger.warning("[RenderEngine] Fallback Reverb failed: %s", e)
|
||||
|
||||
# Process track volume
|
||||
if HAS_PEDALBOARD:
|
||||
try:
|
||||
board = Pedalboard([Gain(gain_db=vol_db)])
|
||||
processed_track = board(track_buffer, sample_rate=self.sample_rate)
|
||||
except Exception:
|
||||
gain_linear = 10 ** (vol_db / 20.0)
|
||||
processed_track = track_buffer * gain_linear
|
||||
else:
|
||||
gain_linear = 10 ** (vol_db / 20.0)
|
||||
processed_track = track_buffer * gain_linear
|
||||
|
||||
# Apply Track Pan
|
||||
if pan != 0.0:
|
||||
# Constant power panning
|
||||
theta = ((np.clip(pan, -1.0, 1.0) + 1.0) / 2.0) * (np.pi / 2.0)
|
||||
processed_track[0, :] *= np.cos(theta)
|
||||
processed_track[1, :] *= np.sin(theta)
|
||||
|
||||
# Mix track to session
|
||||
session_buffer += processed_track
|
||||
|
||||
return session_buffer
|
||||
|
||||
def render_project(self, project_json: dict, output_filepath: str):
|
||||
bpm = project_json["metadata"]["bpm"]
|
||||
time_sig_num = project_json["metadata"].get("time_signature_numerator", 4)
|
||||
main_session = project_json["main_session"]
|
||||
section_store = project_json.get("section_store", {})
|
||||
|
||||
# Compute total project samples
|
||||
total_bars = main_session.get("length_bars", 16.0)
|
||||
total_samples = self.bars_to_samples(total_bars, bpm, time_sig_num)
|
||||
|
||||
# Render main session
|
||||
master_buffer = self.render_session_container(
|
||||
session=main_session,
|
||||
section_store=section_store,
|
||||
bpm=bpm,
|
||||
time_sig_num=time_sig_num,
|
||||
total_samples=total_samples,
|
||||
_cache={},
|
||||
)
|
||||
|
||||
# Normalization to prevent clipping
|
||||
max_peak = np.max(np.abs(master_buffer))
|
||||
if max_peak > 1.0:
|
||||
master_buffer /= max_peak
|
||||
|
||||
# Write final output file
|
||||
sf.write(output_filepath, master_buffer.T, self.sample_rate)
|
||||
return output_filepath
|
||||
@@ -0,0 +1,551 @@
|
||||
import os
|
||||
import struct
|
||||
import subprocess
|
||||
import logging
|
||||
import wave
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SF_TARGET_DIRS = [
|
||||
"/opt/daw_engine/soundfonts",
|
||||
]
|
||||
|
||||
|
||||
def _find_chunk(data, chunk_id, offset=12, max_size=0):
|
||||
pos = offset
|
||||
end = len(data) if max_size == 0 else offset + max_size
|
||||
while pos + 8 <= end:
|
||||
ck_id = data[pos:pos + 4]
|
||||
ck_size = struct.unpack("<I", data[pos + 4:pos + 8])[0]
|
||||
if ck_id == chunk_id:
|
||||
return (pos, ck_id, ck_size, pos + 8)
|
||||
if ck_id == b"LIST" and pos + 12 <= end:
|
||||
list_type = data[pos + 8:pos + 12]
|
||||
# Check if we're looking for a specific list type
|
||||
if chunk_id == b"smpl" and list_type == b"sdta":
|
||||
inner = _find_chunk_in_list(data, chunk_id, pos + 12, ck_size - 4)
|
||||
if inner:
|
||||
return inner
|
||||
elif chunk_id == b"LIST":
|
||||
# When searching for LIST by type, check if it's sdta
|
||||
pass
|
||||
else:
|
||||
inner = _find_chunk_in_list(data, chunk_id, pos + 12, ck_size - 4)
|
||||
if inner:
|
||||
return inner
|
||||
pos += 8 + ck_size
|
||||
if ck_size % 2 == 1:
|
||||
pos += 1
|
||||
return None
|
||||
|
||||
|
||||
def _find_list_of_type(data, list_type_id, offset=12):
|
||||
pos = offset
|
||||
while pos + 12 <= len(data):
|
||||
ck_id = data[pos:pos + 4]
|
||||
ck_size = struct.unpack("<I", data[pos + 4:pos + 8])[0]
|
||||
if ck_id == b"LIST":
|
||||
form_type = data[pos + 8:pos + 12]
|
||||
if form_type == list_type_id:
|
||||
return (pos, ck_id, ck_size, pos + 12)
|
||||
pos += 8 + ck_size
|
||||
if ck_size % 2 == 1:
|
||||
pos += 1
|
||||
return None
|
||||
|
||||
|
||||
def _find_chunk_in_list(data, chunk_id, list_data_offset, list_data_size):
|
||||
pos = list_data_offset
|
||||
end = list_data_offset + list_data_size
|
||||
while pos + 8 <= end:
|
||||
ck_id = data[pos:pos + 4]
|
||||
ck_size = struct.unpack("<I", data[pos + 4:pos + 8])[0]
|
||||
if ck_id == chunk_id:
|
||||
return (pos, ck_id, ck_size, pos + 8)
|
||||
pos += 8 + ck_size
|
||||
if ck_size % 2 == 1:
|
||||
pos += 1
|
||||
return None
|
||||
|
||||
|
||||
def _update_size(data, offset, new_size):
|
||||
return data[:offset] + struct.pack("<I", new_size) + data[offset + 4:]
|
||||
|
||||
|
||||
class SoundFontConverter:
|
||||
def __init__(self, target_dirs=None):
|
||||
self.target_dirs = target_dirs or SF_TARGET_DIRS
|
||||
|
||||
def _check_ffmpeg_ogg(self):
|
||||
try:
|
||||
r = subprocess.run(["ffmpeg", "-encoders"], capture_output=True, text=True, timeout=5)
|
||||
return "libvorbis" in r.stdout
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _encode_sample_ogg(self, pcm: bytes, rate: int, tmp_dir: str) -> bytes:
|
||||
"""Encode a single mono 16-bit PCM slice to an Ogg Vorbis stream."""
|
||||
tmp_wav = os.path.join(tmp_dir, "sample_tmp.wav")
|
||||
tmp_ogg = os.path.join(tmp_dir, "sample_tmp.ogg")
|
||||
try:
|
||||
with open(tmp_wav, "wb") as fw:
|
||||
with wave.open(fw, "wb") as w:
|
||||
w.setnchannels(1)
|
||||
w.setsampwidth(2)
|
||||
w.setframerate(rate)
|
||||
w.writeframes(pcm)
|
||||
subprocess.run([
|
||||
"ffmpeg", "-y", "-i", tmp_wav,
|
||||
"-c:a", "libvorbis", "-q:a", "3",
|
||||
"-f", "ogg", tmp_ogg
|
||||
], capture_output=True, timeout=600, check=True)
|
||||
with open(tmp_ogg, "rb") as fo:
|
||||
return fo.read()
|
||||
except Exception as e:
|
||||
logger.error(f"Sample OGG encode failed: {e}")
|
||||
return b""
|
||||
finally:
|
||||
for p in [tmp_wav, tmp_ogg]:
|
||||
try:
|
||||
if os.path.exists(p):
|
||||
os.remove(p)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _sf2_to_sf3_python(self, sf2_path: str, sf3_path: str) -> bool:
|
||||
has_ogg = self._check_ffmpeg_ogg()
|
||||
if not has_ogg:
|
||||
logger.warning("ffmpeg with libvorbis not available, cannot convert to SF3")
|
||||
return False
|
||||
|
||||
try:
|
||||
with open(sf2_path, "rb") as f:
|
||||
data = f.read()
|
||||
except Exception as e:
|
||||
logger.error(f"Cannot read {sf2_path}: {e}")
|
||||
return False
|
||||
|
||||
if len(data) < 12 or data[:4] != b"RIFF" or data[8:12] != b"sfbk":
|
||||
logger.warning("Not a valid SF2 file")
|
||||
return False
|
||||
|
||||
# Find smpl chunk (PCM sample data) recursively
|
||||
smpl = _find_chunk(data, b"smpl")
|
||||
if smpl is None:
|
||||
logger.warning("No smpl chunk found in SF2")
|
||||
return False
|
||||
|
||||
smpl_head_off, _, smpl_old_size, smpl_data_off = smpl
|
||||
sample_data = data[smpl_data_off:smpl_data_off + smpl_old_size]
|
||||
|
||||
if len(sample_data) < 16:
|
||||
logger.warning("Sample data too small")
|
||||
return False
|
||||
|
||||
# Locate shdr (sample headers) inside the pdta LIST
|
||||
pdta = _find_list_of_type(data, b"pdta", 12)
|
||||
if not pdta:
|
||||
logger.warning("No pdta LIST found")
|
||||
return False
|
||||
_, _, pdta_size, pdta_data_off = pdta
|
||||
shdr = _find_chunk_in_list(data, b"shdr", pdta_data_off, pdta_size - 4)
|
||||
if not shdr:
|
||||
logger.warning("No shdr chunk found")
|
||||
return False
|
||||
_, _, shdr_size, shdr_data_off = shdr
|
||||
if shdr_size <= 0 or shdr_size % 46 != 0:
|
||||
logger.warning(f"Invalid shdr size {shdr_size}")
|
||||
return False
|
||||
n_samples = shdr_size // 46
|
||||
|
||||
# Locate ifil (version) inside the INFO LIST
|
||||
ifil_abs = None
|
||||
info = _find_list_of_type(data, b"INFO", 12)
|
||||
if info:
|
||||
_, _, info_size, info_data_off = info
|
||||
ifil = _find_chunk_in_list(data, b"ifil", info_data_off, info_size - 4)
|
||||
if ifil:
|
||||
ifil_abs = ifil[3]
|
||||
|
||||
import tempfile
|
||||
tmp_dir = tempfile.mkdtemp(prefix="sf3conv_")
|
||||
ogg_parts = []
|
||||
new_shdr = bytearray()
|
||||
byte_offset = 0
|
||||
ogg_bytes = 0
|
||||
try:
|
||||
for i in range(n_samples):
|
||||
base = shdr_data_off + i * 46
|
||||
# shdr layout: name[20] | start(u32) end(u32) loopstart(u32) loopend(u32) samplerate(i32) ...
|
||||
start, end, startloop, endloop, rate = struct.unpack("<IIIIi", data[base + 20:base + 40])
|
||||
sampletype = struct.unpack("<H", data[base + 44:base + 46])[0]
|
||||
ogg_stream = b""
|
||||
if end >= start and start * 2 < len(sample_data):
|
||||
pcm = sample_data[start * 2:(end + 1) * 2]
|
||||
if len(pcm) >= 4:
|
||||
safe_rate = rate if 1000 < rate < 192000 else 44100
|
||||
ogg_stream = self._encode_sample_ogg(pcm, safe_rate, tmp_dir)
|
||||
# SF3: start/end are byte offsets into the concatenated OGG stream.
|
||||
# FluidSynth treats shdr `end` as EXCLUSIVE (reads [start..end-1]),
|
||||
# so end = start + ogg length.
|
||||
new_start = byte_offset
|
||||
new_end = byte_offset + len(ogg_stream)
|
||||
# OGG loop pointers are relative to the individual decompressed sample
|
||||
new_sloop = (startloop - start) if (startloop > start and startloop <= end) else 0
|
||||
new_eloop = (endloop - start) if (endloop > start and endloop <= end) else 0
|
||||
# Mark the sample as Ogg Vorbis compressed (FLUID_SAMPLETYPE_OGG_VORBIS = 0x20)
|
||||
new_stype = sampletype | 0x20
|
||||
new_shdr += data[base:base + 20] # sample name
|
||||
new_shdr += struct.pack("<IIIIi", new_start, new_end, new_sloop, new_eloop, rate)
|
||||
new_shdr += data[base + 40:base + 44] # originalpitch, correction, samplelink
|
||||
new_shdr += struct.pack("<H", new_stype)
|
||||
ogg_parts.append(ogg_stream)
|
||||
ogg_bytes += len(ogg_stream)
|
||||
byte_offset = new_start + len(ogg_stream)
|
||||
if len(ogg_stream) % 2 == 1:
|
||||
ogg_parts.append(b"\x00")
|
||||
byte_offset += 1
|
||||
|
||||
if not ogg_parts:
|
||||
logger.warning("No samples to encode")
|
||||
return False
|
||||
|
||||
ogg_padded = b"".join(ogg_parts)
|
||||
if len(ogg_padded) % 2 == 1:
|
||||
ogg_padded += b"\x00"
|
||||
new_smpl_size = len(ogg_padded)
|
||||
old_padded = smpl_old_size + (1 if smpl_old_size % 2 == 1 else 0)
|
||||
delta = len(ogg_padded) - old_padded
|
||||
|
||||
# Rebuild file: replace smpl chunk and update all sizes
|
||||
out = bytearray()
|
||||
out.extend(data[:smpl_head_off]) # up to smpl chunk header (excl. id)
|
||||
out.extend(b"smpl") # chunk id (required for valid SF3)
|
||||
out.extend(struct.pack("<I", new_smpl_size)) # new smpl size
|
||||
out.extend(ogg_padded) # concatenated per-sample OGG streams
|
||||
rest_off = smpl_head_off + 8 + old_padded
|
||||
rest = bytearray(data[rest_off:])
|
||||
# Patch the shdr sample headers inside the pdta copy
|
||||
shdr_in_rest = shdr_data_off - rest_off
|
||||
if shdr_in_rest < 0 or shdr_in_rest + shdr_size > len(rest):
|
||||
logger.warning("shdr not found after smpl chunk")
|
||||
return False
|
||||
rest[shdr_in_rest:shdr_in_rest + shdr_size] = new_shdr
|
||||
out.extend(rest)
|
||||
|
||||
data_out = bytes(out)
|
||||
|
||||
# SF3 requires version 3.0 so FluidSynth treats it as an SF3 file
|
||||
if ifil_abs is not None and ifil_abs + 4 <= len(data_out):
|
||||
data_out = data_out[:ifil_abs] + struct.pack("<HH", 3, 0) + data_out[ifil_abs + 4:]
|
||||
|
||||
# Find sdta LIST and update its size
|
||||
sdta = _find_list_of_type(data_out, b"sdta", 12)
|
||||
if sdta:
|
||||
lh_off, _, lh_size, ld_off = sdta
|
||||
data_out = _update_size(data_out, lh_off + 4, lh_size + delta)
|
||||
|
||||
# Update RIFF root size
|
||||
new_total = len(data_out) - 8
|
||||
data_out = _update_size(data_out, 4, new_total)
|
||||
|
||||
with open(sf3_path, "wb") as fout:
|
||||
fout.write(data_out)
|
||||
|
||||
# Validate: check that RIFF size matches actual size
|
||||
written = os.path.getsize(sf3_path)
|
||||
parsed_riff = struct.unpack("<I", data_out[4:8])[0]
|
||||
if parsed_riff != written - 8:
|
||||
logger.warning(f"Size mismatch: RIFF says {parsed_riff}, actual is {written - 8}")
|
||||
return False
|
||||
|
||||
logger.info(f"Converted {n_samples} samples -> {ogg_bytes} bytes OGG ({100 * (1 - ogg_bytes / max(len(sample_data), 1)):.0f}% smaller)")
|
||||
return True
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error("Ogg conversion timed out")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Conversion error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
finally:
|
||||
try:
|
||||
import shutil
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _sf3_plays_audio(path: str) -> bool:
|
||||
"""Verify a SoundFont actually loads and renders audible audio (guards
|
||||
against shipping malformed SF3 files that silently play nothing).
|
||||
|
||||
Uses the low-level CFFI binding (new_fluid_synth / write_float) — the
|
||||
high-level Synth() class does not exist in this binding, so it is never
|
||||
used here.
|
||||
"""
|
||||
if not os.path.exists(path):
|
||||
return False
|
||||
try:
|
||||
import fluidsynth as _fs
|
||||
import numpy as np
|
||||
_settings = _fs.new_fluid_settings()
|
||||
_fl = _fs.new_fluid_synth(_settings)
|
||||
try:
|
||||
h = _fs.fluid_synth_sfload(_fl, path.encode("utf-8"), 1)
|
||||
if h < 0:
|
||||
return False
|
||||
_fs.fluid_synth_program_select(_fl, 0, h, 0, 0)
|
||||
_fs.fluid_synth_noteon(_fl, 0, 60, 100)
|
||||
frames = 8820 # 0.2s
|
||||
buf = np.zeros(frames * 2, dtype=np.float32)
|
||||
_fs.fluid_synth_write_float(
|
||||
_fl, frames, buf.ctypes.data, 0, 1,
|
||||
buf.ctypes.data + frames * 4, 0, 1
|
||||
)
|
||||
_fs.fluid_synth_noteoff(_fl, 0, 60)
|
||||
rms = float(np.sqrt(np.mean(buf ** 2)))
|
||||
return rms > 1e-4
|
||||
finally:
|
||||
try:
|
||||
_fs.delete_fluid_synth(_fl)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _find_sf3_converter():
|
||||
for exe in ["fluidsynth", "mscore"]:
|
||||
try:
|
||||
subprocess.run([exe, "--help"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=2.0)
|
||||
return exe
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
return "python"
|
||||
|
||||
def convert_sf2_to_sf3(self, sf2_path: str) -> str:
|
||||
if not os.path.exists(sf2_path):
|
||||
raise FileNotFoundError(f"Source SF2 file not found: {sf2_path}")
|
||||
|
||||
sf3_path = os.path.splitext(sf2_path)[0] + ".sf3"
|
||||
|
||||
# Reuse a working SF3 if it is newer than the SF2 AND actually plays
|
||||
# audio. Malformed SF3s (e.g. produced by an older converter) are
|
||||
# re-converted automatically instead of being shipped silently broken.
|
||||
if os.path.exists(sf3_path) and os.path.getmtime(sf3_path) >= os.path.getmtime(sf2_path):
|
||||
if self._sf3_plays_audio(sf3_path):
|
||||
logger.info(f"SF3 already up-to-date: {sf3_path}")
|
||||
return sf3_path
|
||||
logger.warning(f"Existing SF3 does not play audio, re-converting: {sf3_path}")
|
||||
try:
|
||||
os.remove(sf3_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
converter = self._find_sf3_converter()
|
||||
try:
|
||||
logger.info(f"Converting '{sf2_path}' -> '{sf3_path}' using {converter}...")
|
||||
if converter == "fluidsynth":
|
||||
converter = "python"
|
||||
|
||||
if converter == "mscore":
|
||||
cmd = ["mscore", "-o", sf3_path, sf2_path]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
|
||||
if result.returncode == 0 and os.path.exists(sf3_path):
|
||||
logger.info(f"Created SF3 via mscore: {sf3_path} ({os.path.getsize(sf3_path)/1024/1024:.1f}MB)")
|
||||
return sf3_path
|
||||
converter = "python"
|
||||
|
||||
if converter == "python":
|
||||
if self._sf2_to_sf3_python(sf2_path, sf3_path) and os.path.exists(sf3_path):
|
||||
if self._sf3_plays_audio(sf3_path):
|
||||
size_mb = os.path.getsize(sf3_path) / (1024 * 1024)
|
||||
logger.info(f"Created SF3: {sf3_path} ({size_mb:.2f} MB)")
|
||||
return sf3_path
|
||||
# Conversion produced a broken file — never ship it
|
||||
logger.warning(f"Converted SF3 failed audio verification, removing: {sf3_path}")
|
||||
try:
|
||||
os.remove(sf3_path)
|
||||
except Exception:
|
||||
pass
|
||||
logger.warning(f"Python converter failed for {sf2_path}, returning SF2 path")
|
||||
return sf2_path
|
||||
|
||||
return sf2_path
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error(f"Conversion timed out for {sf2_path}")
|
||||
return sf2_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error converting {sf2_path}: {e}")
|
||||
return sf2_path
|
||||
|
||||
def sf3_to_sf2(self, sf3_path: str, sf2_path: str = None) -> str:
|
||||
"""Convert an SF3 (Ogg Vorbis samples) soundfont into a playable SF2.
|
||||
|
||||
The client FluidSynth WASM cannot decode Ogg Vorbis/SF3 samples, so any
|
||||
SF3 soundfont (converted or uploaded) plays silence. Decompressing to
|
||||
SF2 makes every instrument audible again.
|
||||
"""
|
||||
if not os.path.exists(sf3_path):
|
||||
raise FileNotFoundError(f"Source SF3 file not found: {sf3_path}")
|
||||
|
||||
try:
|
||||
with open(sf3_path, "rb") as f:
|
||||
data = f.read()
|
||||
except Exception as e:
|
||||
raise IOError(f"Cannot read {sf3_path}: {e}")
|
||||
|
||||
if len(data) < 12 or data[:4] != b"RIFF" or data[8:12] != b"sfbk":
|
||||
raise ValueError(f"Not a valid SoundFont file: {sf3_path}")
|
||||
|
||||
smpl = _find_chunk(data, b"smpl")
|
||||
if not smpl:
|
||||
raise ValueError("No smpl chunk found")
|
||||
smpl_head_off, _, smpl_old_size, smpl_data_off = smpl
|
||||
sample_data = data[smpl_data_off:smpl_data_off + smpl_old_size]
|
||||
|
||||
pdta = _find_list_of_type(data, b"pdta", 12)
|
||||
if not pdta:
|
||||
raise ValueError("No pdta LIST found")
|
||||
_, _, pdta_size, pdta_data_off = pdta
|
||||
shdr = _find_chunk_in_list(data, b"shdr", pdta_data_off, pdta_size - 4)
|
||||
if not shdr:
|
||||
raise ValueError("No shdr chunk found")
|
||||
_, _, shdr_size, shdr_data_off = shdr
|
||||
if shdr_size <= 0 or shdr_size % 46 != 0:
|
||||
raise ValueError(f"Invalid shdr size {shdr_size}")
|
||||
n_samples = shdr_size // 46
|
||||
|
||||
ifil_abs = None
|
||||
info = _find_list_of_type(data, b"INFO", 12)
|
||||
if info:
|
||||
_, _, info_size, info_data_off = info
|
||||
ifil = _find_chunk_in_list(data, b"ifil", info_data_off, info_size - 4)
|
||||
if ifil:
|
||||
ifil_abs = ifil[3]
|
||||
|
||||
import tempfile
|
||||
tmp_dir = tempfile.mkdtemp(prefix="sf2conv_")
|
||||
pcm_parts = []
|
||||
new_shdr = bytearray()
|
||||
frame_offset = 0
|
||||
total_pcm_bytes = 0
|
||||
try:
|
||||
for i in range(n_samples):
|
||||
base = shdr_data_off + i * 46
|
||||
name = data[base:base + 20]
|
||||
start, end, loopstart, loopend, rate = struct.unpack("<IIIIi", data[base + 20:base + 40])
|
||||
sampletype = struct.unpack("<H", data[base + 44:base + 46])[0]
|
||||
frames = 0
|
||||
pcm = b""
|
||||
if end > start and start < len(sample_data):
|
||||
# FluidSynth reads the OGG region as [start..end-1]
|
||||
ogg = sample_data[start:min(end, len(sample_data))]
|
||||
if len(ogg) >= 4 and ogg[:4] == b"OggS":
|
||||
pcm = self._decode_sample_ogg(ogg, tmp_dir)
|
||||
frames = len(pcm) // 2
|
||||
new_start = frame_offset
|
||||
new_end = frame_offset + frames
|
||||
# SF3 loop points are relative to the decompressed sample; SF2 needs absolute
|
||||
new_loopstart = loopstart + new_start if (loopstart or loopend) else 0
|
||||
new_loopend = loopend + new_start if (loopstart or loopend) else 0
|
||||
# Clear the Ogg Vorbis flag; keep mono/left/right/linked flags
|
||||
new_stype = sampletype & ~0x20
|
||||
new_shdr += name
|
||||
new_shdr += struct.pack("<IIIIi", new_start, new_end, new_loopstart, new_loopend, rate)
|
||||
new_shdr += data[base + 40:base + 44]
|
||||
new_shdr += struct.pack("<H", new_stype)
|
||||
pcm_parts.append(pcm)
|
||||
frame_offset += frames
|
||||
if frames:
|
||||
total_pcm_bytes += frames * 2
|
||||
|
||||
if total_pcm_bytes == 0:
|
||||
logger.warning("SF3 contained no decodable samples")
|
||||
return sf3_path
|
||||
|
||||
new_smpl_size = total_pcm_bytes
|
||||
if new_smpl_size % 2 == 1:
|
||||
new_smpl_size += 1
|
||||
old_padded = smpl_old_size + (1 if smpl_old_size % 2 == 1 else 0)
|
||||
delta = new_smpl_size - old_padded
|
||||
|
||||
out = bytearray()
|
||||
out.extend(data[:smpl_head_off])
|
||||
out.extend(b"smpl")
|
||||
out.extend(struct.pack("<I", new_smpl_size))
|
||||
for part in pcm_parts:
|
||||
out.extend(part)
|
||||
if total_pcm_bytes % 2 == 1:
|
||||
out.extend(b"\x00")
|
||||
rest_off = smpl_head_off + 8 + old_padded
|
||||
rest = bytearray(data[rest_off:])
|
||||
shdr_in_rest = shdr_data_off - rest_off
|
||||
if shdr_in_rest < 0 or shdr_in_rest + shdr_size > len(rest):
|
||||
logger.warning("shdr not found after smpl chunk")
|
||||
return sf3_path
|
||||
rest[shdr_in_rest:shdr_in_rest + shdr_size] = new_shdr
|
||||
out.extend(rest)
|
||||
|
||||
data_out = bytes(out)
|
||||
|
||||
# Back to SF2 version 2.01
|
||||
if ifil_abs is not None and ifil_abs + 4 <= len(data_out):
|
||||
data_out = data_out[:ifil_abs] + struct.pack("<HH", 2, 1) + data_out[ifil_abs + 4:]
|
||||
|
||||
sdta = _find_list_of_type(data_out, b"sdta", 12)
|
||||
if sdta:
|
||||
lh_off, _, lh_size, ld_off = sdta
|
||||
data_out = _update_size(data_out, lh_off + 4, lh_size + delta)
|
||||
|
||||
data_out = _update_size(data_out, 4, len(data_out) - 8)
|
||||
|
||||
sf2_path = sf2_path or (os.path.splitext(sf3_path)[0] + ".sf2")
|
||||
with open(sf2_path, "wb") as fout:
|
||||
fout.write(data_out)
|
||||
logger.info(f"Converted SF3 -> SF2: {sf2_path} ({os.path.getsize(sf2_path) / 1048576:.1f} MB)")
|
||||
return sf2_path
|
||||
finally:
|
||||
try:
|
||||
import shutil
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _decode_sample_ogg(self, ogg: bytes, tmp_dir: str) -> bytes:
|
||||
"""Decode an Ogg Vorbis stream to mono 16-bit PCM; returns PCM bytes."""
|
||||
tmp_ogg = os.path.join(tmp_dir, "sample.ogg")
|
||||
tmp_pcm = os.path.join(tmp_dir, "sample.pcm")
|
||||
try:
|
||||
with open(tmp_ogg, "wb") as fo:
|
||||
fo.write(ogg)
|
||||
r = subprocess.run(
|
||||
["ffmpeg", "-y", "-v", "error", "-i", tmp_ogg, "-f", "s16le", "-ac", "1", tmp_pcm],
|
||||
capture_output=True, timeout=600)
|
||||
if r.returncode != 0:
|
||||
logger.warning(f"OGG decode failed: {r.stderr.decode(errors='replace')[:120]}")
|
||||
return b""
|
||||
with open(tmp_pcm, "rb") as fp:
|
||||
return fp.read()
|
||||
except Exception as e:
|
||||
logger.warning(f"OGG decode error: {e}")
|
||||
return b""
|
||||
finally:
|
||||
for p in [tmp_ogg, tmp_pcm]:
|
||||
try:
|
||||
if os.path.exists(p):
|
||||
os.remove(p)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def batch_convert_all(self):
|
||||
for sdir in self.target_dirs:
|
||||
if not os.path.isdir(sdir):
|
||||
continue
|
||||
for fname in sorted(os.listdir(sdir)):
|
||||
if fname.lower().endswith(".sf2"):
|
||||
self.convert_sf2_to_sf3(os.path.join(sdir, fname))
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
import os
|
||||
import json
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
from sf2utils.sf2parse import Sf2File
|
||||
HAS_SF2UTILS = True
|
||||
except ImportError:
|
||||
HAS_SF2UTILS = False
|
||||
|
||||
GM_CATEGORIES = [
|
||||
("Piano", range(0, 8)),
|
||||
("Chromatic Percussion", range(8, 16)),
|
||||
("Organ", range(16, 24)),
|
||||
("Guitar", range(24, 32)),
|
||||
("Bass", range(32, 40)),
|
||||
("Strings", range(40, 48)),
|
||||
("Ensemble", range(48, 56)),
|
||||
("Brass", range(56, 64)),
|
||||
("Reed", range(64, 72)),
|
||||
("Pipe", range(72, 80)),
|
||||
("Synth Lead", range(80, 90)),
|
||||
("Synth Pad", range(90, 104)),
|
||||
]
|
||||
|
||||
MAX_CONDENSED_ENTRIES = 50
|
||||
|
||||
|
||||
class SoundFontInspector:
|
||||
def __init__(self, system_sf_dir="/opt/daw_engine/soundfonts", upload_sf_dir=None):
|
||||
self.system_sf_dir = system_sf_dir
|
||||
self.upload_sf_dir = upload_sf_dir
|
||||
self._catalog_cache = None
|
||||
|
||||
def invalidate_catalog_cache(self):
|
||||
self._catalog_cache = None
|
||||
|
||||
def inspect_sf2_file(self, filepath: str) -> dict:
|
||||
if not HAS_SF2UTILS:
|
||||
logger.warning("sf2utils not installed, cannot inspect .sf2 files")
|
||||
return {}
|
||||
if not os.path.exists(filepath):
|
||||
return {}
|
||||
|
||||
try:
|
||||
sf_name = os.path.basename(filepath)
|
||||
sf_id = os.path.splitext(sf_name)[0].lower()
|
||||
|
||||
instruments = []
|
||||
with open(filepath, 'rb') as f:
|
||||
sf2 = Sf2File(f)
|
||||
for preset in sf2.presets:
|
||||
name = preset.name.strip()
|
||||
if name == "EOP" or (preset.bank == 128 and preset.preset == 127):
|
||||
continue
|
||||
instruments.append({
|
||||
"bank": preset.bank,
|
||||
"program": preset.preset,
|
||||
"name": name,
|
||||
"is_percussion": (preset.bank == 128)
|
||||
})
|
||||
|
||||
return {
|
||||
"soundfont_id": sf_id,
|
||||
"filename": sf_name,
|
||||
"total_instruments": len(instruments),
|
||||
"instruments": instruments
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"Skipping corrupted .sf2 file {filepath}: {e}")
|
||||
return {}
|
||||
|
||||
def _scan_directory(self, directory: str) -> dict:
|
||||
catalog = {}
|
||||
if not os.path.isdir(directory):
|
||||
return catalog
|
||||
for fname in os.listdir(directory):
|
||||
if not fname.lower().endswith(('.sf2', '.sf3')):
|
||||
continue
|
||||
full_path = os.path.join(directory, fname)
|
||||
sf_info = self.inspect_sf2_file(full_path)
|
||||
if sf_info and sf_info.get("soundfont_id"):
|
||||
catalog[sf_info["soundfont_id"]] = sf_info
|
||||
return catalog
|
||||
|
||||
def generate_full_catalog(self, output_json_path: str = None) -> dict:
|
||||
catalog = {}
|
||||
catalog.update(self._scan_directory(self.system_sf_dir))
|
||||
if self.upload_sf_dir and os.path.isdir(self.upload_sf_dir):
|
||||
catalog.update(self._scan_directory(self.upload_sf_dir))
|
||||
|
||||
if output_json_path:
|
||||
os.makedirs(os.path.dirname(output_json_path), exist_ok=True)
|
||||
with open(output_json_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(catalog, f, ensure_ascii=False, indent=2)
|
||||
|
||||
self._catalog_cache = catalog
|
||||
return catalog
|
||||
|
||||
def get_catalog(self) -> dict:
|
||||
if self._catalog_cache is not None:
|
||||
return self._catalog_cache
|
||||
return self.generate_full_catalog()
|
||||
|
||||
def get_condensed_catalog_summary(self, catalog: dict = None) -> dict:
|
||||
catalog = catalog if catalog is not None else self.get_catalog()
|
||||
condensed = {}
|
||||
for sf_id, sf_info in catalog.items():
|
||||
instruments = sf_info.get("instruments", [])
|
||||
if not instruments:
|
||||
continue
|
||||
|
||||
selected = []
|
||||
used_programs = set()
|
||||
for cat_name, prog_range in GM_CATEGORIES:
|
||||
cat_members = [
|
||||
inst for inst in instruments
|
||||
if inst["program"] in prog_range and not inst["is_percussion"]
|
||||
]
|
||||
if cat_members:
|
||||
representative = cat_members[0]
|
||||
key = (representative["program"], representative["bank"])
|
||||
if key not in used_programs:
|
||||
used_programs.add(key)
|
||||
selected.append(representative)
|
||||
|
||||
drum_kits = [inst for inst in instruments if inst["is_percussion"]]
|
||||
for dk in drum_kits[:3]:
|
||||
key = (dk["program"], dk["bank"])
|
||||
if key not in used_programs:
|
||||
used_programs.add(key)
|
||||
selected.append(dk)
|
||||
|
||||
if len(selected) > MAX_CONDENSED_ENTRIES:
|
||||
selected = selected[:MAX_CONDENSED_ENTRIES]
|
||||
|
||||
condensed[sf_id] = {
|
||||
"soundfont_id": sf_info["soundfont_id"],
|
||||
"filename": sf_info["filename"],
|
||||
"total_instruments": sf_info["total_instruments"],
|
||||
"condensed_count": len(selected),
|
||||
"instruments": selected
|
||||
}
|
||||
return condensed
|
||||
|
||||
def format_condensed_for_prompt(self) -> str:
|
||||
condensed = self.get_condensed_catalog_summary()
|
||||
lines = []
|
||||
for sf_id, info in condensed.items():
|
||||
lines.append(f"SoundFont ID: '{sf_id}' (File: {info['filename']}):")
|
||||
for inst in info["instruments"]:
|
||||
lines.append(f" - {inst['name']}: bank={inst['bank']}, program={inst['program']}")
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,129 @@
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
import logging
|
||||
import threading
|
||||
from app.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TRACK_FILE = os.path.join(settings.STORAGE_DIR, "sf_scan_state.json")
|
||||
SYSTEM_SF_DIR = "/opt/daw_engine/soundfonts"
|
||||
UPLOAD_SF_DIR = os.path.join(settings.STORAGE_DIR, "soundfonts")
|
||||
|
||||
|
||||
def _file_sig(path: str) -> tuple:
|
||||
s = os.path.getsize(path)
|
||||
m = os.path.getmtime(path)
|
||||
return (s, m)
|
||||
|
||||
|
||||
class SoundFontAutoScanner:
|
||||
def __init__(self, system_sf_dir=SYSTEM_SF_DIR, upload_sf_dir=UPLOAD_SF_DIR):
|
||||
self.system_sf_dir = system_sf_dir
|
||||
self.upload_sf_dir = upload_sf_dir
|
||||
self._catalog = {}
|
||||
self._lock = threading.Lock()
|
||||
self._state = self._load_state()
|
||||
|
||||
def _load_state(self) -> dict:
|
||||
if not os.path.exists(TRACK_FILE):
|
||||
return {}
|
||||
try:
|
||||
with open(TRACK_FILE) as f:
|
||||
return json.load(f)
|
||||
except Exception as e:
|
||||
logger.warning("scan_state load failed: %s", e)
|
||||
return {}
|
||||
|
||||
def _save_state(self):
|
||||
os.makedirs(os.path.dirname(TRACK_FILE), exist_ok=True)
|
||||
with open(TRACK_FILE, "w") as f:
|
||||
json.dump(self._state, f, indent=2)
|
||||
|
||||
def _sf_files(self, directory: str) -> list:
|
||||
if not os.path.isdir(directory):
|
||||
return []
|
||||
out = []
|
||||
for fname in os.listdir(directory):
|
||||
if fname.lower().endswith((".sf2", ".sf3")):
|
||||
out.append((fname, os.path.join(directory, fname)))
|
||||
return out
|
||||
|
||||
def _inspect_single(self, fname: str, full: str, inspector) -> dict:
|
||||
if fname.lower().endswith(".sf2"):
|
||||
sf_info = inspector.inspect_sf2_file(full) or {}
|
||||
else:
|
||||
sf_info = {}
|
||||
if not sf_info.get("soundfont_id"):
|
||||
sf_id = os.path.splitext(fname)[0].lower()
|
||||
sf_info = {
|
||||
"soundfont_id": sf_id,
|
||||
"filename": fname,
|
||||
"total_instruments": 0,
|
||||
"instruments": [],
|
||||
"_sf3": True
|
||||
}
|
||||
return sf_info
|
||||
|
||||
def scan_once(self) -> bool:
|
||||
from app.core.soundfont_inspector import SoundFontInspector
|
||||
inspector = SoundFontInspector(self.system_sf_dir, self.upload_sf_dir)
|
||||
found_new = False
|
||||
dirs = [(self.system_sf_dir, "system")]
|
||||
if self.upload_sf_dir and os.path.isdir(self.upload_sf_dir):
|
||||
dirs.append((self.upload_sf_dir, "upload"))
|
||||
|
||||
for directory, source in dirs:
|
||||
for fname, full in self._sf_files(directory):
|
||||
sf_id = os.path.splitext(fname)[0].lower()
|
||||
key = f"{source}:{sf_id}"
|
||||
sig = _file_sig(full)
|
||||
prev = self._state.get(key)
|
||||
unchanged = prev and prev["size"] == sig[0] and prev["mtime"] == sig[1]
|
||||
if unchanged and sf_id in self._catalog:
|
||||
continue
|
||||
if unchanged:
|
||||
with self._lock:
|
||||
if sf_id in self._catalog:
|
||||
continue
|
||||
sf_info = self._inspect_single(fname, full, inspector)
|
||||
self._catalog[sf_id] = sf_info
|
||||
continue
|
||||
found_new = True
|
||||
logger.info("New/changed SF detected: %s", fname)
|
||||
sf_info = self._inspect_single(fname, full, inspector)
|
||||
with self._lock:
|
||||
self._catalog[sf_id] = sf_info
|
||||
self._state[key] = {"size": sig[0], "mtime": sig[1], "file": fname}
|
||||
|
||||
if found_new:
|
||||
self._save_state()
|
||||
inspector.invalidate_catalog_cache()
|
||||
return found_new
|
||||
|
||||
def scan_loop(self, interval: int = 30, stop_event: threading.Event = None):
|
||||
logger.info("SF auto-scanner started (interval=%ds)", interval)
|
||||
self.scan_once()
|
||||
while not (stop_event and stop_event.is_set()):
|
||||
time.sleep(interval)
|
||||
try:
|
||||
self.scan_once()
|
||||
except Exception as e:
|
||||
logger.error("scan cycle error: %s", e)
|
||||
|
||||
def start_background(self, interval: int = 30) -> threading.Event:
|
||||
ev = threading.Event()
|
||||
t = threading.Thread(target=self.scan_loop, args=(interval, ev), daemon=True)
|
||||
t.start()
|
||||
return ev
|
||||
|
||||
def get_catalog(self) -> dict:
|
||||
with self._lock:
|
||||
return dict(self._catalog)
|
||||
|
||||
def get_instruments(self, sf_id: str) -> list:
|
||||
entry = self._catalog.get(sf_id)
|
||||
if entry:
|
||||
return entry.get("instruments", [])
|
||||
return []
|
||||
+325
-31
@@ -1,77 +1,371 @@
|
||||
# SonicForge Studio VST / VSTi Engine Service (22_CLIENT_DESK.md §3)
|
||||
# SonicForge Studio VST / VSTi Engine Service
|
||||
import os
|
||||
import numpy as np
|
||||
import functools
|
||||
from ctypes import c_char_p
|
||||
|
||||
def midi_note_to_freq(note_number: int) -> float:
|
||||
"""Quy đổi số nốt MIDI (0 - 127) sang tần số Hertz (Hz)."""
|
||||
return 440.0 * (2.0 ** ((note_number - 69) / 12.0))
|
||||
|
||||
def render_midi_events_to_audio(midi_events: list, sr: int = 44100, bpm: float = 120.0, instrument: str = 'synth') -> np.ndarray:
|
||||
"""
|
||||
Tổng hợp mảng âm thanh NumPy Stereo từ sự kiện MIDI Piano Roll (22_CLIENT_DESK.md §3.1 & §3.2).
|
||||
|
||||
Args:
|
||||
midi_events: Danh sách nốt MIDI [{"note": 60, "start_beat": 0, "duration_beats": 1, "velocity": 100}, ...]
|
||||
sr: Tần số lấy mẫu (Sample Rate)
|
||||
bpm: Nhịp BPM của dự án
|
||||
instrument: Loại nhạc cụ tổng hợp
|
||||
|
||||
Returns:
|
||||
np.ndarray: Mảng 2D Stereo Float32 [2, num_samples]
|
||||
"""
|
||||
beat_duration_sec = 60.0 / max(30.0, bpm)
|
||||
max_duration_sec = 2.0
|
||||
|
||||
for event in midi_events:
|
||||
start_beat = event.get('start_beat', 0.0)
|
||||
dur_beats = event.get('duration_beats', 1.0)
|
||||
end_sec = (start_beat + dur_beats) * beat_duration_sec
|
||||
if end_sec > max_duration_sec:
|
||||
max_duration_sec = end_sec
|
||||
|
||||
total_samples = int((max_duration_sec + 0.5) * sr)
|
||||
out_l = np.zeros(total_samples, dtype=np.float32)
|
||||
out_r = np.zeros(total_samples, dtype=np.float32)
|
||||
|
||||
for event in midi_events:
|
||||
note = event.get('note', 60)
|
||||
velocity = event.get('velocity', 100) / 127.0
|
||||
start_beat = event.get('start_beat', 0.0)
|
||||
dur_beats = event.get('duration_beats', 1.0)
|
||||
|
||||
start_sample = int(start_beat * beat_duration_sec * sr)
|
||||
dur_samples = int(dur_beats * beat_duration_sec * sr)
|
||||
end_sample = min(total_samples, start_sample + dur_samples)
|
||||
actual_len = end_sample - start_sample
|
||||
|
||||
if actual_len <= 0 or start_sample >= total_samples:
|
||||
continue
|
||||
|
||||
freq = midi_note_to_freq(note)
|
||||
t = np.arange(actual_len) / float(sr)
|
||||
|
||||
# Synth tone + fundamental harmonics
|
||||
tone = 0.6 * np.sin(2 * np.pi * freq * t) + 0.3 * np.sin(2 * np.pi * freq * 2 * t) + 0.1 * np.sin(2 * np.pi * freq * 3 * t)
|
||||
|
||||
# ADSR Envelope
|
||||
attack = min(int(0.01 * sr), actual_len // 4)
|
||||
release = min(int(0.05 * sr), actual_len // 4)
|
||||
sustain_len = actual_len - attack - release
|
||||
|
||||
env = np.ones(actual_len, dtype=np.float32)
|
||||
if attack > 0:
|
||||
env[:attack] = np.linspace(0.0, 1.0, attack)
|
||||
if release > 0:
|
||||
env[-release:] = np.linspace(1.0, 0.0, release)
|
||||
|
||||
signal = tone * env * velocity
|
||||
|
||||
out_l[start_sample:end_sample] += signal
|
||||
out_r[start_sample:end_sample] += signal
|
||||
|
||||
# Clamping normalization to prevent clipping
|
||||
max_peak = max(np.max(np.abs(out_l)), np.max(np.abs(out_r)))
|
||||
if max_peak > 1.0:
|
||||
out_l /= max_peak
|
||||
out_r /= max_peak
|
||||
|
||||
return np.vstack([out_l, out_r])
|
||||
|
||||
def check_pedalboard_safe():
|
||||
import subprocess, sys
|
||||
try:
|
||||
res = subprocess.run(
|
||||
[sys.executable, "-c", "import pedalboard"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=2.0
|
||||
)
|
||||
return res.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def check_pyfluidsynth_safe():
|
||||
import subprocess, sys
|
||||
try:
|
||||
res = subprocess.run(
|
||||
[sys.executable, "-c", "import fluidsynth"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=2.0
|
||||
)
|
||||
return res.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
HAS_PEDALBOARD = check_pedalboard_safe()
|
||||
HAS_PYFLUIDSYNTH = check_pyfluidsynth_safe()
|
||||
|
||||
def ensure_pyfluidsynth():
|
||||
global HAS_PYFLUIDSYNTH
|
||||
if not HAS_PYFLUIDSYNTH:
|
||||
HAS_PYFLUIDSYNTH = check_pyfluidsynth_safe()
|
||||
return HAS_PYFLUIDSYNTH
|
||||
|
||||
if HAS_PEDALBOARD:
|
||||
try:
|
||||
from pedalboard import VST3Plugin, Pedalboard, Gain, MidiMessage
|
||||
except Exception:
|
||||
HAS_PEDALBOARD = False
|
||||
|
||||
if HAS_PYFLUIDSYNTH:
|
||||
try:
|
||||
import fluidsynth
|
||||
except Exception:
|
||||
HAS_PYFLUIDSYNTH = False
|
||||
|
||||
|
||||
# ── Module-level caches ──
|
||||
_FLUID_CACHE = {} # path → (fluidsynth.FluidSynth, refcount)
|
||||
_PLUGIN_MANAGER_INSTANCE = None
|
||||
_PLUGIN_MANAGER_ARGS = None
|
||||
_SF_INSTRUMENTS_CACHE = {} # sf_id → list[presets]
|
||||
|
||||
def get_plugin_manager(vst_dir="/opt/daw_engine/vst3", sf_dir="/opt/daw_engine/soundfonts", upload_sf_dir=None) -> "PluginManager":
|
||||
"""Singleton: reuse PluginManager when args match, else create new."""
|
||||
global _PLUGIN_MANAGER_INSTANCE, _PLUGIN_MANAGER_ARGS
|
||||
args = (vst_dir, sf_dir, upload_sf_dir)
|
||||
if _PLUGIN_MANAGER_INSTANCE is not None and _PLUGIN_MANAGER_ARGS == args:
|
||||
return _PLUGIN_MANAGER_INSTANCE
|
||||
_PLUGIN_MANAGER_ARGS = args
|
||||
_PLUGIN_MANAGER_INSTANCE = PluginManager(vst_dir, sf_dir, upload_sf_dir)
|
||||
return _PLUGIN_MANAGER_INSTANCE
|
||||
|
||||
def load_soundfont_cached(path: str):
|
||||
"""Return a cached low-level FluidSynth instance for path, incrementing refcount.
|
||||
|
||||
Uses the CFFI binding API (new_fluid_synth / fluid_synth_sfload) — the same
|
||||
API render_engine relies on. The high-level `FluidSynth()`/`Synth()` classes
|
||||
do not exist in this binding, so they are never used here.
|
||||
"""
|
||||
global _FLUID_CACHE
|
||||
if not HAS_PYFLUIDSYNTH:
|
||||
return None
|
||||
if path in _FLUID_CACHE:
|
||||
fl, ref = _FLUID_CACHE[path]
|
||||
_FLUID_CACHE[path] = (fl, ref + 1)
|
||||
return fl
|
||||
try:
|
||||
import fluidsynth as _fs
|
||||
_settings = _fs.new_fluid_settings()
|
||||
_fs.fluid_settings_setnum(_settings, b'synth.sample-rate', 44100.0)
|
||||
fl = _fs.new_fluid_synth(_settings)
|
||||
font_id = _fs.fluid_synth_sfload(fl, path.encode("utf-8"), 1)
|
||||
if font_id < 0:
|
||||
_fs.delete_fluid_synth(fl)
|
||||
return None
|
||||
_fs.fluid_synth_program_select(fl, 0, font_id, 0, 0)
|
||||
_FLUID_CACHE[path] = (fl, 1)
|
||||
return fl
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def release_soundfont(path: str):
|
||||
"""Decrement refcount; delete FluidSynth when count reaches 0."""
|
||||
global _FLUID_CACHE
|
||||
if path not in _FLUID_CACHE:
|
||||
return
|
||||
fl, ref = _FLUID_CACHE[path]
|
||||
if ref <= 1:
|
||||
try:
|
||||
import fluidsynth as _fs
|
||||
_fs.delete_fluid_synth(fl)
|
||||
except Exception:
|
||||
pass
|
||||
del _FLUID_CACHE[path]
|
||||
else:
|
||||
_FLUID_CACHE[path] = (fl, ref - 1)
|
||||
|
||||
class PluginManager:
|
||||
def __init__(self, vst_dir="/opt/daw_engine/vst3", sf_dir="/opt/daw_engine/soundfonts", upload_sf_dir=None):
|
||||
self.vst_dir = vst_dir
|
||||
self.sf_dir = sf_dir
|
||||
self.upload_sf_dir = upload_sf_dir
|
||||
self._sf_scan_cache = None # cache for _scan_soundfonts()
|
||||
|
||||
def _scan_plugins(self) -> dict:
|
||||
plugins = {}
|
||||
if not os.path.isdir(self.vst_dir):
|
||||
return plugins
|
||||
for root, dirs, files in os.walk(self.vst_dir):
|
||||
for file in files:
|
||||
if file.endswith(".vst3") or file.endswith(".so"):
|
||||
plugin_path = os.path.join(root, file)
|
||||
plugin_name = os.path.splitext(file)[0]
|
||||
plugins[plugin_name] = plugin_path
|
||||
return plugins
|
||||
|
||||
def _scan_soundfonts(self) -> list:
|
||||
sf_map = {}
|
||||
dirs = [("system", self.sf_dir)]
|
||||
if self.upload_sf_dir and self.upload_sf_dir != self.sf_dir:
|
||||
dirs.append(("upload", self.upload_sf_dir))
|
||||
|
||||
meta_cache = {}
|
||||
if self.upload_sf_dir and os.path.isdir(self.upload_sf_dir):
|
||||
for f in os.listdir(self.upload_sf_dir):
|
||||
if f.endswith(".meta"):
|
||||
try:
|
||||
import json
|
||||
with open(os.path.join(self.upload_sf_dir, f), "r") as mf:
|
||||
meta_cache[os.path.splitext(f)[0]] = json.load(mf)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for source, d in dirs:
|
||||
if not os.path.isdir(d):
|
||||
continue
|
||||
for f in os.listdir(d):
|
||||
if f.endswith(".sf2") or f.endswith(".sf3"):
|
||||
base_id = os.path.splitext(f)[0]
|
||||
if base_id in sf_map:
|
||||
continue
|
||||
meta = meta_cache.get(base_id, None)
|
||||
if meta:
|
||||
display_name = meta.get("original_name", f)
|
||||
else:
|
||||
short_id = base_id[:8] if len(base_id) > 8 else base_id
|
||||
display_name = f"SoundFont_{short_id}"
|
||||
sf_map[base_id] = {
|
||||
"id": base_id,
|
||||
"name": display_name,
|
||||
"file": f,
|
||||
"display": os.path.splitext(display_name)[0][:40],
|
||||
"source": source
|
||||
}
|
||||
return list(sf_map.values())
|
||||
|
||||
def load_vst(self, plugin_name: str, preset_data: dict = None):
|
||||
if not HAS_PEDALBOARD:
|
||||
return None
|
||||
plugins = self._scan_plugins()
|
||||
if plugin_name not in plugins:
|
||||
return None
|
||||
path = plugins[plugin_name]
|
||||
vst = VST3Plugin(path)
|
||||
if preset_data:
|
||||
for k, v in preset_data.items():
|
||||
try:
|
||||
setattr(vst, k, v)
|
||||
except Exception:
|
||||
pass
|
||||
return vst
|
||||
|
||||
def _scan_soundfonts_cached(self):
|
||||
if self._sf_scan_cache is not None:
|
||||
return self._sf_scan_cache
|
||||
self._sf_scan_cache = self._scan_soundfonts()
|
||||
return self._sf_scan_cache
|
||||
|
||||
def load_soundfont(self, path: str):
|
||||
return load_soundfont_cached(path)
|
||||
|
||||
def list_soundfont_instruments(self, sf_id: str):
|
||||
if not ensure_pyfluidsynth():
|
||||
return []
|
||||
if sf_id in _SF_INSTRUMENTS_CACHE:
|
||||
return _SF_INSTRUMENTS_CACHE[sf_id]
|
||||
search_dirs = []
|
||||
if os.path.isdir(self.sf_dir):
|
||||
search_dirs.append(self.sf_dir)
|
||||
if self.upload_sf_dir and os.path.isdir(self.upload_sf_dir) and self.upload_sf_dir != self.sf_dir:
|
||||
search_dirs.append(self.upload_sf_dir)
|
||||
for d in search_dirs:
|
||||
for f in os.listdir(d):
|
||||
if not (f.endswith(".sf2") or f.endswith(".sf3")):
|
||||
continue
|
||||
base = os.path.splitext(f)[0]
|
||||
if base == sf_id or base == sf_id.replace("sf_", ""):
|
||||
path = os.path.join(d, f)
|
||||
try:
|
||||
import fluidsynth as _fs
|
||||
# Low-level CFFI API (same as render_engine); never use
|
||||
# the high-level Synth() class that this binding lacks.
|
||||
_settings = _fs.new_fluid_settings()
|
||||
_synth = _fs.new_fluid_synth(_settings)
|
||||
try:
|
||||
fid = _fs.fluid_synth_sfload(_synth, path.encode("utf-8"), 1)
|
||||
if fid < 0:
|
||||
continue
|
||||
sfont = _fs.fluid_synth_get_sfont_by_id(_synth, fid)
|
||||
presets = []
|
||||
if sfont:
|
||||
for bank in range(0, 2):
|
||||
for prog_num in range(0, 128):
|
||||
try:
|
||||
preset = _fs.fluid_sfont_get_preset(sfont, bank, prog_num)
|
||||
except Exception:
|
||||
break
|
||||
if preset:
|
||||
try:
|
||||
name_ptr = _fs.fluid_preset_get_name(preset)
|
||||
if name_ptr:
|
||||
if hasattr(_fs, "ffi"):
|
||||
raw = _fs.ffi.string(name_ptr)
|
||||
else:
|
||||
raw = c_char_p(name_ptr).value
|
||||
if raw:
|
||||
presets.append({
|
||||
"bank": bank,
|
||||
"program": prog_num,
|
||||
"name": raw.decode("utf-8", errors="replace")
|
||||
})
|
||||
except Exception:
|
||||
continue
|
||||
_SF_INSTRUMENTS_CACHE[sf_id] = presets[:256]
|
||||
return presets[:256]
|
||||
finally:
|
||||
try:
|
||||
_fs.delete_fluid_synth(_synth)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
import traceback; traceback.print_exc()
|
||||
_SF_INSTRUMENTS_CACHE[sf_id] = []
|
||||
return []
|
||||
|
||||
def list_available(self) -> dict:
|
||||
return {
|
||||
"vst_instruments": [
|
||||
{"id": k, "name": k, "type": "VST3", "has_native_support": HAS_PEDALBOARD}
|
||||
for k in self._scan_plugins().keys()
|
||||
],
|
||||
"soundfonts": self._scan_soundfonts()
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def midi_events_to_messages(midi_events: list, bpm: float, sr: int, bank: int = None, program: int = None) -> list:
|
||||
if not HAS_PEDALBOARD:
|
||||
return []
|
||||
beat_duration_sec = 60.0 / max(30.0, bpm)
|
||||
messages = []
|
||||
if bank is not None:
|
||||
messages.append(MidiMessage(control_change=0, value=bank, sample_offset=0))
|
||||
if program is not None:
|
||||
messages.append(MidiMessage(program_change=program, sample_offset=0))
|
||||
for ev in midi_events:
|
||||
note = ev.get("note", 60)
|
||||
velocity = ev.get("velocity", 100)
|
||||
start_beat = ev.get("start_beat", 0.0)
|
||||
dur_beats = ev.get("duration_beats", 1.0)
|
||||
start_sec = start_beat * beat_duration_sec
|
||||
dur_sec = dur_beats * beat_duration_sec
|
||||
sample_offset = int(start_sec * sr)
|
||||
end_sample_offset = int((start_sec + dur_sec) * sr)
|
||||
messages.append(MidiMessage(note_on=note, velocity=velocity, sample_offset=sample_offset))
|
||||
messages.append(MidiMessage(note_off=note, velocity=0, sample_offset=end_sample_offset))
|
||||
return messages
|
||||
|
||||
@staticmethod
|
||||
def validate_sf2_header(data: bytes) -> bool:
|
||||
if len(data) < 12:
|
||||
return False
|
||||
if data[0:4] != b'RIFF':
|
||||
return False
|
||||
if data[8:12] != b'sfbk':
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
class DecentSamplerManager:
|
||||
def __init__(self, vst_path="/opt/daw_engine/vst3/DecentSampler.vst3"):
|
||||
self.vst_path = vst_path
|
||||
|
||||
def create_decent_sampler_instance(self, dspreset_path: str):
|
||||
if not HAS_PEDALBOARD:
|
||||
raise RuntimeError("pedalboard not available")
|
||||
if not os.path.exists(self.vst_path):
|
||||
raise FileNotFoundError(f"DecentSampler VST3 not found at {self.vst_path}")
|
||||
if not os.path.exists(dspreset_path):
|
||||
raise FileNotFoundError(f"Preset file not found at {dspreset_path}")
|
||||
|
||||
plugin = VST3Plugin(self.vst_path)
|
||||
|
||||
abs_preset = os.path.abspath(dspreset_path)
|
||||
preset_dir = os.path.dirname(abs_preset)
|
||||
cwd_before = os.getcwd()
|
||||
try:
|
||||
os.chdir(preset_dir)
|
||||
plugin.load_preset(abs_preset)
|
||||
finally:
|
||||
os.chdir(cwd_before)
|
||||
|
||||
return plugin
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
+48
-11
@@ -1,8 +1,11 @@
|
||||
import os
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi.responses import HTMLResponse, FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.middleware.gzip import GZipMiddleware
|
||||
from app.config import settings
|
||||
from app.api.v1.audio import router as audio_router
|
||||
from app.api.v1.tasks import router as tasks_router
|
||||
@@ -11,22 +14,43 @@ from app.api.v1.auth import router as auth_router
|
||||
from app.api.v1.admin import router as admin_router
|
||||
from app.api.v1.projects import router as projects_router
|
||||
from app.api.v1.user_config import router as user_config_router
|
||||
from app.api.v1.ai_proxy import router as ai_proxy_router
|
||||
from app.api.v1.ai_presets import router as ai_presets_router
|
||||
from app.api.v1.plugins import router as plugins_router
|
||||
from app.api.v1.media import router as media_router
|
||||
from app.core.auth import seed_admin
|
||||
from app.core.soundfont_scanner import SoundFontAutoScanner
|
||||
|
||||
# Ensure storage directories exist
|
||||
os.makedirs(settings.UPLOADS_DIR, exist_ok=True)
|
||||
os.makedirs(settings.PROCESSED_DIR, exist_ok=True)
|
||||
|
||||
app = FastAPI(title="SonicForge API Engine")
|
||||
_SF_SCANNER_STOP = None
|
||||
|
||||
from fastapi.middleware.gzip import GZipMiddleware
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Startup
|
||||
seed_admin()
|
||||
scanner = SoundFontAutoScanner()
|
||||
global _SF_SCANNER_STOP
|
||||
_SF_SCANNER_STOP = scanner.start_background(interval=30)
|
||||
yield
|
||||
# Shutdown
|
||||
if _SF_SCANNER_STOP is not None:
|
||||
_SF_SCANNER_STOP.set()
|
||||
|
||||
|
||||
app = FastAPI(title="SonicForge API Engine", lifespan=lifespan)
|
||||
|
||||
app.add_middleware(GZipMiddleware, minimum_size=500)
|
||||
|
||||
# Auth is token/cookie based (no cookies required for CORS), so credentials are
|
||||
# disabled — "*" + allow_credentials=True is rejected by browsers anyway.
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_credentials=False,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
@@ -45,11 +69,11 @@ app.include_router(auth_router, prefix="/api/v1/auth", tags=["auth"])
|
||||
app.include_router(admin_router, prefix="/api/v1/admin", tags=["admin"])
|
||||
app.include_router(projects_router, prefix="/api/v1/projects", tags=["projects"])
|
||||
app.include_router(user_config_router, prefix="/api/v1/user", tags=["user_config"])
|
||||
app.include_router(ai_proxy_router, prefix="/api/v1/ai", tags=["ai"])
|
||||
app.include_router(ai_presets_router, prefix="/api/v1/ai", tags=["ai"])
|
||||
app.include_router(plugins_router, prefix="/api/v1/plugins", tags=["plugins"])
|
||||
app.include_router(media_router, prefix="/api/v1/media", tags=["media"])
|
||||
|
||||
# Seed admin user on startup
|
||||
@app.on_event("startup")
|
||||
async def startup_seed_admin():
|
||||
seed_admin()
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def get_index():
|
||||
@@ -57,13 +81,26 @@ async def get_index():
|
||||
if not os.path.exists(index_path):
|
||||
return HTMLResponse(content=f"<h1>SonicForge Studio: index.html not found at {index_path}</h1>", status_code=404)
|
||||
with open(index_path, "r", encoding="utf-8") as file:
|
||||
return HTMLResponse(content=file.read(), status_code=200)
|
||||
resp = HTMLResponse(content=file.read(), status_code=200)
|
||||
# no-cache: index.html PHẢI luôn mới (các bundle JS dùng ?v= để bust) —
|
||||
# nếu browser cache HTML cũ → stamp cũ → tải bundle cũ (bug "không load
|
||||
# được bundle mới" ở incognito — cache heuristic không có Cache-Control).
|
||||
resp.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
return resp
|
||||
|
||||
|
||||
@app.get("/favicon.svg")
|
||||
async def get_favicon():
|
||||
import os
|
||||
favicon_path = os.path.join(settings.TEMPLATES_DIR, "favicon.svg")
|
||||
if os.path.exists(favicon_path):
|
||||
from fastapi.responses import FileResponse
|
||||
return FileResponse(favicon_path, media_type="image/svg+xml")
|
||||
return HTMLResponse(content="", status_code=404)
|
||||
|
||||
|
||||
@app.get("/ai-prompt-generator", response_class=HTMLResponse)
|
||||
async def get_ai_prompt_generator():
|
||||
md_path = os.path.join(settings.BASE_DIR, "md", "49_AI_PROMPT_GENERATOR.md")
|
||||
if not os.path.exists(md_path):
|
||||
return HTMLResponse(content="<h1>File not found</h1>", status_code=404)
|
||||
with open(md_path, "r", encoding="utf-8") as file:
|
||||
return HTMLResponse(content=file.read(), status_code=200)
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "DAWProject",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"project_id": { "type": "string" },
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": { "type": "string" },
|
||||
"bpm": { "type": "number", "minimum": 20.0, "maximum": 999.0, "default": 120.0 },
|
||||
"time_signature_numerator": { "type": "integer", "default": 4 },
|
||||
"time_signature_denominator": { "type": "integer", "default": 4 },
|
||||
"sample_rate": { "type": "integer", "default": 44100 }
|
||||
},
|
||||
"required": ["title", "bpm", "time_signature_numerator", "time_signature_denominator", "sample_rate"]
|
||||
},
|
||||
"main_session": { "$ref": "#/definitions/SessionContainer" },
|
||||
"section_store": {
|
||||
"type": "object",
|
||||
"description": "Auxiliary registry mapping section_id to sub-session containers",
|
||||
"additionalProperties": { "$ref": "#/definitions/SessionContainer" }
|
||||
}
|
||||
},
|
||||
"required": ["project_id", "metadata", "main_session", "section_store"],
|
||||
"definitions": {
|
||||
"SessionContainer": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"is_root": { "type": "boolean" },
|
||||
"length_bars": { "type": "number", "description": "Computed or manually set total length in bars" },
|
||||
"auto_compute_length": { "type": "boolean", "default": true },
|
||||
"tracks": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/Track" }
|
||||
}
|
||||
},
|
||||
"required": ["id", "is_root", "tracks"]
|
||||
},
|
||||
"Track": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["AUDIO", "MIDI", "SECTION"] },
|
||||
"color": { "type": ["string", "null"], "default": null },
|
||||
"volume_db": { "type": "number", "default": 0.0 },
|
||||
"pan": { "type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0 },
|
||||
"mute": { "type": "boolean", "default": false },
|
||||
"solo": { "type": "boolean", "default": false },
|
||||
"fx_chain": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/FXPlugin" }
|
||||
},
|
||||
"synth_engine": { "$ref": "#/definitions/SynthPlugin" },
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/TimelineItem" }
|
||||
}
|
||||
},
|
||||
"required": ["id", "name", "type", "items"]
|
||||
},
|
||||
"TimelineItem": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["AUDIO_ITEM", "MIDI_ITEM", "SECTION_ITEM"] },
|
||||
"start_bar": { "type": "number", "description": "Global timeline position where the item starts" },
|
||||
"duration_bars": { "type": "number", "description": "Visible duration on the track timeline in bars" },
|
||||
"clip_start_offset_bars": { "type": "number", "description": "Internal start offset inside the source buffer/item" },
|
||||
"source_data": {
|
||||
"type": "object",
|
||||
"oneOf": [
|
||||
{ "$ref": "#/definitions/AudioSourceData" },
|
||||
{ "$ref": "#/definitions/MIDISourceData" },
|
||||
{ "$ref": "#/definitions/SectionSourceData" }
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": ["id", "type", "start_bar", "duration_bars", "clip_start_offset_bars", "source_data"]
|
||||
},
|
||||
"AudioSourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"audio_file_url": { "type": "string" },
|
||||
"sample_rate": { "type": "integer" },
|
||||
"channels": { "type": "integer" },
|
||||
"gain": { "type": "number", "default": 1.0 }
|
||||
},
|
||||
"required": ["audio_file_url"]
|
||||
},
|
||||
"MIDISourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"total_buffer_bars": { "type": "number", "default": 8.0 },
|
||||
"notes": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/MIDINote" }
|
||||
}
|
||||
},
|
||||
"required": ["total_buffer_bars", "notes"]
|
||||
},
|
||||
"SectionSourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"referenced_section_id": { "type": "string", "description": "Pointer to section_store key" }
|
||||
},
|
||||
"required": ["referenced_section_id"]
|
||||
},
|
||||
"MIDINote": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
|
||||
"start_beat": { "type": "number", "description": "Beat offset relative to the start of the source buffer (bar 0)" },
|
||||
"duration_beats": { "type": "number" },
|
||||
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0, "default": 0.8 },
|
||||
"pan": { "type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0 }
|
||||
},
|
||||
"required": ["id", "pitch", "start_beat", "duration_beats", "velocity"]
|
||||
},
|
||||
"FXPlugin": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"plugin_id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"bypass": { "type": "boolean", "default": false },
|
||||
"parameters": { "type": "object" }
|
||||
}
|
||||
},
|
||||
"SynthPlugin": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"plugin_id": { "type": "string" },
|
||||
"preset_id": { "type": "string" },
|
||||
"parameters": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+42
-2
@@ -5,12 +5,18 @@ import time
|
||||
from typing import Optional, Dict, Any, List
|
||||
from app.config import settings
|
||||
|
||||
DB_PATH = os.path.join(settings.STORAGE_DIR, "sonicforge.db")
|
||||
# Default DB lives in storage/; tests override via SONICFORGE_DB_PATH so the
|
||||
# dev database is never touched by the test suite.
|
||||
DB_PATH = os.getenv("SONICFORGE_DB_PATH") or os.path.join(settings.STORAGE_DIR, "sonicforge.db")
|
||||
|
||||
def get_db_connection():
|
||||
os.makedirs(settings.STORAGE_DIR, exist_ok=True)
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
# WAL improves concurrent read/write; FK enforcement makes quota/backup
|
||||
# cleanup consistent when users are deleted.
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA foreign_keys=ON")
|
||||
return conn
|
||||
|
||||
def init_db():
|
||||
@@ -65,8 +71,42 @@ def init_db():
|
||||
);
|
||||
""")
|
||||
|
||||
# Migration: thêm cột backup nếu chưa tồn tại
|
||||
try:
|
||||
cursor.execute("ALTER TABLE projects ADD COLUMN is_backup INTEGER DEFAULT 0")
|
||||
except Exception:
|
||||
pass # column already exists
|
||||
try:
|
||||
cursor.execute("ALTER TABLE projects ADD COLUMN original_id TEXT DEFAULT NULL")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Bảng Project Backups (snapshot riêng, không lẫn với projects chính)
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS project_backups (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id TEXT NOT NULL,
|
||||
project_id TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
data_json TEXT NOT NULL,
|
||||
size_bytes INTEGER DEFAULT 0,
|
||||
created_at REAL NOT NULL,
|
||||
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
""")
|
||||
|
||||
# Placeholder user for anonymous autosave: projects are saved with
|
||||
# user_id='anonymous' when no token is present, so the FK must resolve.
|
||||
cursor.execute("SELECT id FROM users WHERE id = 'anonymous'")
|
||||
if not cursor.fetchone():
|
||||
import secrets as _secrets
|
||||
cursor.execute("""
|
||||
INSERT OR IGNORE INTO users (id, username, email, hashed_password, role, must_change_password, created_at, is_active)
|
||||
VALUES ('anonymous', 'anonymous', 'anonymous@local', ?, 'standard', 0, ?, 0)
|
||||
""", (_secrets.token_hex(32), time.time()))
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
|
||||
# Tự động khởi tạo DB khi module được import
|
||||
init_db()
|
||||
|
||||
@@ -47,3 +47,9 @@ body {
|
||||
font-size: 11px;
|
||||
backdrop-filter: blur(4px);
|
||||
}
|
||||
|
||||
/* Disable mouse pointer events on SVG icons to prevent SVGAnimatedString className type errors in event listeners */
|
||||
svg, i[data-lucide] {
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
|
||||
+19658
-1104
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+1163
-9215
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,526 @@
|
||||
// SonicForge Studio - AI Gateway & Function Routing
|
||||
// LLM Gateway with Function Calling / Structured Outputs (28_AI_PANEL.md §2)
|
||||
|
||||
const AIGateway = (function() {
|
||||
const DEFAULT_TOOLS = [{
|
||||
name: 'set_selection', description: 'Chọn vùng timeline', parameters: { type: 'object', properties: { start_bar: { type: 'number' }, end_bar: { type: 'number' }, start_time: { type: 'number' }, end_time: { type: 'number' }, length_bars: { type: 'number' } } }
|
||||
}, {
|
||||
name: 'cut_audio', description: 'Cắt audio, snap zero-crossing, tạo track mới', parameters: { type: 'object', properties: { track_id: { type: 'string' }, start_time: { type: 'number' }, end_time: { type: 'number' }, start_bar: { type: 'number' }, end_bar: { type: 'number' }, length_bars: { type: 'number' }, snap_silence: { type: 'boolean' }, new_track_name: { type: 'string' } } }
|
||||
}, {
|
||||
name: 'create_track', description: 'Tạo track mới', parameters: { type: 'object', properties: { name: { type: 'string' }, type: { type: 'string', enum: ['audio', 'midi'] } }, required: ['name'] }
|
||||
}, {
|
||||
name: 'delete_track', description: 'Xóa track', parameters: { type: 'object', properties: { track_id: { type: 'string' } } }
|
||||
}, {
|
||||
name: 'rename_track', description: 'Đổi tên track', parameters: { type: 'object', properties: { track_id: { type: 'string' }, name: { type: 'string' } }, required: ['track_id', 'name'] }
|
||||
}, {
|
||||
name: 'add_clip', description: 'Thêm clip rỗng vào track', parameters: { type: 'object', properties: { track_id: { type: 'string' }, start_time: { type: 'number' }, duration_seconds: { type: 'number' }, start_bar: { type: 'number' }, length_bars: { type: 'number' }, name: { type: 'string' } } }
|
||||
}, {
|
||||
name: 'remove_clip', description: 'Xóa clip khỏi track', parameters: { type: 'object', properties: { track_id: { type: 'string' }, clip_id: { type: 'string' } }, required: ['clip_id'] }
|
||||
}, {
|
||||
name: 'set_track_volume', description: 'Chỉnh âm lượng dB', parameters: { type: 'object', properties: { track_id: { type: 'string' }, volume_db: { type: 'number' } }, required: ['volume_db'] }
|
||||
}, {
|
||||
name: 'set_track_pan', description: 'Chỉnh pan trái/phải', parameters: { type: 'object', properties: { track_id: { type: 'string' }, pan: { type: 'integer' } }, required: ['pan'] }
|
||||
}, {
|
||||
name: 'toggle_mute', description: 'Mute/unmute track', parameters: { type: 'object', properties: { track_id: { type: 'string' } } }
|
||||
}, {
|
||||
name: 'toggle_solo', description: 'Solo/unsolo track', parameters: { type: 'object', properties: { track_id: { type: 'string' } } }
|
||||
}, {
|
||||
name: 'set_bpm', description: 'Thay đổi BPM', parameters: { type: 'object', properties: { bpm: { type: 'number' } }, required: ['bpm'] }
|
||||
}, {
|
||||
name: 'set_playhead', description: 'Di chuyển playhead', parameters: { type: 'object', properties: { time: { type: 'number' }, bar: { type: 'number' } } }
|
||||
}, {
|
||||
name: 'add_marker', description: 'Thêm marker', parameters: { type: 'object', properties: { track_id: { type: 'string' }, time: { type: 'number' }, label: { type: 'string' } } }
|
||||
}, {
|
||||
name: 'process_audio_dsp', description: 'Xử lý DSP: normalize/invert/gain/pitch', parameters: { type: 'object', properties: { track_id: { type: 'string' }, action: { type: 'string', enum: ['normalize', 'invert_phase', 'gain', 'pitch_shift'] }, params: { type: 'object' } }, required: ['track_id', 'action'] }
|
||||
}, {
|
||||
name: 'create_midi_item', description: 'Tạo MIDI item trên track', parameters: { type: 'object', properties: { track_id: { type: 'string' }, start_bar: { type: 'number' }, length_bars: { type: 'number' } }, required: ['track_id', 'start_bar', 'length_bars'] }
|
||||
}, {
|
||||
name: 'modify_midi_notes', description: 'Sửa note MIDI trong item', parameters: { type: 'object', properties: { item_id: { type: 'string' }, notes: { type: 'array', items: { type: 'object', properties: { pitch: { type: 'string' }, start_time: { type: 'number' }, duration: { type: 'number' }, velocity: { type: 'integer', minimum: 0, maximum: 127 } }, required: ['pitch', 'start_time', 'duration'] } } }, required: ['item_id', 'notes'] }
|
||||
}, {
|
||||
name: 'select_item', description: 'Chọn clip/item theo tên', parameters: { type: 'object', properties: { track_id: { type: 'string' }, item_name: { type: 'string' }, select_all: { type: 'boolean' } } }
|
||||
}, {
|
||||
name: 'scan_track', description: 'Phân tích track: BPM, SR, kênh', parameters: { type: 'object', properties: { track_id: { type: 'string' }, set_tempo: { type: 'boolean' } } }
|
||||
}, {
|
||||
name: 'fade_in', description: 'Fade-in clip (0.5s đến max)', parameters: { type: 'object', properties: { track_id: { type: 'string' }, duration_seconds: { type: 'number' }, clip_index: { type: 'number', description: 'Chỉ số của clip trên track (1-based, ví dụ: 1 cho clip 1, 2 cho clip 2)' }, clip_id: { type: 'string', description: 'ID của clip cụ thể' } } }
|
||||
}, {
|
||||
name: 'export_audio', description: 'Xuất file WAV/MP3/OGG và tải về', parameters: { type: 'object', properties: { track_id: { type: 'string' }, format: { type: 'string', enum: ['wav', 'mp3', 'ogg'] }, sample_rate: { type: 'string', enum: ['22500', '44100'] }, bit_depth: { type: 'string', enum: ['8', '16', '24'] }, quality: { type: 'string', enum: ['44khz', 'lossless'] }, channels: { type: 'string', enum: ['mono', 'stereo'] }, start_time: { type: 'number' }, end_time: { type: 'number' }, start_bar: { type: 'number' }, length_bars: { type: 'number' } }, required: ['format'] }
|
||||
}, {
|
||||
name: 'fade_out', description: 'Fade-out clip (0.5s đến max)', parameters: { type: 'object', properties: { track_id: { type: 'string' }, duration_seconds: { type: 'number' }, clip_index: { type: 'number', description: 'Chỉ số của clip trên track (1-based, ví dụ: 1 cho clip 1, 2 cho clip 2)' }, clip_id: { type: 'string', description: 'ID của clip cụ thể' } } }
|
||||
}, {
|
||||
name: 'generate_multitrack_midi',
|
||||
description: 'Generates multi-track MIDI data along with SoundFont Program configurations for each track.',
|
||||
parameters: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
composition_title: { type: 'string', description: 'Title of the musical piece (e.g., Epic Orchestra Intro 8-Bars)' },
|
||||
bpm: { type: 'integer' },
|
||||
total_bars: { type: 'integer', description: 'Total length of the composition in bars. You MUST populate all bars with notes.' },
|
||||
tracks: {
|
||||
type: 'array',
|
||||
description: 'CRITICAL: Array of instrument tracks. You MUST generate exactly the number of tracks requested by the user. Every track in this array MUST contain a full sequence of notes that spans the entire duration of the piece (from start_beat 0.0 to total_bars * 4.0).',
|
||||
items: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
track_name: { type: 'string', description: 'Track name (e.g., String Ensemble, Epic Brass, Taiko Drums)' },
|
||||
instrument_type: { type: 'string', enum: ['STRINGS', 'BRASS', 'SYNTH', 'PERCUSSION', 'DRUMS'] },
|
||||
soundfont_id: { type: 'string', description: "ID of the SoundFont to use (e.g. 'generaluser_gs')" },
|
||||
soundfont_bank: { type: 'integer', default: 0, description: 'MIDI Bank code: 0 for melodic instruments, 128 for Drums/Percussion' },
|
||||
soundfont_program: { type: 'integer', description: 'MIDI Program Number 0-127 matching the instrument name in the SoundFont catalog' },
|
||||
notes: {
|
||||
type: 'array',
|
||||
description: 'CRITICAL: Array of MIDI notes. You MUST write notes completely filling all bars from bar 0 (beat 0.0) up to the final bar (beat total_bars * 4.0). Do NOT leave empty bars or stop early. Fill the entire duration of the composition with continuous musical notes.',
|
||||
items: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
pitch: { type: 'integer', description: 'MIDI note pitch from 0 to 127 (e.g., C4 = 60, C3 = 48)' },
|
||||
start_beat: { type: 'number', description: 'Note start position in beats (from 0.0 to total_bars * 4.0)' },
|
||||
duration_beats: { type: 'number', description: 'Note length in beats (e.g., Quarter note = 1.0, Eighth note = 0.5)' },
|
||||
velocity: { type: 'number', description: 'Keypress velocity intensity from 0.0 to 1.0' }
|
||||
},
|
||||
required: ['pitch', 'start_beat', 'duration_beats', 'velocity']
|
||||
}
|
||||
}
|
||||
},
|
||||
required: ['track_name', 'instrument_type', 'soundfont_id', 'soundfont_bank', 'soundfont_program', 'notes']
|
||||
}
|
||||
}
|
||||
},
|
||||
required: ['composition_title', 'bpm', 'total_bars', 'tracks']
|
||||
}
|
||||
}];
|
||||
|
||||
const REARRANGE_TOOL_SPEC = {
|
||||
type: 'function',
|
||||
function: {
|
||||
name: 'rearrange_midi_melody',
|
||||
description: 'Accepts source MIDI notes and rearranges/re-harmonizes them into a new musical variation while preserving the core melody. Use this when the user asks to rearrange, remix, or create variations of an existing MIDI item.',
|
||||
parameters: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
rearrange_title: { type: 'string', description: 'Title/description of the rearranged version (e.g. "Jazz Variation", "Dark Orchestral Remix")' },
|
||||
soundfont_id: { type: 'string', default: 'generaluser_gs' },
|
||||
soundfont_bank: { type: 'integer', default: 0 },
|
||||
soundfont_program: { type: 'integer', default: 0 },
|
||||
rearranged_notes: {
|
||||
type: 'array',
|
||||
description: 'Array of rearranged MIDI notes. Keep the same total_duration_beats as the original unless user explicitly requests length change.',
|
||||
items: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
pitch: { type: 'integer', description: 'MIDI note pitch 0-127 (C4=60)' },
|
||||
start_beat: { type: 'number', description: 'Note start position in beats from 0.0' },
|
||||
duration_beats: { type: 'number', description: 'Note length in beats (quarter=1.0)' },
|
||||
velocity: { type: 'number', description: 'Velocity 0.0-1.0' }
|
||||
},
|
||||
required: ['pitch', 'start_beat', 'duration_beats', 'velocity']
|
||||
}
|
||||
}
|
||||
},
|
||||
required: ['rearrange_title', 'rearranged_notes']
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// ai_midi_rearrange_specification.md §3 — Function Tool Schema hỗ trợ 2 mode:
|
||||
// SIMILAR_VARIATION (biến tấu cùng độ dài) / EXTEND_CONTINUATION (viết tiếp
|
||||
// các bar sau). AI trả gói dữ liệu có vị trí target trên Timeline.
|
||||
const REARRANGE_EXTEND_TOOL_SPEC = {
|
||||
type: 'function',
|
||||
function: {
|
||||
name: 'rearrange_or_extend_midi_melody',
|
||||
description: 'Analyzes source MIDI melody data and returns either a variation (Variation) or continuation (Extend) based on user instructions.',
|
||||
parameters: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
mode: {
|
||||
type: 'string',
|
||||
enum: ['SIMILAR_VARIATION', 'EXTEND_CONTINUATION'],
|
||||
description: "Mode: 'SIMILAR_VARIATION' (new arrangement of equal length) or 'EXTEND_CONTINUATION' (writes subsequent bars)."
|
||||
},
|
||||
composition_title: {
|
||||
type: 'string',
|
||||
description: 'Short title describing the new melody style (e.g., Jazz Swing Variation, Epic Extension Part 2)'
|
||||
},
|
||||
target_start_bar: {
|
||||
type: 'number',
|
||||
description: 'Starting bar number for the generated notes on the Timeline'
|
||||
},
|
||||
target_duration_bars: {
|
||||
type: 'number',
|
||||
description: 'Total bar duration covered by the generated sequence'
|
||||
},
|
||||
soundfont_id: { type: 'string', default: 'generaluser_gs' },
|
||||
soundfont_bank: { type: 'integer', default: 0 },
|
||||
soundfont_program: { type: 'integer', default: 0 },
|
||||
generated_notes: {
|
||||
type: 'array',
|
||||
description: 'Array of AI-generated MIDI notes.',
|
||||
items: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
pitch: { type: 'integer', minimum: 0, maximum: 127 },
|
||||
start_beat: { type: 'number', description: 'Starting beat position relative to beat 0.0 of the generated item' },
|
||||
duration_beats: { type: 'number', minimum: 0.1 },
|
||||
velocity: { type: 'number', minimum: 0.0, maximum: 1.0 }
|
||||
},
|
||||
required: ['pitch', 'start_beat', 'duration_beats', 'velocity']
|
||||
}
|
||||
}
|
||||
},
|
||||
required: ['mode', 'composition_title', 'target_start_bar', 'target_duration_bars', 'generated_notes']
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const REARRANGE_SCENARIOS = [
|
||||
{
|
||||
id: 'arpeggio',
|
||||
keywords: ['arpeggio', 'arp', 'broken chord', 'broken chords', 'shimmering sequence'],
|
||||
description: 'Arpeggio variation',
|
||||
technique: 'Subdivide sustained chord notes (longer than 0.5 beats) into sequential 0.25-beat arpeggiated interval steps using octave/triad jumps. Create a driving, shimmering sequence.'
|
||||
},
|
||||
{
|
||||
id: 'harmonies',
|
||||
keywords: ['harmony', 'harmonize', 'harmonies', '3rd', '3rds', 'duet', 'chord voicing', 'voicing'],
|
||||
description: 'Add parallel harmonies',
|
||||
technique: 'For each source pitch P, insert parallel harmonizing notes at pitch P+3 or P+4 (Major/Minor 3rds) matching the same start_beat. Create richer, fuller texture resembling two instruments playing in duet.'
|
||||
},
|
||||
{
|
||||
id: 'syncopation',
|
||||
keywords: ['syncopation', 'syncopate', 'syncopated', 'off-beat', 'off beat', 'funk', 'latin', 'rnb', 'r&b'],
|
||||
description: 'Syncopation / off-beat feel',
|
||||
technique: 'Shift start_beat alignment off strong beats (0.0, 1.0, 2.0, 3.0) onto off-beats (0.5, 1.5, 2.5, 3.5). Create a rhythmic, syncopated feel.'
|
||||
},
|
||||
{
|
||||
id: 'walking_bass',
|
||||
keywords: ['walking bass', 'bassline', 'bass line', 'walking'],
|
||||
description: 'Walking bass line',
|
||||
technique: 'Lower pitches to bass register (36-48 range). Build continuous quarter-note steps (1.0 beat duration) following the chord outline. Create an improvisational walking bassline.'
|
||||
},
|
||||
{
|
||||
id: 'jazz_swing',
|
||||
keywords: ['jazz', 'swing', 'bebop', 'bop', 'jazz swing'],
|
||||
description: 'Jazz Swing style',
|
||||
technique: 'Apply Jazz Swing characteristics: off-beat syncopation, 7th/9th chord extensions, rhythmic variations, and swung eighth notes (triplet feel). Preserve the core melodic outline while adding jazz harmony.'
|
||||
},
|
||||
{
|
||||
id: 'synthwave',
|
||||
keywords: ['synthwave', '80s', 'retro', 'synth wave', 'outrun', 'retrowave'],
|
||||
description: 'Synthwave / 80s style',
|
||||
technique: 'Transform into 80s Synthwave style: driving 8th-note bass arpeggios, gated reverb snare, analog synth lead with portamento, and pulsating chord pads. Use octave jumps in the bass.'
|
||||
},
|
||||
{
|
||||
id: 'cinematic',
|
||||
keywords: ['cinematic', 'orchestral', 'epic', 'film score', 'movie', 'hollywood', 'symphonic'],
|
||||
description: 'Cinematic Orchestral style',
|
||||
technique: 'Transform into dramatic Cinematic Orchestral style: swelling dynamics, brass stabs on downbeats, string ostinato patterns, taiko percussion hits on strong beats. Layer multiple octaves for epic width.'
|
||||
},
|
||||
{
|
||||
id: 'simplify',
|
||||
keywords: ['simplify', 'minimal', 'strip', 'downbeat', 'reduce', 'sparse'],
|
||||
description: 'Simplify / strip down',
|
||||
technique: 'Keep only the downbeat notes (start_beat at 0.0, 1.0, 2.0, 3.0, etc.). Strip out embellishments, passing tones, and grace notes. Reduce chord voicings to root and 5th only.'
|
||||
},
|
||||
{
|
||||
id: 'passing_tones',
|
||||
keywords: ['passing tone', 'chromatic', 'chromaticism', 'smooth', 'approach note'],
|
||||
description: 'Add passing tones / chromaticism',
|
||||
technique: 'Insert passing tones and chromatic approach notes between chord tones to smooth out the melodic progression. Use half-step and whole-step approach notes targeting chord tones on strong beats.'
|
||||
}
|
||||
];
|
||||
|
||||
function detectRearrangeScenario(prompt) {
|
||||
if (!prompt) return null;
|
||||
const lower = prompt.toLowerCase();
|
||||
const matched = [];
|
||||
for (const sc of REARRANGE_SCENARIOS) {
|
||||
const hits = sc.keywords.filter(kw => lower.includes(kw.toLowerCase()));
|
||||
if (hits.length > 0) {
|
||||
matched.push({ scenario: sc, hitCount: hits.length });
|
||||
}
|
||||
}
|
||||
matched.sort((a, b) => b.hitCount - a.hitCount);
|
||||
return matched.length > 0 ? matched[0].scenario : null;
|
||||
}
|
||||
|
||||
function buildRearrangeMessage(prompt, sourceContext) {
|
||||
const notesJson = JSON.stringify(sourceContext.notes, null, 2);
|
||||
const scenario = detectRearrangeScenario(prompt);
|
||||
const techniqueSection = scenario
|
||||
? `\nDETECTED SCENARIO: ${scenario.description}\nSPECIFIC TECHNIQUE REQUIRED: ${scenario.technique}\n`
|
||||
: '';
|
||||
|
||||
const rules = [
|
||||
'1. Use the rearrange_midi_melody tool to return the rearranged notes.',
|
||||
'2. PRESERVE the core melodic outline and overall structure unless the user explicitly asks for a complete transformation.',
|
||||
'3. Keep the total duration (' + sourceContext.total_beats + ' beats) the same unless user requests a different length.',
|
||||
'4. The rearranged_notes array MUST contain notes with pitch (0-127), start_beat (0.0 to ' + sourceContext.total_beats + '.0), duration_beats, and velocity (0.0-1.0).',
|
||||
'5. Start beats should remain within 0-' + sourceContext.total_beats + ' range.',
|
||||
'6. You may add, remove, or modify notes to achieve the requested style.',
|
||||
'7. ALWAYS return a rearrange_title describing what was created (e.g. "Jazz Swing Variation of Piano Lead").'
|
||||
];
|
||||
|
||||
return [
|
||||
{ role: 'system', content: `You are a professional Music Composer & Arranger.
|
||||
|
||||
SOURCE MIDI CONTEXT:
|
||||
- Track: ${sourceContext.track_name}
|
||||
- Item: ${sourceContext.item_name}
|
||||
- Duration: ${sourceContext.duration_bars} bars (${sourceContext.total_beats} beats)
|
||||
- BPM: ${sourceContext.bpm}
|
||||
- Time Signature: 4/4
|
||||
- Total notes: ${sourceContext.total_notes}
|
||||
|
||||
ORIGINAL MELODY NOTES (pitch, note_name, start_beat, duration_beats, velocity):
|
||||
${notesJson}
|
||||
${techniqueSection}
|
||||
STRICT REARRANGE RULES:
|
||||
${rules.join('\n')}` },
|
||||
{ role: 'user', content: `${prompt}\n\nRearrange the source MIDI notes above according to this request. Return the result via the rearrange_midi_melody tool.` }
|
||||
];
|
||||
}
|
||||
|
||||
function parseOrigin(urlStr) {
|
||||
try { const u = new URL(urlStr); return `${u.protocol}//${u.hostname}${u.port ? ':'+u.port : ''}`; } catch (_) { return null; }
|
||||
}
|
||||
|
||||
function isLocalhost(urlStr) {
|
||||
try {
|
||||
const u = new URL(urlStr);
|
||||
return u.hostname === 'localhost' || u.hostname === '127.0.0.1' || u.hostname === '0.0.0.0' || u.hostname === '::1';
|
||||
} catch (_) { return false; }
|
||||
}
|
||||
|
||||
async function callLLM({ provider, model, apiKey, baseUrl, messages, tools, toolChoice }) {
|
||||
const base = baseUrl.replace(/\/$/, '');
|
||||
const url = `${base}/chat/completions`;
|
||||
const origin = window.location.origin;
|
||||
const urlOrigin = parseOrigin(url);
|
||||
const appOrigin = parseOrigin(origin);
|
||||
const sameOrigin = urlOrigin === appOrigin;
|
||||
const targetIsLocal = isLocalhost(url);
|
||||
|
||||
const headers = {
|
||||
'Content-Type': 'application/json',
|
||||
...(apiKey ? { 'Authorization': `Bearer ${apiKey}` } : {})
|
||||
};
|
||||
|
||||
const body = {
|
||||
model,
|
||||
messages,
|
||||
stream: false,
|
||||
...(tools && tools.length > 0 ? { tools: tools.map(t => ({ type: 'function', function: t })) } : {}),
|
||||
...(toolChoice ? { tool_choice: toolChoice } : {})
|
||||
};
|
||||
|
||||
let response;
|
||||
|
||||
if (sameOrigin) {
|
||||
response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: JSON.stringify(body)
|
||||
});
|
||||
} else if (targetIsLocal && !isLocalhost(origin)) {
|
||||
throw new Error(`AI provider local (${url}) không khả dụng từ domain từ xa (${origin}).\nHãy dùng provider từ xa (OpenAI, Anthropic...) hoặc dùng CORS plugin trình duyệt.`);
|
||||
} else {
|
||||
response = await fetch(`${origin}/api/v1/ai/proxy`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
...(localStorage.getItem('sonic_token') ? { 'X-Auth-Token': localStorage.getItem('sonic_token') } : {})
|
||||
},
|
||||
body: JSON.stringify({ url, headers, body })
|
||||
});
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const errText = await response.text();
|
||||
let detail = errText;
|
||||
try { const j = JSON.parse(errText); if (j.detail) detail = j.detail; } catch (_) {}
|
||||
throw new Error(detail);
|
||||
}
|
||||
|
||||
return await response.json();
|
||||
}
|
||||
|
||||
function extractFunctionCalls(completion) {
|
||||
const calls = [];
|
||||
const choice = completion.choices && completion.choices[0];
|
||||
if (!choice) return calls;
|
||||
|
||||
const msg = choice.message;
|
||||
if (msg.tool_calls && Array.isArray(msg.tool_calls)) {
|
||||
for (const tc of msg.tool_calls) {
|
||||
if (tc.type === 'function' && tc.function) {
|
||||
let args = {};
|
||||
try { args = JSON.parse(tc.function.arguments || '{}'); } catch (e) { args = { raw: tc.function.arguments }; }
|
||||
calls.push({
|
||||
id: tc.id,
|
||||
name: tc.function.name,
|
||||
arguments: args
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if (msg.function_call) {
|
||||
let args = {};
|
||||
try { args = JSON.parse(msg.function_call.arguments || '{}'); } catch (e) { args = { raw: msg.function_call.arguments }; }
|
||||
calls.push({
|
||||
id: 'call_' + Date.now(),
|
||||
name: msg.function_call.name,
|
||||
arguments: args
|
||||
});
|
||||
}
|
||||
return calls;
|
||||
}
|
||||
|
||||
function buildCatalogPromptSection() {
|
||||
const catalog = window.__soundfontCatalog;
|
||||
if (!catalog || !catalog.condensed_catalog) return '';
|
||||
const lines = [];
|
||||
for (const [sfId, info] of Object.entries(catalog.condensed_catalog)) {
|
||||
lines.push(`SoundFont ID: '${sfId}' (File: ${info.filename}):`);
|
||||
for (const inst of info.instruments || []) {
|
||||
lines.push(` - ${inst.name}: bank=${inst.bank}, program=${inst.program}`);
|
||||
}
|
||||
}
|
||||
if (lines.length === 0) return '';
|
||||
return `\n=== SOUNDFONT INSTRUMENT CATALOG ===\nYou have the following SoundFont instruments available on the system:\n${lines.join('\n')}\n\nMANDATORY RULES WHEN CREATING TRACKS WITH generate_multitrack_midi:\n1. You MUST look up the catalog above and fill in the correct soundfont_id, soundfont_bank, and soundfont_program for each track.\n2. Melodic instruments (Piano, Strings, Brass, etc.) MUST use soundfont_bank=0.\n3. Drums and Percussion MUST use soundfont_bank=128.\n4. Example: For \"Brass horns\", use soundfont_id="generaluser_gs", soundfont_bank=0, soundfont_program=56.\n5. Example: For \"Drum kit\", use soundfont_id="generaluser_gs", soundfont_bank=128, soundfont_program=0.\n`;
|
||||
}
|
||||
|
||||
function buildUserMessage(prompt, context, systemInstruction = '') {
|
||||
const contextStr = JSON.stringify(context, null, 2);
|
||||
const toolNames = DEFAULT_TOOLS.map(t => ` - ${t.name}: ${t.description}`).join('\n');
|
||||
const catalogSection = buildCatalogPromptSection();
|
||||
return [
|
||||
{ role: 'system', content: `Bạn là trợ lý điều khiển DAW chuyên nghiệp.
|
||||
Nhiệm vụ của bạn là phân tích yêu cầu của người dùng và chuyển đổi thành danh sách các function calls tương ứng.
|
||||
${systemInstruction ? `\nHướng dẫn tạo nhạc đặc biệt từ Preset:\n${systemInstruction}\n` : ''}
|
||||
${catalogSection}
|
||||
|
||||
=== HƯỚNG DẪN SOẠN NHẠC MIDI / MIDI COMPOSITION RULES ===
|
||||
KHI NGƯỜI DÙNG YÊU CẦU TẠO NHẠC / COMPOSITION RULES:
|
||||
1. FULL TRACKS & BARS: If the user requests X tracks and Y bars, you MUST generate exactly X tracks. Each track MUST contain a continuous sequence of MIDI notes starting from beat 0.0 and stretching all the way to beat Y * 4.0 (the end of the composition).
|
||||
2. NO EARLY STOPPING: Do NOT stop early or leave empty bars at the end or in the middle. Every track must be fully populated with notes throughout the entire duration.
|
||||
3. EXPRESS MELODY & EMOTION: The generated MIDI notes (pitch, start_beat, duration_beats, velocity) must express the requested musical emotion (e.g., happy, sad, epic, energetic, melancholic). Use rich harmonies/chords for backing tracks (Strings, Pads, Piano) and expressive, rhythmic melodies for Lead/Solo tracks. Do NOT write single repeating notes or overly sparse patterns unless explicitly asked.
|
||||
4. VIẾT ĐẦY ĐỦ CÁC NOTE: Bạn phải viết đầy đủ các note cho TẤT CẢ các tracks được yêu cầu, và trải dài trong SUỐT chiều dài số bars yêu cầu (ví dụ: yêu cầu 8 bars và 6 tracks thì phải tạo đủ 6 tracks, mỗi track phải có các note MIDI bắt đầu từ beat 0.0 kéo dài liên tục đến beat 32.0 (8 bars * 4 beat/bar)).
|
||||
|
||||
QUAN TRỌNG:
|
||||
1. Bạn đang hoạt động ở chế độ một lượt (one-shot). Hãy trả về TẤT CẢ các function calls cần thiết để thực hiện toàn bộ các bước trong yêu cầu của người dùng trong một phản hồi duy nhất. Đừng thực hiện từng bước qua nhiều lượt chat.
|
||||
2. Có thể gọi nhiều function cùng một lúc (gọi song song/nối tiếp). Chúng sẽ được thực thi theo thứ tự bạn trả về.
|
||||
3. Khi người dùng yêu cầu chọn và cắt/sao chép/copy một đoạn nhạc từ track cũ để tạo đoạn nhạc mới (bằng lệnh 'cut_audio'), và sau đó yêu cầu xử lý tiếp đoạn nhạc mới tạo đó (ví dụ: 'sau đó fade in đoạn đó', 'chỉnh âm lượng đoạn đó', 'xuất mp3 đoạn đó'...), thì tất cả các lệnh xử lý tiếp theo này (như 'fade_in', 'export_audio', 'set_track_volume') PHẢI để trống tham số 'track_id' (hoặc truyền null/không truyền) để hệ thống tự động áp dụng lên track mới vừa được tạo ra. KHÔNG ĐƯỢC dùng 'track_id' của track gốc ban đầu cho các lệnh xử lý phía sau.
|
||||
Ví dụ: "Hãy chọn và copy từ bar 4 đến bar 12 của track 1 sau đó fade in clip đó 3s, xuất ra mp3" -> Bạn phải trả về đồng thời 3 cuộc gọi hàm theo thứ tự:
|
||||
- cut_audio({"track_id": "1", "start_bar": 4, "end_bar": 12})
|
||||
- fade_in({"duration_seconds": 3}) (không truyền track_id)
|
||||
- export_audio({"format": "mp3"}) (không truyền track_id)
|
||||
4. Bar 0 đại diện cho bar đầu tiên trên timeline.` },
|
||||
{ role: 'user', content: `Ngữ cảnh DAW hiện tại:\n${contextStr}\n\nYêu cầu người dùng: ${prompt}` }
|
||||
];
|
||||
}
|
||||
|
||||
function buildAIPromptContext(dawState) {
|
||||
const tracks = (dawState.tracks || []).map(t => {
|
||||
const clips = t.clips && t.clips.length > 0 ? t.clips : (t.buffer ? [{ id: 'default_' + t.id, name: t.name, startTime: t.startTime || 0, duration: t.buffer.duration }] : []);
|
||||
const se = t.synth_engine || null;
|
||||
return {
|
||||
id: t.id,
|
||||
name: t.name,
|
||||
type: t.buffer ? 'audio' : 'empty',
|
||||
hasBuffer: !!t.buffer,
|
||||
muted: t.muted,
|
||||
solo: t.solo,
|
||||
volumeDb: t.volumeDb ?? 0,
|
||||
pan: t.pan ?? 0,
|
||||
synth_engine: se ? { type: se.type, plugin_id: se.plugin_id, soundfont_bank: se.soundfont_bank, soundfont_program: se.soundfont_program, soundfont_id: se.soundfont_id } : undefined,
|
||||
clips: clips.map(c => ({ id: c.id, name: c.name, startTime: parseFloat((c.startTime || 0).toFixed(3)), duration: parseFloat((c.buffer ? c.buffer.duration : 0).toFixed(3)) }))
|
||||
};
|
||||
});
|
||||
return {
|
||||
tempo: parseInt(dawState.bpm || '120'),
|
||||
timeSignature: '4/4',
|
||||
selectedTrackId: dawState.selectedTrackId || null,
|
||||
playheadPosition: parseFloat((dawState.currentTime || 0).toFixed(3)),
|
||||
selection: (dawState.selLeft !== null && dawState.selRight !== null && dawState.selRight > dawState.selLeft) ? {
|
||||
start: parseFloat(dawState.selLeft.toFixed(3)),
|
||||
end: parseFloat(dawState.selRight.toFixed(3)),
|
||||
length: parseFloat((dawState.selRight - dawState.selLeft).toFixed(3))
|
||||
} : null,
|
||||
tracks
|
||||
};
|
||||
}
|
||||
|
||||
async function executeAIPrompt({ prompt, provider, model, apiKey, baseUrl, dawContext, tools, systemInstruction }) {
|
||||
const messages = buildUserMessage(prompt, dawContext, systemInstruction);
|
||||
const toolList = tools || DEFAULT_TOOLS;
|
||||
|
||||
const completion = await callLLM({
|
||||
provider,
|
||||
model,
|
||||
apiKey,
|
||||
baseUrl,
|
||||
messages,
|
||||
tools: toolList,
|
||||
toolChoice: 'auto'
|
||||
});
|
||||
|
||||
if (completion && completion.error) {
|
||||
const errMsg = completion.error.message || completion.error.code || JSON.stringify(completion.error);
|
||||
throw new Error(`AI Provider error: ${errMsg}`);
|
||||
}
|
||||
|
||||
const functionCalls = extractFunctionCalls(completion);
|
||||
const textResponse = completion.choices && completion.choices[0] && completion.choices[0].message && completion.choices[0].message.content
|
||||
? completion.choices[0].message.content
|
||||
: '';
|
||||
|
||||
return {
|
||||
functionCalls,
|
||||
textResponse,
|
||||
raw: completion
|
||||
};
|
||||
}
|
||||
|
||||
async function createMidiItem(args) {
|
||||
return fetch('/api/audio_editor', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ action: 'add_midi', ...args })
|
||||
}).then(r => r.json());
|
||||
}
|
||||
|
||||
async function modifyMidiNotes(args) {
|
||||
return fetch('/api/audio_editor', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ action: 'modify_midi_notes', ...args })
|
||||
}).then(r => r.json());
|
||||
}
|
||||
|
||||
async function processAIDSP(args) {
|
||||
return fetch('/api/ai_dsp_engine', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ action: 'process_ai_dsp', ...args })
|
||||
}).then(r => r.json());
|
||||
}
|
||||
|
||||
return {
|
||||
DEFAULT_TOOLS,
|
||||
REARRANGE_TOOL_SPEC,
|
||||
REARRANGE_EXTEND_TOOL_SPEC,
|
||||
REARRANGE_SCENARIOS,
|
||||
detectRearrangeScenario,
|
||||
buildRearrangeMessage,
|
||||
callLLM,
|
||||
extractFunctionCalls,
|
||||
buildUserMessage,
|
||||
buildAIPromptContext,
|
||||
executeAIPrompt,
|
||||
createMidiItem,
|
||||
modifyMidiNotes,
|
||||
processAIDSP
|
||||
};
|
||||
})();
|
||||
|
||||
window.executeAIPrompt = AIGateway.executeAIPrompt;
|
||||
window.AIGateway = AIGateway;
|
||||
@@ -41,17 +41,57 @@ window.API_BASE_URL = window.API_BASE_URL || window.location.origin;
|
||||
updateUserQuota: (userId, storageLimitMb, maxTracks = 16) => apiRequest(`/api/v1/admin/quotas/${userId}`, { method: 'PUT', body: JSON.stringify({ storage_limit_mb: storageLimitMb, max_tracks: maxTracks }) }),
|
||||
updateUserRole: (userId, role, isActive = true) => apiRequest(`/api/v1/admin/users/${userId}/role`, { method: 'PUT', body: JSON.stringify({ role, is_active: isActive }) }),
|
||||
deleteUser: (userId) => apiRequest(`/api/v1/admin/users/${userId}`, { method: 'DELETE' }),
|
||||
createUser: (username, email, password, role = 'standard') => apiRequest('/api/v1/admin/users', { method: 'POST', body: JSON.stringify({ username, email, password, role }) }),
|
||||
|
||||
saveTempProject: (dataJson) => apiRequest('/api/v1/projects/temp', { method: 'POST', body: JSON.stringify({ data_json: dataJson }) }),
|
||||
getTempProject: () => apiRequest('/api/v1/projects/temp', { method: 'GET' }),
|
||||
saveCloudProject: (name, dataJson) => apiRequest('/api/v1/projects/cloud', { method: 'POST', body: JSON.stringify({ name, data_json: dataJson }) }),
|
||||
listCloudProjects: () => apiRequest('/api/v1/projects/cloud', { method: 'GET' }),
|
||||
getCloudProject: (projectId) => apiRequest(`/api/v1/projects/cloud/${projectId}`, { method: 'GET' }),
|
||||
deleteCloudProject: (projectId) => apiRequest(`/api/v1/projects/cloud/${projectId}`, { method: 'DELETE' }),
|
||||
updateCloudProject: (projectId, name, dataJson) => apiRequest(`/api/v1/projects/cloud/${projectId}`, { method: 'PUT', body: JSON.stringify({ name, data_json: dataJson }) }),
|
||||
listMyFiles: (activeFileIds) => apiRequest('/api/v1/audio/my-files', { method: 'POST', body: JSON.stringify({ active_file_ids: activeFileIds }) }),
|
||||
deleteMyFile: (fileId) => apiRequest(`/api/v1/audio/my-files/${fileId}`, { method: 'DELETE' }),
|
||||
|
||||
aiScan: (trackId, fileId, minLoopDuration = 2.0, maxLoopDuration = 6.0) => apiRequest('/api/v1/audio/ai-scan', { method: 'POST', body: JSON.stringify({ track_id: trackId, file_id: fileId, min_loop_duration: minLoopDuration, max_loop_duration: maxLoopDuration }) }),
|
||||
aiCut: (sourceTrackId, fileId, selectionStart, selectionEnd) => apiRequest('/api/v1/audio/ai-cut', { method: 'POST', body: JSON.stringify({ source_track_id: sourceTrackId, file_id: fileId, selection_start: selectionStart, selection_end: selectionEnd }) }),
|
||||
runPythonTool: (toolType, trackId, fileId, timePos = 0.0, freq = 440.0, duration = 2.0, waveType = "sine") => apiRequest('/api/v1/audio/python-tool', { method: 'POST', body: JSON.stringify({ tool_type: toolType, track_id: trackId, file_id: fileId, time_pos: timePos, freq: freq, duration: duration, wave_type: waveType }) }),
|
||||
|
||||
getAIConfigs: () => apiRequest('/api/v1/user/config/ai', { method: 'GET' }),
|
||||
saveAIConfigs: (providers) => apiRequest('/api/v1/user/config/ai', { method: 'POST', body: JSON.stringify({ providers }) })
|
||||
saveAIConfigs: (providers) => apiRequest('/api/v1/user/config/ai', { method: 'POST', body: JSON.stringify({ providers }) }),
|
||||
|
||||
getPreferences: () => apiRequest('/api/v1/user/preferences', { method: 'GET' }),
|
||||
savePreferences: (prefs) => apiRequest('/api/v1/user/preferences', { method: 'POST', body: JSON.stringify({ preferences: prefs }) }),
|
||||
|
||||
listPlugins: () => apiRequest('/api/v1/plugins/available', { method: 'GET' }),
|
||||
getSoundfontCatalog: () => apiRequest('/api/v1/plugins/soundfonts/catalog', { method: 'GET' }),
|
||||
listDefaultSoundfonts: () => apiRequest('/api/v1/plugins/default-soundfonts', { method: 'GET' }),
|
||||
listSoundfontInstruments: (sfId) => apiRequest(`/api/v1/plugins/soundfont-instruments/${sfId}`, { method: 'GET' }),
|
||||
getAIPresets: () => apiRequest('/api/v1/ai/presets', { method: 'GET' }),
|
||||
saveAIPreset: (preset) => apiRequest('/api/v1/ai/presets', { method: 'POST', body: JSON.stringify(preset) }),
|
||||
deleteAIPreset: (presetId) => apiRequest(`/api/v1/ai/presets/${presetId}`, { method: 'DELETE' }),
|
||||
|
||||
createBackup: (projectId) => apiRequest(`/api/v1/projects/cloud/${projectId}/backup`, { method: 'POST' }),
|
||||
listBackups: (projectId) => apiRequest(`/api/v1/projects/cloud/${projectId}/backups`, { method: 'GET' }),
|
||||
deleteBackup: (backupId) => apiRequest(`/api/v1/projects/cloud/backups/${backupId}`, { method: 'DELETE' }),
|
||||
cleanupBackups: (keep) => apiRequest('/api/v1/projects/cloud/backups/cleanup', { method: 'POST', body: JSON.stringify({ keep }) }),
|
||||
|
||||
renderProject: (projectJson, outputFilename) => apiRequest('/api/v1/plugins/render', { method: 'POST', body: JSON.stringify({ project_json: projectJson, output_filename: outputFilename }) }),
|
||||
deleteSoundFont: (sfId) => apiRequest(`/api/v1/plugins/soundfont/${sfId}`, { method: 'DELETE' }),
|
||||
uploadSoundFont: async (file) => {
|
||||
const formData = new FormData();
|
||||
formData.append('file', file);
|
||||
const token = localStorage.getItem('sonic_token') || '';
|
||||
const resp = await fetch(`${window.API_BASE_URL}/api/v1/plugins/upload-soundfont`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${token}` },
|
||||
body: formData
|
||||
});
|
||||
if (!resp.ok) {
|
||||
const err = await resp.json().catch(() => ({}));
|
||||
throw new Error(err.detail || 'Upload failed');
|
||||
}
|
||||
return resp.json();
|
||||
}
|
||||
};
|
||||
})();
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
// SonicForge Studio - DAW Command Dispatcher
|
||||
// Command Pattern & Undo/Redo Engine (28_AI_PANEL.md §1 & §2)
|
||||
|
||||
const DAWCommandDispatcher = (function() {
|
||||
const MAX_HISTORY = 50;
|
||||
const history = [];
|
||||
let historyIndex = -1;
|
||||
|
||||
function pushHistory(entry) {
|
||||
history.push(entry);
|
||||
if (history.length > MAX_HISTORY) history.shift();
|
||||
historyIndex = history.length - 1;
|
||||
}
|
||||
|
||||
function undo() {
|
||||
if (historyIndex < 0) return null;
|
||||
const entry = history[historyIndex];
|
||||
historyIndex--;
|
||||
return entry;
|
||||
}
|
||||
|
||||
function redo() {
|
||||
if (historyIndex >= history.length - 1) return null;
|
||||
historyIndex++;
|
||||
const entry = history[historyIndex];
|
||||
return entry;
|
||||
}
|
||||
|
||||
function canUndo() { return historyIndex >= 0; }
|
||||
function canRedo() { return historyIndex < history.length - 1; }
|
||||
|
||||
const registry = {};
|
||||
|
||||
function register(name, handler) {
|
||||
registry[name] = handler;
|
||||
}
|
||||
|
||||
function execute(name, args) {
|
||||
if (!registry[name]) {
|
||||
return { success: false, error: `Unknown command: ${name}` };
|
||||
}
|
||||
const result = registry[name](args);
|
||||
pushHistory({ name, args, result, timestamp: Date.now() });
|
||||
return result;
|
||||
}
|
||||
|
||||
function getHistory() { return history; }
|
||||
function getHistoryIndex() { return historyIndex; }
|
||||
|
||||
function registerDAWCommands(api) {
|
||||
register('CREATE_TRACK', (args) => api.createTrack(args));
|
||||
register('DELETE_TRACK', (args) => api.deleteTrack(args));
|
||||
register('ADD_CLIP', (args) => api.addClip(args));
|
||||
register('REMOVE_CLIP', (args) => api.removeClip(args));
|
||||
register('SET_TRACK_VOLUME', (args) => api.setTrackVolume(args));
|
||||
register('SET_TRACK_PAN', (args) => api.setTrackPan(args));
|
||||
register('TOGGLE_MUTE', (args) => api.toggleMute(args));
|
||||
register('TOGGLE_SOLO', (args) => api.toggleSolo(args));
|
||||
register('PROCESS_AUDIO_DSP', (args) => api.processAudioDsp(args));
|
||||
register('RENAME_TRACK', (args) => api.renameTrack(args));
|
||||
register('SCAN_TRACK', (args) => api.scanTrack(args));
|
||||
register('FADE_IN', (args) => api.fadeIn(args));
|
||||
register('FADE_OUT', (args) => api.fadeOut(args));
|
||||
register('CUT_AUDIO', (args) => api.cutAudio(args));
|
||||
register('SET_SELECTION', (args) => api.setSelection(args));
|
||||
register('EXPORT_AUDIO', (args) => api.exportAudio(args));
|
||||
register('SET_BPM', (args) => api.setBpm(args));
|
||||
register('SET_PLAYHEAD', (args) => api.setPlayhead(args));
|
||||
register('SELECT_ITEM', (args) => api.selectItem(args));
|
||||
register('ADD_MARKER', (args) => api.addMarker(args));
|
||||
register('CREATE_MIDI_ITEM', (args) => api.createMidiItem(args));
|
||||
register('MODIFY_MIDI_NOTES', (args) => api.modifyMidiNotes(args));
|
||||
register('PROCESS_AI_DSP', (args) => api.processAudioDsp(args));
|
||||
register('GENERATE_MULTITRACK_MIDI', (args) => api.generateMultitrackMidi(args));
|
||||
register('REARRANGE_MIDI_MELODY', (args) => api.rearrangeMidiMelody(args));
|
||||
}
|
||||
|
||||
return {
|
||||
register,
|
||||
execute,
|
||||
undo,
|
||||
redo,
|
||||
canUndo,
|
||||
canRedo,
|
||||
pushHistory,
|
||||
getHistory,
|
||||
getHistoryIndex,
|
||||
registerDAWCommands
|
||||
};
|
||||
})();
|
||||
|
||||
window.DAWCommandDispatcher = DAWCommandDispatcher;
|
||||
@@ -0,0 +1,37 @@
|
||||
(function () {
|
||||
if (window.__FluidSynthReady) return;
|
||||
|
||||
var FLUIDSYNTH_JS_URL;
|
||||
var FLUIDSYNTH_WASM_URL;
|
||||
|
||||
if (window.__FLUIDSYNTH_CDN) {
|
||||
FLUIDSYNTH_JS_URL = window.__FLUIDSYNTH_CDN;
|
||||
} else if (window.location.hostname === 'localhost' || window.location.hostname === '127.0.0.1') {
|
||||
FLUIDSYNTH_JS_URL = 'https://cdn.jsdelivr.net/npm/@enikey87/fluidsynth-emscripten@0.1.1/dist/libfluidsynth-2.3.0-sf3.js';
|
||||
} else {
|
||||
FLUIDSYNTH_JS_URL = '/static/js/vendor/libfluidsynth-2.3.0-sf3.js';
|
||||
}
|
||||
|
||||
FLUIDSYNTH_WASM_URL = FLUIDSYNTH_JS_URL.replace(/\.js$/, '.wasm');
|
||||
|
||||
window.__FluidSynthLocateWasm = function () { return FLUIDSYNTH_WASM_URL; };
|
||||
|
||||
window.__FluidSynthReady = new Promise(function (resolve) {
|
||||
var script = document.createElement('script');
|
||||
script.type = 'module';
|
||||
script.textContent = [
|
||||
'import FluidsynthModule from "' + FLUIDSYNTH_JS_URL + '";',
|
||||
'window.__FluidSynthModuleFactory = FluidsynthModule;',
|
||||
'console.log("[FluidSynth] Loaded:", "' + FLUIDSYNTH_JS_URL + '");'
|
||||
].join('\n');
|
||||
document.head.appendChild(script);
|
||||
|
||||
var check = setInterval(function () {
|
||||
if (window.__FluidSynthModuleFactory) {
|
||||
clearInterval(check);
|
||||
resolve();
|
||||
}
|
||||
}, 50);
|
||||
setTimeout(function () { clearInterval(check); resolve(); }, 15000);
|
||||
});
|
||||
})();
|
||||
@@ -0,0 +1,77 @@
|
||||
// SonicForge Studio Ghost Note Extractor Service
|
||||
(function() {
|
||||
function extractGhostLayers(activeTracks, targetTrackId, targetItemId, bpm) {
|
||||
if (!activeTracks || !targetItemId) return [];
|
||||
|
||||
const secondsPerBeat = 60.0 / (parseInt(bpm) || 120);
|
||||
|
||||
let targetItem = null;
|
||||
for (var i = 0; i < activeTracks.length; i++) {
|
||||
var t = activeTracks[i];
|
||||
var found = (t.midiItems || []).find(function(m) { return m.id === targetItemId; });
|
||||
if (found) { targetItem = found; break; }
|
||||
}
|
||||
if (!targetItem) return [];
|
||||
|
||||
const windowStartBeat = targetItem.startTime / secondsPerBeat;
|
||||
const windowEndBeat = (targetItem.startTime + targetItem.duration) / secondsPerBeat;
|
||||
|
||||
const ghostLayers = [];
|
||||
|
||||
for (var i = 0; i < activeTracks.length; i++) {
|
||||
var track = activeTracks[i];
|
||||
if (!track.midiItems || !track.midiItems.length) continue;
|
||||
if (track.muted) continue;
|
||||
|
||||
var isSameTrack = track.id === targetTrackId;
|
||||
var trackGhostNotes = [];
|
||||
|
||||
for (var j = 0; j < track.midiItems.length; j++) {
|
||||
var item = track.midiItems[j];
|
||||
if (item.id === targetItemId) continue;
|
||||
|
||||
var itemStartBeat = item.startTime / secondsPerBeat;
|
||||
var itemEndBeat = (item.startTime + item.duration) / secondsPerBeat;
|
||||
|
||||
// Same-track items always contribute notes (show whole track); cross-track only when overlapping the window
|
||||
if (itemStartBeat >= windowEndBeat || itemEndBeat < windowStartBeat) {
|
||||
if (!isSameTrack) continue;
|
||||
}
|
||||
|
||||
var notes = item.notes || [];
|
||||
for (var k = 0; k < notes.length; k++) {
|
||||
var note = notes[k];
|
||||
var noteAbsStart = itemStartBeat + (note.start_beat || 0);
|
||||
var noteAbsEnd = noteAbsStart + (note.duration_beats || 1);
|
||||
|
||||
if (noteAbsStart >= windowEndBeat && !isSameTrack) continue;
|
||||
|
||||
trackGhostNotes.push({
|
||||
id: 'ghost_' + (note.id || Math.random().toString(36).substr(2, 9)),
|
||||
pitch: note.pitch,
|
||||
relative_start_beat: noteAbsStart - windowStartBeat,
|
||||
duration_beats: (note.duration_beats || 1),
|
||||
velocity: note.velocity,
|
||||
item_id: item.id,
|
||||
original_track_name: track.name,
|
||||
original_track_color: track.color || '#888888'
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (trackGhostNotes.length > 0) {
|
||||
ghostLayers.push({
|
||||
track_id: track.id,
|
||||
track_name: track.name,
|
||||
track_color: track.color || '#6b7280',
|
||||
isSameTrack: isSameTrack,
|
||||
notes: trackGhostNotes
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return ghostLayers;
|
||||
}
|
||||
|
||||
window.SonicGhost = { extractGhostLayers: extractGhostLayers };
|
||||
})();
|
||||
@@ -0,0 +1,60 @@
|
||||
const SonicMidiExtractor = (function() {
|
||||
const NOTE_NAMES = ['C','C#','D','D#','E','F','F#','G','G#','A','A#','B'];
|
||||
|
||||
function midiPitchToNoteName(pitch) {
|
||||
const note = NOTE_NAMES[pitch % 12];
|
||||
const octave = Math.floor(pitch / 12) - 1;
|
||||
return note + octave;
|
||||
}
|
||||
|
||||
function extractSelectedMIDIContext(tracks, selectedItemId, bpm) {
|
||||
let targetItem = null;
|
||||
let targetTrack = null;
|
||||
|
||||
for (const track of tracks) {
|
||||
const items = track.midiItems || [];
|
||||
const item = items.find(i => i.id === selectedItemId);
|
||||
if (item) { targetItem = item; targetTrack = track; break; }
|
||||
}
|
||||
|
||||
if (!targetItem) {
|
||||
throw new Error('Please select a MIDI Item on the Timeline before requesting a Rearrangement!');
|
||||
}
|
||||
|
||||
const notes = targetItem.notes || [];
|
||||
if (notes.length === 0) {
|
||||
throw new Error('Selected MIDI item has no notes to rearrange.');
|
||||
}
|
||||
|
||||
const compactNotes = notes.map(n => ({
|
||||
pitch: n.pitch,
|
||||
note_name: midiPitchToNoteName(n.pitch),
|
||||
start_beat: parseFloat((n.start_beat || 0).toFixed(2)),
|
||||
duration_beats: parseFloat((n.duration_beats || 1).toFixed(2)),
|
||||
velocity: parseFloat((n.velocity || 0.8).toFixed(2))
|
||||
}));
|
||||
|
||||
const totalDurationBeats = compactNotes.reduce((max, n) => Math.max(max, n.start_beat + n.duration_beats), 0);
|
||||
|
||||
return {
|
||||
track_name: targetTrack.name,
|
||||
track_id: targetTrack.id,
|
||||
item_id: targetItem.id,
|
||||
item_name: targetItem.name,
|
||||
duration_bars: targetItem.length_bars || Math.ceil(totalDurationBeats / 4),
|
||||
total_beats: Math.ceil(totalDurationBeats),
|
||||
bpm: parseInt(bpm || '120'),
|
||||
total_notes: compactNotes.length,
|
||||
notes: compactNotes,
|
||||
instrument: targetTrack.synth_engine ? {
|
||||
soundfont_id: targetTrack.synth_engine.soundfont_id || '',
|
||||
soundfont_bank: targetTrack.synth_engine.soundfont_bank ?? 0,
|
||||
soundfont_program: targetTrack.synth_engine.soundfont_program ?? 0
|
||||
} : null
|
||||
};
|
||||
}
|
||||
|
||||
return { extractSelectedMIDIContext, midiPitchToNoteName };
|
||||
})();
|
||||
|
||||
window.SonicMidiExtractor = SonicMidiExtractor;
|
||||
@@ -0,0 +1,43 @@
|
||||
(function () {
|
||||
'use strict';
|
||||
window.SonicPianoRoll = {
|
||||
getParentTrackByItemId: function (itemId, tracks) {
|
||||
if (!tracks || !itemId) return null;
|
||||
for (var i = 0; i < tracks.length; i++) {
|
||||
var items = tracks[i].midiItems || [];
|
||||
for (var j = 0; j < items.length; j++) {
|
||||
if (items[j].id === itemId) return tracks[i];
|
||||
}
|
||||
}
|
||||
return null;
|
||||
},
|
||||
getParentTrackIdByItemId: function (itemId, tracks) {
|
||||
var trk = this.getParentTrackByItemId(itemId, tracks);
|
||||
return trk ? trk.id : null;
|
||||
},
|
||||
buildActiveScope: function (itemId, tracks) {
|
||||
var parentTrack = this.getParentTrackByItemId(itemId, tracks);
|
||||
if (!parentTrack) return null;
|
||||
var trackIndex = -1;
|
||||
for (var i = 0; i < tracks.length; i++) {
|
||||
if (tracks[i].id === parentTrack.id) { trackIndex = i; break; }
|
||||
}
|
||||
return {
|
||||
item_id: itemId,
|
||||
parent_track_id: parentTrack.id,
|
||||
midi_channel: trackIndex >= 0 ? trackIndex % 16 : 0,
|
||||
current_synth_engine: parentTrack.synth_engine || null,
|
||||
instrument_program: parentTrack.instrumentProgram,
|
||||
instrument_name: parentTrack.instrumentName
|
||||
};
|
||||
},
|
||||
getTrackMidiChannel: function (track, tracks) {
|
||||
if (track && track.midiChannel !== undefined) return track.midiChannel;
|
||||
if (!track || !tracks) return 0;
|
||||
for (var i = 0; i < tracks.length; i++) {
|
||||
if (tracks[i].id === track.id) return i % 16;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
};
|
||||
})();
|
||||
@@ -0,0 +1,261 @@
|
||||
const PromptTemplateManager = (function() {
|
||||
const DEFAULT_PRESETS = [
|
||||
{
|
||||
id: "preset_epic_orchestra_intro",
|
||||
name: "Epic Orchestra Intro (8 Bars)",
|
||||
keywords: ["epic orchestra", "epic orchestral", "hoành tráng", "nhạc phim epic"],
|
||||
category: "Orchestral / Film Score",
|
||||
default_bars: 8,
|
||||
default_bpm: 130,
|
||||
default_scale: "C Minor",
|
||||
system_instruction_template: "You are a professional film composer. Create a powerful, dramatic 8-bar orchestral intro. Write notes continuously across all 8 bars (from beat 0.0 to 32.0) for every track. Use whole notes, half notes, or quarter notes to maintain a clean layout without overloading the sequence. The required structure to return via the `generate_multitrack_midi` tool consists of 3 tracks: 1. Strings: plays smooth legato chord changes (one chord per 1 or 2 bars continuously). 2. Brass Theme: plays a swelling simple melodic line in the C3-C5 range spanning all 8 bars. 3. Epic Percussion: hits heavily on beats 1 and 3 throughout the entire 8 bars.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-23T16:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_pop_piano_chords",
|
||||
name: "Pop Piano Chords (4 Bars)",
|
||||
keywords: ["pop piano", "piano chords", "ballad piano", "hợp âm piano"],
|
||||
category: "Pop / Ballad",
|
||||
default_bars: 4,
|
||||
default_bpm: 90,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional Pop Piano player. Generate a beautiful 4-bar piano chord progression (e.g. C - G - Am - F) with pleasant chord voicing and simple accompaniment. Return the MIDI notes via the `generate_multitrack_midi` function on a track named 'Pop Piano'. Write notes continuously across all 4 bars (from beat 0.0 to 16.0). Use whole/half/quarter notes continuously to ensure every bar has piano chords playing.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-23T16:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_cyberpunk_synth",
|
||||
name: "Cyberpunk Synthwave (8 Bars)",
|
||||
keywords: ["cyberpunk synth", "synthwave", "cyberpunk", "futuristic synth"],
|
||||
category: "Electronic / Synthwave",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "A Minor",
|
||||
system_instruction_template: "You are a Synthwave producer. Generate a driving 8-bar cyberpunk synth theme. Return MIDI notes via `generate_multitrack_midi` containing: 1. Synth Bass: plays eighth notes on pitch A1, C2, G1 continuously across all 8 bars (from beat 0.0 to 32.0). 2. Synth Lead: plays a simple, melodic line in the high register (C4-E5) continuously across all 8 bars (from beat 0.0 to 32.0).",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-23T16:00:00Z"
|
||||
},
|
||||
// ── Rearrange & Variation Scenarios ──
|
||||
{
|
||||
id: "preset_rearrange_arpeggio",
|
||||
name: "Arpeggio Variation",
|
||||
keywords: ["arpeggio", "arp", "broken chord", "arpeggiate"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional arranger. Rearrange the source MIDI notes into an arpeggiated variation. Subdivide sustained chord notes into sequential 0.25-beat arpeggiated steps with octave/triad jumps. Create a driving, shimmering sequence while preserving the underlying chord progression.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_harmonies",
|
||||
name: "Add 3rd/4th Harmonies",
|
||||
keywords: ["harmony", "harmonize", "harmonies", "3rd", "duet", "chord voicing"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional arranger. Add parallel harmonizing notes to the source melody. For each source pitch P, insert notes at P+3 or P+4 (Major/Minor 3rds) at the same start_beat. Create a richer, fuller texture resembling two instruments playing in duet.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_syncopation",
|
||||
name: "Syncopation / Off-beat",
|
||||
keywords: ["syncopation", "syncopate", "syncopated", "off-beat", "off beat", "funk", "latin"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional arranger. Apply syncopation to the source melody. Shift start_beat alignment off strong beats (0.0, 1.0, 2.0, 3.0) onto off-beats (0.5, 1.5, 2.5, 3.5). Preserve the core melodic outline while creating a rhythmic, syncopated feel.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_walking_bass",
|
||||
name: "Walking Bass Line",
|
||||
keywords: ["walking bass", "bassline", "walking"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional arranger. Transform the source melody into a walking bass line. Lower pitches to bass register (36-48 range). Build continuous quarter-note steps following the chord outline. Create an improvisational walking bassline matching the original chord progression.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_jazz",
|
||||
name: "Jazz Swing Rearrangement",
|
||||
keywords: ["jazz", "swing", "jazz swing", "bebop"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional Jazz arranger. Rearrange the source melody into a rhythmic Jazz Swing style. Apply: off-beat syncopation, 7th/9th chord extensions, swung eighth notes (triplet feel), and rhythmic variations while preserving the core melodic outline.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_synthwave",
|
||||
name: "Synthwave / 80s Style",
|
||||
keywords: ["synthwave", "80s", "retro", "synth wave", "retrowave", "outrun"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "A Minor",
|
||||
system_instruction_template: "You are a Synthwave producer. Transform the source melody into an 80s Synthwave style. Apply: driving 8th-note bass arpeggios, analog synth lead with portamento, pulsating chord pads, and octave jumps in the bass.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_cinematic",
|
||||
name: "Cinematic Orchestral",
|
||||
keywords: ["cinematic", "orchestral", "epic", "film score", "symphonic", "hollywood"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 130,
|
||||
default_scale: "C Minor",
|
||||
system_instruction_template: "You are a cinematic composer. Transform the source melody into a dramatic Cinematic Orchestral arrangement. Apply: swelling dynamics, brass stabs on downbeats, string ostinato patterns, taiko percussion hits. Layer multiple octaves for epic width.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_simplify",
|
||||
name: "Simplify / Strip Down",
|
||||
keywords: ["simplify", "minimal", "strip down", "sparse", "reduce"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional arranger. Simplify the source melody: keep only downbeat notes (start_beat at 0.0, 1.0, 2.0, 3.0). Strip out embellishments, passing tones, and grace notes. Reduce chord voicings to root and 5th only.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
},
|
||||
{
|
||||
id: "preset_rearrange_chromatic",
|
||||
name: "Passing Tones & Chromaticism",
|
||||
keywords: ["passing tone", "chromatic", "chromaticism", "smooth", "approach note"],
|
||||
category: "Rearrange / Variation",
|
||||
default_bars: 8,
|
||||
default_bpm: 120,
|
||||
default_scale: "C Major",
|
||||
system_instruction_template: "You are a professional arranger. Add passing tones and chromatic approach notes to the source melody. Insert half-step and whole-step approach notes targeting chord tones on strong beats. Smooth out the melodic progression while preserving the harmonic framework.",
|
||||
is_user_defined: false,
|
||||
is_favorite: false,
|
||||
created_at: "2026-07-28T11:00:00Z"
|
||||
}
|
||||
];
|
||||
|
||||
const STORAGE_KEY = 'daw_ai_prompt_presets';
|
||||
|
||||
function PromptTemplateManager() {
|
||||
this.presets = [];
|
||||
this.loadPresets();
|
||||
}
|
||||
|
||||
PromptTemplateManager.prototype.loadPresets = function() {
|
||||
try {
|
||||
const localData = localStorage.getItem(STORAGE_KEY);
|
||||
if (localData) {
|
||||
const parsed = JSON.parse(localData);
|
||||
const userPresets = parsed.filter(p => p.is_user_defined);
|
||||
this.presets = [...DEFAULT_PRESETS, ...userPresets];
|
||||
} else {
|
||||
this.presets = [...DEFAULT_PRESETS];
|
||||
this.savePresets();
|
||||
}
|
||||
} catch (_) {
|
||||
this.presets = [...DEFAULT_PRESETS];
|
||||
}
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.savePresets = function() {
|
||||
const userData = this.presets.filter(p => p.is_user_defined);
|
||||
localStorage.setItem(STORAGE_KEY, JSON.stringify(userData));
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.getPresets = function() {
|
||||
return this.presets;
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.matchPreset = function(userQuery) {
|
||||
if (!userQuery) return null;
|
||||
const queryLower = userQuery.toLowerCase();
|
||||
let bestMatch = null;
|
||||
let bestScore = 0;
|
||||
|
||||
for (const preset of this.presets) {
|
||||
for (const kw of preset.keywords) {
|
||||
const kwLower = kw.toLowerCase();
|
||||
if (queryLower === kwLower) {
|
||||
if (3 > bestScore) {
|
||||
bestScore = 3;
|
||||
bestMatch = { preset, score: 3 };
|
||||
}
|
||||
} else if (queryLower.includes(kwLower)) {
|
||||
if (2 > bestScore) {
|
||||
bestScore = 2;
|
||||
bestMatch = { preset, score: 2 };
|
||||
}
|
||||
} else if (kwLower.includes(queryLower)) {
|
||||
if (1 > bestScore) {
|
||||
bestScore = 1;
|
||||
bestMatch = { preset, score: 1 };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return bestMatch;
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.saveUserPreset = function(presetObject) {
|
||||
const index = this.presets.findIndex(p => p.id === presetObject.id);
|
||||
if (index >= 0) {
|
||||
this.presets[index] = presetObject;
|
||||
} else {
|
||||
this.presets.push(presetObject);
|
||||
}
|
||||
this.savePresets();
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.deletePreset = function(id) {
|
||||
this.presets = this.presets.filter(p => p.id !== id);
|
||||
this.savePresets();
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.getContextualSuggestions = function(hasSelectedItem) {
|
||||
if (hasSelectedItem) {
|
||||
return this.presets.filter(p => p.category === 'Rearrange / Variation' || p.is_favorite);
|
||||
} else {
|
||||
return this.presets.filter(p => p.category !== 'Rearrange / Variation');
|
||||
}
|
||||
};
|
||||
|
||||
PromptTemplateManager.prototype.toggleFavorite = function(id) {
|
||||
const preset = this.presets.find(p => p.id === id);
|
||||
if (preset) {
|
||||
preset.is_favorite = !preset.is_favorite;
|
||||
if (preset.is_user_defined) {
|
||||
this.savePresets();
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
return PromptTemplateManager;
|
||||
})();
|
||||
|
||||
window.PromptTemplateManager = PromptTemplateManager;
|
||||
window.DEFAULT_PRESETS = (new PromptTemplateManager()).getPresets();
|
||||
@@ -0,0 +1,851 @@
|
||||
(function () {
|
||||
const RENDER_BLOCK = 512;
|
||||
const QUEUE_TARGET = 16;
|
||||
let _audioCtx = null;
|
||||
let _fluidModule = null;
|
||||
let _synthPtr = null;
|
||||
let _settingsPtr = null;
|
||||
let _workletNode = null;
|
||||
let _initialized = false;
|
||||
let _initPromise = null;
|
||||
let _currentSfId = null;
|
||||
let _sfHandleMap = new Map();
|
||||
let _channels = Array.from({ length: 16 }, () => ({ bank: 0, program: 0, isPercussion: false }));
|
||||
let _nextMelodicChannel = 0;
|
||||
let _sustainStates = new Array(16).fill(false);
|
||||
let _engineChMap = {};
|
||||
let _activeNotes = {};
|
||||
let _leftBufPtr = null;
|
||||
let _rightBufPtr = null;
|
||||
let _renderTimer = null;
|
||||
let _pendingNoteTimers = [];
|
||||
let _loadedFonts = {};
|
||||
let _activeOscillators = {};
|
||||
let _gainNode = null;
|
||||
let _pendingOutputDestination = null;
|
||||
let _outputDestination = null; // cache đích route — dedupe swap dư giữa stream
|
||||
let _validPercCache = {}; // { sfId: [bank, prog] | null } — preset percussion hợp lệ
|
||||
let _sfLoadFailAt = {}; // { sfId: timestamp } — cooldown 10s sau load fail
|
||||
let _scheduledNotes = [];
|
||||
let _loadPromises = {};
|
||||
let _sfloadSeq = 0;
|
||||
|
||||
const getCtx = function () {
|
||||
if (_audioCtx) {
|
||||
if (!_gainNode) {
|
||||
_gainNode = _audioCtx.createGain();
|
||||
_gainNode.gain.value = 0.3;
|
||||
_outputDestination = _pendingOutputDestination || (window.masterBus ? window.masterBus.input : _audioCtx.destination);
|
||||
_gainNode.connect(_outputDestination);
|
||||
}
|
||||
return _audioCtx;
|
||||
}
|
||||
if (typeof getAudioContext === 'function') {
|
||||
var ctx = getAudioContext();
|
||||
if (!_gainNode) {
|
||||
_gainNode = ctx.createGain();
|
||||
_gainNode.gain.value = 0.3;
|
||||
_outputDestination = _pendingOutputDestination || (window.masterBus ? window.masterBus.input : ctx.destination);
|
||||
_gainNode.connect(_outputDestination);
|
||||
}
|
||||
return ctx;
|
||||
}
|
||||
if (!window.__sharedAudioCtx) {
|
||||
window.__sharedAudioCtx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
}
|
||||
if (window.__sharedAudioCtx.state === 'suspended') {
|
||||
window.__sharedAudioCtx.resume();
|
||||
}
|
||||
if (!_gainNode) {
|
||||
_gainNode = window.__sharedAudioCtx.createGain();
|
||||
_gainNode.gain.value = 0.3;
|
||||
_outputDestination = _pendingOutputDestination || window.__sharedAudioCtx.destination;
|
||||
_gainNode.connect(_outputDestination);
|
||||
}
|
||||
return window.__sharedAudioCtx;
|
||||
};
|
||||
|
||||
const SonicSF = {
|
||||
loadedFonts: _loadedFonts,
|
||||
|
||||
// Route the shared FluidSynth output through a per-track node (e.g. the
|
||||
// track's gainNode) so the track's FX chain / fader / pan affect the
|
||||
// soundfont instrument. Pass null to restore the default master-bus route.
|
||||
setOutputDestination: function (node) {
|
||||
try {
|
||||
if (_gainNode) {
|
||||
const dest = node || (window.masterBus ? window.masterBus.input : ((_audioCtx || window.__sharedAudioCtx).destination));
|
||||
// DEDUPE: đích không đổi → KHÔNG disconnect/reconnect.
|
||||
// Swap dư giữa dòng notes đang phát (applyAllTrackMuteSolo →
|
||||
// updateSfRouting gọi lại cùng đích sfEntry sau noteon đầu)
|
||||
// làm ScriptProcessor xuất buffer uninitialized → NaN →
|
||||
// 11 biquad "state is bad" → CÂM (mọi log: state-bad nổ
|
||||
// ngay sau setOutputDestination lần 2).
|
||||
if (dest === _outputDestination) return;
|
||||
_gainNode.disconnect();
|
||||
_gainNode.connect(dest);
|
||||
_outputDestination = dest;
|
||||
console.log('[SonicSF] setOutputDestination to:', node ? 'track node (sfEntry)' : 'masterBus.input');
|
||||
} else {
|
||||
_pendingOutputDestination = node || null;
|
||||
console.log('[SonicSF] setOutputDestination pending:', node ? 'track node (sfEntry)' : 'null');
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn('[SonicSF] setOutputDestination error:', e);
|
||||
}
|
||||
},
|
||||
getOutputNode: function () {
|
||||
return _gainNode;
|
||||
},
|
||||
|
||||
init: async function (audioContext) {
|
||||
if (_initialized && _fluidModule) return;
|
||||
if (_initPromise) return _initPromise;
|
||||
_initPromise = (async () => {
|
||||
if (!window.__FluidSynthModuleFactory) {
|
||||
console.log("[SonicSF] Waiting for FluidSynth WASM module to load...");
|
||||
await window.__FluidSynthReady;
|
||||
if (!window.__FluidSynthModuleFactory) {
|
||||
console.warn("[SonicSF] FluidSynth WASM still not available after waiting.");
|
||||
_initPromise = null;
|
||||
return;
|
||||
}
|
||||
}
|
||||
try {
|
||||
_audioCtx = audioContext;
|
||||
if (_audioCtx.state === 'suspended') await _audioCtx.resume();
|
||||
// Create gain node for master bus routing before any node connections
|
||||
if (!_gainNode) {
|
||||
_gainNode = _audioCtx.createGain();
|
||||
_gainNode.gain.value = 0.3;
|
||||
_outputDestination = _pendingOutputDestination || (window.masterBus ? window.masterBus.input : _audioCtx.destination);
|
||||
_gainNode.connect(_outputDestination);
|
||||
}
|
||||
|
||||
console.log("[SonicSF] AudioCtx state:", _audioCtx.state, "sampleRate:", _audioCtx.sampleRate);
|
||||
|
||||
// ── Renderer selection ──
|
||||
// ScriptProcessor is the default and FINAL choice: it is
|
||||
// pull-based (onaudioprocess is invoked by the audio thread),
|
||||
// so it cannot starve when the main thread is busy (font
|
||||
// loading, WASM decode, UI). The AudioWorklet push model
|
||||
// (setInterval on the main thread) starves under load and
|
||||
// produces SILENCE on this machine — repeatedly confirmed.
|
||||
// The deprecation console warning is purely cosmetic.
|
||||
var _useScriptNode = true;
|
||||
try {
|
||||
await _audioCtx.audioWorklet.addModule('/static/js/worklets/fluidsynth-bridge.js?v=202608031240');
|
||||
console.log("[SonicSF] Worklet registered OK (unused)");
|
||||
} catch (e) {
|
||||
console.warn("[SonicSF] Worklet reg failed:", e);
|
||||
}
|
||||
|
||||
console.log("[SonicSF] Initializing FluidSynth WASM Engine...");
|
||||
var TOTAL_MEMORY = 256 * 1024 * 1024;
|
||||
_fluidModule = await window.__FluidSynthModuleFactory({
|
||||
locateFile: function (path) {
|
||||
if (path.endsWith('.wasm')) {
|
||||
return window.__FluidSynthLocateWasm ? window.__FluidSynthLocateWasm() : path;
|
||||
}
|
||||
return path;
|
||||
},
|
||||
TOTAL_MEMORY: TOTAL_MEMORY,
|
||||
printErr: function (msg) {
|
||||
// "No preset found on channel" is FluidSynth's
|
||||
// expected notice when a soundfont simply has no
|
||||
// preset for a bank (e.g. bank 128 on a melodic-only
|
||||
// font) — the note is just silent, not an error.
|
||||
if (msg && (msg.indexOf('No preset found on channel') !== -1 || msg.indexOf('There is no preset with bank number') !== -1)) return;
|
||||
console.warn('[FluidSynth:err]', msg);
|
||||
}
|
||||
});
|
||||
|
||||
if (!_fluidModule || !_fluidModule._new_fluid_settings) {
|
||||
throw new Error("FluidSynth WASM module loaded but API missing");
|
||||
}
|
||||
|
||||
_settingsPtr = _fluidModule._new_fluid_settings();
|
||||
_fluidModule._fluid_settings_setnum(_settingsPtr, "synth.sample-rate", _audioCtx.sampleRate || 44100);
|
||||
_fluidModule._fluid_settings_setnum(_settingsPtr, "synth.gain", 1.0);
|
||||
_fluidModule._fluid_settings_setnum(_settingsPtr, "synth.polyphony", 256);
|
||||
_fluidModule._fluid_settings_setint(_settingsPtr, "synth.verbose", 0);
|
||||
_fluidModule._fluid_settings_setint(_settingsPtr, "synth.ladspa.active", 0);
|
||||
_fluidModule._fluid_settings_setstr(_settingsPtr, "player.timing-source", "audio");
|
||||
console.log("[SonicSF] FluidSynth settings configured");
|
||||
|
||||
_synthPtr = _fluidModule._new_fluid_synth(_settingsPtr);
|
||||
if (!_synthPtr) throw new Error("Failed to create FluidSynth synthesizer");
|
||||
_fluidModule._fluid_synth_set_gain(_synthPtr, 1.0);
|
||||
|
||||
try { _fluidModule.FS.mkdir('/soundfonts'); } catch (e) {}
|
||||
|
||||
_leftBufPtr = _fluidModule._malloc(RENDER_BLOCK * 4);
|
||||
_rightBufPtr = _fluidModule._malloc(RENDER_BLOCK * 4);
|
||||
|
||||
if (!_useScriptNode) {
|
||||
try {
|
||||
// Force stereo output regardless of the device's
|
||||
// channel count — FluidSynth renders stereo, and a
|
||||
// mono output would crash the worklet (out[1] undefined).
|
||||
_workletNode = new AudioWorkletNode(_audioCtx, 'fluidsynth-bridge', {
|
||||
numberOfOutputs: 1,
|
||||
outputChannelCount: [2],
|
||||
channelCount: 2,
|
||||
channelCountMode: 'explicit'
|
||||
});
|
||||
_workletNode.connect(_gainNode);
|
||||
console.log("[SonicSF] AudioWorklet node connected via gain");
|
||||
_startRenderLoop();
|
||||
} catch (e) {
|
||||
console.warn("[SonicSF] AudioWorkletNode failed:", e);
|
||||
_useScriptNode = true;
|
||||
}
|
||||
}
|
||||
if (_useScriptNode) {
|
||||
var spBufSz = 2048;
|
||||
var spn = _audioCtx.createScriptProcessor(spBufSz, 0, 2);
|
||||
var lp = _fluidModule._malloc(spBufSz * 4);
|
||||
var rp = _fluidModule._malloc(spBufSz * 4);
|
||||
// Heap WASM có thể realloc khi load SoundFont lớn (SGM-V2.01
|
||||
// ~300MB) → lp/rp DANGLE → đọc vùng nhớ đã free → NaN/garbage
|
||||
// → master chain "state is bad" → CÂM + stuck. Theo dõi
|
||||
// buffer + re-malloc khi đổi.
|
||||
var _heapBufRef = _fluidModule.HEAPU8.buffer;
|
||||
spn.onaudioprocess = function (e) {
|
||||
var left = e.outputBuffer.getChannelData(0);
|
||||
var right = e.outputBuffer.getChannelData(1);
|
||||
var sz = left.length;
|
||||
try {
|
||||
if (_fluidModule.HEAPU8.buffer !== _heapBufRef) {
|
||||
try { _fluidModule._free(lp); _fluidModule._free(rp); } catch (er2) {}
|
||||
lp = _fluidModule._malloc(sz * 4);
|
||||
rp = _fluidModule._malloc(sz * 4);
|
||||
_heapBufRef = _fluidModule.HEAPU8.buffer;
|
||||
}
|
||||
_fluidModule._fluid_synth_write_float(_synthPtr, sz, lp, 0, 1, rp, 0, 1);
|
||||
var hf = _fluidModule.HEAPF32;
|
||||
var lpb = lp >> 2, rpb = rp >> 2;
|
||||
for (var si = 0; si < sz; si++) {
|
||||
// NaN sweep: mẫu NaN/Inf → 0 (chain biquad
|
||||
// KHÔNG BAO GIỜ được nhận NaN → không state-bad).
|
||||
var L = hf[lpb + si], R = hf[rpb + si];
|
||||
left[si] = isFinite(L) ? L : 0;
|
||||
right[si] = isFinite(R) ? R : 0;
|
||||
}
|
||||
} catch (er) {}
|
||||
};
|
||||
spn.connect(_gainNode);
|
||||
_workletNode = spn;
|
||||
console.log("[SonicSF] ScriptProcessorNode connected via gain (buf:", spBufSz, ")");
|
||||
}
|
||||
|
||||
_initialized = true;
|
||||
console.log("[SonicSF] FluidSynth WASM Engine initialized.");
|
||||
} catch (e) {
|
||||
console.error("[SonicSF] FluidSynth init failed:", e);
|
||||
_initPromise = null;
|
||||
_cleanupFluid();
|
||||
}
|
||||
})();
|
||||
return _initPromise;
|
||||
},
|
||||
|
||||
_allocCStr: function (str) {
|
||||
var ptr = _fluidModule._malloc(str.length + 1);
|
||||
for (var i = 0; i < str.length; i++) {
|
||||
_fluidModule.HEAPU8[ptr + i] = str.charCodeAt(i);
|
||||
}
|
||||
_fluidModule.HEAPU8[ptr + str.length] = 0;
|
||||
return ptr;
|
||||
},
|
||||
|
||||
_tryLoadSFL: function (buf, ext) {
|
||||
var fname = '/' + ext + '_' + (++_sfloadSeq) + '_' + Date.now();
|
||||
try { _fluidModule.FS.unlink(fname); } catch (e) {}
|
||||
_fluidModule.FS.writeFile(fname, new Uint8Array(buf));
|
||||
var cPath = this._allocCStr(fname);
|
||||
// reset_presets = 0: loading a NEW soundfont must NOT reset the
|
||||
// presets already selected on other channels. With 1, FluidSynth
|
||||
// re-points every channel to the new font's preset 0, so loading a
|
||||
// second instrument silently changes the first one's sound
|
||||
// (decay/loop envelope…).
|
||||
var handle = _fluidModule._fluid_synth_sfload(_synthPtr, cPath, 0);
|
||||
_fluidModule._free(cPath);
|
||||
try { _fluidModule.FS.unlink(fname); } catch (e) {}
|
||||
return handle;
|
||||
},
|
||||
|
||||
loadSoundFont: async function (sfId) {
|
||||
if (!_initialized || !_fluidModule) return false;
|
||||
if (_currentSfId === sfId) return true;
|
||||
if (_sfHandleMap.has(sfId)) {
|
||||
_currentSfId = sfId;
|
||||
return true;
|
||||
}
|
||||
// Deduplicate concurrent loads: rapid key presses (or several armed
|
||||
// tracks) all call loadSoundFont for the same font before the first
|
||||
// load resolves. Without this, the same soundfont is sfload'd several
|
||||
// times (handles 1,2,3,4…) — wasting the 256MB WASM heap and stalling
|
||||
// notes until each load finishes (audible lag, then silence).
|
||||
if (!_loadPromises[sfId]) {
|
||||
_loadPromises[sfId] = this._doLoadSoundFont(sfId).then(function (ok) {
|
||||
// Do NOT cache failures: a transient error (network hiccup,
|
||||
// memory pressure) must not permanently kill the instrument —
|
||||
// the next note retries the load and recovers.
|
||||
if (!ok) delete _loadPromises[sfId];
|
||||
return ok;
|
||||
});
|
||||
}
|
||||
return _loadPromises[sfId];
|
||||
},
|
||||
|
||||
_doLoadSoundFont: async function (sfId) {
|
||||
try {
|
||||
// KHÔNG unload SF cũ khi sfload SF mới: unload làm handle cũ
|
||||
// thành rác trong khi channel state vẫn trỏ tới → program_select
|
||||
// bị skip (progAlreadySet) → noteon trên handle đã unload →
|
||||
// "Instrument not found ... substituted prog 0". Heap 256MB đủ
|
||||
// cho vài SF (SGM + latin = 2 handle — log OK). SF cũ khi cần
|
||||
// lại chỉ được sfload lại nếu map bị xóa (không xảy ra ở đây).
|
||||
var cache = window.SonicSFStorage;
|
||||
var buf = cache ? await cache.getBuffer(sfId) : null;
|
||||
if (buf) {
|
||||
var cachedOk = this._tryLoadSFL(buf, '.sf3');
|
||||
if (cachedOk === -1) cachedOk = this._tryLoadSFL(buf, '.sf2');
|
||||
if (cachedOk !== -1) {
|
||||
_sfHandleMap.set(sfId, cachedOk);
|
||||
_currentSfId = sfId;
|
||||
_loadedFonts[sfId] = true;
|
||||
console.log("[SonicSF] SoundFont loaded from cache:", sfId, "handle:", cachedOk);
|
||||
return true;
|
||||
}
|
||||
// Stale/corrupt cache (e.g. old SF3 buffers the WASM can't
|
||||
// decode) — drop it and re-download from the server.
|
||||
console.warn("[SonicSF] Cached SoundFont unplayable, re-downloading:", sfId);
|
||||
try { await cache.saveBuffer(sfId, null); } catch (e2) {}
|
||||
}
|
||||
var url = "/api/v1/plugins/soundfonts/download/" + encodeURIComponent(sfId) + "?t=" + Date.now();
|
||||
var resp = await fetch(url);
|
||||
if (!resp.ok) {
|
||||
// Fallback: font bundled theo deployment (static/soundfonts —
|
||||
// serve qua /soundfonts/{f} — catalog default-soundfonts).
|
||||
var url2 = "/soundfonts/" + encodeURIComponent(sfId.replace(/^sf_/, '')) + "?t=" + Date.now();
|
||||
var resp2 = await fetch(url2);
|
||||
if (!resp2.ok) {
|
||||
console.warn("[SonicSF] SoundFont not found:", sfId);
|
||||
return false;
|
||||
}
|
||||
resp = resp2;
|
||||
}
|
||||
buf = await resp.arrayBuffer();
|
||||
if (cache) await cache.saveBuffer(sfId, buf);
|
||||
var sfHandle = this._tryLoadSFL(buf, '.sf3');
|
||||
if (sfHandle === -1) {
|
||||
console.warn("[SonicSF] sfload .sf3 failed, trying .sf2 for", sfId);
|
||||
sfHandle = this._tryLoadSFL(buf, '.sf2');
|
||||
}
|
||||
if (sfHandle === -1) {
|
||||
console.error("[SonicSF] FluidSynth failed to parse SoundFont:", sfId);
|
||||
return false;
|
||||
}
|
||||
_sfHandleMap.set(sfId, sfHandle);
|
||||
_currentSfId = sfId;
|
||||
_loadedFonts[sfId] = true;
|
||||
console.log("[SonicSF] SoundFont loaded:", sfId, "handle:", sfHandle);
|
||||
return true;
|
||||
} catch (e) {
|
||||
console.warn("[SonicSF] loadSoundFont error:", e);
|
||||
return false;
|
||||
}
|
||||
},
|
||||
selectInstrument: async function (channel, bank, program, sfId) {
|
||||
if (!_initialized || !_fluidModule) {
|
||||
var ctx = (typeof getAudioContext === 'function') ? getAudioContext() : new (window.AudioContext || window.webkitAudioContext)();
|
||||
await this.init(ctx);
|
||||
}
|
||||
if (sfId) {
|
||||
var ok = await this.loadSoundFont(sfId);
|
||||
if (!ok) return;
|
||||
}
|
||||
var engKey = (sfId || '') + ':' + bank + ':' + program;
|
||||
if (channel === undefined || channel === null) {
|
||||
if (!_engineChMap[engKey]) {
|
||||
var allocCh = this.allocateChannel(bank);
|
||||
_engineChMap[engKey] = allocCh;
|
||||
channel = allocCh;
|
||||
} else {
|
||||
channel = _engineChMap[engKey];
|
||||
}
|
||||
} else if (!_engineChMap[engKey]) {
|
||||
_engineChMap[engKey] = channel;
|
||||
}
|
||||
var sfHandle = _sfHandleMap.get(sfId);
|
||||
if (sfHandle !== undefined) {
|
||||
try {
|
||||
_fluidModule._fluid_synth_program_select(_synthPtr, channel, sfHandle, bank, program);
|
||||
} catch (e) {}
|
||||
} else {
|
||||
try { _fluidModule._fluid_synth_bank_select(_synthPtr, channel, bank); } catch (e) {}
|
||||
try { _fluidModule._fluid_synth_program_change(_synthPtr, channel, program); } catch (e) {}
|
||||
}
|
||||
_channels[channel].bank = bank;
|
||||
_channels[channel].program = program;
|
||||
_channels[channel].isPercussion = (bank === 128);
|
||||
_channels[channel].sfId = sfId;
|
||||
},
|
||||
|
||||
controllerChange: function (channel, controller, value) {
|
||||
if (channel < 0 || channel > 15) return;
|
||||
if (_initialized && _fluidModule) {
|
||||
try {
|
||||
_fluidModule._fluid_synth_cc(_synthPtr, channel, controller, value);
|
||||
} catch (e) {}
|
||||
}
|
||||
if (controller === 0) {
|
||||
_channels[channel].bank = value;
|
||||
_channels[channel].isPercussion = (value === 128);
|
||||
}
|
||||
if (controller === 64) {
|
||||
_sustainStates[channel] = value >= 64;
|
||||
}
|
||||
},
|
||||
|
||||
programChange: function (channel, program) {
|
||||
if (channel < 0 || channel > 15) return;
|
||||
if (_initialized && _fluidModule) {
|
||||
try {
|
||||
_fluidModule._fluid_synth_program_change(_synthPtr, channel, program);
|
||||
} catch (e) {}
|
||||
}
|
||||
_channels[channel].program = program;
|
||||
},
|
||||
|
||||
allocateChannel: function (bank) {
|
||||
if (bank === 128) return 9;
|
||||
var ch = _nextMelodicChannel % 9;
|
||||
_nextMelodicChannel = (_nextMelodicChannel + 1) % 9;
|
||||
return ch;
|
||||
},
|
||||
|
||||
applyAITrackInstrument: function (bank, program, synthEngine) {
|
||||
if (synthEngine) {
|
||||
bank = bank !== undefined ? bank : (synthEngine.soundfont_bank || 0);
|
||||
program = program !== undefined ? program : (synthEngine.soundfont_program || 0);
|
||||
}
|
||||
var sfId = synthEngine && synthEngine.soundfont_id;
|
||||
var engKey = (sfId || '') + ':' + bank + ':' + program;
|
||||
if (!_engineChMap[engKey]) {
|
||||
var channel = this.allocateChannel(bank);
|
||||
_engineChMap[engKey] = channel;
|
||||
}
|
||||
this.selectInstrument(_engineChMap[engKey], bank, program, sfId);
|
||||
return _engineChMap[engKey];
|
||||
},
|
||||
|
||||
getChannelState: function (channel) {
|
||||
if (channel < 0 || channel > 15) return null;
|
||||
return { bank: _channels[channel].bank, program: _channels[channel].program, isPercussion: _channels[channel].isPercussion };
|
||||
},
|
||||
|
||||
sustainActive: function (channel) {
|
||||
if (channel < 0 || channel > 15) return false;
|
||||
return _sustainStates[channel];
|
||||
},
|
||||
|
||||
pitchBend: function (channel, value) {
|
||||
if (channel < 0 || channel > 15) return;
|
||||
if (_initialized && _fluidModule) {
|
||||
try {
|
||||
_fluidModule._fluid_synth_pitch_bend(_synthPtr, channel, value);
|
||||
} catch (e) {}
|
||||
}
|
||||
},
|
||||
|
||||
stopNote: function (channel, pitch) {
|
||||
if (channel < 0 || channel > 15) return;
|
||||
if (_initialized && _fluidModule) {
|
||||
var key = channel + ':' + pitch;
|
||||
var mappedChs = _activeNotes[key];
|
||||
if (mappedChs === undefined) mappedChs = [channel];
|
||||
for (var i = 0; i < mappedChs.length; i++) {
|
||||
try { _fluidModule._fluid_synth_noteoff(_synthPtr, mappedChs[i], pitch); } catch (e) {}
|
||||
}
|
||||
delete _activeNotes[key];
|
||||
}
|
||||
},
|
||||
|
||||
playNote: function (note, velocity, durationMs, startTime, program, destinationNode, channel, synthEngine) {
|
||||
if (!_initialized || !_fluidModule) {
|
||||
this._lazyInit();
|
||||
this._playNoteFallback(note, velocity, 2000, startTime, program, destinationNode, channel, synthEngine);
|
||||
return;
|
||||
}
|
||||
this._playNoteFluid(note, velocity, durationMs, startTime, program, channel, synthEngine);
|
||||
},
|
||||
|
||||
_lazyInit: async function () {
|
||||
if (_initialized && _fluidModule) return;
|
||||
if (_initPromise) return;
|
||||
try {
|
||||
var ctx = (typeof getAudioContext === 'function') ? getAudioContext() : new (window.AudioContext || window.webkitAudioContext)();
|
||||
if (ctx.state === 'suspended') await ctx.resume();
|
||||
await this.init(ctx);
|
||||
} catch (e) {}
|
||||
},
|
||||
|
||||
_playNoteFluid: function (note, velocity, durationMs, startTime, program, channel, synthEngine) {
|
||||
var parsedPitch = parseInt(note);
|
||||
var midiPitch = isNaN(parsedPitch) ? 60 : Math.min(127, Math.max(0, parsedPitch));
|
||||
var rawVel = (typeof velocity === 'number' && isFinite(velocity)) ? (velocity > 1 ? velocity : velocity * 127) : 100;
|
||||
if (isNaN(rawVel)) rawVel = 100;
|
||||
var midiVel = Math.min(127, Math.max(1, Math.floor(rawVel)));
|
||||
var _origChannel = channel;
|
||||
var usedBank = 0, usedProg = 0;
|
||||
if (synthEngine) {
|
||||
usedBank = synthEngine.soundfont_bank || 0;
|
||||
usedProg = synthEngine.soundfont_program || 0;
|
||||
if (channel === undefined) {
|
||||
var engKey = (synthEngine.soundfont_id || '') + ':' + usedBank + ':' + usedProg;
|
||||
var mappedCh = _engineChMap[engKey];
|
||||
if (mappedCh !== undefined) {
|
||||
channel = mappedCh;
|
||||
} else {
|
||||
channel = usedBank === 128 ? 9 : 0;
|
||||
}
|
||||
}
|
||||
if (channel === undefined) channel = (usedBank === 128 ? 9 : 0);
|
||||
} else if (program !== undefined) {
|
||||
usedProg = program;
|
||||
if (channel === undefined) channel = 0;
|
||||
} else {
|
||||
// No instrument configured: silent — no FluidSynth, no oscillator.
|
||||
return;
|
||||
}
|
||||
if (channel === undefined) channel = (usedBank === 128 ? 9 : 0);
|
||||
var ch = channel;
|
||||
var ctx = getCtx();
|
||||
var now = ctx.currentTime;
|
||||
var delay = (typeof startTime === 'number' && startTime > now) ? (startTime - now) : 0;
|
||||
var durSec = (durationMs || 500) / 1000;
|
||||
var scheduledNote = { on: null, off: null };
|
||||
var self = this;
|
||||
var doNote = function () {
|
||||
try {
|
||||
var finalBank = parseInt(usedBank);
|
||||
if (isNaN(finalBank) || !isFinite(finalBank)) finalBank = 0;
|
||||
var finalProg = parseInt(usedProg);
|
||||
if (isNaN(finalProg) || !isFinite(finalProg)) finalProg = 0;
|
||||
var finalSfId = synthEngine ? synthEngine.soundfont_id : undefined;
|
||||
var cachedCh = _channels[ch];
|
||||
// The note's own synth engine (track instrument) is
|
||||
// authoritative. Channel state is only a cache: it must never
|
||||
// mask the track's instrument, otherwise multi-track ARM or a
|
||||
// re-picked instrument plays the wrong soundfont. Without an
|
||||
// engine, fall back to the soundfont configured on the channel.
|
||||
if (!synthEngine && cachedCh && cachedCh.sfId !== undefined) {
|
||||
finalBank = parseInt(cachedCh.bank) || 0;
|
||||
finalProg = parseInt(cachedCh.program) || 0;
|
||||
finalSfId = cachedCh.sfId;
|
||||
}
|
||||
// Ensure the soundfont is actually loaded before the note plays.
|
||||
// Quick instrument pick on a track does not pre-load it, so load
|
||||
// lazily here and retry the note once the font is ready.
|
||||
if (finalSfId && !_sfHandleMap.has(finalSfId)) {
|
||||
// Cooldown lỗi: font 404 → KHÔNG spam fetch mỗi note (10s)
|
||||
// — note chạy thẳng fallback để CÓ ÂM.
|
||||
var _lastFail = _sfLoadFailAt[finalSfId] || 0;
|
||||
if (Date.now() - _lastFail < 10000) {
|
||||
try { self._playNoteFallback(note, velocity, durationMs, startTime, program, null, channel, synthEngine); } catch (e) {}
|
||||
return;
|
||||
}
|
||||
console.log('[SonicSF] soundfont not loaded yet, loading:', finalSfId);
|
||||
self.loadSoundFont(finalSfId).then(function (ok) {
|
||||
console.log('[SonicSF] loadSoundFont result:', ok, 'for:', finalSfId);
|
||||
if (ok) {
|
||||
doNote();
|
||||
} else {
|
||||
// Font KHÔNG tải được (404/format) → KHÔNG drop note
|
||||
// câm lặng ("bỏ qua WASM") — fallback oscillator.
|
||||
_sfLoadFailAt[finalSfId] = Date.now();
|
||||
try { self._playNoteFallback(note, velocity, durationMs, startTime, program, null, channel, synthEngine); } catch (e) {}
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
// Program change at note time, not call time — ensures correct
|
||||
// instrument for each item regardless of processing order.
|
||||
// Skip if the channel already has this exact instrument (avoids
|
||||
// per-note soundfont reloads that cause audible crackle/glitches).
|
||||
// ⚠️ Chỉ skip khi handle SF vẫn CÒN HỢP LỆ trong map — nếu
|
||||
// không → vẫn program_select lại (tránh dùng handle đã unload).
|
||||
var progAlreadySet = cachedCh && cachedCh.program === finalProg && cachedCh.bank === finalBank && cachedCh.sfId === finalSfId
|
||||
&& (finalSfId ? _sfHandleMap.has(finalSfId) : true);
|
||||
if ((synthEngine || program !== undefined) && !progAlreadySet) {
|
||||
var sfHandle = finalSfId ? _sfHandleMap.get(finalSfId) : undefined;
|
||||
console.log('[SonicSF] selectProgram for channel:', ch, 'sfHandle:', sfHandle, 'bank:', finalBank, 'prog:', finalProg);
|
||||
if (sfHandle !== undefined) {
|
||||
try {
|
||||
// Percussion (bank 128): tìm preset HỢP LỆ trong
|
||||
// font — quét bank 128 + bank 0 (0-127) MỘT LẦN,
|
||||
// cache theo sfId. Trước đây chỉ thử 4 preset cố
|
||||
// định → font không có → cache channel = (128,0)
|
||||
// INVALID → note sau skip re-select (progAlreadySet)
|
||||
// → noteon preset rỗng = CÂM ("1 âm đầu rồi câm").
|
||||
if (finalBank === 128) {
|
||||
var _vKey = finalSfId || ('h' + sfHandle);
|
||||
if (_validPercCache[_vKey] === undefined) {
|
||||
var _found = null;
|
||||
for (var _b = 0; _b < 2 && !_found; _b++) {
|
||||
var _bk = _b === 0 ? 128 : 0;
|
||||
for (var _p = 0; _p < 128 && !_found; _p++) {
|
||||
try {
|
||||
if (_fluidModule._fluid_synth_program_select(_synthPtr, 9, sfHandle, _bk, _p) === 0) {
|
||||
_found = [_bk, _p];
|
||||
}
|
||||
} catch (e) {}
|
||||
}
|
||||
}
|
||||
_validPercCache[_vKey] = _found;
|
||||
}
|
||||
if (_validPercCache[_vKey]) {
|
||||
finalBank = _validPercCache[_vKey][0];
|
||||
finalProg = _validPercCache[_vKey][1];
|
||||
}
|
||||
}
|
||||
var _selRet = _fluidModule._fluid_synth_program_select(_synthPtr, ch, sfHandle, finalBank, finalProg);
|
||||
} catch (e) {}
|
||||
} else {
|
||||
try { _fluidModule._fluid_synth_bank_select(_synthPtr, ch, finalBank); } catch (e) {}
|
||||
try { _fluidModule._fluid_synth_program_change(_synthPtr, ch, finalProg); } catch (e) {}
|
||||
}
|
||||
if (!_channels[ch]) _channels[ch] = {};
|
||||
_channels[ch].bank = finalBank;
|
||||
_channels[ch].program = finalProg;
|
||||
_channels[ch].sfId = finalSfId;
|
||||
}
|
||||
console.log('[SonicSF] noteon channel:', ch, 'pitch:', midiPitch, 'vel:', midiVel, 'sfId:', finalSfId);
|
||||
_fluidModule._fluid_synth_noteon(_synthPtr, ch, midiPitch, midiVel);
|
||||
var noteMapKey = (_origChannel !== undefined ? _origChannel : 0) + ':' + midiPitch;
|
||||
if (!_activeNotes[noteMapKey]) _activeNotes[noteMapKey] = [];
|
||||
if (_activeNotes[noteMapKey].indexOf(ch) === -1) _activeNotes[noteMapKey].push(ch);
|
||||
if (durationMs > 0 && durationMs < 60000) {
|
||||
scheduledNote.off = setTimeout(function () {
|
||||
try {
|
||||
_fluidModule._fluid_synth_noteoff(_synthPtr, ch, midiPitch);
|
||||
var arr = _activeNotes[noteMapKey];
|
||||
if (arr) {
|
||||
var idx = arr.indexOf(ch);
|
||||
if (idx >= 0) arr.splice(idx, 1);
|
||||
if (arr.length === 0) delete _activeNotes[noteMapKey];
|
||||
}
|
||||
} catch (e) {}
|
||||
}, durationMs);
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn("[SonicSF] FluidSynth noteOn error:", e);
|
||||
self._playNoteFallback(note, velocity, durationMs, startTime, program, null, channel, synthEngine);
|
||||
}
|
||||
};
|
||||
if (delay > 0) {
|
||||
scheduledNote.on = setTimeout(doNote, delay * 1000);
|
||||
_scheduledNotes.push(scheduledNote);
|
||||
} else {
|
||||
doNote();
|
||||
}
|
||||
},
|
||||
|
||||
// Hủy mọi note-on được schedule (tương lai) + note-off mọi notes đang
|
||||
// ngân — gọi khi STOP/PAUSE để hết "âm thanh bị stuck" (note-on chưa
|
||||
// bắn vẫn bắn sau khi dừng; notes durationMs>=60000 không có note-off
|
||||
// tự động → ngân vô hạn → VU master nhảy dù không play).
|
||||
panic: function () {
|
||||
_scheduledNotes.forEach(function (sn) { if (sn.on) { clearTimeout(sn.on); sn.on = null; } });
|
||||
_scheduledNotes = [];
|
||||
if (_initialized && _fluidModule) {
|
||||
// noteoff TỪNG note đang ngân (binding _fluid_synth_noteoff chắc
|
||||
// chắn tồn tại — đã dùng cho duration hết) — all_notes_off có
|
||||
// thể không có trong WASM exports (catch nuốt → notes kẹt).
|
||||
Object.keys(_activeNotes).forEach(function (key) {
|
||||
var parts = key.split(':');
|
||||
var pitch = parseInt(parts[1], 10);
|
||||
(_activeNotes[key] || []).forEach(function (ch) {
|
||||
try { _fluidModule._fluid_synth_noteoff(_synthPtr, ch, pitch); } catch (e) {}
|
||||
});
|
||||
});
|
||||
try {
|
||||
for (var c = 0; c < 16; c++) _fluidModule._fluid_synth_all_notes_off(_synthPtr, c);
|
||||
} catch (e) {}
|
||||
}
|
||||
_activeNotes = {};
|
||||
},
|
||||
|
||||
_playNoteFallback: function (note, velocity, durationMs, startTime, program, destinationNode, channel, synthEngine) {
|
||||
var ctx = getCtx();
|
||||
var freq = 440 * Math.pow(2, (note - 69) / 12);
|
||||
if (freq <= 0 || isNaN(freq)) return null;
|
||||
if (synthEngine) {
|
||||
var ch = channel !== undefined ? channel : (synthEngine.soundfont_bank === 128 ? 9 : 0);
|
||||
this.controllerChange(ch, 0, synthEngine.soundfont_bank || 0);
|
||||
this.programChange(ch, synthEngine.soundfont_program || 0);
|
||||
if (channel === undefined) channel = ch;
|
||||
if (program === undefined) program = synthEngine.soundfont_program;
|
||||
}
|
||||
var osc = ctx.createOscillator();
|
||||
var noteGain = ctx.createGain();
|
||||
var oscType = 'triangle';
|
||||
var attackTime = 0.03, decayTime = 0.1, sustainLevel = 0.5, releaseTime = 0.2, volFactor = 0.25;
|
||||
var prog = program !== undefined ? parseInt(program) : 0;
|
||||
// CHỈ dùng cache channel khi KHÔNG có program/synthEngine được
|
||||
// truyền — trước đây override program của track bằng cache channel
|
||||
// (bị track khác cùng channel ghi đè → preview note vẽ mới mang
|
||||
// nhạc cụ của track TRƯỚC).
|
||||
if (program === undefined && channel !== undefined && channel >= 0 && channel < 16) {
|
||||
prog = _channels[channel].program || prog;
|
||||
}
|
||||
if (prog >= 0 && prog <= 7) { oscType = 'sine'; decayTime = 0.3; sustainLevel = 0.1; releaseTime = 0.2; }
|
||||
else if (prog >= 8 && prog <= 15) { oscType = 'sine'; decayTime = 0.1; sustainLevel = 0.0; releaseTime = 0.1; }
|
||||
else if (prog >= 16 && prog <= 23) { oscType = 'sine'; attackTime = 0.05; sustainLevel = 0.8; releaseTime = 0.1; }
|
||||
else if (prog >= 24 && prog <= 31) { oscType = 'triangle'; decayTime = 0.4; sustainLevel = 0.2; releaseTime = 0.3; }
|
||||
else if (prog >= 32 && prog <= 39) { oscType = 'triangle'; attackTime = 0.02; decayTime = 0.2; sustainLevel = 0.6; releaseTime = 0.2; }
|
||||
else if (prog >= 40 && prog <= 47) { oscType = 'sawtooth'; attackTime = 0.15; sustainLevel = 0.8; releaseTime = 0.5; volFactor = 0.15; }
|
||||
else if (prog >= 48 && prog <= 55) { oscType = 'sawtooth'; attackTime = 0.2; sustainLevel = 0.8; releaseTime = 0.6; volFactor = 0.12; }
|
||||
else if (prog >= 56 && prog <= 63) { oscType = 'sawtooth'; attackTime = 0.08; sustainLevel = 0.7; releaseTime = 0.3; volFactor = 0.15; }
|
||||
else if (prog >= 64 && prog <= 71) { oscType = 'square'; attackTime = 0.05; sustainLevel = 0.6; releaseTime = 0.2; volFactor = 0.15; }
|
||||
else if (prog >= 72 && prog <= 79) { oscType = 'sine'; attackTime = 0.1; sustainLevel = 0.7; releaseTime = 0.3; volFactor = 0.2; }
|
||||
else if (prog >= 80 && prog <= 119) { oscType = 'sawtooth'; attackTime = 0.05; sustainLevel = 0.6; releaseTime = 0.4; volFactor = 0.15; }
|
||||
osc.type = oscType;
|
||||
osc.frequency.setValueAtTime(freq, 0);
|
||||
var startAt = startTime !== undefined ? startTime : ctx.currentTime;
|
||||
var durSec = durationMs / 1000;
|
||||
var vel = (typeof velocity === 'number' && isFinite(velocity) && !isNaN(velocity)) ? (velocity > 1 ? velocity / 127 : velocity) : 0.8;
|
||||
if (isNaN(vel)) vel = 0.8;
|
||||
var targetGain = vel * volFactor;
|
||||
noteGain.gain.setValueAtTime(0, startAt);
|
||||
noteGain.gain.linearRampToValueAtTime(targetGain, startAt + attackTime);
|
||||
noteGain.gain.linearRampToValueAtTime(targetGain * sustainLevel, startAt + attackTime + decayTime);
|
||||
var releaseStart = startAt + Math.max(attackTime + decayTime, durSec);
|
||||
noteGain.gain.linearRampToValueAtTime(targetGain * sustainLevel, releaseStart);
|
||||
noteGain.gain.linearRampToValueAtTime(0, releaseStart + releaseTime);
|
||||
osc.connect(noteGain);
|
||||
noteGain.connect(destinationNode || _gainNode || ctx.destination);
|
||||
osc.start(startAt);
|
||||
var stopAt = releaseStart + releaseTime + 0.02;
|
||||
osc.stop(stopAt);
|
||||
var oscId = note + '_' + Date.now() + '_' + Math.random();
|
||||
_activeOscillators[oscId] = { osc: osc, gain: noteGain };
|
||||
setTimeout(function () { delete _activeOscillators[oscId]; }, (stopAt - ctx.currentTime) * 1000 + 100);
|
||||
return osc;
|
||||
},
|
||||
|
||||
stopAll: function () {
|
||||
if (_initialized && _fluidModule) {
|
||||
// noteoff từng note đang ngân (binding chắc chắn tồn tại) —
|
||||
// phòng all_notes_off không có trong WASM exports.
|
||||
Object.keys(_activeNotes).forEach(function (key) {
|
||||
var parts = key.split(':');
|
||||
var pitch = parseInt(parts[1], 10);
|
||||
(_activeNotes[key] || []).forEach(function (ch) {
|
||||
try { _fluidModule._fluid_synth_noteoff(_synthPtr, ch, pitch); } catch (e) {}
|
||||
});
|
||||
});
|
||||
for (var ch = 0; ch < 16; ch++) {
|
||||
try { _fluidModule._fluid_synth_all_notes_off(_synthPtr, ch); } catch (e) {}
|
||||
}
|
||||
}
|
||||
while (_scheduledNotes.length > 0) {
|
||||
var n = _scheduledNotes.pop();
|
||||
if (n.on) { clearTimeout(n.on); n.on = null; }
|
||||
if (n.off) { clearTimeout(n.off); n.off = null; }
|
||||
}
|
||||
var ctx = getCtx();
|
||||
var now = ctx.currentTime;
|
||||
Object.values(_activeOscillators).forEach(function (entry) {
|
||||
try {
|
||||
if (entry.gain) { entry.gain.gain.cancelScheduledValues(now); entry.gain.gain.setValueAtTime(0, now); }
|
||||
if (entry.osc) try { entry.osc.stop(now); } catch (e) {}
|
||||
} catch (e) {}
|
||||
});
|
||||
Object.keys(_activeOscillators).forEach(function (k) { delete _activeOscillators[k]; });
|
||||
_activeNotes = {};
|
||||
},
|
||||
|
||||
saveToIndexedDB: async function (name, arrayBuffer) {
|
||||
if (window.SonicStorage && window.SonicStorage.saveToIndexedDB) {
|
||||
await window.SonicStorage.saveToIndexedDB('soundfont_' + name, arrayBuffer);
|
||||
}
|
||||
},
|
||||
|
||||
loadFromIndexedDB: async function (name) {
|
||||
if (window.SonicStorage && window.SonicStorage.loadFromIndexedDB) {
|
||||
return await window.SonicStorage.loadFromIndexedDB('soundfont_' + name);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
function _startRenderLoop() {
|
||||
if (_renderTimer) return;
|
||||
var Module = _fluidModule;
|
||||
var synth = _synthPtr;
|
||||
var node = _workletNode;
|
||||
var leftPtr = _leftBufPtr;
|
||||
var rightPtr = _rightBufPtr;
|
||||
var block = RENDER_BLOCK;
|
||||
var maxQueue = QUEUE_TARGET;
|
||||
var queueDepth = 0;
|
||||
var lastTick = performance.now();
|
||||
var frameMs = (block / _audioCtx.sampleRate) * 1000;
|
||||
|
||||
// Track consumption by wall-clock time instead of async messages — immune
|
||||
// to message-latency races that could underrun (silence gaps → crackle).
|
||||
function pushFrame() {
|
||||
if (!Module || !synth || !node) return;
|
||||
try {
|
||||
var lpb = leftPtr >> 2;
|
||||
var rpb = rightPtr >> 2;
|
||||
Module.HEAPF32.fill(0, lpb, lpb + block);
|
||||
Module.HEAPF32.fill(0, rpb, rpb + block);
|
||||
Module._fluid_synth_write_float(synth, block, leftPtr, 0, 1, rightPtr, 0, 1);
|
||||
var leftArr = new Float32Array(Module.HEAPF32.subarray(lpb, lpb + block));
|
||||
var rightArr = new Float32Array(Module.HEAPF32.subarray(rpb, rpb + block));
|
||||
node.port.postMessage({ type: 'PCM', L: leftArr, R: rightArr }, [leftArr.buffer, rightArr.buffer]);
|
||||
queueDepth++;
|
||||
} catch (e) { console.warn("[SonicSF] pushFrame error:", e); }
|
||||
}
|
||||
|
||||
function fillLoop() {
|
||||
if (!Module || !synth || !node || !_initialized) {
|
||||
_renderTimer = null;
|
||||
return;
|
||||
}
|
||||
var now = performance.now();
|
||||
queueDepth = Math.max(0, queueDepth - (now - lastTick) / frameMs);
|
||||
lastTick = now;
|
||||
var needed = Math.min(maxQueue - queueDepth, maxQueue);
|
||||
for (var i = 0; i < needed; i++) {
|
||||
pushFrame();
|
||||
}
|
||||
}
|
||||
|
||||
_renderTimer = setInterval(fillLoop, Math.max(4, frameMs * 0.5));
|
||||
}
|
||||
|
||||
function _stopRenderLoop() {
|
||||
if (_renderTimer) {
|
||||
clearInterval(_renderTimer);
|
||||
_renderTimer = null;
|
||||
}
|
||||
}
|
||||
|
||||
function _cleanupFluid() {
|
||||
_stopRenderLoop();
|
||||
if (_leftBufPtr && _fluidModule) { try { _fluidModule._free(_leftBufPtr); } catch (e) {} _leftBufPtr = null; }
|
||||
if (_rightBufPtr && _fluidModule) { try { _fluidModule._free(_rightBufPtr); } catch (e) {} _rightBufPtr = null; }
|
||||
if (_synthPtr && _fluidModule) { try { _fluidModule._delete_fluid_synth(_synthPtr); } catch (e) {} _synthPtr = null; }
|
||||
if (_settingsPtr && _fluidModule) { try { _fluidModule._delete_fluid_settings(_settingsPtr); } catch (e) {} _settingsPtr = null; }
|
||||
_fluidModule = null;
|
||||
_initialized = false;
|
||||
}
|
||||
|
||||
window.SonicSF = SonicSF;
|
||||
})();
|
||||
@@ -0,0 +1,56 @@
|
||||
(function () {
|
||||
'use strict';
|
||||
|
||||
class SoundFontStorage {
|
||||
constructor() {
|
||||
this.dbName = "DAW_SoundFont_Cache";
|
||||
this.storeName = "sf3_buffers";
|
||||
}
|
||||
|
||||
async openDB() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const request = indexedDB.open(this.dbName, 1);
|
||||
request.onupgradeneeded = (e) => {
|
||||
const db = e.target.result;
|
||||
if (!db.objectStoreNames.contains(this.storeName)) {
|
||||
db.createObjectStore(this.storeName);
|
||||
}
|
||||
};
|
||||
request.onsuccess = () => resolve(request.result);
|
||||
request.onerror = () => reject(request.error);
|
||||
});
|
||||
}
|
||||
|
||||
async getBuffer(sfId) {
|
||||
try {
|
||||
const db = await this.openDB();
|
||||
return await new Promise((resolve) => {
|
||||
const tx = db.transaction(this.storeName, "readonly");
|
||||
const store = tx.objectStore(this.storeName);
|
||||
const req = store.get(sfId);
|
||||
req.onsuccess = () => resolve(req.result || null);
|
||||
req.onerror = () => resolve(null);
|
||||
});
|
||||
} catch (e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async saveBuffer(sfId, arrayBuffer) {
|
||||
try {
|
||||
const db = await this.openDB();
|
||||
return await new Promise((resolve, reject) => {
|
||||
const tx = db.transaction(this.storeName, "readwrite");
|
||||
const store = tx.objectStore(this.storeName);
|
||||
const req = store.put(arrayBuffer, sfId);
|
||||
req.onsuccess = () => resolve(true);
|
||||
req.onerror = () => reject(req.error);
|
||||
});
|
||||
} catch (e) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
window.SonicSFStorage = new SoundFontStorage();
|
||||
})();
|
||||
File diff suppressed because one or more lines are too long
@@ -3,12 +3,11 @@
|
||||
const SFS_VERSION = "1.0.0";
|
||||
|
||||
function exportProjectToSFS(projectState) {
|
||||
const sfsBundle = {
|
||||
format: "SONICFORGE_STUDIO_PROJECT",
|
||||
version: SFS_VERSION,
|
||||
timestamp: Date.now(),
|
||||
domain: window.location.origin,
|
||||
project: {
|
||||
let projectObj = {};
|
||||
if (projectState.main_session) {
|
||||
projectObj = projectState;
|
||||
} else {
|
||||
projectObj = {
|
||||
id: projectState.id || `proj_${Date.now()}`,
|
||||
name: projectState.name || "Dự án mới",
|
||||
tracks: (projectState.tracks || []).map(t => ({
|
||||
@@ -22,9 +21,21 @@
|
||||
solo: t.solo,
|
||||
color: t.color,
|
||||
markers: t.markers || [],
|
||||
serverFileId: t.serverFileId || null
|
||||
serverFileId: t.serverFileId || null,
|
||||
clips: t.clips || [],
|
||||
sections: t.sections || [],
|
||||
midiItems: t.midiItems || [],
|
||||
channelInfo: t.channelInfo || null
|
||||
}))
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
const sfsBundle = {
|
||||
format: "SONICFORGE_STUDIO_PROJECT",
|
||||
version: SFS_VERSION,
|
||||
timestamp: Date.now(),
|
||||
domain: window.location.origin,
|
||||
project: projectObj
|
||||
};
|
||||
|
||||
const jsonStr = JSON.stringify(sfsBundle, null, 2);
|
||||
@@ -33,7 +44,8 @@
|
||||
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = `${(projectState.name || 'project').replace(/\s+/g, '_')}.sfs`;
|
||||
const displayName = projectObj.metadata?.title || projectObj.name || 'project';
|
||||
a.download = `${displayName.replace(/\s+/g, '_')}.sfs`;
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
@@ -48,12 +60,14 @@
|
||||
}
|
||||
|
||||
let autoSaveTimer = null;
|
||||
let lastGetProjectStateCallback = null;
|
||||
function scheduleTempAutoSave(getProjectStateCallback) {
|
||||
if (getProjectStateCallback) lastGetProjectStateCallback = getProjectStateCallback;
|
||||
if (autoSaveTimer) clearTimeout(autoSaveTimer);
|
||||
autoSaveTimer = setTimeout(async () => {
|
||||
try {
|
||||
const state = getProjectStateCallback();
|
||||
if (!state || !state.tracks || state.tracks.length === 0) return;
|
||||
const state = lastGetProjectStateCallback ? lastGetProjectStateCallback() : null;
|
||||
if (!state || (!state.tracks && !state.main_session)) return;
|
||||
const dataJson = JSON.stringify(state);
|
||||
localStorage.setItem('sonic_temp_project', dataJson);
|
||||
if (window.SonicAPI && localStorage.getItem('sonic_token')) {
|
||||
@@ -64,10 +78,26 @@
|
||||
}
|
||||
}, 2000);
|
||||
}
|
||||
// Lưu NGAY (bỏ debounce 2s) — dùng cho thay đổi cần bền vững tức thì (đổi màu track)
|
||||
async function flushTempAutoSave() {
|
||||
if (autoSaveTimer) { clearTimeout(autoSaveTimer); autoSaveTimer = null; }
|
||||
try {
|
||||
const state = lastGetProjectStateCallback ? lastGetProjectStateCallback() : null;
|
||||
if (!state || (!state.tracks && !state.main_session)) return;
|
||||
const dataJson = JSON.stringify(state);
|
||||
localStorage.setItem('sonic_temp_project', dataJson);
|
||||
if (window.SonicAPI && localStorage.getItem('sonic_token')) {
|
||||
await window.SonicAPI.saveTempProject(dataJson).catch(() => {});
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn("Flush temp project warning:", e);
|
||||
}
|
||||
}
|
||||
|
||||
window.SonicStorage = {
|
||||
exportProjectToSFS,
|
||||
importProjectFromSFSFile,
|
||||
scheduleTempAutoSave
|
||||
scheduleTempAutoSave,
|
||||
flushTempAutoSave
|
||||
};
|
||||
})();
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
// SonicForge Studio - Unified Undo/Redo Engine
|
||||
// Handles all undoable actions across MAIN SESSION and SECTION-TAB
|
||||
|
||||
const UndoRedoEngine = (function() {
|
||||
const MAX_HISTORY = 50;
|
||||
const history = [];
|
||||
let historyIndex = -1;
|
||||
|
||||
function push(entry) {
|
||||
history.push(entry);
|
||||
if (history.length > MAX_HISTORY) {
|
||||
history.shift();
|
||||
}
|
||||
historyIndex = history.length - 1;
|
||||
}
|
||||
|
||||
function undo() {
|
||||
if (historyIndex < 0) return null;
|
||||
const entry = history[historyIndex];
|
||||
historyIndex--;
|
||||
return entry;
|
||||
}
|
||||
|
||||
function redo() {
|
||||
if (historyIndex >= history.length - 1) return null;
|
||||
historyIndex++;
|
||||
const entry = history[historyIndex];
|
||||
return entry;
|
||||
}
|
||||
|
||||
function canUndo() { return historyIndex >= 0; }
|
||||
function canRedo() { return historyIndex < history.length - 1; }
|
||||
|
||||
function clear() {
|
||||
history.length = 0;
|
||||
historyIndex = -1;
|
||||
}
|
||||
|
||||
function execute(entry) {
|
||||
// entry: { type, scope, label, before, after, undo, redo }
|
||||
// Trim future history if we're not at the end
|
||||
if (historyIndex < history.length - 1) {
|
||||
history.splice(historyIndex + 1);
|
||||
}
|
||||
push(entry);
|
||||
return entry;
|
||||
}
|
||||
|
||||
function getStatus() {
|
||||
return {
|
||||
canUndo: canUndo(),
|
||||
canRedo: canRedo(),
|
||||
undoCount: historyIndex + 1,
|
||||
redoCount: history.length - historyIndex - 1,
|
||||
lastAction: history[historyIndex]?.type || null,
|
||||
lastLabel: history[historyIndex]?.label || null
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
push,
|
||||
undo,
|
||||
redo,
|
||||
canUndo,
|
||||
canRedo,
|
||||
clear,
|
||||
execute,
|
||||
getStatus,
|
||||
history,
|
||||
historyIndex
|
||||
};
|
||||
})();
|
||||
|
||||
window.UndoRedoEngine = UndoRedoEngine;
|
||||
+16
File diff suppressed because one or more lines are too long
Binary file not shown.
@@ -0,0 +1,44 @@
|
||||
class FluidSynthBridge extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.leftQ = [];
|
||||
this.rightQ = [];
|
||||
this.called = 0;
|
||||
this.port.onmessage = (e) => {
|
||||
const d = e.data;
|
||||
if (d.type === 'PCM') {
|
||||
this.leftQ.push(d.L);
|
||||
this.rightQ.push(d.R);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
process(inputs, outputs) {
|
||||
const out = outputs[0];
|
||||
if (!out || out.length === 0) return true;
|
||||
this.called++;
|
||||
const numCh = out.length;
|
||||
const len = out[0].length;
|
||||
const qL = this.leftQ;
|
||||
const qR = this.rightQ;
|
||||
let fi = 0;
|
||||
let si = 0;
|
||||
// Handle any output channel count (mono devices produce 1 channel, so
|
||||
// out[1] may be undefined — never write into a missing channel).
|
||||
for (let i = 0; i < len; i++) {
|
||||
if (fi >= qL.length) {
|
||||
for (let c = 0; c < numCh; c++) out[c][i] = 0;
|
||||
continue;
|
||||
}
|
||||
for (let c = 0; c < numCh; c++) {
|
||||
out[c][i] = c % 2 === 0 ? qL[fi][si] : qR[fi][si];
|
||||
}
|
||||
si++;
|
||||
if (si >= qL[fi].length) { fi++; si = 0; }
|
||||
}
|
||||
if (fi > 0) { this.leftQ.splice(0, fi); this.rightQ.splice(0, fi); }
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor('fluidsynth-bridge', FluidSynthBridge);
|
||||
@@ -0,0 +1,31 @@
|
||||
class PCMRecorderProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.bufferSize = 4096;
|
||||
this.buffer = new Float32Array(this.bufferSize);
|
||||
this.bufferIndex = 0;
|
||||
}
|
||||
|
||||
process(inputs, outputs, parameters) {
|
||||
const input = inputs[0];
|
||||
if (input && input.length > 0) {
|
||||
const inputChannel = input[0]; // Mono Channel 0
|
||||
|
||||
for (let i = 0; i < inputChannel.length; i++) {
|
||||
this.buffer[this.bufferIndex++] = inputChannel[i];
|
||||
|
||||
// When Ring-Buffer fills, send Float32Array to Main Thread
|
||||
if (this.bufferIndex >= this.bufferSize) {
|
||||
this.port.postMessage({
|
||||
type: 'PCM_DATA',
|
||||
buffer: this.buffer.slice(0, this.bufferSize)
|
||||
});
|
||||
this.bufferIndex = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
return true; // Keep worklet active
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor('pcm-recorder-processor', PCMRecorderProcessor);
|
||||
@@ -0,0 +1 @@
|
||||
8d26e2b55e73579d1bb3c37b4878f1845ef9cbf50a8e4ee6f7deaa2ab80db32d
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
{"original_name": "weedsgm3.sf2", "uuid": "518e850f-a5d3-4790-b1f9-0c90c203c524", "file": "518e850f-a5d3-4790-b1f9-0c90c203c524.sf2"}
|
||||
Binary file not shown.
@@ -303,3 +303,27 @@ def cleanup_expired_files_task(max_age_hours: int = 24):
|
||||
"cleaned_size_mb": round(cleaned_size / (1024 * 1024), 2),
|
||||
"max_age_hours": max_age_hours
|
||||
}
|
||||
|
||||
|
||||
@celery_app.task
|
||||
def render_project_task(project_id: str, project_name: str, project_json_str: str, sample_rate: int = 44100):
|
||||
"""
|
||||
Task Celery để kết xuất dự án ngoại tuyến (Offline Project Mixdown) áp dụng specs 30_DAW_ARCHITECT.md.
|
||||
"""
|
||||
import json
|
||||
from app.core.render_engine import PythonRenderEngine
|
||||
|
||||
project_json = json.loads(project_json_str)
|
||||
output_filename = f"{project_id}_render.wav"
|
||||
output_path = os.path.join(settings.PROCESSED_DIR, output_filename)
|
||||
|
||||
engine = PythonRenderEngine(sample_rate=sample_rate)
|
||||
engine.render_project(project_json, output_path)
|
||||
|
||||
return {
|
||||
"project_id": project_id,
|
||||
"project_name": project_name,
|
||||
"success": True,
|
||||
"output_file_id": output_filename
|
||||
}
|
||||
|
||||
|
||||
+388
-21
@@ -1,5 +1,6 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="vi">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
@@ -7,44 +8,410 @@
|
||||
<link rel="icon" type="image/svg+xml" href="/favicon.svg">
|
||||
<script src="https://cdn.tailwindcss.com"></script>
|
||||
<script src="https://unpkg.com/lucide@latest"></script>
|
||||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.2/css/all.min.css" crossorigin="anonymous" referrerpolicy="no-referrer">
|
||||
<script src="https://unpkg.com/react@18.3.1/umd/react.production.min.js"></script>
|
||||
<script src="https://unpkg.com/react-dom@18.3.1/umd/react-dom.production.min.js"></script>
|
||||
<script src="/static/js/services/api.js"></script>
|
||||
<script src="/static/js/services/audioEngine.js"></script>
|
||||
<script src="/static/js/services/storage.js"></script>
|
||||
<script src="/static/js/app.precompiled.js" defer></script>
|
||||
<script src="/static/js/services/fluidsynthLoader.js?v=202607271245"></script>
|
||||
<script src="/static/js/services/api.js?v=202607271016"></script>
|
||||
<script src="/static/js/services/audioEngine.js?v=202607271016"></script>
|
||||
<script src="/static/js/services/storage.js?v=202608038200"></script>
|
||||
<script src="/static/js/services/soundfontStorage.js?v=202607271016"></script>
|
||||
<script src="/static/js/services/soundfontPlayer.js?v=202608060630"></script>
|
||||
<script src="/static/js/services/aiGateway.js?v=202608037200"></script>
|
||||
<script src="/static/js/services/dawCommandDispatcher.js?v=202607271016"></script>
|
||||
<script src="/static/js/services/pianoRollTabService.js?v=202607272044"></script>
|
||||
<script src="/static/js/services/ghostNoteExtractor.js?v=202607271727"></script>
|
||||
<script src="/static/js/services/midiExtractor.js?v=202607281052"></script>
|
||||
<script src="/static/js/services/promptTemplateManager.js?v=202607281039"></script>
|
||||
<script src="/static/js/services/undoRedoEngine.js?v=202607290941"></script>
|
||||
<script src="/static/js/app.precompiled.js?v=202608061030" defer></script>
|
||||
<link rel="stylesheet" href="/static/css/styles.css?v=202607271016">
|
||||
<style>
|
||||
body {
|
||||
background-color: #1a1a1a;
|
||||
color: #c0c0c0;
|
||||
:root {
|
||||
--right-sidebar-width: 320px;
|
||||
--bottom-strip-height: 220px;
|
||||
--top-bar-height: 80px;
|
||||
--status-bar-height: 25px;
|
||||
--panel-border-color: #2a2a2a;
|
||||
}
|
||||
|
||||
body {
|
||||
background-color: #1a1a1a;
|
||||
color: #c0c0c0;
|
||||
font-family: 'Inter', system-ui, -apple-system, sans-serif;
|
||||
overflow: hidden;
|
||||
user-select: none;
|
||||
}
|
||||
.daw-bg { background-color: #1e1e1e; }
|
||||
.daw-panel { background-color: #262626; }
|
||||
.daw-header { background-color: #2e2e2e; }
|
||||
.daw-border { border-color: #181818; }
|
||||
.daw-track-active { background-color: #333333; }
|
||||
::-webkit-scrollbar { width: 10px; height: 10px; }
|
||||
::-webkit-scrollbar-track { background: #141414; }
|
||||
::-webkit-scrollbar-thumb { background: #3a3a3a; border: 2px solid #141414; border-radius: 4px; }
|
||||
::-webkit-scrollbar-thumb:hover { background: #4a4a4a; }
|
||||
.knob-container { position: relative; width: 28px; height: 28px; }
|
||||
.knob-dial { transform-origin: center; transition: transform 0.1s ease; }
|
||||
|
||||
.daw-bg {
|
||||
background-color: #1e1e1e;
|
||||
}
|
||||
|
||||
.daw-panel {
|
||||
background-color: #262626;
|
||||
}
|
||||
|
||||
.daw-header {
|
||||
background-color: #2e2e2e;
|
||||
}
|
||||
|
||||
.daw-border {
|
||||
border-color: #181818;
|
||||
}
|
||||
|
||||
.daw-track-active {
|
||||
background-color: #333333;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar {
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-track {
|
||||
background: #141414;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-thumb {
|
||||
background: #3a3a3a;
|
||||
border: 2px solid #141414;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-thumb:hover {
|
||||
background: #4a4a4a;
|
||||
}
|
||||
|
||||
.knob-container {
|
||||
position: relative;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
}
|
||||
|
||||
.knob-dial {
|
||||
transform-origin: center;
|
||||
transition: transform 0.1s ease;
|
||||
}
|
||||
|
||||
.selection-interactive-box {
|
||||
min-width: 4px;
|
||||
}
|
||||
|
||||
.no-scrollbar {
|
||||
scrollbar-width: none; /* Firefox */
|
||||
-ms-overflow-style: none; /* IE 10+ */
|
||||
scrollbar-width: none;
|
||||
/* Firefox */
|
||||
-ms-overflow-style: none;
|
||||
/* IE 10+ */
|
||||
}
|
||||
|
||||
.no-scrollbar::-webkit-scrollbar {
|
||||
display: none; /* Safari and Chrome */
|
||||
display: none;
|
||||
/* Safari and Chrome */
|
||||
}
|
||||
|
||||
/* Fullscreen Fixed App Shell */
|
||||
.daw-app-shell {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
width: 100vw;
|
||||
height: 100vh;
|
||||
overflow: hidden;
|
||||
background-color: #121212;
|
||||
color: #e0e0e0;
|
||||
}
|
||||
|
||||
/* Middle Section holding Main Workspace and Right Sidebar */
|
||||
.daw-body-container {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
height: calc(100vh - var(--top-bar-height) - var(--bottom-strip-height) - var(--status-bar-height));
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* Auto-expanding Main Workspace */
|
||||
.daw-main-workspace {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* Right Sidebar with Width controlled via CSS Variable */
|
||||
.daw-right-sidebar {
|
||||
width: var(--right-sidebar-width);
|
||||
min-width: 200px;
|
||||
max-width: 600px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
background-color: #1a1a1a;
|
||||
border-left: 1px solid var(--panel-border-color);
|
||||
}
|
||||
|
||||
/* Vertically stacked child Panels in Right Sidebar */
|
||||
.sidebar-panel {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
background: #1e1e1e;
|
||||
border-bottom: 1px solid var(--panel-border-color);
|
||||
}
|
||||
|
||||
#panel-media-explorer {
|
||||
height: 50%;
|
||||
/* Default 50/50 split */
|
||||
min-height: 100px;
|
||||
}
|
||||
|
||||
#panel-ai {
|
||||
flex: 1;
|
||||
/* Fills remaining height */
|
||||
min-height: 100px;
|
||||
}
|
||||
|
||||
/* BOTTOM ROW: Enables Horizontal Scrolling */
|
||||
.daw-bottom-strip {
|
||||
height: var(--bottom-strip-height);
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 8px;
|
||||
overflow-x: auto;
|
||||
/* Enables horizontal scroll when panels overflow */
|
||||
overflow-y: hidden;
|
||||
background-color: #161616;
|
||||
border-top: 1px solid var(--panel-border-color);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* Optimized custom horizontal scrollbar for DAW styling */
|
||||
.daw-bottom-strip::-webkit-scrollbar {
|
||||
height: 8px;
|
||||
}
|
||||
|
||||
.daw-bottom-strip::-webkit-scrollbar-thumb {
|
||||
background: #3a3a3a;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.daw-bottom-strip::-webkit-scrollbar-thumb:hover {
|
||||
background: #00ffcc;
|
||||
}
|
||||
|
||||
/* Sub-panels inside the bottom strip */
|
||||
.bottom-panel {
|
||||
flex: 0 0 auto;
|
||||
/* Prevents shrinking, locks content dimensions */
|
||||
width: 320px;
|
||||
height: 100%;
|
||||
background-color: #222;
|
||||
border: 1px solid #333;
|
||||
border-radius: 6px;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
/* RESIZER HANDLES */
|
||||
.resizer-col-handle {
|
||||
width: 5px;
|
||||
cursor: ew-resize;
|
||||
/* Horizontal resize cursor */
|
||||
background: transparent;
|
||||
transition: background 0.2s;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
.resizer-col-handle:hover,
|
||||
.resizer-col-handle:active {
|
||||
background: #00ffcc;
|
||||
}
|
||||
|
||||
.resizer-row-handle {
|
||||
height: 5px;
|
||||
cursor: ns-resize;
|
||||
/* Vertical resize cursor */
|
||||
background: transparent;
|
||||
transition: background 0.2s;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
.resizer-row-handle:hover,
|
||||
.resizer-row-handle:active {
|
||||
background: #00ffcc;
|
||||
}
|
||||
|
||||
/* Disable mouse pointer events on SVG icons to prevent SVGAnimatedString className type errors in event listeners */
|
||||
svg, i[data-lucide] {
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
/* Ozone Mastering Module Styles */
|
||||
.oz-panel {
|
||||
background: linear-gradient(180deg, #131822 0%, #0d1118 100%);
|
||||
border: 1px solid rgba(255, 255, 255, 0.07);
|
||||
}
|
||||
.oz-card {
|
||||
background: #18202c;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
transition: all 0.15s ease;
|
||||
}
|
||||
.oz-card-active {
|
||||
background: #1e2a3a;
|
||||
border-color: #38bdf8;
|
||||
box-shadow: 0 0 12px rgba(56, 189, 248, 0.2);
|
||||
}
|
||||
.knob-container { touch-action: none; user-select: none; }
|
||||
.knob-dial {
|
||||
transition: transform 0.05s ease-out;
|
||||
box-shadow: inset 0 1px 3px rgba(255,255,255,0.15), 0 3px 8px rgba(0,0,0,0.8);
|
||||
}
|
||||
.oz-scrollbar::-webkit-scrollbar { width: 4px; height: 4px; }
|
||||
.oz-scrollbar::-webkit-scrollbar-track { background: #080b12; }
|
||||
.oz-scrollbar::-webkit-scrollbar-thumb { background: #1e293b; border-radius: 2px; }
|
||||
.oz-scrollbar::-webkit-scrollbar-thumb:hover { background: #334155; }
|
||||
.oz-font-mono { font-family: 'JetBrains Mono', monospace; }
|
||||
|
||||
/* Master & Track Strip Console Styles (from md/47, md/48) */
|
||||
.strip-bg {
|
||||
background: linear-gradient(180deg, #2f2f2f 0%, #222222 100%);
|
||||
border: 1px solid #141414;
|
||||
box-shadow: inset 1px 1px 0 rgba(255,255,255,0.08), 0 8px 24px rgba(0,0,0,0.8);
|
||||
}
|
||||
.strip-bg-track {
|
||||
background: linear-gradient(180deg, #2c2c2c 0%, #1e1e1e 100%);
|
||||
border: 1px solid #111111;
|
||||
box-shadow: inset 1px 1px 0 rgba(255,255,255,0.08), 0 10px 30px rgba(0,0,0,0.85);
|
||||
}
|
||||
|
||||
.fader-track {
|
||||
background: #121212;
|
||||
box-shadow: inset 1px 1px 3px rgba(0,0,0,0.9), 1px 1px 0 rgba(255,255,255,0.05);
|
||||
}
|
||||
.fader-track-bg {
|
||||
background: #101010;
|
||||
box-shadow: inset 1px 1px 3px rgba(0,0,0,0.9), 1px 1px 0 rgba(255,255,255,0.05);
|
||||
}
|
||||
|
||||
/* DAW Metallic Button */
|
||||
.btn-daw {
|
||||
background: linear-gradient(180deg, #444 0%, #2a2a2a 100%);
|
||||
border: 1px solid #1a1a1a;
|
||||
box-shadow: inset 0 1px 0 rgba(255,255,255,0.15), 0 2px 4px rgba(0,0,0,0.5);
|
||||
transition: all 0.1s ease;
|
||||
}
|
||||
.btn-daw:hover {
|
||||
background: linear-gradient(180deg, #555 0%, #333 100%);
|
||||
}
|
||||
.btn-daw:active {
|
||||
box-shadow: inset 0 2px 4px rgba(0,0,0,0.8);
|
||||
transform: translateY(0.5px);
|
||||
}
|
||||
|
||||
/* Teal Active State (FX & Power Button) */
|
||||
.btn-teal-active {
|
||||
background: linear-gradient(180deg, #10b981 0%, #047857 100%) !important;
|
||||
border-color: #34d399 !important;
|
||||
color: #ffffff !important;
|
||||
box-shadow: 0 0 10px rgba(16, 185, 129, 0.6) !important;
|
||||
}
|
||||
|
||||
/* Mute Active State */
|
||||
.btn-mute-active {
|
||||
background: linear-gradient(180deg, #ef4444 0%, #991b1b 100%) !important;
|
||||
border-color: #f87171 !important;
|
||||
color: #ffffff !important;
|
||||
box-shadow: 0 0 8px rgba(239, 68, 68, 0.6) !important;
|
||||
}
|
||||
|
||||
/* Solo Active State */
|
||||
.btn-solo-active {
|
||||
background: linear-gradient(180deg, #eab308 0%, #854d0e 100%) !important;
|
||||
border-color: #fde047 !important;
|
||||
color: #000000 !important;
|
||||
box-shadow: 0 0 8px rgba(234, 179, 8, 0.6) !important;
|
||||
}
|
||||
|
||||
/* Mono Active State */
|
||||
.btn-mono-active {
|
||||
background: linear-gradient(180deg, #3b82f6 0%, #1e40af 100%) !important;
|
||||
border-color: #60a5fa !important;
|
||||
color: #ffffff !important;
|
||||
}
|
||||
|
||||
/* Record Arm Active State */
|
||||
.btn-arm-active {
|
||||
background: radial-gradient(circle, #ef4444 0%, #7f1d1d 100%) !important;
|
||||
border-color: #fca5a5 !important;
|
||||
box-shadow: 0 0 12px rgba(239, 68, 68, 0.9), inset 0 1px 2px rgba(255,255,255,0.8) !important;
|
||||
animation: pulse-red 1.2s infinite alternate;
|
||||
}
|
||||
|
||||
@keyframes pulse-red {
|
||||
0% { box-shadow: 0 0 6px rgba(239, 68, 68, 0.6); }
|
||||
100% { box-shadow: 0 0 16px rgba(239, 68, 68, 1); }
|
||||
}
|
||||
|
||||
/* Custom Vertical Range Fader Slider */
|
||||
input[type=range].fader-slider {
|
||||
-webkit-appearance: none;
|
||||
appearance: none;
|
||||
writing-mode: vertical-lr;
|
||||
direction: rtl;
|
||||
background: transparent;
|
||||
cursor: pointer;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
flex: 1;
|
||||
min-height: 20px;
|
||||
width: 100%;
|
||||
}
|
||||
input[type=range].fader-slider::-webkit-slider-thumb {
|
||||
-webkit-appearance: none;
|
||||
height: 24px;
|
||||
width: 22px;
|
||||
background: linear-gradient(180deg, #e2e8f0 0%, #64748b 45%, #1e293b 50%, #94a3b8 100%);
|
||||
border: 1px solid #000000;
|
||||
border-radius: 2px;
|
||||
box-shadow: 0 4px 8px rgba(0,0,0,0.9), inset 0 1px 1px rgba(255,255,255,0.9);
|
||||
}
|
||||
input[type=range].fader-slider::-moz-range-thumb {
|
||||
height: 24px;
|
||||
width: 22px;
|
||||
background: linear-gradient(180deg, #e2e8f0 0%, #64748b 45%, #1e293b 50%, #94a3b8 100%);
|
||||
border: 1px solid #000000;
|
||||
border-radius: 2px;
|
||||
box-shadow: 0 4px 8px rgba(0,0,0,0.9), inset 0 1px 1px rgba(255,255,255,0.9);
|
||||
}
|
||||
|
||||
/* Media Explorer: horizontal volume slider + selected file row */
|
||||
input[type=range].me-fader-slider {
|
||||
-webkit-appearance: none;
|
||||
appearance: none;
|
||||
background: #111;
|
||||
height: 6px;
|
||||
border-radius: 3px;
|
||||
border: 1px solid #555;
|
||||
cursor: pointer;
|
||||
}
|
||||
input[type=range].me-fader-slider::-webkit-slider-thumb {
|
||||
-webkit-appearance: none;
|
||||
height: 16px;
|
||||
width: 12px;
|
||||
background: linear-gradient(180deg, #e2e8f0 0%, #64748b 50%, #1e293b 100%);
|
||||
border: 1px solid #000;
|
||||
border-radius: 2px;
|
||||
box-shadow: 0 2px 4px rgba(0,0,0,0.5);
|
||||
cursor: pointer;
|
||||
}
|
||||
.file-row-selected {
|
||||
background-color: #3399ff !important;
|
||||
color: #ffffff !important;
|
||||
}
|
||||
.file-row-selected .file-icon { color: #ffffff !important; }
|
||||
</style>
|
||||
</head>
|
||||
|
||||
<body class="h-screen w-screen flex flex-col">
|
||||
<div id="root" class="h-full w-full flex flex-col"></div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
@@ -0,0 +1,20 @@
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, "/app")
|
||||
|
||||
try:
|
||||
import fluidsynth
|
||||
print("fluidsynth import successful.")
|
||||
|
||||
fl = fluidsynth.Synth()
|
||||
# Try to load a pre-existing system SF3
|
||||
sf3_path = "/opt/daw_engine/soundfonts/Equinox_Grand_Pianos.sf3"
|
||||
print(f"Checking if {sf3_path} exists: {os.path.exists(sf3_path)}")
|
||||
if os.path.exists(sf3_path):
|
||||
h = fl.sfload(sf3_path)
|
||||
print(f"Loaded {sf3_path}, handle: {h}")
|
||||
else:
|
||||
print("Equinox_Grand_Pianos.sf3 not found.")
|
||||
except Exception as e:
|
||||
print(f"Failed: {e}")
|
||||
@@ -0,0 +1,66 @@
|
||||
import os
|
||||
import sys
|
||||
import logging
|
||||
|
||||
# Ensure app is in path
|
||||
sys.path.insert(0, "/app")
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger("test_sf_convert")
|
||||
|
||||
from app.core.soundfont_converter import SoundFontConverter
|
||||
|
||||
def test():
|
||||
sf2_dir = "/app/app/storage/soundfonts"
|
||||
sf2_files = [os.path.join(sf2_dir, f) for f in os.listdir(sf2_dir) if f.endswith(".sf2") and "_decomp" not in f]
|
||||
if not sf2_files:
|
||||
logger.error("No SF2 files found in /app/app/storage/soundfonts")
|
||||
return
|
||||
|
||||
sf2_path = sf2_files[0]
|
||||
logger.info(f"Testing with SF2 file: {sf2_path}")
|
||||
|
||||
converter = SoundFontConverter()
|
||||
|
||||
# Check ffmpeg encoder support
|
||||
has_ogg = converter._check_ffmpeg_ogg()
|
||||
logger.info(f"ffmpeg with libvorbis available: {has_ogg}")
|
||||
|
||||
# Convert SF2 -> SF3
|
||||
sf3_path = sf2_path.replace(".sf2", ".sf3")
|
||||
if os.path.exists(sf3_path):
|
||||
os.remove(sf3_path)
|
||||
|
||||
logger.info("Converting SF2 -> SF3...")
|
||||
result_path = converter.convert_sf2_to_sf3(sf2_path)
|
||||
logger.info(f"Result path from convert_sf2_to_sf3: {result_path}")
|
||||
|
||||
if result_path.endswith(".sf3"):
|
||||
logger.info(f"SF3 file exists: {os.path.exists(sf3_path)}")
|
||||
if os.path.exists(sf3_path):
|
||||
logger.info(f"SF3 size: {os.path.getsize(sf3_path)} bytes")
|
||||
# Verify if it plays audio
|
||||
plays = converter._sf3_plays_audio(sf3_path)
|
||||
logger.info(f"SF3 plays audio (pyfluidsynth verify): {plays}")
|
||||
|
||||
# Now test decompression back to SF2
|
||||
decomp_sf2 = sf3_path.replace(".sf3", "_decomp.sf2")
|
||||
if os.path.exists(decomp_sf2):
|
||||
os.remove(decomp_sf2)
|
||||
|
||||
logger.info("Decompressing SF3 -> SF2...")
|
||||
try:
|
||||
decomp_result = converter.sf3_to_sf2(sf3_path, decomp_sf2)
|
||||
logger.info(f"Decompress result path: {decomp_result}")
|
||||
if os.path.exists(decomp_sf2):
|
||||
logger.info(f"Decompressed SF2 size: {os.path.getsize(decomp_sf2)} bytes")
|
||||
# Check if it plays
|
||||
decomp_plays = converter._sf3_plays_audio(decomp_sf2)
|
||||
logger.info(f"Decompressed SF2 plays audio: {decomp_plays}")
|
||||
except Exception as e:
|
||||
logger.error(f"Decompression failed: {e}", exc_info=True)
|
||||
else:
|
||||
logger.warning("Conversion did not produce an SF3 path.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test()
|
||||
@@ -0,0 +1,14 @@
|
||||
// Rebuild app.precompiled.js from app.jsx using @babel/standalone (avoids the
|
||||
// Babel 8 ESM-only CLI conflict). Mirrors package.json's build script:
|
||||
// babel app/static/js/app.jsx --config-file ./babel.config.json -o app/static/js/app.precompiled.js
|
||||
import * as Babel from '@babel/standalone';
|
||||
import { readFileSync, writeFileSync } from 'fs';
|
||||
|
||||
const src = readFileSync('app/static/js/app.jsx', 'utf8');
|
||||
const out = Babel.transform(src, {
|
||||
presets: ['react'],
|
||||
filename: 'app.jsx',
|
||||
sourceType: 'script',
|
||||
}).code;
|
||||
writeFileSync('app/static/js/app.precompiled.js', out);
|
||||
console.log('BUILD OK', out.length, 'bytes');
|
||||
@@ -10,6 +10,9 @@ services:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- .:/app
|
||||
- /home/locpham/daw_assets/vst3:/opt/daw_engine/vst3
|
||||
- /home/locpham/daw_assets/soundfonts:/opt/daw_engine/soundfonts
|
||||
- /home/locpham/daw_assets/pianobook:/opt/daw_engine/samples/pianobook
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379/0
|
||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
||||
@@ -22,6 +25,9 @@ services:
|
||||
command: celery -A app.tasks.worker.celery_app worker --loglevel=info
|
||||
volumes:
|
||||
- .:/app
|
||||
- /home/locpham/daw_assets/vst3:/opt/daw_engine/vst3
|
||||
- /home/locpham/daw_assets/soundfonts:/opt/daw_engine/soundfonts
|
||||
- /home/locpham/daw_assets/pianobook:/opt/daw_engine/samples/pianobook
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379/0
|
||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
||||
@@ -34,6 +40,9 @@ services:
|
||||
command: celery -A app.tasks.worker.celery_app beat --loglevel=info
|
||||
volumes:
|
||||
- .:/app
|
||||
- /home/locpham/daw_assets/vst3:/opt/daw_engine/vst3
|
||||
- /home/locpham/daw_assets/soundfonts:/opt/daw_engine/soundfonts
|
||||
- /home/locpham/daw_assets/pianobook:/opt/daw_engine/samples/pianobook
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379/0
|
||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Kế hoạch Thực hiện: Tính năng DAW mới (Track FX, Section Preview & Piano Roll Edit Shortcuts)
|
||||
|
||||
Bản kế hoạch này mô tả thiết kế kỹ thuật và các bước thực hiện các tính năng nâng cao liên quan đến xử lý hiệu ứng âm thanh (FX Chain) song song giữa Web Audio Client và Python Backend, kết xuất trực quan Section item trên Timeline, và bổ sung các phím tắt biên tập thông minh trong Piano Roll.
|
||||
|
||||
## Điểm lưu ý từ Người dùng
|
||||
1. **Kiến trúc hiệu ứng Song song (Dual FX Architecture)**:
|
||||
- **Phía Client (Trình duyệt)**: Vẫn phải sử dụng Web Audio API để xử lý hiệu ứng thời gian thực (Real-time Preview) khi nhấn Play nghe thử.
|
||||
- **Phía Server (Backend)**: Sử dụng các thư viện Python (`Pedalboard`, `pydub`, `pysndfx`, `Librosa`...) để xử lý hiệu ứng khi xuất bản kết xuất (Offline Export / Render).
|
||||
2. **Timeline Playback**: Khi Playhead trên Main Session quét qua Section item, trình phát nhạc sẽ phát nội dung của Section item đó giống như đang nghe thử trên Section-tab.
|
||||
3. **Sửa lỗi Release Note**: Khi click chọn nốt mới để kéo thả, các nốt cũ đã chọn trước đó phải được giải phóng hoàn toàn và không di chuyển theo nốt mới.
|
||||
|
||||
---
|
||||
|
||||
## Phân tích Ưu/Nhược điểm & Giải pháp Tối ưu hóa (Dual FX)
|
||||
|
||||
### 1. Ưu điểm (Pros)
|
||||
* **Trải nghiệm Tức thì (Zero Latency)**: Người dùng nghe thấy hiệu ứng ngay lập tức khi kéo nút hoặc đổi chế độ mà không cần đợi gửi file lên server xử lý.
|
||||
* **Giảm tải cực lớn cho Server**: Trình duyệt tự gánh vác phần giải mã và xử lý DSP thời gian thực trên thiết bị người dùng, máy chủ chỉ cần chạy xử lý khi người dùng xuất bản (Export).
|
||||
* **Chất lượng Master tuyệt đối**: Bản xuất ra (Render) trên Backend sử dụng thuật toán C++ tối ưu cao của `pedalboard` hoặc các bộ lọc khoa học của `scipy` giúp chất lượng âm thanh đạt chuẩn phòng thu, không bị méo tiếng do giới hạn tài nguyên của trình duyệt.
|
||||
|
||||
### 2. Nhược điểm (Cons)
|
||||
* **Độ lệch âm thanh (Sound Discrepancy)**: Bản nghe thử trên trình duyệt (Web Audio API) và bản xuất ra (Python Backend) có thể có khác biệt nhỏ về màu sắc âm thanh nếu thuật toán tạo Chorus/Reverb khác nhau.
|
||||
* **Trùng lặp mã nguồn (Code Duplication)**: Phải viết logic xử lý hiệu ứng ở cả 2 ngôn ngữ (JavaScript trên Client và Python trên Server).
|
||||
|
||||
### 3. Giải pháp Tối ưu hóa tốt nhất (Optimal Strategies)
|
||||
Để tối ưu hóa hiệu suất và chất lượng, chúng ta áp dụng các giải pháp sau:
|
||||
* **Đồng bộ hóa tham số (Unified Parameters)**: Cả Client và Server đều đọc chung các tham số hiệu ứng từ tệp JSON cấu hình dự án (ví dụ: `chorus_rate = 1.5 Hz`, `reverb_room_size = 0.5`).
|
||||
* **Đồng hóa thuật toán (DSP Matching)**:
|
||||
* *Chorus*: Cài đặt LFO sine điều tần thời gian trễ đồng bộ ở cả 2 bên.
|
||||
* *Reverb*:
|
||||
* **Client**: Sử dụng `ConvolverNode` nạp bộ đệm xung tự sinh (synthetic impulse response) dài 2 giây từ thuật toán nhiễu trắng phân rã lũy thừa.
|
||||
* **Server**: Sử dụng `pedalboard.Reverb` hoặc thuật toán chập tín hiệu phân rã tương đương trong Python.
|
||||
* **Tối ưu hóa tài nguyên phía Server**:
|
||||
* Không nạp lại SoundFont hoặc thư viện nhiều lần; sử dụng cơ chế singleton hoặc caching cho các module xử lý hiệu ứng của Python.
|
||||
|
||||
---
|
||||
|
||||
## Đề xuất Thay đổi cụ thể
|
||||
|
||||
### 1. Kiến trúc hiệu ứng Track FX (Chorus & Reverb)
|
||||
|
||||
#### Phía Client (Web Audio API)
|
||||
Chúng ta sẽ bổ sung chuỗi liên kết hiệu ứng âm thanh trực tiếp vào nút nguồn của từng track trong [app.jsx](file:///home/locpham/SonicForgeStudio/app/static/js/app.jsx). Bất cứ nguồn âm thanh nào đi qua track (cả clip tiếng động lẫn âm thanh MIDI qua SoundFont) đều chịu tác dụng của hiệu ứng:
|
||||
- **Chorus**: Sử dụng một dry gain và một wet gain kết nối tới LFO-modulated delay node. LFO dao động hình sin tần số `1.5 Hz` với độ lệch delay `2 ms` tạo tiếng đồng ca.
|
||||
- **Reverb**: Sử dụng dry gain, wet gain và `ConvolverNode` chứa bộ đệm phản hồi âm thanh nhân tạo tự sinh dài 2 giây từ thuật toán nhiễu trắng phân rã mũ.
|
||||
- **Giao diện Chọn hiệu ứng**: Khi người dùng nhấn nút **FX: None** trên Track Header, một popup selector sẽ hiển thị để người dùng chọn: **None / Chorus / Reverb**, cập nhật thuộc tính `track.fxType`.
|
||||
|
||||
#### Phía Server (Python Rendering)
|
||||
Khi kết xuất dự án trong [render_engine.py](file:///home/locpham/SonicForgeStudio/app/core/render_engine.py), chúng ta đọc thuộc tính `fx_type` từ track:
|
||||
- Nếu `track.fx_type == 'chorus'`: Sử dụng lớp `Chorus` của `pedalboard` (nếu có thư viện), hoặc dùng `scipy.signal` để tạo dịch pha điều tần chậm.
|
||||
- Nếu `track.fx_type == 'reverb'`: Sử dụng lớp `Reverb` của `pedalboard`, hoặc sử dụng chập phản hồi âm để tạo vang.
|
||||
|
||||
---
|
||||
|
||||
### 2. Vẽ lại trực quan Section item (Waveform nền & Note màu)
|
||||
|
||||
Cập nhật thuật toán vẽ Section item trên Canvas Timeline trong [app.jsx](file:///home/locpham/SonicForgeStudio/app/static/js/app.jsx):
|
||||
- **Phông nền Waveform**: Duyệt qua tối đa 4 track con trong `sec.tracks`. Nếu track con chứa audio clip có `buffer` dữ liệu, lấy mẫu thu gọn 100 peaks và vẽ thành đồ thị cột phân rã đối xứng màu xanh cyan làm nền bên trong phân vùng của Section.
|
||||
- **Vẽ note MIDI theo màu sắc**: Quét các MIDI item trong các track con. Vẽ các note nhạc đè lên waveform với màu sắc tương trưng cho thứ tự track con (ví dụ: track 1 dùng màu vàng hổ phách `#fbbf24`, track 2 dùng màu tím `#a78bfa`, v.v.), giúp người dùng nhận diện nhanh cấu trúc hòa âm.
|
||||
- **Phát Section Item trên Main Session**: Đảm bảo thuật toán lập lịch phát âm thanh trong `startTrackPlayback` quét qua toàn bộ clip và nốt nhạc con của Section item để lên lịch phát nhạc đồng bộ khi playhead quét qua.
|
||||
|
||||
---
|
||||
|
||||
### 3. Phím tắt Nâng cao trong Piano Roll
|
||||
|
||||
Chúng ta sẽ chỉnh sửa các hàm xử lý sự kiện chuột của canvas Piano Roll trong [app.jsx](file:///home/locpham/SonicForgeStudio/app/static/js/app.jsx):
|
||||
|
||||
#### Giải phóng nốt cũ khi chọn nốt mới (Sửa lỗi Drag & Release)
|
||||
- Trong `handleGridMouseDown`, khi người dùng click vào một nốt mới nằm ngoài vùng chọn hiện tại, chúng ta sẽ cập nhật danh sách chọn lựa bằng một biến cục bộ đồng bộ `nextSelectedIds = [clickedNote.id]` thay vì sử dụng state bất đồng bộ `selectedNoteIds`. Điều này đảm bảo danh sách nốt bị dịch chuyển (`selectedNotesOffset`) chỉ chứa duy nhất nốt mới click, giải phóng hoàn toàn nốt cũ.
|
||||
|
||||
#### Sao chép nốt nhanh (Ctrl + Click + Drag)
|
||||
- Trong `handleGridMouseDown`, nếu nhấn chuột trái đồng thời đè phím `Ctrl` trên một nốt nhạc:
|
||||
- Tạo các bản sao nhân bản (cloned copies) của tất cả các nốt đang được chọn với ID ngẫu nhiên mới.
|
||||
- Thêm các nốt nhân bản này vào danh sách `notes` của tab.
|
||||
- Chuyển trạng thái `draggedNote` sang chế độ di chuyển (`move`) áp dụng trực tiếp lên các bản sao mới này, giữ nguyên các nốt gốc ở vị trí cũ.
|
||||
|
||||
#### Giãn/Thu tỷ lệ thời gian các nốt (Alt + Drag Resize)
|
||||
- Khi người dùng nhấn giữ phím `Alt` và kéo cạnh phải (resize edge) của một nốt trong nhóm đang được chọn:
|
||||
- Xác định thời điểm bắt đầu của nốt đầu tiên trong nhóm tuyển chọn (`firstStartBeat`) và điểm kết thúc ban đầu của nốt bị kéo (`originalDraggedEndBeat`).
|
||||
- Tính toán tỷ lệ co giãn thời gian:
|
||||
$$\text{scaleFactor} = \frac{\text{newDraggedEndBeat} - \text{firstStartBeat}}{\text{originalDraggedEndBeat} - \text{firstStartBeat}}$$
|
||||
- Cập nhật thời điểm bắt đầu (`start_beat`) và thời lượng (`duration_beats`) của tất cả các nốt nhạc được chọn có thời điểm bắt đầu nhỏ hơn hoặc bằng điểm kết thúc ban đầu của nốt bị kéo bằng cách nhân với `scaleFactor`.
|
||||
- Các nốt nhạc nằm sau vị trí kéo (ví dụ nốt thứ 4) sẽ được giữ nguyên không đổi.
|
||||
|
||||
#### Vẽ nhiều nốt bằng cách di chuột (Brush/Drag to Draw)
|
||||
- Khi đang kéo vẽ nốt mới, lưu vết mảng các cao độ (pitch/row) đã đi qua trong `draggedNote.visitedPitches`.
|
||||
- Khi di chuyển chuột qua cao độ mới, thêm cao độ đó vào danh sách và phân bổ đều tổng khoảng cách kéo (`beat - startOffsetBeat`) thành các nốt nhạc nối tiếp nhau, mỗi nốt có thời lượng bằng `totalSpan / visitedPitches.length`.
|
||||
|
||||
---
|
||||
|
||||
## Kế hoạch Kiểm thử
|
||||
|
||||
### 1. Kiểm thử hiệu ứng âm thanh FX
|
||||
- **Thời gian thực (Client)**: Kích hoạt Chorus/Reverb trên track, bấm Play nghe thử để xác nhận tiếng vang/tiếng đồng ca chạy mượt mà.
|
||||
- **Kết xuất (Backend)**: Mixdown dự án có track bật Chorus hoặc Reverb, kiểm tra file wav đầu ra xem hiệu ứng có được áp dụng chuẩn.
|
||||
|
||||
### 2. Kiểm thử vẽ trực quan & Phát nhạc Section
|
||||
- Thêm nốt nhạc và audio clip vào Section, nhấn lưu. Kiểm tra xem Section item hiển thị đúng dạng sóng và nốt màu.
|
||||
- Phát nhạc trên Main Session, kiểm tra xem khi playhead đi qua Section item thì âm thanh của Section có phát ra đúng nhịp.
|
||||
|
||||
### 3. Kiểm thử phím tắt Piano Roll
|
||||
- Kiểm tra click chọn nốt mới để di chuyển xem nốt cũ có được giải phóng hoàn toàn và không di chuyển theo.
|
||||
- Giữ `Ctrl` kéo nốt để sao chép.
|
||||
- Giữ `Alt` kéo giãn nhóm nốt.
|
||||
- Di chuột chéo để vẽ chuỗi nốt bậc thang (Brush tool).
|
||||
@@ -0,0 +1,63 @@
|
||||
# Implementation Plan: Project Management, Save As, and File Management inside Profile
|
||||
|
||||
We will add robust cloud/local project management, a custom "Save Project" name modal, a "Save As..." dialog offering server/local options, and a comprehensive file and project manager inside the User Profile Modal.
|
||||
|
||||
## User Review Required
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The profile modal will now contain three tabs: Account, Cloud Projects, and My Uploaded Files. Unused files (those not in the current session tracks or any saved projects) can be deleted by the user to free up quota storage.
|
||||
>
|
||||
> **Save As...** will trigger a modal allowing the user to type a new name and save it to either the server or export locally as a `.sfs` file.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### Backend APIs
|
||||
|
||||
#### [MODIFY] [projects.py](file:///home/locpham/SonicForgeStudio/app/api/v1/projects.py)
|
||||
- **`GET /cloud/{project_id}`**: Retrieves a specific user cloud project.
|
||||
- **`DELETE /cloud/{project_id}`**: Deletes a specific user cloud project.
|
||||
- **`PUT /cloud/{project_id}`**: Updates/overwrites an existing user cloud project.
|
||||
|
||||
#### [MODIFY] [audio.py](file:///home/locpham/SonicForgeStudio/app/api/v1/audio.py)
|
||||
- **`POST /upload`**, **`run_python_dsp_tool`** (for synth), and **`ai_cut_audio`**: Prefix file IDs with `user_{user_id}_` to establish file ownership and quota tracking securely.
|
||||
- **`POST /my-files`**: Lists all files starting with `user_{user_id}_` on the server disk. Identifies if they are referenced in the active project session or any database project records to compute their `is_in_use` status.
|
||||
- **`DELETE /my-files/{file_id}`**: Deletes a user's uploaded/generated file from the server uploads and processed directories after verifying ownership.
|
||||
|
||||
---
|
||||
|
||||
### Frontend Services & UI
|
||||
|
||||
#### [MODIFY] [api.js](file:///home/locpham/SonicForgeStudio/app/static/js/services/api.js)
|
||||
- Expose APIs for fetching, deleting, and updating cloud projects.
|
||||
- Expose APIs for listing and deleting user audio files.
|
||||
|
||||
#### [MODIFY] [app.jsx](file:///home/locpham/SonicForgeStudio/app/static/js/app.jsx)
|
||||
- **State Additions**:
|
||||
- `currentProjectId`: Tracks the ID of the loaded cloud project (synced with localStorage).
|
||||
- `saveProjectModalOpen`, `saveAsModalOpen`: Controls the new custom modals.
|
||||
- **Save Project Modal**:
|
||||
- Modal with an input for project name, used when saving a project that doesn't have a name yet.
|
||||
- **Save As Modal**:
|
||||
- Allows choosing to save under a new name either locally (.sfs file) or on the server.
|
||||
- **Profile Modal Extensions**:
|
||||
- Add Tabs: **Account Settings**, **Cloud Projects**, **My Uploaded Files**.
|
||||
- **Cloud Projects Tab**: Displays saved projects with load (open DAW project) and delete options.
|
||||
- **My Uploaded Files Tab**: Displays files with sizes, creation dates, usage badges, individual delete buttons, and a global "Clean Up Unused Files" button.
|
||||
|
||||
---
|
||||
|
||||
## Verification Plan
|
||||
|
||||
### Automated Tests
|
||||
- Run backend lint and sanity checks.
|
||||
```bash
|
||||
python -m flake8 app/api/v1/projects.py app/api/v1/audio.py
|
||||
```
|
||||
|
||||
### Manual Verification
|
||||
1. Create a new project, press Save, verify the custom input modal appears.
|
||||
2. Upload some files, check the Profile -> My Uploaded Files tab. Verify the files are listed as "In Use".
|
||||
3. Remove a track containing a file, verify the file changes to "Not In Use". Press delete to free up quota.
|
||||
4. Click File -> Save As... and select either Cloud or Local .sfs and verify name updates and downloads.
|
||||
@@ -0,0 +1,220 @@
|
||||
Để tích hợp **AI Panel** điều khiển dự án DAW (Digital Audio Workstation) của bạn dựa trên yêu cầu từ Prompt, hệ thống cần được thiết kế theo kiến trúc **Copilot Agent / Command Orchestrator**.
|
||||
|
||||
Thay vì để AI can thiệp trực tiếp vào dữ liệu dự án, AI sẽ đóng vai trò **Bộ não điều hướng (Brain)**: Nhận nhận ngôn ngữ tự nhiên từ Prompt $\rightarrow$ Dịch ra định dạng lệnh chuẩn dạng JSON/Function Call $\rightarrow$ Gửi về cho **DAW Core Engine** thực thi.
|
||||
|
||||
Dưới đây là thiết kế kiến trúc, kỹ thuật và thuật toán chi tiết cho hệ thống này.
|
||||
|
||||
---
|
||||
|
||||
## 1. Kiến trúc Tổng thể (System Architecture)
|
||||
|
||||
Hệ thống nên chia làm 4 lớp (Layers) rõ ràng:
|
||||
|
||||
```text
|
||||
[ User Interface (AI Panel) ]
|
||||
│ (User Prompt + Current App Context)
|
||||
▼
|
||||
[ LLM Gateway & Function Routing ]
|
||||
│ (JSON Intent / Function Calling)
|
||||
▼
|
||||
[ DAW Command Manager / State Engine ] ──► [ Local Python DSP Server ] (Xử lý âm thanh/DSP nặng)
|
||||
│ (Mutation Protocol)
|
||||
▼
|
||||
[ Frontend State Engine (JS/Canvas) ] (Cập nhật UI & MIDI Viewport)
|
||||
|
||||
```
|
||||
|
||||
### A. AI Panel UI & Model Gateway
|
||||
|
||||
* **UI Controls:** Nơi nhập Prompt, nút chọn Provider (OpenAI, Anthropic, Gemini, Local Ollama) & Model (GPT-4o, Claude Sonnet, Llama 3, v.v.).
|
||||
* **Context Aggregator:** Thu thập **Trạng thái hiện tại của DAW (State)** truyền kèm vào Prompt để AI "hiểu" ngữ cảnh (VD: track nào đang chọn, Tempo bao nhiêu, đang chọn khoảng thời gian nào).
|
||||
|
||||
### B. Command Manager & State Engine (Bộ điều khiển chính)
|
||||
|
||||
* Không cho LLM viết trực tiếp vào State của DAW. LLM chỉ trả về một danh sách các **Action Descriptor (Lệnh hành động)**.
|
||||
* **Command Pattern & Undo/Redo Engine:** Mọi hành động AI trả về đều đi qua Dispatcher để có thể `Undo` (`Ctrl + Z`) dễ dàng.
|
||||
|
||||
---
|
||||
|
||||
## 2. Kỹ thuật "Function Calling / Structured Outputs" (Cốt lõi)
|
||||
|
||||
Để AI trả về chính xác lệnh điều khiển hệ thống mà không nói "lan man", bạn áp dụng kỹ thuật **Function Calling / Schema Enforcement**.
|
||||
|
||||
### Định nghĩa các Tool / Command Protocol cho AI:
|
||||
|
||||
Bạn sẽ định nghĩa danh sách API dưới dạng **JSON Schema** cho AI:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{
|
||||
"name": "create_track",
|
||||
"description": "Tạo một track mới trong dự án",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["audio", "midi"] }
|
||||
},
|
||||
"required": ["name", "type"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "add_midi_item",
|
||||
"description": "Thêm một MIDI item/clip vào track",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"track_id": { "type": "string" },
|
||||
"start_bar": { "type": "number" },
|
||||
"length_bars": { "type": "number" }
|
||||
},
|
||||
"required": ["track_id", "start_bar", "length_bars"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "modify_midi_notes",
|
||||
"description": "Thêm, chỉnh sửa hoặc xóa các note MIDI trong item",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"item_id": { "type": "string" },
|
||||
"notes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pitch": { "type": "string", "description": "VD: C4, D#3, F5" },
|
||||
"start_time": { "type": "number", "description": "Tính bằng Bar hoặc Giây" },
|
||||
"duration": { "type": "number" },
|
||||
"velocity": { "type": "integer", "minimum": 0, "maximum": 127 }
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": ["item_id", "notes"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "process_audio_dsp",
|
||||
"description": "Gửi yêu cầu chỉnh sửa âm thanh sang Python DSP Backend",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"track_id": { "type": "string" },
|
||||
"action": { "type": "string", "enum": ["normalize", "invert_phase", "gain", "pitch_shift"] },
|
||||
"params": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Thuật toán Context Injection (Bơm ngữ cảnh DAW vào Prompt)
|
||||
|
||||
AI không thể sửa MIDI hay tạo Track chính xác nếu không biết trạng thái hiện tại. Do đó, trước khi gửi prompt của người dùng lên LLM, hệ thống phải chạy **Thuật toán đóng gói Context**:
|
||||
|
||||
### Thuật toán đóng gói Context:
|
||||
|
||||
```javascript
|
||||
function buildAIPromptContext(userPrompt) {
|
||||
const currentState = {
|
||||
tempo: dawState.bpm,
|
||||
timeSignature: dawState.timeSignature,
|
||||
selectedTrackId: dawState.activeTrackId,
|
||||
selectedItemId: dawState.activeItemId,
|
||||
playheadPosition: dawState.playheadTime,
|
||||
tracks: dawState.tracks.map(t => ({
|
||||
id: t.id,
|
||||
name: t.name,
|
||||
type: t.type,
|
||||
itemsCount: t.items.length
|
||||
}))
|
||||
};
|
||||
|
||||
return {
|
||||
system_instruction: "Bạn là trợ lý AI cho DAW. Hãy phân tích yêu cầu người dùng và phản hồi BẰNG DẠNG FUNCTION CALLS phù hợp với Context dự án.",
|
||||
daw_context: currentState,
|
||||
user_prompt: userPrompt
|
||||
};
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### Ví dụ xử lý kịch bản thực tế:
|
||||
|
||||
* **User Prompt:** *"Thêm một hợp âm C Major (Đô trưởng) vào đầu track 2 dạng MIDI item dài 2 bar"*
|
||||
* **LLM nhận Prompt + Context** $\rightarrow$ **AI trả về JSON:**
|
||||
```json
|
||||
[
|
||||
{
|
||||
"function": "add_midi_item",
|
||||
"args": { "track_id": "track_02", "start_bar": 1, "length_bars": 2 }
|
||||
},
|
||||
{
|
||||
"function": "modify_midi_notes",
|
||||
"args": {
|
||||
"item_id": "new_created_item_id",
|
||||
"notes": [
|
||||
{ "pitch": "C4", "start_time": 0, "duration": 2, "velocity": 100 },
|
||||
{ "pitch": "E4", "start_time": 0, "duration": 2, "velocity": 100 },
|
||||
{ "pitch": "G4", "start_time": 0, "duration": 2, "velocity": 100 }
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4. Phân chia xử lý giữa Backend (Python) và Frontend (JS)
|
||||
|
||||
Vì dự án của bạn là **Hybrid (HTML5/JS Frontend + Python Server)**, luồng công việc sẽ phân chia như sau:
|
||||
|
||||
| Thao tác | Đơn vị xử lý | Mô tả thuật toán / Kỹ thuật |
|
||||
| --- | --- | --- |
|
||||
| **1. Chỉnh sửa MIDI (Thêm/Sửa note, Item)** | **Frontend (JS)** | Sửa mảng JSON chứa dữ liệu MIDI trên JS State $\rightarrow$ Gọi hàm `requestAnimationFrame()` để render lại Piano Roll / Canvas. **Không cần gửi về Python** để tối ưu tốc độ realtime. |
|
||||
| **2. Chỉnh sửa Audio (DSP, Pitch Shift, Cut/Norm)** | **Backend (Python)** | JS gửi request kèm file path/audio buffer sang Python. Python dùng `librosa` / `pydub` / `scipy` thực hiện tính toán DSP $\rightarrow$ Trả về file WAV mới hoặc Array Buffer mới cho Frontend render lại Waveform. |
|
||||
| **3. Tạo nhạc tự động (Music Generation)** | **Backend (Python)** | Nếu Prompt yêu cầu *"Tạo 1 đoạn Beat Lofi 8 bar"*, Python gọi các model AI chuyên biệt local (như **MusicGen**, **AudioLDM**) để tạo ra file Audio / File MIDI thực tế $\rightarrow$ Trả về Frontend. |
|
||||
|
||||
---
|
||||
|
||||
## 5. UI/UX cho phần AI Panel (Dựa trên Ảnh Mockup)
|
||||
|
||||
Để UI ở phần dưới hiển thị tốt như hình bạn đính kèm:
|
||||
|
||||
1. **Nút bấm Chọn Provider & Model:**
|
||||
* Mở Menu Popup (Popover) cho phép chọn:
|
||||
* **Provider:** OpenAI, Anthropic, Google Gemini, Ollama (Local).
|
||||
* **Model:** GPT-4o, Claude 3.5 Sonnet, Llama 3.
|
||||
* **API Key input** (lưu ở `localStorage`).
|
||||
|
||||
|
||||
|
||||
|
||||
2. **Luồng chỉ báo phản hồi (Visual Feedback):**
|
||||
* Khi AI đang suy luận: Hiện trạng thái `AI đang phân tích lệnh...`.
|
||||
* Khi AI thực thi xong: Hiển thị danh sách các hành động đã làm, ví dụ:
|
||||
* `[v] Đã tạo Track 3 (MIDI)`
|
||||
* `[v] Đã thêm 4 MIDI Notes (C4, E4, G4, B4)`
|
||||
|
||||
|
||||
* Nút **Undo** nhanh ngay trên AI Panel nếu AI thực hiện chưa đúng ý.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 6. Lộ trình triển khai khuyến nghị
|
||||
|
||||
1. **Bước 1 (Protocol):** Viết lớp `DAWCommandDispatcher` ở Frontend JS để có thể gọi các hàm dạng `Dispatcher.execute('CREATE_TRACK', payload)` trước.
|
||||
2. **Bước 2 (LLM Integration):** Tích hợp SDK API (OpenAI/Gemini/Claude) vào JS/Python, cấu hình `tools` (Function Calling).
|
||||
3. **Bước 3 (Context Injection):** Viết hàm xuất cấu trúc `dawState` hiện tại ra JSON ngắn gọn để làm Context cho Prompt.
|
||||
4. **Bước 4 (MIDI & Audio Execution):** Map kết quả Function Calling từ AI trả về vào `DAWCommandDispatcher` để cập nhật UI & gửi lệnh DSP qua Python.
|
||||
@@ -0,0 +1,333 @@
|
||||
|
||||
# DAW UI LAYOUT & PANEL SYSTEM ARCHITECTURE
|
||||
|
||||
This document details the interface layout solution (UI Layout Architecture), HTML/CSS structure, and interaction algorithms (Resizing, Scrolling) for a Hybrid DAW system, supporting responsive flexible scaling across Panels and the bottom dock strip.
|
||||
|
||||
---
|
||||
|
||||
## 1. Overall Layout Diagram (Grid Structure)
|
||||
|
||||
The application interface is structured around 3 main axes following an App Shell model (`Viewport Locked 100vh`):
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Top Navigation & Transport Toolbar (Fixed Top Bar) │
|
||||
├───────────────────────────────────────────────────────────┬──────────────────────┤
|
||||
│ │ RIGHT COLUMN │
|
||||
│ MAIN WORKSPACE │ (RIGHT SIDEBAR) │
|
||||
│ ┌───────────────────────┬───────────────────────────────┐ │ ┌──────────────────┐ │
|
||||
│ │ Track Control Panels │ Timeline / Audio Viewport │ │ │ Media Explorer │ │
|
||||
│ │ (Track List) │ (Beat Grid & Waveforms) │ │ │ (Dynamic Height) │ │
|
||||
│ │ │ │ │ ├──────────────────┤ │
|
||||
│ │ │ │ │ │ AI Panel │ │
|
||||
│ │ │ │ │ │ (Dynamic Height) │ │
|
||||
│ └───────────────────────┴───────────────────────────────┘ │ └──────────────────┘ │
|
||||
├───────────────────────────────────────────────────────────┴──────────────────────┤
|
||||
│ BOTTOM DOCK PANEL STRIP - Horizontal Scroll (Overflow-X Auto) │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Export Panel │ │ DSP Tools │ │ Panel 03 │ │ Panel 04... │ ──────► │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
├──────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Status Bar (Fixed Bottom Status) │
|
||||
└──────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Layout Region Details
|
||||
|
||||
### A. Main Workspace (Center Region)
|
||||
|
||||
* **Function:** Contains the track list (Track Controls), timeline ruler (Timeline Ruler), and audio/MIDI display areas (Audio Waveform & Piano Roll Clip Grid).
|
||||
* **Behavior:** Auto-expands (`flex-grow: 1`) to fill the remaining screen space after subtracting the width of the Right Sidebar and the height of the Bottom Panel.
|
||||
|
||||
### B. Bottom Panel Dock Strip (Bottom Row)
|
||||
|
||||
* **Technical Specifications:**
|
||||
* **Flexible Horizontal Scroll:** The container has a fixed height (e.g., `220px`), using `overflow-x: auto` and `display: flex`.
|
||||
* **Sub-Panels:** Houses a list of independent Card/Tile tools (Export Panel, Python DSP Tools Panel, Selection Panel, FX Panel, etc.).
|
||||
* **No Shrinking (`flex-shrink: 0`):** Each Sub-Panel is configured with `flex-shrink: 0` and a minimum width (`min-width: 280px - 350px`). When the combined width of all panels exceeds the screen width, a horizontal scrollbar appears automatically.
|
||||
|
||||
|
||||
|
||||
### C. Right Resizable Sidebar (Multi-Panel Right Column)
|
||||
|
||||
* **Technical Specifications:**
|
||||
* **Width Resizing:** The entire right column can be resized by dragging its left border (Border Left Drag Handle) to expand or collapse the visible space of the Main Workspace.
|
||||
* **Vertical Stacking:** Houses stacked child panels (e.g., Media Explorer, AI Panel, Inspector, etc.).
|
||||
* **Independent Height Resizing:** Horizontal splitters (Horizontal Splitter / Resizer Handle) sit between stacked child panels, allowing users to drag up/down to adjust height ratios between panels.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. HTML & CSS Framework Implementation
|
||||
|
||||
### HTML Core Structure
|
||||
|
||||
```html
|
||||
<div class="daw-app-shell">
|
||||
<!-- Top Toolbar -->
|
||||
<header class="daw-top-bar">...</header>
|
||||
|
||||
<!-- Body Middle Container -->
|
||||
<div class="daw-body-container">
|
||||
|
||||
<!-- Main Center Viewport -->
|
||||
<main class="daw-main-workspace">
|
||||
<div class="track-headers-column">...</div>
|
||||
<div class="timeline-canvas-viewport">...</div>
|
||||
</main>
|
||||
|
||||
<!-- Vertical Resizer Handle (Adjusts Right Sidebar Width) -->
|
||||
<div class="resizer-col-handle" id="col-resizer"></div>
|
||||
|
||||
<!-- Right Sidebar Container -->
|
||||
<aside class="daw-right-sidebar" id="right-sidebar">
|
||||
|
||||
<!-- Panel 1: Media Explorer -->
|
||||
<div class="sidebar-panel" id="panel-media-explorer">
|
||||
<div class="panel-header">Media Explorer</div>
|
||||
<div class="panel-content">...</div>
|
||||
</div>
|
||||
|
||||
<!-- Horizontal Resizer Handle (Adjusts Panel Heights inside the Column) -->
|
||||
<div class="resizer-row-handle" id="row-resizer-1"></div>
|
||||
|
||||
<!-- Panel 2: AI Panel -->
|
||||
<div class="sidebar-panel" id="panel-ai">
|
||||
<div class="panel-header">AI Panel</div>
|
||||
<div class="panel-content">...</div>
|
||||
</div>
|
||||
|
||||
</aside>
|
||||
</div>
|
||||
|
||||
<!-- Bottom Panel Strip (Horizontal Scroll Container) -->
|
||||
<footer class="daw-bottom-strip">
|
||||
<div class="bottom-panel">Export Panel</div>
|
||||
<div class="bottom-panel">Python DSP Tools Panel</div>
|
||||
<div class="bottom-panel">Selection Panel</div>
|
||||
<div class="bottom-panel">Plugin FX Rack Panel</div>
|
||||
<div class="bottom-panel">MIDI Event List Panel</div>
|
||||
</footer>
|
||||
|
||||
<!-- Status Bar -->
|
||||
<div class="daw-status-bar">...</div>
|
||||
</div>
|
||||
|
||||
```
|
||||
|
||||
### CSS System Architecture
|
||||
|
||||
```css
|
||||
:root {
|
||||
--right-sidebar-width: 320px;
|
||||
--bottom-strip-height: 220px;
|
||||
--top-bar-height: 80px;
|
||||
--status-bar-height: 25px;
|
||||
--panel-border-color: #2a2a2a;
|
||||
}
|
||||
|
||||
/* Fullscreen Fixed App Shell */
|
||||
.daw-app-shell {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
width: 100vw;
|
||||
height: 100vh;
|
||||
overflow: hidden;
|
||||
background-color: #121212;
|
||||
color: #e0e0e0;
|
||||
}
|
||||
|
||||
/* Middle Section holding Main Workspace and Right Sidebar */
|
||||
.daw-body-container {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
height: calc(100vh - var(--top-bar-height) - var(--bottom-strip-height) - var(--status-bar-height));
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* Auto-expanding Main Workspace */
|
||||
.daw-main-workspace {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* Right Sidebar with Width controlled via CSS Variable */
|
||||
.daw-right-sidebar {
|
||||
width: var(--right-sidebar-width);
|
||||
min-width: 200px;
|
||||
max-width: 600px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
background-color: #1a1a1a;
|
||||
border-left: 1px solid var(--panel-border-color);
|
||||
}
|
||||
|
||||
/* Vertically stacked child Panels in Right Sidebar */
|
||||
.sidebar-panel {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
background: #1e1e1e;
|
||||
border-bottom: 1px solid var(--panel-border-color);
|
||||
}
|
||||
|
||||
#panel-media-explorer {
|
||||
height: 50%; /* Default 50/50 split */
|
||||
min-height: 100px;
|
||||
}
|
||||
|
||||
#panel-ai {
|
||||
flex: 1; /* Fills remaining height */
|
||||
min-height: 100px;
|
||||
}
|
||||
|
||||
/* BOTTOM ROW: Enables Horizontal Scrolling */
|
||||
.daw-bottom-strip {
|
||||
height: var(--bottom-strip-height);
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 8px;
|
||||
overflow-x: auto; /* Enables horizontal scroll when panels overflow */
|
||||
overflow-y: hidden;
|
||||
background-color: #161616;
|
||||
border-top: 1px solid var(--panel-border-color);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* Optimized custom horizontal scrollbar for DAW styling */
|
||||
.daw-bottom-strip::-webkit-scrollbar {
|
||||
height: 8px;
|
||||
}
|
||||
.daw-bottom-strip::-webkit-scrollbar-thumb {
|
||||
background: #3a3a3a;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.daw-bottom-strip::-webkit-scrollbar-thumb:hover {
|
||||
background: #00ffcc;
|
||||
}
|
||||
|
||||
/* Sub-panels inside the bottom strip */
|
||||
.bottom-panel {
|
||||
flex: 0 0 auto; /* Prevents shrinking, locks content dimensions */
|
||||
width: 320px;
|
||||
height: 100%;
|
||||
background-color: #222;
|
||||
border: 1px solid #333;
|
||||
border-radius: 6px;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
/* RESIZER HANDLES */
|
||||
.resizer-col-handle {
|
||||
width: 5px;
|
||||
cursor: ew-resize; /* Horizontal resize cursor */
|
||||
background: transparent;
|
||||
transition: background 0.2s;
|
||||
z-index: 10;
|
||||
}
|
||||
.resizer-col-handle:hover,
|
||||
.resizer-col-handle:active {
|
||||
background: #00ffcc;
|
||||
}
|
||||
|
||||
.resizer-row-handle {
|
||||
height: 5px;
|
||||
cursor: ns-resize; /* Vertical resize cursor */
|
||||
background: transparent;
|
||||
transition: background 0.2s;
|
||||
z-index: 10;
|
||||
}
|
||||
.resizer-row-handle:hover,
|
||||
.resizer-row-handle:active {
|
||||
background: #00ffcc;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Interaction Algorithms (JS Resizing Logic)
|
||||
|
||||
To handle smooth resizing without stuttering or dropped events when dragging over `iframe` or `canvas` elements, the algorithms rely on `pointerdown`, `pointermove`, and `pointerup` events.
|
||||
|
||||
### A. Right Sidebar Width Resizing Algorithm (Horizontal Resizer)
|
||||
|
||||
```javascript
|
||||
const colResizer = document.getElementById('col-resizer');
|
||||
const rightSidebar = document.getElementById('right-sidebar');
|
||||
|
||||
colResizer.addEventListener('pointerdown', (e) => {
|
||||
e.preventDefault();
|
||||
colResizer.setPointerCapture(e.pointerId);
|
||||
|
||||
const startX = e.clientX;
|
||||
const startWidth = rightSidebar.getBoundingClientRect().width;
|
||||
|
||||
const onPointerMove = (moveEvent) => {
|
||||
// Delta calculation: dragging left increases width, dragging right decreases width
|
||||
const deltaX = startX - moveEvent.clientX;
|
||||
const newWidth = Math.max(200, Math.min(600, startWidth + deltaX));
|
||||
|
||||
document.documentElement.style.setProperty('--right-sidebar-width', `${newWidth}px`);
|
||||
};
|
||||
|
||||
const onPointerUp = (upEvent) => {
|
||||
colResizer.releasePointerCapture(upEvent.pointerId);
|
||||
colResizer.removeEventListener('pointermove', onPointerMove);
|
||||
colResizer.removeEventListener('pointerup', onPointerUp);
|
||||
};
|
||||
|
||||
colResizer.addEventListener('pointermove', onPointerMove);
|
||||
colResizer.addEventListener('pointerup', onPointerUp);
|
||||
});
|
||||
|
||||
```
|
||||
|
||||
### B. Right Sidebar Panel Height Resizing Algorithm (Vertical Resizer)
|
||||
|
||||
```javascript
|
||||
const rowResizer = document.getElementById('row-resizer-1');
|
||||
const topPanel = document.getElementById('panel-media-explorer');
|
||||
|
||||
rowResizer.addEventListener('pointerdown', (e) => {
|
||||
e.preventDefault();
|
||||
rowResizer.setPointerCapture(e.pointerId);
|
||||
|
||||
const startY = e.clientY;
|
||||
const startHeight = topPanel.getBoundingClientRect().height;
|
||||
|
||||
const onPointerMove = (moveEvent) => {
|
||||
const deltaY = moveEvent.clientY - startY;
|
||||
const newHeight = Math.max(100, startHeight + deltaY);
|
||||
|
||||
topPanel.style.height = `${newHeight}px`;
|
||||
topPanel.style.flex = 'none'; // Switch from flex ratio to fixed px during drag
|
||||
};
|
||||
|
||||
const onPointerUp = (upEvent) => {
|
||||
rowResizer.releasePointerCapture(upEvent.pointerId);
|
||||
rowResizer.removeEventListener('pointermove', onPointerMove);
|
||||
rowResizer.removeEventListener('pointerup', onPointerUp);
|
||||
};
|
||||
|
||||
colResizer.addEventListener('pointermove', onPointerMove);
|
||||
colResizer.addEventListener('pointerup', onPointerUp);
|
||||
});
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Summary of Solution Advantages
|
||||
|
||||
* **Native Horizontal Scrolling:** The bottom Dock area flexibly accommodates an unlimited number of Panels. Users can scroll horizontally (`Shift + Mouse Wheel`) or use a trackpad to browse panels easily.
|
||||
* **Smooth & Accurate Resizing:** Utilizing Pointer Capture ensures drag interactions do not drop or break even when the cursor moves rapidly beyond the Resizer handle's bounds.
|
||||
* **Standardized CSS Variables:** Enables easy persistence of layout states (`Width`/`Height`) to the browser's `localStorage`, restoring the user's custom layout configuration on app reload.
|
||||
@@ -0,0 +1,760 @@
|
||||
Here is the complete translation and conversion of the document into a clean, professionally formatted Markdown layout:
|
||||
|
||||
# ARCHITECTURAL, TECHNICAL, AND ALGORITHMIC SPECIFICATION
|
||||
|
||||
## Hybrid Web-Based Digital Audio Workstation (DAW) with Nested Section Architecture and Non-Destructive Timeline Mechanics
|
||||
|
||||
---
|
||||
|
||||
### 1. System Overview & Architecture Design
|
||||
|
||||
#### 1.1 High-Level Architecture Topology
|
||||
|
||||
The system follows a hybrid Client-Server architecture designed for real-time Web-based audio production, composition, and high-performance offline DSP rendering.
|
||||
|
||||
* **Frontend Client (HTML5 / Vanilla JS / Web Audio API / HTML5 Canvas)**
|
||||
* **UI Layer:** HTML5 Canvas / Web Components for high-FPS multi-lane timeline rendering, Piano Roll canvas, Sample Editor, and Sub-Tab navigation.
|
||||
* **Audio Engine Layer:** Web Audio API `AudioContext` graph, Custom `AudioWorklet` Processors (WebAssembly/JS) for real-time synthesis, playback scheduling, sample playback, and latency-compensated signal routing.
|
||||
* **State Management Engine:** Immutable/Reactive Central State Store handling Session tree hierarchy, Section Store registries, Undo/Redo stack, and view-state context isolation.
|
||||
|
||||
|
||||
* **Backend Server (Python Engine)**
|
||||
* **RESTful / WebSocket API:** Event-driven client communication layer (FastAPI or AIOHTTP).
|
||||
* **DSP / Rendering Engine:** Python-based audio processing (`numpy`, `scipy`, `pyo`, `pedalboard`) for offline stem bouncing, high-fidelity export, sample processing, and optional VST/VSTi hosting/bridging.
|
||||
|
||||
|
||||
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| FRONTEND (HTML5/JS) |
|
||||
| |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | UI & View State System | |
|
||||
| | +---------------------+ +----------------------+ +--------------------+ | |
|
||||
| | | Main Session Canvas | | Section-Tab View | | Piano Roll View | | |
|
||||
| | +---------------------+ +----------------------+ +--------------------+ | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | Central Data State Store | |
|
||||
| | [Project Model] ---> [Section Store] ---> [Item Clip Metadata] | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | Audio & Clock Engine | |
|
||||
| | +------------------------+ +------------------+ +---------------------+ | |
|
||||
| | | Precision Scheduler | | Web Audio Graph | | AudioWorklet Synth | | |
|
||||
| | | (Lookahead Timer) | | AudioNode Router | | / WebAssembly Core | | |
|
||||
| | +------------------------+ +------------------+ +---------------------+ | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
+------------------------------------------^----------------------------------------+
|
||||
| WebSocket / REST API
|
||||
+------------------------------------------v----------------------------------------+
|
||||
| BACKEND SERVER (PYTHON) |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | FastAPI / WebSocket Handler | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | DSP Engine (Pedalboard / Numpy / Scipy) - Offline Render, Audio Export | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
| | VST / VSTi Hosting Bridge & Plugin State Persistence | |
|
||||
| +-----------------------------------------------------------------------------+ |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Detailed Data Schemas (JSON Specification)
|
||||
|
||||
#### 2.1 Project Root Schema (`project_schema.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "DAWProject",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"project_id": { "type": "string", "format": "uuid" },
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": { "type": "string" },
|
||||
"bpm": { "type": "number", "minimum": 20.0, "maximum": 999.0, "default": 120.0 },
|
||||
"time_signature_numerator": { "type": "integer", "default": 4 },
|
||||
"time_signature_denominator": { "type": "integer", "default": 4 },
|
||||
"sample_rate": { "type": "integer", "default": 44100 }
|
||||
},
|
||||
"required": ["title", "bpm", "time_signature_numerator", "time_signature_denominator", "sample_rate"]
|
||||
},
|
||||
"main_session": { "$ref": "#/definitions/SessionContainer" },
|
||||
"section_store": {
|
||||
"type": "object",
|
||||
"description": "Auxiliary registry mapping section_id to sub-session containers",
|
||||
"additionalProperties": { "$ref": "#/definitions/SessionContainer" }
|
||||
}
|
||||
},
|
||||
"required": ["project_id", "metadata", "main_session", "section_store"],
|
||||
"definitions": {
|
||||
"SessionContainer": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"is_root": { "type": "boolean" },
|
||||
"length_bars": { "type": "number", "description": "Computed or manually set total length in bars" },
|
||||
"auto_compute_length": { "type": "boolean", "default": true },
|
||||
"tracks": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/Track" }
|
||||
}
|
||||
},
|
||||
"required": ["id", "is_root", "tracks"]
|
||||
},
|
||||
"Track": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["AUDIO", "MIDI", "SECTION"] },
|
||||
"volume_db": { "type": "number", "default": 0.0 },
|
||||
"pan": { "type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0 },
|
||||
"mute": { "type": "boolean", "default": false },
|
||||
"solo": { "type": "boolean", "default": false },
|
||||
"fx_chain": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/FXPlugin" }
|
||||
},
|
||||
"synth_engine": { "$ref": "#/definitions/SynthPlugin" },
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/TimelineItem" }
|
||||
}
|
||||
},
|
||||
"required": ["id", "name", "type", "items"]
|
||||
},
|
||||
"TimelineItem": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"type": { "type": "string", "enum": ["AUDIO_ITEM", "MIDI_ITEM", "SECTION_ITEM"] },
|
||||
"start_bar": { "type": "number", "description": "Global timeline position where the item starts" },
|
||||
"duration_bars": { "type": "number", "description": "Visible duration on the track timeline in bars" },
|
||||
"clip_start_offset_bars": { "type": "number", "description": "Internal start offset inside the source buffer/item" },
|
||||
"source_data": {
|
||||
"type": "object",
|
||||
"oneOf": [
|
||||
{ "$ref": "#/definitions/AudioSourceData" },
|
||||
{ "$ref": "#/definitions/MIDISourceData" },
|
||||
{ "$ref": "#/definitions/SectionSourceData" }
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": ["id", "type", "start_bar", "duration_bars", "clip_start_offset_bars", "source_data"]
|
||||
},
|
||||
"AudioSourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"audio_file_url": { "type": "string" },
|
||||
"sample_rate": { "type": "integer" },
|
||||
"channels": { "type": "integer" },
|
||||
"gain": { "type": "number", "default": 1.0 }
|
||||
},
|
||||
"required": ["audio_file_url"]
|
||||
},
|
||||
"MIDISourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"total_buffer_bars": { "type": "number", "default": 8.0 },
|
||||
"notes": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/definitions/MIDINote" }
|
||||
}
|
||||
},
|
||||
"required": ["total_buffer_bars", "notes"]
|
||||
},
|
||||
"SectionSourceData": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"referenced_section_id": { "type": "string", "description": "Pointer to section_store key" }
|
||||
},
|
||||
"required": ["referenced_section_id"]
|
||||
},
|
||||
"MIDINote": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
|
||||
"start_beat": { "type": "number", "description": "Beat offset relative to the start of the source buffer (bar 0)" },
|
||||
"duration_beats": { "type": "number" },
|
||||
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0, "default": 0.8 },
|
||||
"pan": { "type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0 }
|
||||
},
|
||||
"required": ["id", "pitch", "start_beat", "duration_beats", "velocity"]
|
||||
},
|
||||
"FXPlugin": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"plugin_id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"bypass": { "type": "boolean", "default": false },
|
||||
"parameters": { "type": "object" }
|
||||
}
|
||||
},
|
||||
"SynthPlugin": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"plugin_id": { "type": "string" },
|
||||
"preset_id": { "type": "string" },
|
||||
"parameters": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. UI, Tab Navigation & View State Management
|
||||
|
||||
#### 3.1 Tab Context Model, Pinning Rules & Close Prevention Hierarchy
|
||||
|
||||
The application manages view tabs dynamically while maintaining strict lifecycle integrity:
|
||||
|
||||
* **Main Session Tab (Fixed / Pinned):** Always pinned at index 0 (`is_closeable: false`). It cannot be closed under any circumstances.
|
||||
* **Sub-Tabs (Section-Tab, Piano Roll Tab, Audio Sample Editor Sub-Tab):** Dynamic views (`is_closeable: true`).
|
||||
* **Parent-Child Tab Dependency Rules:**
|
||||
* A Section-Tab represents an intermediate sub-session.
|
||||
* When a user opens a child item (e.g., a `MIDIItem` or `AudioItem` inside a Section-Tab) into a Piano Roll Tab or Audio Sample Editor Sub-Tab, a parent-child context lineage is registered.
|
||||
* **Close Block Rule:** A Section-Tab cannot be closed while any of its child items are currently open in active sub-tabs. Attempting to close the parent Section-Tab displays a block notice highlighting open child editors.
|
||||
|
||||
|
||||
|
||||
```text
|
||||
+---------------------------------------+
|
||||
| Tab Navigation Controller |
|
||||
+-------------------+-------------------+
|
||||
|
|
||||
+--------------------------------+--------------------------------+
|
||||
| (Pinned / Uncloseable) | (Dynamic / Closable) | (Dynamic / Closable)
|
||||
+--------v--------+ +--------v--------+ +--------v--------+
|
||||
| MAIN SESSION | | SECTION TAB | | PIANO ROLL TAB |
|
||||
| (Root Context) | | (Sub-Session) | | (Item Context) |
|
||||
| | | [Parent Context] | [Child Context]|
|
||||
+-----------------+ +--------+--------+ +--------+--------+
|
||||
| |
|
||||
+---- Depends on child closure ---+
|
||||
|
||||
```
|
||||
|
||||
##### State Object Schema with Tab Dependency Tracking:
|
||||
|
||||
```json
|
||||
{
|
||||
"active_tab_id": "tab_pr_1",
|
||||
"open_tabs": [
|
||||
{
|
||||
"tab_id": "tab_root",
|
||||
"title": "MAIN SESSION",
|
||||
"type": "MAIN_SESSION",
|
||||
"target_id": "main",
|
||||
"is_closeable": false,
|
||||
"parent_tab_id": null
|
||||
},
|
||||
{
|
||||
"tab_id": "tab_sec_1",
|
||||
"title": "Section: Verse 1",
|
||||
"type": "SECTION_TAB",
|
||||
"target_id": "Section_01",
|
||||
"is_closeable": true,
|
||||
"parent_tab_id": "tab_root"
|
||||
},
|
||||
{
|
||||
"tab_id": "tab_pr_1",
|
||||
"title": "Piano Roll: Bassline",
|
||||
"type": "PIANO_ROLL",
|
||||
"target_id": "ItemMIDI_Bassline",
|
||||
"is_closeable": true,
|
||||
"parent_tab_id": "tab_sec_1"
|
||||
}
|
||||
],
|
||||
"piano_roll_state": {
|
||||
"target_item_id": "ItemMIDI_Bassline",
|
||||
"viewport_start_bar": 0.0,
|
||||
"viewport_bar_width": 8.0,
|
||||
"scroll_y_pitch": 60,
|
||||
"snap_resolution": "1/16",
|
||||
"note_selection": []
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 3.2 Piano Roll View Canvas Layout & Interaction Spec
|
||||
|
||||
* **Top Navigation Rule Pane (Bars/Beats Bar):**
|
||||
* Displays bars from $0$ to $N$ (where $N = \text{total\_buffer\_bars}$, e.g., 8 bars).
|
||||
* Highlights active clip visibility bounds (e.g., Bar 4.0 to Bar 6.0 shaded with active overlay, exterior bars dimmed).
|
||||
|
||||
|
||||
* **Left Piano Keybed:**
|
||||
* Anchored vertically, spans pitches $0$ (C-1) through $127$ (G9).
|
||||
* Draws standard 88 key / 128 key pattern with distinct black key visually offset bars and pitch labeling ($C3$, $C4$, etc.).
|
||||
|
||||
|
||||
* **Note Grid Canvas (Right Pane):**
|
||||
* Synced to vertical pitch scroll and horizontal beat zoom.
|
||||
* **Row Background Rendering:** Black key rows are assigned darker background fill color `#1A1A1E`, white key rows use `#25252A`.
|
||||
* **Snap Grid Lines:** Rendered dynamically based on selected snap mode: Free, 1/1 Bar, 1/2 Beat, 1/4 Beat, 1/8 Beat, 1/16 Beat, 1/32 Beat.
|
||||
|
||||
|
||||
* **Bottom Controller Pane (CC / Velocity / Pan Lane):**
|
||||
* Synchronized horizontally with note grid.
|
||||
* Displays vertical stem bars per note representing properties (Velocity, Pan). Allows click-and-drag line shaping or direct stem adjustment.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 4. Audio & Synth Engine Routing Architecture (Web Audio API)
|
||||
|
||||
#### 4.1 Real-Time Signal Flow Graph
|
||||
|
||||
```text
|
||||
[MIDI Scheduler] ---> [AudioWorklet / Virtual Synth Engine]
|
||||
|
|
||||
v (Audio Buffer / Stream)
|
||||
[Audio Sample Playback Node] ----> [Track Channel FX Chain]
|
||||
|
|
||||
v
|
||||
[Track Gain / Pan Node]
|
||||
|
|
||||
v
|
||||
+---------------------+---------------------+
|
||||
| |
|
||||
v (If inside Section) v (If Direct Track)
|
||||
[Section Sub-Mix Bus] [Main Master Mixer Bus]
|
||||
| |
|
||||
+-------------------->----------------------+
|
||||
|
|
||||
v
|
||||
[Web Audio Destination]
|
||||
|
||||
```
|
||||
|
||||
#### 4.2 Web Audio Node Architecture Specifications
|
||||
|
||||
* **AudioTrack Node Structure:**
|
||||
```javascript
|
||||
TrackAudioGraph = {
|
||||
inputNode: GainNode,
|
||||
fxChain: [ BiquadFilterNode, DelayNode, ConvolverNode ],
|
||||
panNode: StereoPannerNode,
|
||||
outputGainNode: GainNode,
|
||||
connect(destination) { ... }
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
* **Section Bus Graph Routing:**
|
||||
* Each Section in Section-tab Store instantiates an intermediate `GainNode` sub-mixer (`SectionBus`).
|
||||
* Tracks within the Section connect their final outputs to `SectionBus`.
|
||||
* When a `SectionItem` is placed on a Main Session track, the `SectionBus` output is routed into the Main Session track's input node, preserving non-destructive DSP processing hierarchies.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 5. Core Mathematical & Technical Algorithms
|
||||
|
||||
#### 5.1 Algorithm 1: Non-Destructive Item Slicing & Offset Playback Math
|
||||
|
||||
##### Mathematical Formulation
|
||||
|
||||
Let:
|
||||
|
||||
* $T_{\text{global}}$ = Current global playback time in seconds on the main timeline.
|
||||
* $\text{BPM}$ = Beats Per Minute of the project.
|
||||
* $\text{TS}_{\text{num}}$ = Time Signature Numerator (e.g., 4 beats per bar).
|
||||
* $S_{\text{item}}$ = Item start position in global bars ($\text{start\_bar}$).
|
||||
* $L_{\text{item}}$ = Item visible length on timeline in bars ($\text{duration\_bars}$).
|
||||
* $O_{\text{item}}$ = Source internal start offset in bars ($\text{clip\_start\_offset\_bars}$).
|
||||
|
||||
Bar to Time Conversion Factor:
|
||||
|
||||
$$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$
|
||||
|
||||
$$\text{SecondsPerBar} = \text{SecondsPerBeat} \times \text{TS}_{\text{num}}$$
|
||||
|
||||
Item Global Time Bounds:
|
||||
|
||||
$$T_{\text{start}} = S_{\text{item}} \times \text{SecondsPerBar}$$
|
||||
|
||||
$$T_{\text{end}} = (S_{\text{item}} + L_{\text{item}}) \times \text{SecondsPerBar}$$
|
||||
|
||||
Active Playback Slicing Condition: An item is active if and only if:
|
||||
|
||||
$$T_{\text{start}} \le T_{\text{global}} < T_{\text{end}}$$
|
||||
|
||||
Local Item Buffer Time Mapping ($T_{\text{local}}$): When $T_{\text{global}}$ falls within $[T_{\text{start}}, T_{\text{end}}]$, the corresponding time $T_{\text{local\_bars}}$ relative to the internal source clip buffer (0 to $\text{BufferLength}$) is:
|
||||
|
||||
$$T_{\text{local\_bars}} = \frac{T_{\text{global}} - T_{\text{start}}}{\text{SecondsPerBar}} + O_{\text{item}}$$
|
||||
|
||||
MIDI Note Slicing & Filtering Rule: For a MIDI note $N$ inside the item source with start beat $N_{\text{start\_beat}}$ and length $N_{\text{dur\_beat}}$ (converted to internal bar metric $N_{\text{bar\_start}} = \frac{N_{\text{start\_beat}}}{\text{TS}_{\text{num}}}$, $N_{\text{bar\_dur}} = \frac{N_{\text{dur\_beat}}}{\text{TS}_{\text{num}}}$):
|
||||
|
||||
The note is triggered during main playback if and only if:
|
||||
|
||||
$$N_{\text{bar\_start}} \ge O_{\text{item}} \quad \text{AND} \quad N_{\text{bar\_start}} < (O_{\text{item}} + L_{\text{item}})$$
|
||||
|
||||
##### Pseudocode Implementation
|
||||
|
||||
```javascript
|
||||
function getActiveMIDINotesForPlayback(item, currentGlobalBar, timeSigNum) {
|
||||
const itemStartBar = item.start_bar;
|
||||
const itemEndBar = item.start_bar + item.duration_bars;
|
||||
const offsetBar = item.clip_start_offset_bars;
|
||||
|
||||
// Check if playback cursor is inside visible item clip
|
||||
if (currentGlobalBar < itemStartBar || currentGlobalBar >= itemEndBar) {
|
||||
return []; // Item inactive
|
||||
}
|
||||
|
||||
const activeNotes = [];
|
||||
const internalWindowStartBar = offsetBar;
|
||||
const internalWindowEndBar = offsetBar + item.duration_bars;
|
||||
|
||||
for (const note of item.source_data.notes) {
|
||||
const noteStartBar = note.start_beat / timeSigNum;
|
||||
const noteEndBar = noteStartBar + (note.duration_beats / timeSigNum);
|
||||
|
||||
// Filter notes outside the non-destructive visible window
|
||||
if (noteStartBar >= internalWindowStartBar && noteStartBar < internalWindowEndBar) {
|
||||
// Calculate playback time relative to global session
|
||||
const relativeBarInItem = noteStartBar - internalWindowStartBar;
|
||||
const targetGlobalBar = itemStartBar + relativeBarInItem;
|
||||
|
||||
activeNotes.push({
|
||||
note: note,
|
||||
scheduledGlobalBar: targetGlobalBar
|
||||
});
|
||||
}
|
||||
}
|
||||
return activeNotes;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.2 Algorithm 2: Dynamic Section Length Calculation Algorithm
|
||||
|
||||
When `auto_compute_length` is enabled for a Section, its total duration in bars $L_{\text{section}}$ is dynamically evaluated from the boundary bounds of all child items across all tracks inside that Section.
|
||||
|
||||
##### Mathematical Formulation
|
||||
|
||||
Let $T$ be the set of tracks in the section, and $I(t)$ be the set of items in track $t$.
|
||||
|
||||
$$L_{\text{section}} = \max_{t \in T} \left( \max_{i \in I(t)} \left( i.\text{start\_bar} + i.\text{duration\_bars} \right) \right)$$
|
||||
|
||||
If $I(t)$ is empty for all $t$, then $L_{\text{section}} = 4.0$ (default baseline minimum).
|
||||
|
||||
##### Implementation Architecture
|
||||
|
||||
```javascript
|
||||
function recomputeSectionLength(sectionContainer) {
|
||||
if (!sectionContainer.auto_compute_length) {
|
||||
return sectionContainer.length_bars;
|
||||
}
|
||||
|
||||
let maxEndBar = 0.0;
|
||||
|
||||
for (const track of sectionContainer.tracks) {
|
||||
for (const item of track.items) {
|
||||
const itemEndBar = item.start_bar + item.duration_bars;
|
||||
if (itemEndBar > maxEndBar) {
|
||||
maxEndBar = itemEndBar;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Enforce baseline grid quantization rounding (e.g. minimum 1 bar)
|
||||
const computedLength = Math.max(1.0, Math.ceil(maxEndBar));
|
||||
sectionContainer.length_bars = computedLength;
|
||||
|
||||
return computedLength;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.3 Algorithm 3: Piano Roll Grid Mapping & Quantization Math
|
||||
|
||||
##### Grid Coordinate Transformation Formulae
|
||||
|
||||
Let:
|
||||
|
||||
* $X_{\text{px}}$ = Pixel X-coordinate on Piano Roll Canvas.
|
||||
* $Y_{\text{px}}$ = Pixel Y-coordinate on Piano Roll Canvas.
|
||||
* $\text{Zoom}_x$ = Pixels per Beat.
|
||||
* $\text{NoteHeight}$ = Height in pixels per pitch key row (e.g., 18px).
|
||||
* $\text{Scroll}_x$ = Horizontal scroll offset in beats.
|
||||
* $\text{Scroll}_y$ = Vertical scroll top note pitch (e.g., pitch 127 down to 0).
|
||||
|
||||
Beat to Canvas Pixel Conversion:
|
||||
|
||||
$$X_{\text{px}} = (\text{Beat} - \text{Scroll}_x) \times \text{Zoom}_x$$
|
||||
|
||||
$$\text{Beat} = \frac{X_{\text{px}}}{\text{Zoom}_x} + \text{Scroll}_x$$
|
||||
|
||||
Pitch to Canvas Pixel Conversion:
|
||||
|
||||
$$Y_{\text{px}} = (127 - \text{Pitch} - \text{Scroll}_y) \times \text{NoteHeight}$$
|
||||
|
||||
$$\text{Pitch} = 127 - \left\lfloor \frac{Y_{\text{px}}}{\text{NoteHeight}} \right\rfloor - \text{Scroll}_y$$
|
||||
|
||||
##### Quantization (Snap To Grid) Math
|
||||
|
||||
Let $Q$ be the snap unit in beats (e.g., $1/4 \text{ bar} = 1.0 \text{ beat}$, $1/16 \text{ note} = 0.25 \text{ beat}$). Given raw unquantized beat $B_{\text{raw}}$:
|
||||
|
||||
$$B_{\text{quantized}} = \text{round}\left(\frac{B_{\text{raw}}}{Q}\right) \times Q$$
|
||||
|
||||
#### 5.4 Algorithm 4: Tab Close Dependency & Lifecycle Validation Algorithm
|
||||
|
||||
This algorithm validates whether a tab close request can be fulfilled, enforcing the fixed Main Session constraint and preventing parent Section tab closures while child editor sub-tabs remain active.
|
||||
|
||||
```javascript
|
||||
function requestCloseTab(tabIdToClose, stateStore) {
|
||||
const targetTab = stateStore.open_tabs.find(tab => tab.tab_id === tabIdToClose);
|
||||
if (!targetTab) {
|
||||
return { success: false, reason: "TAB_NOT_FOUND" };
|
||||
}
|
||||
|
||||
// 1. Rule: Main Session cannot be closed
|
||||
if (!targetTab.is_closeable || targetTab.type === 'MAIN_SESSION') {
|
||||
return { success: false, reason: "CANNOT_CLOSE_MAIN_SESSION" };
|
||||
}
|
||||
|
||||
// 2. Rule: Section Tab cannot be closed if child tabs are active
|
||||
if (targetTab.type === 'SECTION_TAB') {
|
||||
const activeChildTabs = stateStore.open_tabs.filter(
|
||||
tab => tab.parent_tab_id === targetTab.tab_id
|
||||
);
|
||||
|
||||
if (activeChildTabs.length > 0) {
|
||||
return {
|
||||
success: false,
|
||||
reason: "SECTION_HAS_ACTIVE_CHILD_EDITORS",
|
||||
activeChildTabs: activeChildTabs.map(t => ({ id: t.tab_id, title: t.title }))
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Execution: Perform clean tab shutdown and update active context
|
||||
const updatedTabs = stateStore.open_tabs.filter(tab => tab.tab_id !== tabIdToClose);
|
||||
|
||||
// Fallback active tab selection if current active tab is being closed
|
||||
let nextActiveTabId = stateStore.active_tab_id;
|
||||
if (stateStore.active_tab_id === tabIdToClose) {
|
||||
// Fallback to parent tab, or default to main session (index 0)
|
||||
nextActiveTabId = targetTab.parent_tab_id || updatedTabs[0].tab_id;
|
||||
}
|
||||
|
||||
stateStore.open_tabs = updatedTabs;
|
||||
stateStore.active_tab_id = nextActiveTabId;
|
||||
|
||||
return { success: true, nextActiveTabId: nextActiveTabId };
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.5 Algorithm 5: Sample-Accurate Lookahead MIDI & Audio Scheduler
|
||||
|
||||
Web Audio API timing operates on a high-precision hardware audio clock (`audioContext.currentTime`). JavaScript timers (`setTimeout`/`setInterval`) lack frame accuracy. The Lookahead Scheduler combines JS interval ticks with Web Audio precision scheduling.
|
||||
|
||||
```text
|
||||
Lookahead Window (e.g. 100ms)
|
||||
|-------------------------------------------|
|
||||
| AudioContext Time: 10.0s |
|
||||
| Schedule horizon: 10.1s |
|
||||
| |
|
||||
| [Event 1 @ 10.02s] -> Scheduled in WebAudio
|
||||
| [Event 2 @ 10.08s] -> Scheduled in WebAudio
|
||||
|___________________________________________|
|
||||
|
||||
```
|
||||
|
||||
##### Scheduler Specification
|
||||
|
||||
```javascript
|
||||
class PrecisionAudioScheduler {
|
||||
constructor(audioCtx, lookaheadMs = 25.0, scheduleAheadTimeSec = 0.1) {
|
||||
this.audioCtx = audioCtx;
|
||||
this.lookaheadMs = lookaheadMs; // Frequency of timer evaluation
|
||||
this.scheduleAheadTime = scheduleAheadTimeSec; // How far ahead to queue WebAudio events
|
||||
this.nextNoteBeat = 0.0;
|
||||
this.currentBeat = 0.0;
|
||||
this.bpm = 120.0;
|
||||
this.timerId = null;
|
||||
}
|
||||
|
||||
beatToTime(beat) {
|
||||
const secondsPerBeat = 60.0 / this.bpm;
|
||||
return beat * secondsPerBeat;
|
||||
}
|
||||
|
||||
timeToBeat(timeSec) {
|
||||
const secondsPerBeat = 60.0 / this.bpm;
|
||||
return timeSec / secondsPerBeat;
|
||||
}
|
||||
|
||||
schedulerTick(activeSession) {
|
||||
const currentTime = this.audioCtx.currentTime;
|
||||
const horizonTime = currentTime + this.scheduleAheadTime;
|
||||
|
||||
// Traverse session items and find notes falling within [currentTime, horizonTime]
|
||||
const pendingEvents = activeSession.getEventsInTimeRange(
|
||||
this.timeToBeat(currentTime),
|
||||
this.timeToBeat(horizonTime)
|
||||
);
|
||||
|
||||
for (const evt of pendingEvents) {
|
||||
if (!evt.scheduled) {
|
||||
const preciseAudioTime = currentTime + this.beatToTime(evt.targetBeat - this.currentBeat);
|
||||
this.triggerWebAudioEvent(evt, preciseAudioTime);
|
||||
evt.scheduled = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
triggerWebAudioEvent(evt, exactAudioTime) {
|
||||
if (evt.type === 'MIDI_NOTE_ON') {
|
||||
const synthNode = evt.trackSynthNode;
|
||||
synthNode.noteOn(evt.note.pitch, evt.note.velocity, exactAudioTime);
|
||||
synthNode.noteOff(evt.note.pitch, exactAudioTime + this.beatToTime(evt.note.duration_beats));
|
||||
} else if (evt.type === 'AUDIO_CLIP') {
|
||||
const sourceNode = this.audioCtx.createBufferSource();
|
||||
sourceNode.buffer = evt.audioBuffer;
|
||||
sourceNode.connect(evt.trackGainNode);
|
||||
sourceNode.start(exactAudioTime, evt.offsetSec, evt.durationSec);
|
||||
}
|
||||
}
|
||||
|
||||
start(session) {
|
||||
this.timerId = setInterval(() => this.schedulerTick(session), this.lookaheadMs);
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (this.timerId) clearInterval(this.timerId);
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### 5.6 Algorithm 6: Playhead UI Rendering Sync Loop
|
||||
|
||||
UI Playhead rendering uses `requestAnimationFrame` and queries `audioContext.currentTime` directly to prevent visual jitter or lag.
|
||||
|
||||
$$\text{Current Beat UI} = \frac{\text{audioCtx.currentTime} - \text{PlaybackStartTimeSec}}{\text{SecondsPerBeat}}$$
|
||||
|
||||
$$\text{Pixel Position X} = (\text{Current Beat UI} - \text{ViewportStartBeat}) \times \text{Zoom}_x$$
|
||||
|
||||
---
|
||||
|
||||
### 6. Backend Python Server Architecture & Offline Render Spec
|
||||
|
||||
#### 6.1 Server Architecture Framework
|
||||
|
||||
* **Framework:** FastAPI with Async WebSocket endpoints for real-time state synchronization.
|
||||
* **DSP Engine:** `pedalboard` (Spotify's Python Audio Processing Library) and `numpy` for multi-track mixing, high-quality audio resampling, and plugin hosting.
|
||||
|
||||
#### 6.2 Python Offline Stem Bouncing Engine Specification (`render_engine.py`)
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
from pedalboard import Pedalboard, Gain, Reverb, Compressor
|
||||
import soundfile as sf
|
||||
|
||||
class PythonRenderEngine:
|
||||
def __init__(self, sample_rate=44100):
|
||||
self.sample_rate = sample_rate
|
||||
|
||||
def bars_to_samples(self, bars: float, bpm: float, time_sig_num: int) -> int:
|
||||
seconds_per_beat = 60.0 / bpm
|
||||
seconds_per_bar = seconds_per_beat * time_sig_num
|
||||
return int(bars * seconds_per_bar * self.sample_rate)
|
||||
|
||||
def render_project(self, project_json: dict, output_filepath: str):
|
||||
bpm = project_json["metadata"]["bpm"]
|
||||
time_sig_num = project_json["metadata"]["time_signature_numerator"]
|
||||
main_session = project_json["main_session"]
|
||||
|
||||
# 1. Compute total project samples
|
||||
total_bars = main_session.get("length_bars", 16.0)
|
||||
total_samples = self.bars_to_samples(total_bars, bpm, time_sig_num)
|
||||
|
||||
# Stereo Master Buffer
|
||||
master_buffer = np.zeros((2, total_samples), dtype=np.float32)
|
||||
|
||||
# 2. Iterate and process main tracks
|
||||
for track in main_session["tracks"]:
|
||||
track_type = track["type"]
|
||||
track_buffer = np.zeros((2, total_samples), dtype=np.float32)
|
||||
|
||||
for item in track["items"]:
|
||||
start_sample = self.bars_to_samples(item["start_bar"], bpm, time_sig_num)
|
||||
dur_samples = self.bars_to_samples(item["duration_bars"], bpm, time_sig_num)
|
||||
offset_sample = self.bars_to_samples(item["clip_start_offset_bars"], bpm, time_sig_num)
|
||||
|
||||
if item["type"] == "AUDIO_ITEM":
|
||||
# Load audio source sample array
|
||||
audio_data, sr = sf.read(item["source_data"]["audio_file_url"], dtype='float32')
|
||||
audio_data = audio_data.T # Shape: (channels, samples)
|
||||
|
||||
# Apply non-destructive trimming offset
|
||||
sliced_audio = audio_data[:, offset_sample : offset_sample + dur_samples]
|
||||
|
||||
# Accumulate into track buffer with bounds checks
|
||||
end_sample = min(start_sample + sliced_audio.shape[1], total_samples)
|
||||
actual_len = end_sample - start_sample
|
||||
track_buffer[:, start_sample:end_sample] += sliced_audio[:, :actual_len]
|
||||
|
||||
# Apply Track Gain and FX Chain via Pedalboard
|
||||
board = Pedalboard([Gain(gain_db=track.get("volume_db", 0.0))])
|
||||
processed_track = board(track_buffer, sample_rate=self.sample_rate)
|
||||
|
||||
# Mix down to Master
|
||||
master_buffer += processed_track
|
||||
|
||||
# 3. Write final output file
|
||||
sf.write(output_filepath, master_buffer.T, self.sample_rate)
|
||||
return output_filepath
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Execution Context & Sub-Tab Lifecycle Matrix
|
||||
|
||||
| Context Tab Type | Scope Identifier | View Boundaries | Is Closeable | Close Dependency Conditions | Audio Routing Target |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **MAIN SESSION** | Root | Full Master Timeline ($0 \to N$ Bars) | No | Pinned permanently; cannot be closed | WebAudio Hardware Destination |
|
||||
| **SECTION TAB** | Section_ID | Dynamic Section Bounds ($0 \to L_{\text{section}}$) | Yes | Blocked if any child editor sub-tabs are open | Target Section Bus Gain Node |
|
||||
| **PIANO ROLL** | MIDIItem_ID | Item Source Length Bounds ($0 \to N_{\text{buffer}}$) | Yes | Can close freely; notifies parent Section tab | Track Instrument Synth Engine |
|
||||
| **SAMPLE EDITOR** | AudioItem_ID | Sample Buffer Waveform ($0 \to T_{\text{sample}}$) | Yes | Can close freely; notifies parent Section tab | Track Audio Node Router |
|
||||
|
||||
---
|
||||
|
||||
### 8. Summary of Non-Destructive Slice & Tab Lifecycle Validation
|
||||
|
||||
* **Tab Close Prevention Test:**
|
||||
1. `MAIN SESSION` close request is rejected immediately (`CANNOT_CLOSE_MAIN_SESSION`).
|
||||
2. `Section_01` tab has an active child editor tab (`Piano Roll: Bassline`).
|
||||
3. Request to close `Section_01` tab returns `SECTION_HAS_ACTIVE_CHILD_EDITORS`.
|
||||
4. User closes `Piano Roll: Bassline` tab first.
|
||||
5. Subsequent close request for `Section_01` succeeds and cleans up UI context.
|
||||
|
||||
|
||||
* **8-Bar Source with 2-Bar Visible Crop Test:**
|
||||
1. Given `MIDIItem` length = 8 bars ($0 \dots 8$).
|
||||
2. User drags left boundary to Bar 4 and right boundary to Bar 6.
|
||||
3. `start_bar = 4.0` (Global Session Placement), `duration_bars = 2.0`, `clip_start_offset_bars = 4.0`.
|
||||
4. Transport reaches global Bar 4.0 $\to$ scheduler evaluates internal bounds $[4.0, 6.0)$ and triggers only visible notes while preserving complete 8-bar non-destructive source.
|
||||
@@ -0,0 +1,426 @@
|
||||
# TECHNICAL SPECIFICATION: CLIENT-SIDE REAL-TIME RECORDING ENGINE
|
||||
|
||||
## Browser-Based Microphone & Hardware MIDI Keyboard Recording Module
|
||||
|
||||
---
|
||||
|
||||
## 1. System Overview
|
||||
|
||||
The Client-side recording module enables the DAW to capture live audio signals directly from Microphone/Line-in interfaces (via the Web MediaDevices API) and keypress events from Hardware MIDI Keyboards/Controllers (via the Web MIDI API) in real time. The module operates with low latency and includes hardware latency compensation.
|
||||
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| CLIENT BROWSER |
|
||||
| |
|
||||
| +-------------------------+ +-----------------------------+ |
|
||||
| | Hardware MIDI Keyboard | | Live Microphone / Line-In | |
|
||||
| +------------+------------+ +--------------+--------------+ |
|
||||
| | | |
|
||||
| v (Web MIDI API) v (getUserMedia) |
|
||||
| +------------+------------+ +--------------+--------------+ |
|
||||
| | Web MIDI Input Handler | | MediaStreamAudioSourceNode | |
|
||||
| +------------+------------+ +--------------+--------------+ |
|
||||
| | | |
|
||||
| +------------------+ | |
|
||||
| | | v |
|
||||
| v v +--------------+--------------+ |
|
||||
| +------------+-----+ +---------+-----------+ | Track Input Gain Node | |
|
||||
| | Event Clock / | | WebAudio Virtual | +--------------+--------------+ |
|
||||
| | Latency Engine | | Synth Engine | | |
|
||||
| +------------+-----+ +---------+-----------+ +--------+--------+ |
|
||||
| | | | | |
|
||||
| v v v v |
|
||||
| +------------+-----+ (Live Sound) +-------+-------+ +-------+------+ |
|
||||
| | Recorded MIDI | | AudioWorklet | | Monitoring | |
|
||||
| | Buffer | | Ring-Buffer | | Switch | |
|
||||
| +------------+-----+ | Recorder | +-------+------+ |
|
||||
| | +-------+-------+ | |
|
||||
| v | v |
|
||||
| [ Timeline MIDI ] v [ Master Mix ] |
|
||||
| [ Item Creation ] +-------+-------+ |
|
||||
| | Float32 PCM | |
|
||||
| | Audio Buffer | |
|
||||
| +-------+-------+ |
|
||||
| | |
|
||||
| v |
|
||||
| [ Timeline Audio] |
|
||||
| [ Item Creation ] |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Hardware I/O & API Contracts
|
||||
|
||||
### 2.1 MediaDevices (Microphone Capture)
|
||||
|
||||
**Permission Request:** Uses `navigator.mediaDevices.getUserMedia` configured to disable automatic browser processing DSP algorithms to capture pure, unprocessed audio signals:
|
||||
|
||||
```javascript
|
||||
const audioConstraints = {
|
||||
audio: {
|
||||
deviceId: selectedDeviceId ? { exact: selectedDeviceId } : undefined,
|
||||
echoCancellation: false, // Disables echo cancellation to prevent instrument sound distortion
|
||||
noiseSuppression: false, // Disables automatic noise suppression to preserve full frequency range
|
||||
autoGainControl: false, // Disables Automatic Gain Control (AGC)
|
||||
latency: 0 // Requests minimal latency from OS audio driver
|
||||
}
|
||||
};
|
||||
|
||||
```
|
||||
|
||||
### 2.2 Web MIDI API Integration
|
||||
|
||||
**Device Enumeration & Listener Assignment:**
|
||||
|
||||
* Uses `navigator.requestMIDIAccess({ sysex: false })` to scan for USB-connected keyboard devices.
|
||||
* **Timestamp Precision:** Obtains event timestamps from `MIDIMessageEvent.timeStamp` (as a `DOMHighResTimeStamp` in microseconds) and synchronizes them with `AudioContext.currentTime`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recording Lifecycle & State Machine
|
||||
|
||||
```text
|
||||
[IDLE] ───► (User Arms Track) ───► [ARMED] ───► (Press Rec + Play) ───► [COUNT-IN / PRE-ROLL]
|
||||
|
|
||||
[STOP & COMMIT] ◄─── (Press Stop) ◄─── [RECORDING IN PROGRESS] ◄───────────────+
|
||||
|
||||
```
|
||||
|
||||
* **Arming Phase (Record Enable):**
|
||||
* The user selects an input source and activates the Arm (R) button on the target track.
|
||||
* Initializes the input level meter (VU Meter Canvas) to display input volume levels in real time.
|
||||
|
||||
|
||||
* **Pre-Roll / Count-In Phase:**
|
||||
* Transport triggers the metronome count-in (e.g., 1 Bar = 4 beats). The metronome plays click sounds based on project BPM.
|
||||
* The engine does not write data to the Timeline yet, but begins reading the input buffer to prepare memory buffers.
|
||||
|
||||
|
||||
* **Recording Phase:**
|
||||
* Once the transport passes the Start Bar boundary, incoming MIDI key events or PCM Float32 audio samples are written into the active recording buffer memory.
|
||||
* Canvas UI displays real-time visual feedback, rendering waveforms or MIDI note blocks dynamically.
|
||||
|
||||
|
||||
* **Stop & Commit Phase:**
|
||||
* Pressing Stop halts the recording process.
|
||||
* Converts temporary memory buffers into a structured `MIDIItem` or `AudioItem`.
|
||||
* Inserts the new Item onto the target track within the Main Session or Section tab.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Structures
|
||||
|
||||
### 4.1 Live MIDI Event Buffer Element Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pitch": { "type": "integer", "minimum": 0, "maximum": 127 },
|
||||
"start_beat": { "type": "number", "description": "Start position in beats on the timeline" },
|
||||
"duration_beats": { "type": "number", "description": "Keypress duration in beats" },
|
||||
"velocity": { "type": "number", "minimum": 0.0, "maximum": 1.0 },
|
||||
"channel": { "type": "integer", "default": 0 }
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### 4.2 Recording Track Input Configuration State
|
||||
|
||||
```json
|
||||
{
|
||||
"track_id": "track_midi_01",
|
||||
"is_armed": true,
|
||||
"monitoring_enabled": true,
|
||||
"input_source": {
|
||||
"device_type": "MIDI_KEYBOARD",
|
||||
"device_id": "midi_input_usb_keyboard_0",
|
||||
"channel": 1
|
||||
},
|
||||
"input_gain_db": 0.0,
|
||||
"latency_offset_ms": 12.5
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Core Algorithms & Latency Compensation
|
||||
|
||||
### 5.1 Algorithm 1: Hardware Latency Compensation Formula
|
||||
|
||||
When recording, the physical moment a key is pressed or sound enters the microphone is inherently delayed relative to speaker output due to input buffers ($L_{\text{input}}$) and output buffers ($L_{\text{output}}$).
|
||||
|
||||
#### Mathematical Formulation
|
||||
|
||||
Let:
|
||||
|
||||
* $T_{\text{audio\_ctx}}$ = Current timestamp in seconds on the `AudioContext` clock (`audioCtx.currentTime`).
|
||||
* $T_{\text{rec\_start}}$ = Recording start timestamp in seconds.
|
||||
* $\text{BPM}$ = Song tempo (Beats Per Minute).
|
||||
* $\text{TS}_{\text{num}}$ = Time Signature Numerator (beats per bar).
|
||||
* $\text{Bar}_{\text{start}}$ = Target timeline start bar for recording.
|
||||
* $L_{\text{comp}}$ = Total hardware latency offset ($L_{\text{input}} + L_{\text{output}} + L_{\text{user\_offset}}$) in seconds.
|
||||
|
||||
**Actual Elapsed Audio Time ($T_{\text{elapsed}}$):**
|
||||
|
||||
$$T_{\text{elapsed}} = \max\left(0, T_{\text{audio\_ctx}} - T_{\text{rec\_start}} - L_{\text{comp}}\right)$$
|
||||
|
||||
**Audio Time to Beat Conversion ($\text{Beat}_{\text{current}}$):**
|
||||
|
||||
$$\text{SecondsPerBeat} = \frac{60.0}{\text{BPM}}$$
|
||||
|
||||
$$\text{Beat}_{\text{current}} = \frac{T_{\text{elapsed}}}{\text{SecondsPerBeat}} + \left(\text{Bar}_{\text{start}} \times \text{TS}_{\text{num}}\right)$$
|
||||
|
||||
**Timeline Placement Mapping:**
|
||||
|
||||
$$\text{StartBeat}_{\text{item}} = \text{Beat}_{\text{current}}$$
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Algorithm 2: AudioWorklet PCM Ring-Buffer Processor
|
||||
|
||||
To prevent audio glitches or missing PCM frames when the browser's main thread is processing heavy UI renders, microphone recording runs inside an `AudioWorkletProcessor`:
|
||||
|
||||
```javascript
|
||||
// public/processors/pcm-recorder-processor.js
|
||||
class PCMRecorderProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.bufferSize = 4096;
|
||||
this.buffer = new Float32Array(this.bufferSize);
|
||||
this.bufferIndex = 0;
|
||||
}
|
||||
|
||||
process(inputs, outputs, parameters) {
|
||||
const input = inputs[0];
|
||||
if (input && input.length > 0) {
|
||||
const inputChannel = input[0]; // Mono Channel 0
|
||||
|
||||
for (let i = 0; i < inputChannel.length; i++) {
|
||||
this.buffer[this.bufferIndex++] = inputChannel[i];
|
||||
|
||||
// When Ring-Buffer fills, send Float32Array to Main Thread
|
||||
if (this.bufferIndex >= this.bufferSize) {
|
||||
this.port.postMessage({
|
||||
type: 'PCM_DATA',
|
||||
buffer: this.buffer.slice(0, this.bufferSize)
|
||||
});
|
||||
this.bufferIndex = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
return true; // Keep worklet active
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor('pcm-recorder-processor', PCMRecorderProcessor);
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5.3 Algorithm 3: Client MIDIRecorder Class Implementation
|
||||
|
||||
```javascript
|
||||
class ClientMIDIRecorder {
|
||||
constructor(audioContext, bpm = 120, timeSigNumerator = 4) {
|
||||
this.audioCtx = audioContext;
|
||||
this.bpm = bpm;
|
||||
this.timeSigNum = timeSigNumerator;
|
||||
this.isRecording = false;
|
||||
|
||||
this.activeNotes = new Map(); // Store pitch -> { noteId, startBeat, velocity }
|
||||
this.recordedNotes = [];
|
||||
this.recStartAudioTime = 0.0;
|
||||
this.recStartBar = 0.0;
|
||||
|
||||
// Compute round-trip browser latency
|
||||
this.latencyCompSec = (this.audioCtx.baseLatency || 0) + (this.audioCtx.outputLatency || 0);
|
||||
}
|
||||
|
||||
start(startBar = 0.0) {
|
||||
this.isRecording = true;
|
||||
this.recordedNotes = [];
|
||||
this.activeNotes.clear();
|
||||
this.recStartBar = startBar;
|
||||
this.recStartAudioTime = this.audioCtx.currentTime;
|
||||
|
||||
this.bindMIDIInputs();
|
||||
}
|
||||
|
||||
bindMIDIInputs() {
|
||||
if (navigator.requestMIDIAccess) {
|
||||
navigator.requestMIDIAccess().then(midiAccess => {
|
||||
for (let input of midiAccess.inputs.values()) {
|
||||
input.onmidimessage = (event) => this.handleMIDIMessage(event);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
handleMIDIMessage(event) {
|
||||
if (!this.isRecording) return;
|
||||
|
||||
const [status, pitch, velocity] = event.data;
|
||||
const command = status >> 4;
|
||||
|
||||
// Apply latency compensation formula
|
||||
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
|
||||
const secondsPerBeat = 60.0 / this.bpm;
|
||||
const currentBeat = (currentTimeSec / secondsPerBeat) + (this.recStartBar * this.timeSigNum);
|
||||
|
||||
// Command 0x9: Note On
|
||||
if (command === 0x9 && velocity > 0) {
|
||||
const noteId = `rec_${Date.now()}_${pitch}`;
|
||||
this.activeNotes.set(pitch, {
|
||||
id: noteId,
|
||||
pitch: pitch,
|
||||
start_beat: currentBeat,
|
||||
velocity: velocity / 127.0
|
||||
});
|
||||
}
|
||||
// Command 0x8: Note Off (or Note On with velocity = 0)
|
||||
else if (command === 0x8 || (command === 0x9 && velocity === 0)) {
|
||||
if (this.activeNotes.has(pitch)) {
|
||||
const note = this.activeNotes.get(pitch);
|
||||
const durationBeats = Math.max(0.125, currentBeat - note.start_beat); // Min 1/32 note
|
||||
|
||||
this.recordedNotes.push({
|
||||
id: note.id,
|
||||
pitch: note.pitch,
|
||||
start_beat: note.start_beat,
|
||||
duration_beats: durationBeats,
|
||||
velocity: note.velocity,
|
||||
pan: 0.0
|
||||
});
|
||||
|
||||
this.activeNotes.delete(pitch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
stop() {
|
||||
this.isRecording = false;
|
||||
|
||||
// Flush remaining active keypresses when stop is triggered
|
||||
const currentTimeSec = Math.max(0, this.audioCtx.currentTime - this.recStartAudioTime - this.latencyCompSec);
|
||||
const currentBeat = (currentTimeSec / (60.0 / this.bpm)) + (this.recStartBar * this.timeSigNum);
|
||||
|
||||
for (let [pitch, note] of this.activeNotes.entries()) {
|
||||
this.recordedNotes.push({
|
||||
id: note.id,
|
||||
pitch: note.pitch,
|
||||
start_beat: note.start_beat,
|
||||
duration_beats: Math.max(0.25, currentBeat - note.start_beat),
|
||||
velocity: note.velocity,
|
||||
pan: 0.0
|
||||
});
|
||||
}
|
||||
this.activeNotes.clear();
|
||||
return this.recordedNotes;
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5.4 Algorithm 4: Client AudioRecorder & AudioBuffer Splicing Class Implementation
|
||||
|
||||
```javascript
|
||||
class ClientAudioRecorder {
|
||||
constructor(audioContext) {
|
||||
this.audioCtx = audioContext;
|
||||
this.mediaStream = null;
|
||||
this.sourceNode = null;
|
||||
this.workletNode = null;
|
||||
this.pcmChunks = [];
|
||||
this.isRecording = false;
|
||||
}
|
||||
|
||||
async initializeInput(deviceId = null) {
|
||||
const constraints = {
|
||||
audio: {
|
||||
deviceId: deviceId ? { exact: deviceId } : undefined,
|
||||
echoCancellation: false,
|
||||
noiseSuppression: false,
|
||||
autoGainControl: false
|
||||
}
|
||||
};
|
||||
this.mediaStream = await navigator.mediaDevices.getUserMedia(constraints);
|
||||
this.sourceNode = this.audioCtx.createMediaStreamSource(this.mediaStream);
|
||||
}
|
||||
|
||||
async start(destinationTrackGainNode, enableMonitoring = true) {
|
||||
this.pcmChunks = [];
|
||||
this.isRecording = true;
|
||||
|
||||
// Load Worklet Processor Module
|
||||
await this.audioCtx.audioWorklet.addModule('/processors/pcm-recorder-processor.js');
|
||||
this.workletNode = new AudioWorkletNode(this.audioCtx, 'pcm-recorder-processor');
|
||||
|
||||
// Receive PCM data streams from AudioWorklet
|
||||
this.workletNode.port.onmessage = (event) => {
|
||||
if (this.isRecording && event.data.type === 'PCM_DATA') {
|
||||
this.pcmChunks.push(new Float32Array(event.data.buffer));
|
||||
}
|
||||
};
|
||||
|
||||
// Route Audio Nodes
|
||||
this.sourceNode.connect(this.workletNode);
|
||||
|
||||
// Enable Live Input Monitoring if requested
|
||||
if (enableMonitoring) {
|
||||
this.sourceNode.connect(destinationTrackGainNode);
|
||||
}
|
||||
}
|
||||
|
||||
async stop() {
|
||||
this.isRecording = false;
|
||||
|
||||
if (this.sourceNode && this.workletNode) {
|
||||
this.sourceNode.disconnect(this.workletNode);
|
||||
}
|
||||
|
||||
// Concatenate PCM Float32Array chunks into a single AudioBuffer
|
||||
const totalSamples = this.pcmChunks.reduce((sum, chunk) => sum + chunk.length, 0);
|
||||
if (totalSamples === 0) return null;
|
||||
|
||||
const audioBuffer = this.audioCtx.createBuffer(1, totalSamples, this.audioCtx.sampleRate);
|
||||
const channelData = audioBuffer.getChannelData(0);
|
||||
|
||||
let offset = 0;
|
||||
for (const chunk of this.pcmChunks) {
|
||||
channelData.set(chunk, offset);
|
||||
offset += chunk.length;
|
||||
}
|
||||
|
||||
return audioBuffer; // Return compiled AudioBuffer for timeline insertion
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. UI Components & User Interactions
|
||||
|
||||
* **Track Header Arming Controls:**
|
||||
* **[R] Button (Arm Track):** Highlights red when armed for recording on the target track.
|
||||
* **[I] Button (Input Monitor):** Toggles live monitoring for incoming Microphone or Synth audio during performance.
|
||||
* **Input Selector Dropdown:** Allows selection of available Microphone devices or USB Hardware MIDI Keyboards.
|
||||
|
||||
|
||||
* **Real-time VU Meter Component:**
|
||||
* Displays input signal gain level from $-60\text{ dB}$ to $0\text{ dB}$. Displays red clipping indicators when signal levels exceed $0\text{ dBFS}$.
|
||||
|
||||
|
||||
* **Live Waveform & MIDI Preview Rendering:**
|
||||
* **Microphone Recording:** The canvas UI renders incoming waveform signals progressing along the Playhead position in real time.
|
||||
* **MIDI Performance:** Rectangular note blocks (green/orange) appear at note-on trigger events and extend until key release (note-off).
|
||||
@@ -0,0 +1,237 @@
|
||||
Here is the clean, nicely formatted Markdown version of the technical specification document:
|
||||
|
||||
# TECHNICAL INSTALLATION & INTEGRATION GUIDE FOR SOUNDFONT / VSTI IN DAW
|
||||
|
||||
This document provides a detailed technical architecture model for integrating SoundFonts, WebAssembly Plugins (Client), and Native VSTi/AU (Server). It clearly delineates components pre-installed by the Developer (Coder) versus those open for User uploads and additions.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architectural Distribution Overview (Developer vs. User)
|
||||
|
||||
| Plugin / Asset Category | Processing Location | Installed By | Storage & Management Method | Security & Safety Profile |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Default SoundFont (`.sf2`)** | Client (Wasm) | Coder | Static Assets hosted on Web Server / CDN | Extremely High |
|
||||
| **User Custom SoundFont (`.sf2`)** | Client (Wasm) | User | Browser `IndexedDB` or User Cloud Storage | Extremely High (Runs inside Wasm Sandbox) |
|
||||
| **WebAssembly Synths (WAMs)** | Client (JS/Wasm) | Coder | Bundled within Frontend Source Code | Extremely High |
|
||||
| **Core Server VSTi (Vital, Surge...)** | Server (Python) | Coder | System Directory inside Docker/Linux Container | High (Controlled binary footprint) |
|
||||
| **User Custom VST3 / Preset** | Server (Python) | User (Restricted) | Stores `.vst3` files or `.fxp`/`.json` on Container | High Security Risk (Requires Sandboxing) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Client-Side Integration Tech (Browser / WebAssembly)
|
||||
|
||||
The Client-Side handles zero-latency real-time composition and audio previews.
|
||||
|
||||
### 2.1 Coder Pre-bundled Assets
|
||||
|
||||
* **Static SoundFont Hosting:**
|
||||
* The developer places standard `.sf2` files (such as `GeneralUser_GS.sf2`) into the `public/soundfonts/` directory or hosts them via CDN.
|
||||
* Upon application startup, default SoundFonts are queried via REST API:
|
||||
```http
|
||||
GET /api/v1/assets/default-soundfonts
|
||||
|
||||
```
|
||||
|
||||
|
||||
```json
|
||||
[
|
||||
{ "id": "sf_generaluser", "name": "GeneralUser GS v1.471", "size_mb": 31.2, "url": "/soundfonts/GeneralUser.sf2" },
|
||||
{ "id": "sf_sso", "name": "Sonatina Symphonic Orchestra", "size_mb": 95.0, "url": "/soundfonts/SSO.sf2" }
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
* **FluidSynth WebAssembly Engine Integration:**
|
||||
* Compiles FluidSynth C/C++ code into WebAssembly (`fluidsynth.wasm` + `fluidsynth.js`) using Emscripten.
|
||||
* Alternatively, leverages open JavaScript wrappers such as `@soundfont/player` or `SpessaSynth`.
|
||||
|
||||
|
||||
|
||||
### 2.2 Allowing User Custom SoundFont (`.sf2`) Uploads
|
||||
|
||||
Delivers a flexible user experience without overloading server storage:
|
||||
|
||||
* **Upload Mechanism & Local Cache (`IndexedDB`):**
|
||||
* Users drag and drop `.sf2` files directly into the DAW interface.
|
||||
* JavaScript reads the file as an `ArrayBuffer` via the `FileReader` API.
|
||||
* The file persists directly within the browser's local `IndexedDB` cache for immediate reuse across sessions without re-uploading to the server.
|
||||
|
||||
|
||||
* **Dynamic Injection into WebAssembly Memory:**
|
||||
```javascript
|
||||
// Client-side JavaScript snippet
|
||||
async function loadUserSoundFont(fileBuffer) {
|
||||
const uint8Array = new Uint8Array(fileBuffer);
|
||||
// Write buffer straight into Emscripten FluidSynth Virtual File System (MEMFS)
|
||||
Module.FS.writeFile('/user_font.sf2', uint8Array);
|
||||
|
||||
// Call Wasm C-function to load bank
|
||||
const sfont_id = Module._fluid_synth_sfload(synthInstance, '/user_font.sf2', 1);
|
||||
console.log(`User SoundFont loaded successfully with ID: ${sfont_id}`);
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Server-Side Integration Tech (Python Backend Engine)
|
||||
|
||||
The Server-Side executes high-resolution offline WAV rendering when an operator triggers the Export / Bounce workflow.
|
||||
|
||||
### 3.1 Server Environment Installed by Coder
|
||||
|
||||
The developer configures the Server environment (or Docker Container) with pre-installed Native C++ libraries and Python utilities.
|
||||
|
||||
1. **Server Base `Dockerfile` Configuration:**
|
||||
```dockerfile
|
||||
FROM python:3.10-slim
|
||||
|
||||
# Install Linux audio libraries
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fluidsynth \
|
||||
libfluidsynth-dev \
|
||||
libasound2-dev \
|
||||
libjack-jackd2-dev \
|
||||
build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Initialize directories for Native VST3 and system SoundFonts
|
||||
RUN mkdir -p /opt/daw_engine/vst3 \
|
||||
&& mkdir -p /opt/daw_engine/soundfonts
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
```
|
||||
|
||||
|
||||
2. **Pre-installing Native VST3 Plugins:**
|
||||
Places 64-bit Linux `.vst3` binary builds of open-source synths inside `/opt/daw_engine/vst3/`:
|
||||
* `/opt/daw_engine/vst3/Vital.vst3`
|
||||
* `/opt/daw_engine/vst3/Surge XT.vst3`
|
||||
* `/opt/daw_engine/vst3/Dexed.vst3`
|
||||
|
||||
|
||||
3. **Python Backend Integration via Spotify `pedalboard`:**
|
||||
```python
|
||||
# render_engine/vst_loader.py
|
||||
import os
|
||||
from pedalboard import VST3Plugin, Pedalboard
|
||||
|
||||
class PluginManager:
|
||||
def __init__(self, vst_dir="/opt/daw_engine/vst3"):
|
||||
self.vst_dir = vst_dir
|
||||
self.available_plugins = self._scan_plugins()
|
||||
|
||||
def _scan_plugins(self):
|
||||
plugins = {}
|
||||
for root, dirs, files in os.walk(self.vst_dir):
|
||||
for file in files:
|
||||
if file.endswith(".vst3") or file.endswith(".so"):
|
||||
plugin_path = os.path.join(root, file)
|
||||
plugin_name = os.path.splitext(file)[0]
|
||||
plugins[plugin_name] = plugin_path
|
||||
return plugins
|
||||
|
||||
def load_vst(self, plugin_name: str, preset_data: dict = None) -> VST3Plugin:
|
||||
if plugin_name not in self.available_plugins:
|
||||
raise FileNotFoundError(f"VST3 Plugin '{plugin_name}' not found on server.")
|
||||
|
||||
path = self.available_plugins[plugin_name]
|
||||
vst_instance = VST3Plugin(path)
|
||||
|
||||
# Inject parameters if provided
|
||||
if preset_data:
|
||||
for param_name, param_value in preset_data.items():
|
||||
setattr(vst_instance, param_name, param_value)
|
||||
|
||||
return vst_instance
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 3.2 Handling User Custom Plugins / Presets
|
||||
|
||||
#### Option 1: User Presets / Patches Uploads (**RECOMMENDED - Safe**)
|
||||
|
||||
* **Implementation:** The backend locks native VST3 installations to common open engines (Vital, Dexed, Surge XT). Users upload lightweight preset patches like `.vitalbank`, `.syx` (DX7 patches), `.fxp`, or JSON parameter states.
|
||||
* **Workflow:**
|
||||
1. User selects the Vital Synth on the Client UI.
|
||||
2. User clicks "Import Preset" $\rightarrow$ Uploads a `.vital` file or JSON parameter bundle.
|
||||
3. Server parses JSON parameters and injects them directly into the VST3 instance via `pedalboard` during render execution.
|
||||
|
||||
|
||||
* **Benefits:** Absolutely safe, minimal footprint, zero security vulnerabilities to the host infrastructure.
|
||||
|
||||
#### Option 2: User Native Binary VST3 Uploads (**HIGH RISK - Requires Isolation**)
|
||||
|
||||
* **Risk:** A `.vst3` file contains executable machine code (`.so` Shared Object on Linux). Accepting arbitrary uploads grants 100% vector exposure to Remote Code Execution (RCE) attacks.
|
||||
* **Technical Mitigation (If Mandatory):**
|
||||
* **Sandboxing Isolation:** Every user Export/Render request runs inside an isolated, short-lived container (Ephemeral Docker / Firejail / gVisor) stripped of `root` privileges and completely isolated from external internet interfaces.
|
||||
* **Time-To-Live (TTL):** User `.vst3` binaries persist inside temporary directories `/tmp/user_sessions/{user_id}/` and purge automatically upon render job completion.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4. API Specification for SoundFonts & Plugins
|
||||
|
||||
### 4.1 OpenAPI Endpoint Spec for Frontend
|
||||
|
||||
```yaml
|
||||
/api/v1/plugins/available:
|
||||
get:
|
||||
summary: Query available VSTi engines and SoundFont resources on the Server
|
||||
responses:
|
||||
200:
|
||||
content:
|
||||
application/json:
|
||||
example:
|
||||
vst_instruments:
|
||||
- id: "vst_vital"
|
||||
name: "Vital Wavetable Synth"
|
||||
type: "VST3"
|
||||
has_native_support: true
|
||||
- id: "vst_dexed"
|
||||
name: "Dexed FM Synth"
|
||||
type: "VST3"
|
||||
has_native_support: true
|
||||
soundfonts:
|
||||
- id: "sf_generaluser"
|
||||
name: "GeneralUser GS"
|
||||
file: "GeneralUser.sf2"
|
||||
|
||||
/api/v1/projects/render:
|
||||
post:
|
||||
summary: Trigger offline DAW Project rendering to WAV on the Server
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ProjectSchema'
|
||||
responses:
|
||||
200:
|
||||
description: Returns the URL pointing to the rendered WAV file
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Development Team Best Practices Summary
|
||||
|
||||
* **SoundFont (`.sf2`):**
|
||||
* **For Users:** Encourage unrestricted local uploads on the Client (Browser). Store assets in `IndexedDB` to ensure optimal real-time performance without straining server resources.
|
||||
* **For Developers:** Supply 1–2 default General MIDI (GM) SoundFont banks (`GeneralUser_GS.sf2`) bundled on both Client and Server.
|
||||
|
||||
|
||||
* **VSTi Instruments:**
|
||||
* **For Developers:** Pre-install top open-source Linux-native synths on the Server (Vital, Surge XT, Dexed, OB-Xd).
|
||||
* **For Users:** Do **not** allow direct `.vst3` binary uploads to the production server. Instead, permit users to upload Presets / Patches / JSON parameters for the supported synth models. This guarantees 100% security while saving storage and network bandwidth.
|
||||
@@ -0,0 +1,367 @@
|
||||
# TECHNICAL SPECIFICATION: AI MIDI GENERATOR TOOL & PROMPT PRESET SYSTEM
|
||||
|
||||
---
|
||||
|
||||
## 1. System Overview
|
||||
|
||||
The AI Copilot system integrated within the DAW provides two core capabilities:
|
||||
|
||||
* **AI Tool Call (Function Calling):** Receives user requests, triggers the automated music generation engine, and returns a list of `Tracks` and `MIDIItems` complying with the JSON schema structure for direct placement onto the DAW Timeline.
|
||||
* **Prompt Template Engine (Preset Library):** Manages a prompt template directory categorized by genre, mood, and song structure. Upon detecting specific keywords (e.g., *"epic orchestra"*), the engine automatically looks up and expands the raw user prompt into a System Context Standard Prompt containing music theory definitions (pitch range, scales, rhythm patterns, chord progressions, and voicing) prior to dispatching to the LLM.
|
||||
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| USER INTERFACE |
|
||||
| |
|
||||
| [ Prompt Bar / Copilot UI ] <---> [ Prompt Template Preset Manager (CRUD) ] |
|
||||
| | | |
|
||||
| | Input: "Write 8 bars of | Matched Preset: |
|
||||
| | epic orchestra MIDI notes..." | "Epic Orchestra Intro Spec" |
|
||||
| v v |
|
||||
| +---------------------------------------------------------------------------+ |
|
||||
| | Prompt Context Expander Engine | |
|
||||
| +--------------------------------------------------+------------------------+ |
|
||||
| | |
|
||||
+------------------------------------------------------|----------------------------+
|
||||
| Extended Prompt Payload
|
||||
v
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| AI LLM ENGINE |
|
||||
| |
|
||||
| Tool Calling Execution: `generate_multitrack_midi()` |
|
||||
| Output: Structured JSON Payload (Multi-track 8-bar notes) |
|
||||
+------------------------------------------------------|----------------------------+
|
||||
| Validated JSON Output
|
||||
v
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| DAW CLIENT STATE ENGINE |
|
||||
| |
|
||||
| - Parses JSON Payload |
|
||||
| - Spawns/Finds Tracks (Strings, Brass, Synth, Percussion, Drums) |
|
||||
| - Injects `MIDIItem` into `Main Session` / `Section Store` |
|
||||
| - Re-renders Canvas & Piano Roll UI |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Prompt Template & Preset Engine
|
||||
|
||||
To eliminate the need to re-type lengthy system instructions, the application provides a Preset Manager stored as JSON format in `LocalStorage` / `IndexedDB` on the client side or within the server database.
|
||||
|
||||
### 2.1 Prompt Preset Schema (`preset_schema.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "preset_epic_orchestra_intro",
|
||||
"name": "Epic Orchestra Intro (8 Bars)",
|
||||
"keywords": ["epic orchestra", "epic orchestral", "hoành tráng", "nhạc phim epic"],
|
||||
"category": "Orchestral / Film Score",
|
||||
"default_bars": 8,
|
||||
"default_bpm": 130,
|
||||
"default_scale": "C Minor",
|
||||
"system_instruction_template": "You are a professional Epic Orchestral film composer. Create a powerful, dramatic 8-bar intro composition.\nThe required structure to return via the `generate_multitrack_midi` tool consists of 5 tracks:\n1. Strings Ensemble: Plays staccato 16th notes in the low register (C2, G2) driving the rhythm (Ostinato).\n2. Brass Section: Plays the main swelling melodic theme (Horn/Trumpet swells) in the C3-C5 range.\n3. Epic Percussion / Taiko: Hits heavily on beats 1 and 3, featuring a snare roll accent at bars 4 and 8.\n4. Synth Bass/Pad: Holds smooth octave foundation layers (Legato).\n5. Orchestral Drums/Cymbals: Crashes on bar 1 and bar 5.\nEnsure the duration of each track is precisely 8 bars (32 beats).",
|
||||
"is_user_defined": false,
|
||||
"created_at": "2026-07-23T16:00:00Z"
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### 2.2 Intent Detection & Auto-Expansion Flow
|
||||
|
||||
When a user submits a prompt message inside the AI Copilot UI:
|
||||
|
||||
```javascript
|
||||
class PromptTemplateManager {
|
||||
constructor() {
|
||||
this.presets = [];
|
||||
this.loadPresets();
|
||||
}
|
||||
|
||||
// Loads preset list from LocalStorage or API
|
||||
async loadPresets() {
|
||||
const localData = localStorage.getItem('daw_ai_prompt_presets');
|
||||
if (localData) {
|
||||
this.presets = JSON.parse(localData);
|
||||
} else {
|
||||
this.presets = DEFAULT_PRESETS; // Developer default fallback templates
|
||||
this.savePresets();
|
||||
}
|
||||
}
|
||||
|
||||
// Matches queries against preset keywords automatically
|
||||
matchPreset(userQuery) {
|
||||
const queryLower = userQuery.toLowerCase();
|
||||
for (const preset of this.presets) {
|
||||
const hasKeyword = preset.keywords.some(kw => queryLower.includes(kw.toLowerCase()));
|
||||
if (hasKeyword) {
|
||||
return preset;
|
||||
}
|
||||
}
|
||||
return null; // Fallback to raw user prompt if no keyword matches
|
||||
}
|
||||
|
||||
// CRUD methods for user-defined presets
|
||||
saveUserPreset(presetObject) {
|
||||
const index = this.presets.findIndex(p => p.id === presetObject.id);
|
||||
if (index >= 0) {
|
||||
this.presets[index] = presetObject;
|
||||
} else {
|
||||
this.presets.push(presetObject);
|
||||
}
|
||||
this.savePresets();
|
||||
}
|
||||
|
||||
savePresets() {
|
||||
localStorage.setItem('daw_ai_prompt_presets', JSON.stringify(this.presets));
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Function Calling Specification
|
||||
|
||||
Defines the Tool/Function schema passed to the LLM API (OpenAI / Local LLM) to enforce strict structured JSON output.
|
||||
|
||||
### 3.1 Function Tool Schema (`tools_spec.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "generate_multitrack_midi",
|
||||
"description": "Generates multi-track MIDI data based on genre, bar duration, and requested instruments list.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"composition_title": {
|
||||
"type": "string",
|
||||
"description": "Title of the musical piece (e.g., Epic Orchestra Intro 8-Bars)"
|
||||
},
|
||||
"bpm": {
|
||||
"type": "integer",
|
||||
"default": 120
|
||||
},
|
||||
"total_bars": {
|
||||
"type": "integer",
|
||||
"default": 8
|
||||
},
|
||||
"tracks": {
|
||||
"type": "array",
|
||||
"description": "Array of instrument tracks along with their corresponding MIDI notes",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"track_name": {
|
||||
"type": "string",
|
||||
"description": "Track name (e.g., String Ensemble, Epic Brass, Taiko Drums)"
|
||||
},
|
||||
"instrument_type": {
|
||||
"type": "string",
|
||||
"enum": ["STRINGS", "BRASS", "SYNTH", "PERCUSSION", "DRUMS"]
|
||||
},
|
||||
"notes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pitch": {
|
||||
"type": "integer",
|
||||
"description": "MIDI note pitch from 0 to 127 (e.g., C4 = 60, C3 = 48)"
|
||||
},
|
||||
"start_beat": {
|
||||
"type": "number",
|
||||
"description": "Note start position in beats (from 0.0 to total_bars * 4.0)"
|
||||
},
|
||||
"duration_beats": {
|
||||
"type": "number",
|
||||
"description": "Note length in beats (e.g., Quarter note = 1.0, Eighth note = 0.5)"
|
||||
},
|
||||
"velocity": {
|
||||
"type": "number",
|
||||
"description": "Keypress velocity intensity from 0.0 to 1.0",
|
||||
"default": 0.8
|
||||
}
|
||||
},
|
||||
"required": ["pitch", "start_beat", "duration_beats", "velocity"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": ["track_name", "instrument_type", "notes"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": ["composition_title", "bpm", "total_bars", "tracks"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. AI Response Payload Example
|
||||
|
||||
Below is an example JSON payload returned by the AI following the execution of `generate_multitrack_midi` for an 8-bar Epic Orchestra request:
|
||||
|
||||
```json
|
||||
{
|
||||
"tool_call": "generate_multitrack_midi",
|
||||
"result": {
|
||||
"composition_title": "Epic Orchestra Intro",
|
||||
"bpm": 130,
|
||||
"total_bars": 8,
|
||||
"tracks": [
|
||||
{
|
||||
"track_name": "String Ensemble (Staccato)",
|
||||
"instrument_type": "STRINGS",
|
||||
"notes": [
|
||||
{ "pitch": 36, "start_beat": 0.0, "duration_beats": 0.25, "velocity": 0.9 },
|
||||
{ "pitch": 36, "start_beat": 0.5, "duration_beats": 0.25, "velocity": 0.85 },
|
||||
{ "pitch": 48, "start_beat": 1.0, "duration_beats": 0.25, "velocity": 0.95 },
|
||||
{ "pitch": 36, "start_beat": 1.5, "duration_beats": 0.25, "velocity": 0.8 }
|
||||
]
|
||||
},
|
||||
{
|
||||
"track_name": "French Horns & Brass",
|
||||
"instrument_type": "BRASS",
|
||||
"notes": [
|
||||
{ "pitch": 60, "start_beat": 0.0, "duration_beats": 2.0, "velocity": 0.95 },
|
||||
{ "pitch": 63, "start_beat": 2.0, "duration_beats": 2.0, "velocity": 0.9 },
|
||||
{ "pitch": 67, "start_beat": 4.0, "duration_beats": 4.0, "velocity": 1.0 }
|
||||
]
|
||||
},
|
||||
{
|
||||
"track_name": "Epic Synth Lead",
|
||||
"instrument_type": "SYNTH",
|
||||
"notes": [
|
||||
{ "pitch": 72, "start_beat": 4.0, "duration_beats": 1.0, "velocity": 0.85 },
|
||||
{ "pitch": 75, "start_beat": 5.0, "duration_beats": 1.0, "velocity": 0.85 }
|
||||
]
|
||||
},
|
||||
{
|
||||
"track_name": "Taiko & Percussion",
|
||||
"instrument_type": "PERCUSSION",
|
||||
"notes": [
|
||||
{ "pitch": 36, "start_beat": 0.0, "duration_beats": 0.5, "velocity": 1.0 },
|
||||
{ "pitch": 36, "start_beat": 2.0, "duration_beats": 0.5, "velocity": 0.95 },
|
||||
{ "pitch": 38, "start_beat": 3.5, "duration_beats": 0.25, "velocity": 0.8 }
|
||||
]
|
||||
},
|
||||
{
|
||||
"track_name": "Orchestral Cymbals",
|
||||
"instrument_type": "DRUMS",
|
||||
"notes": [
|
||||
{ "pitch": 49, "start_beat": 0.0, "duration_beats": 4.0, "velocity": 0.9 },
|
||||
{ "pitch": 49, "start_beat": 16.0, "duration_beats": 4.0, "velocity": 1.0 }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. DAW State Ingestion Logic
|
||||
|
||||
When the client receives the AI JSON payload, the `ingestAIGeneratedMIDI()` function executes the following pipeline:
|
||||
|
||||
1. Spawns or matches corresponding tracks within `main_session`.
|
||||
2. Generates 8-bar `MIDIItem` objects containing the note lists.
|
||||
3. Triggers UI canvas and timeline re-renders.
|
||||
|
||||
```javascript
|
||||
function ingestAIGeneratedMIDI(aiPayload, sessionState) {
|
||||
const { composition_title, bpm, total_bars, tracks } = aiPayload.result;
|
||||
|
||||
// 1. Update project BPM if specified
|
||||
if (bpm) sessionState.metadata.bpm = bpm;
|
||||
|
||||
// 2. Iterate through generated tracks
|
||||
tracks.forEach((aiTrack) => {
|
||||
// Match existing track or instantiate a new one
|
||||
let targetTrack = sessionState.main_session.tracks.find(
|
||||
t => t.name.toLowerCase() === aiTrack.track_name.toLowerCase()
|
||||
);
|
||||
|
||||
if (!targetTrack) {
|
||||
targetTrack = {
|
||||
id: `track_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`,
|
||||
name: aiTrack.track_name,
|
||||
type: "MIDI",
|
||||
volume_db: 0.0,
|
||||
pan: 0.0,
|
||||
mute: false,
|
||||
solo: false,
|
||||
items: []
|
||||
};
|
||||
sessionState.main_session.tracks.push(targetTrack);
|
||||
}
|
||||
|
||||
// 3. Create 8-bar MIDIItem
|
||||
const newMidiItem = {
|
||||
id: `item_ai_${Date.now()}_${Math.random().toString(36).substr(2, 5)}`,
|
||||
name: `${composition_title} - ${aiTrack.track_name}`,
|
||||
type: "MIDI_ITEM",
|
||||
start_bar: 0.0, // Placed at Timeline start or active Playhead position
|
||||
duration_bars: total_bars,
|
||||
clip_start_offset_bars: 0.0,
|
||||
source_data: {
|
||||
total_buffer_bars: total_bars,
|
||||
notes: aiTrack.notes.map((note, index) => ({
|
||||
id: `note_ai_${Date.now()}_${index}`,
|
||||
pitch: note.pitch,
|
||||
start_beat: note.start_beat,
|
||||
duration_beats: note.duration_beats,
|
||||
velocity: note.velocity,
|
||||
pan: 0.0
|
||||
}))
|
||||
}
|
||||
};
|
||||
|
||||
// 4. Append Item to Track
|
||||
targetTrack.items.push(newMidiItem);
|
||||
});
|
||||
|
||||
// 5. Trigger Timeline & Piano Roll UI Re-render Event
|
||||
window.dispatchEvent(new CustomEvent('DAW_STATE_UPDATED', { detail: sessionState }));
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Prompt Preset Manager UI Layout
|
||||
|
||||
```text
|
||||
+-------------------------------------------------------------------------------+
|
||||
| AI PROMPT PRESET MANAGER [ + New ]|
|
||||
+-------------------------------------------------------------------------------+
|
||||
| SEARCH: [ epic orchestra ] FILTER: [ Orchestral v ]|
|
||||
| |
|
||||
| Preset Name Keywords matched Default Bars Actions |
|
||||
| --------------------------------------------------------------------------- |
|
||||
| [★] Epic Orchestra Intro epic, orchestra, tráng 8 Bars [Edit][Del] |
|
||||
| [ ] Pop Piano Chords pop, piano, chord 4 Bars [Edit][Del] |
|
||||
| [ ] Cyberpunk Synth Synth synth, synthwave, 80s 8 Bars [Edit][Del] |
|
||||
| |
|
||||
+-------------------------------------------------------------------------------+
|
||||
| EDIT PRESET: Epic Orchestra Intro |
|
||||
| |
|
||||
| Keyword Triggers (comma-separated): |
|
||||
| [ epic orchestra, hoành tráng, nhạc phim epic ] |
|
||||
| |
|
||||
| System Instruction / Music Rules: |
|
||||
| +---------------------------------------------------------------------------+ |
|
||||
| | Compose an 8-bar epic orchestral theme featuring Brass, staccato | |
|
||||
| | Strings, Taiko percussion, and background Synth bass... | |
|
||||
| +---------------------------------------------------------------------------+ |
|
||||
| [ CANCEL ] [ SAVE ]
|
||||
+-------------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
@@ -0,0 +1,344 @@
|
||||
# TECHNICAL PLAN: DECENT SAMPLER + PIANOBOOK INSTALLATION AND SOUNDFONT MAPPING EXTRACTION FOR AI AGENT
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Overview
|
||||
|
||||
The system needs to fulfill two core requirements:
|
||||
|
||||
* **Integrate DecentSampler & Pianobook on Linux Server:**
|
||||
* Install `DecentSampler.vst3` (Linux 64-bit) into the Docker Server environment.
|
||||
* Structure the Pianobook sample library directories (`.dspreset` + `.wav` files).
|
||||
* Integrate Preset Loading into Python `pedalboard` for offline rendering steps.
|
||||
|
||||
|
||||
* **Resolve the AI Agent's "Instrument Information Blindness" regarding SoundFont (`.sf2`):**
|
||||
* *Current State:* The AI Agent and the system only load raw `.sf2` files without knowing what instruments are contained within (which Bank, which Program/Patch number, or what the instrument names are).
|
||||
* *Solution:*
|
||||
* Build a **SoundFont Inspection Engine** using Python (`sf2utils`) to scan `.sf2` files upon upload/scan and extract the instrument catalog table (Bank, Program/Preset ID, Instrument Name).
|
||||
* Export a Catalog table (`soundfont_catalog.json`) and pass this context to the AI Agent.
|
||||
* Update the AI Tool Schema so the AI accurately passes `soundfont_id`, `bank`, and `program` (MIDI Program Change) when creating a Track.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 2. DecentSampler + Pianobook Installation Plan (Server Backend)
|
||||
|
||||
### 2.1 Installing DecentSampler Linux Native VST3 in Docker
|
||||
|
||||
1. **Download DecentSampler Linux VST3:**
|
||||
* Download the official Linux 64-bit build from the DecentSampler website (`DecentSampler_Linux_x64.tar.gz` or `.vst3` file).
|
||||
|
||||
|
||||
2. **Server Directory Structure:**
|
||||
```text
|
||||
/opt/daw_engine/
|
||||
├── vst3/
|
||||
│ └── DecentSampler.vst3/ <-- Native Linux VST3 Binary
|
||||
├── soundfonts/
|
||||
│ ├── GeneralUser_GS.sf2
|
||||
│ └── SGM-V2.01.sf2
|
||||
└── samples/
|
||||
└── pianobook/ <-- Pianobook Sample Libraries
|
||||
├── salamander_grand_piano/
|
||||
│ ├── salamander_piano.dspreset
|
||||
│ └── samples/ (*.wav)
|
||||
└── acoustic_guitar/
|
||||
├── guitar.dspreset
|
||||
└── samples/ (*.wav)
|
||||
|
||||
```
|
||||
|
||||
|
||||
3. **Additions to `Dockerfile`:**
|
||||
```dockerfile
|
||||
# Install audio system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
libgl1-mesa-glx \
|
||||
libfreetype6 \
|
||||
libcurl4 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy DecentSampler VST3 to Server
|
||||
COPY ./vst_plugins/DecentSampler.vst3 /opt/daw_engine/vst3/DecentSampler.vst3
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 2.2 Integrating DecentSampler into the Python Engine (`app/core/vst_engine.py`)
|
||||
|
||||
The `pedalboard` library supports loading VST3 plugins and preset files for DecentSampler:
|
||||
|
||||
```python
|
||||
import os
|
||||
from pedalboard import VST3Plugin
|
||||
|
||||
class DecentSamplerManager:
|
||||
def __init__(self, vst_path="/opt/daw_engine/vst3/DecentSampler.vst3"):
|
||||
self.vst_path = vst_path
|
||||
|
||||
def create_decent_sampler_instance(self, dspreset_path: str) -> VST3Plugin:
|
||||
"""
|
||||
Instantiates VST3 DecentSampler and loads the Pianobook sample preset (.dspreset) file.
|
||||
"""
|
||||
if not os.path.exists(self.vst_path):
|
||||
raise FileNotFoundError(f"DecentSampler VST3 not found at {self.vst_path}")
|
||||
|
||||
plugin = VST3Plugin(self.vst_path)
|
||||
|
||||
# Load the Pianobook preset file into DecentSampler VST3
|
||||
if os.path.exists(dspreset_path):
|
||||
plugin.load_preset(dspreset_path)
|
||||
|
||||
return plugin
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Designing the SoundFont Inspection Engine (Bank/Program Extraction)
|
||||
|
||||
Every SoundFont (`.sf2`) file is a collection of Presets (or Programs). For the AI Agent to know what sound presets exist inside the `.sf2` file, the Backend must scan and parse the `.sf2` file.
|
||||
|
||||
### 3.1 Adding Python SoundFont Inspection Libraries
|
||||
|
||||
Add to `requirements.txt`:
|
||||
|
||||
```text
|
||||
sf2utils>=0.9.0
|
||||
mido>=1.3.0
|
||||
|
||||
```
|
||||
|
||||
### 3.2 Building the SoundFont Metadata Inspection Service (`app/core/soundfont_inspector.py`)
|
||||
|
||||
```python
|
||||
import os
|
||||
import json
|
||||
from sf2utils.sf2parse import Sf2File
|
||||
|
||||
class SoundFontInspector:
|
||||
def __init__(self, sf_dir="/opt/daw_engine/soundfonts"):
|
||||
self.sf_dir = sf_dir
|
||||
|
||||
def inspect_sf2_file(self, filepath: str) -> dict:
|
||||
"""
|
||||
Parses an .sf2 file and returns a complete instrument catalog (Bank, Program, Instrument Name).
|
||||
"""
|
||||
if not os.path.exists(filepath):
|
||||
return {}
|
||||
|
||||
sf_name = os.path.basename(filepath)
|
||||
sf_id = os.path.splitext(sf_name)[0].lower()
|
||||
|
||||
instruments = []
|
||||
with open(filepath, 'rb') as f:
|
||||
sf2 = Sf2File(f)
|
||||
for preset in sf2.presets:
|
||||
# Ignore EOP (End of Header) preset
|
||||
if preset.name.strip() == "EOP" or (preset.bank == 128 and preset.preset == 127):
|
||||
continue
|
||||
|
||||
instruments.append({
|
||||
"bank": preset.bank, # Bank number (0 = General MIDI Standard, 128 = Percussion/Drums)
|
||||
"program": preset.preset, # Program/Patch number (0-127)
|
||||
"name": preset.name.strip(), # Instrument name (e.g. "Stereo Grand", "Violin", "Brass Section")
|
||||
"is_percussion": (preset.bank == 128)
|
||||
})
|
||||
|
||||
return {
|
||||
"soundfont_id": sf_id,
|
||||
"filename": sf_name,
|
||||
"total_instruments": len(instruments),
|
||||
"instruments": instruments
|
||||
}
|
||||
|
||||
def generate_full_catalog(self, output_json_path="/opt/daw_engine/soundfont_catalog.json"):
|
||||
"""
|
||||
Scans all .sf2 files in the directory and builds a Catalog JSON for the AI Agent.
|
||||
"""
|
||||
catalog = {}
|
||||
for root, dirs, files in os.walk(self.sf_dir):
|
||||
for file in files:
|
||||
if file.endswith(('.sf2', '.SF2')):
|
||||
full_path = os.path.join(root, file)
|
||||
sf_info = self.inspect_sf2_file(full_path)
|
||||
catalog[sf_info["soundfont_id"]] = sf_info
|
||||
|
||||
with open(output_json_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(catalog, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return catalog
|
||||
|
||||
```
|
||||
|
||||
### 3.3 Catalog File Output Structure (`soundfont_catalog.json`)
|
||||
|
||||
This JSON file serves as an Instrument Dictionary for the AI Agent:
|
||||
|
||||
```json
|
||||
{
|
||||
"generaluser_gs": {
|
||||
"soundfont_id": "generaluser_gs",
|
||||
"filename": "GeneralUser_GS.sf2",
|
||||
"total_instruments": 128,
|
||||
"instruments": [
|
||||
{ "bank": 0, "program": 0, "name": "Stereo Grand Piano", "is_percussion": false },
|
||||
{ "bank": 0, "program": 19, "name": "Church Organ", "is_percussion": false },
|
||||
{ "bank": 0, "program": 40, "name": "Violin Ensemble", "is_percussion": false },
|
||||
{ "bank": 0, "program": 56, "name": "Trumpet", "is_percussion": false },
|
||||
{ "bank": 128, "program": 0, "name": "Standard Drum Kit", "is_percussion": true }
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Guiding the AI Agent in Instrument Selection & Accurate Note Loading
|
||||
|
||||
When the user types: *"Create a Piano track and a Strings section track for 8 bars"*, the AI Agent needs to know precisely which soundfont, bank, and program numbers to assign to the Tracks.
|
||||
|
||||
### 4.1 Updating the AI Tool Function Spec (`tools_spec.json`)
|
||||
|
||||
Add `soundfont_bank` and `soundfont_program` fields to the Tool Schema sent to the AI:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "generate_multitrack_midi",
|
||||
"description": "Generates multi-track MIDI data along with appropriate SoundFont Program configurations.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"tracks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"track_name": { "type": "string" },
|
||||
"instrument_type": { "type": "string", "enum": ["PIANO", "STRINGS", "BRASS", "SYNTH", "DRUMS"] },
|
||||
"soundfont_id": {
|
||||
"type": "string",
|
||||
"description": "ID of the SoundFont to use (e.g. 'generaluser_gs')"
|
||||
},
|
||||
"soundfont_bank": {
|
||||
"type": "integer",
|
||||
"default": 0,
|
||||
"description": "MIDI Bank code (0 for melodic instruments, 128 for Drums)"
|
||||
},
|
||||
"soundfont_program": {
|
||||
"type": "integer",
|
||||
"description": "MIDI Program Number (0-127) corresponding to the instrument name in the Catalog"
|
||||
},
|
||||
"notes": { "type": "array", "items": { "type": "object" } }
|
||||
},
|
||||
"required": ["track_name", "soundfont_id", "soundfont_bank", "soundfont_program", "notes"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### 4.2 Injecting the Catalog into the AI Agent's Context Prompt (Prompt Template)
|
||||
|
||||
Before sending the user query to the LLM, the system reads `soundfont_catalog.json` and injects a condensed catalog table into the System Instruction:
|
||||
|
||||
```python
|
||||
# System Context Prompt Injector
|
||||
def build_ai_system_instruction(catalog_data: dict) -> str:
|
||||
sf_summary = []
|
||||
for sf_id, sf_info in catalog_data.items():
|
||||
sf_summary.append(f"SoundFont ID: '{sf_id}' (File: {sf_info['filename']}):")
|
||||
for inst in sf_info['instruments'][:20]: # Inject primary instrument lists
|
||||
sf_summary.append(
|
||||
f" - [{inst['name']}]: bank={inst['bank']}, program={inst['program']}"
|
||||
)
|
||||
|
||||
catalog_context = "\n".join(sf_summary)
|
||||
|
||||
system_instruction = f"""
|
||||
You are an AI Copilot for a DAW. Below is the Catalog of available SoundFonts on the system:
|
||||
|
||||
{catalog_context}
|
||||
|
||||
MANDATORY RULES WHEN CREATING TRACKS:
|
||||
1. When creating any track, you MUST look up the catalog above and fill in the correct `soundfont_id`, `soundfont_bank`, and `soundfont_program`.
|
||||
2. Example: If the user requests "Piano", select soundfont_id="generaluser_gs", soundfont_bank=0, soundfont_program=0 ("Stereo Grand Piano").
|
||||
3. If the user requests "Violin/Strings", select soundfont_bank=0, soundfont_program=40 ("Violin Ensemble").
|
||||
4. If the user requests "Drums", select soundfont_bank=128, soundfont_program=0 ("Standard Drum Kit").
|
||||
"""
|
||||
return system_instruction
|
||||
|
||||
```
|
||||
|
||||
### 4.3 Applying Program Change on Client & Server Render Layers
|
||||
|
||||
#### A. Client Browser Side (FluidSynth Wasm / SoundFont Player)
|
||||
|
||||
Upon receiving JSON from the AI, the Frontend invokes the Bank and Program selection function to trigger the correct sound:
|
||||
|
||||
```javascript
|
||||
// Client-side Javascript (spessasynth / fluidsynth.wasm)
|
||||
function applyAITrackInstrument(trackId, soundfontBank, soundfontProgram) {
|
||||
const channel = getTrackMIDIChannel(trackId);
|
||||
|
||||
// Send MIDI Bank Select (CC 0)
|
||||
synthInstance.controllerChange(channel, 0, soundfontBank);
|
||||
|
||||
// Send MIDI Program Change
|
||||
synthInstance.programChange(channel, soundfontProgram);
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### B. Server Offline Render Side (`app/core/render_engine.py`)
|
||||
|
||||
When rendering to a WAV file, Python inserts a MIDI Program Change event ahead of the Track's note sequence:
|
||||
|
||||
```python
|
||||
import mido
|
||||
|
||||
def create_midi_track_with_program(notes_data, bank=0, program=0):
|
||||
midi_track = mido.MidiTrack()
|
||||
|
||||
# 1. Insert Bank Select (Control Change 0)
|
||||
midi_track.append(mido.Message('control_change', channel=0, control=0, value=bank, time=0))
|
||||
|
||||
# 2. Insert Program Change (Instrument Sound Selection)
|
||||
midi_track.append(mido.Message('program_change', channel=0, program=program, time=0))
|
||||
|
||||
# 3. Insert MIDI notes generated by AI
|
||||
for note in notes_data:
|
||||
start_tick = int(note['start_beat'] * 480) # 480 ticks per beat
|
||||
dur_tick = int(note['duration_beats'] * 480)
|
||||
pitch = int(note['pitch'])
|
||||
vel = int(note['velocity'] * 127)
|
||||
|
||||
midi_track.append(mido.Message('note_on', note=pitch, velocity=vel, time=start_tick))
|
||||
midi_track.append(mido.Message('note_off', note=pitch, velocity=0, time=dur_tick))
|
||||
|
||||
return midi_track
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Action Checklist
|
||||
|
||||
* [ ] **Step 1:** Download Linux 64-bit `DecentSampler.vst3` and copy it into `/opt/daw_engine/vst3/`.
|
||||
* [ ] **Step 2:** Download Pianobook sound libraries (e.g. Salamander Grand Piano) and extract them to `/opt/daw_engine/samples/pianobook/`.
|
||||
* [ ] **Step 3:** Add `sf2utils` to `requirements.txt` and install it in Docker.
|
||||
* [ ] **Step 4:** Create `app/core/soundfont_inspector.py` to automatically scan all `.sf2` files in the project and generate `soundfont_catalog.json`.
|
||||
* [ ] **Step 5:** Build API Endpoint `GET /api/v1/plugins/soundfonts/catalog` returning the extracted instrument catalog.
|
||||
* [ ] **Step 6:** Update the Prompt Template and AI Tool Schema to support `soundfont_bank` & `soundfont_program`.
|
||||
* [ ] **Step 7:** End-to-End Verification: Type Prompt *"Generate 8 bars of Brass horn music"* $\rightarrow$ AI reads Catalog and selects Program `56` $\rightarrow$ Web Audio & Server Render produce the correct Brass horn instrument sound.
|
||||
@@ -0,0 +1,115 @@
|
||||
# INTEGRATION & OPERATIONAL GUIDE: SOUNDFONT & VST3 ENGINE SYSTEM
|
||||
|
||||
This document outlines the workflow for connecting and operating the designed technical methods and modules across the entire DAW system, clearly categorized by system integration steps.
|
||||
|
||||
---
|
||||
|
||||
## 1. System Environment & Storage Setup
|
||||
|
||||
### A. Server & Docker Directory Structure
|
||||
|
||||
* **System SoundFont Directory (`/opt/daw_engine/soundfonts/`):** Stores system default `.sf2` files (e.g., `GeneralUser_GS.sf2`, `SGM-V2.01.sf2`).
|
||||
* **User Upload Directory (`app/storage/uploads/soundfonts/`):** Stores `.sf2` files uploaded by users via the web interface.
|
||||
* **VST3 & Pianobook Directories (`/opt/daw_engine/vst3/`, `/opt/daw_engine/samples/pianobook/`):** Contains the `DecentSampler.vst3` binary along with the directory structure holding `.dspreset` sample files and `samples/*.wav` subdirectories.
|
||||
|
||||
### B. System Dependencies
|
||||
|
||||
Ensure the `Dockerfile`/`Virtualenv` has installed the `libcurl4` system library (mandatory for DecentSampler) and the Python package `sf2utils>=0.9.0`.
|
||||
|
||||
---
|
||||
|
||||
## 2. SoundFont Catalog Operational Lifecycle
|
||||
|
||||
### A. First Startup (Lazy Initialization)
|
||||
|
||||
* When the Server boots, the Catalog is not generated immediately to prevent slowing down the app boot time.
|
||||
* When the Frontend dispatches its first request to the API Endpoint `GET /api/v1/plugins/soundfonts/catalog`, the Backend triggers `SoundFontInspector` to simultaneously scan both system and upload directories.
|
||||
* The extracted data is categorized into 2 versions:
|
||||
* **Full Catalog:** Designed for the UI to display the complete list of instruments.
|
||||
* **Condensed Catalog:** A summary (maximum 40–50 representative instruments categorized under General MIDI groups such as Piano, Brass, Drums, etc.) specifically tailored for the AI Agent.
|
||||
|
||||
|
||||
* The parsed data is cached in memory (Memory Cache) for subsequent queries.
|
||||
|
||||
### B. Cache Invalidation on User Upload
|
||||
|
||||
* Once the upload handling endpoint successfully saves an uploaded `.sf2` file to the upload directory:
|
||||
* Automatically invokes the `invalidate_catalog_cache()` method to purge the memory cache.
|
||||
* Triggers a Background Task calling the catalog initialization function to incrementally scan the new file without blocking the user's HTTP response.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. AI Copilot Integration Workflow (AI Gateway & System Prompt)
|
||||
|
||||
### A. Initial Instrument Catalog Load (Frontend Startup)
|
||||
|
||||
* As soon as the Web application launches (`app.jsx`), the Frontend proactively calls the API to fetch the Catalog.
|
||||
* Extracts the `condensed_catalog` section and persists it into the application's global state (Global State).
|
||||
|
||||
### B. Automated Prompt Context Injection
|
||||
|
||||
* When a user submits an interaction command to the AI:
|
||||
* The System Instruction generator reads the `condensed_catalog` and converts it into a concise text description of available instruments (including name, bank code, and program code).
|
||||
* Enforces the rule that the AI must assign `soundfont_bank: 0` for melodic instruments and `soundfont_bank: 128` for Drum Kits.
|
||||
|
||||
|
||||
|
||||
### C. Function Calling Schema Definition
|
||||
|
||||
* When dispatching requests to the LLM, the Tools list configuring `generate_multitrack_midi` includes 3 mandatory fields for every Track: `soundfont_id`, `soundfont_bank`, and `soundfont_program`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Real-time Client-Side Instrument Switching (Browser Playback)
|
||||
|
||||
### A. Listening for AI Responses
|
||||
|
||||
* When the AI successfully completes a Function Call and returns a JSON payload containing musical notes alongside `soundfont_bank` & `soundfont_program` parameters for each Track:
|
||||
* The Client allocates each Track to a corresponding MIDI Channel (Channels 0 through 8 for standard instruments, fixed Channel 9 for Drum Kits).
|
||||
|
||||
|
||||
|
||||
### B. Applying Real-Time Program Changes
|
||||
|
||||
* Calls the `applyAITrackInstrument` method on the Client's SoundFont Player module.
|
||||
* The module dispatches a Control Change (CC 0) signal to select the Bank, followed by a Program Change event to the designated MIDI channel to immediately play the newly selected instrument sound inside the browser.
|
||||
|
||||
---
|
||||
|
||||
## 5. Server-Side Offline Render Workflow (Audio Export)
|
||||
|
||||
When a user clicks "Export WAV" or "Bounce Track", the processing pipeline on the Server executes as follows:
|
||||
|
||||
### A. Reading Track Metadata
|
||||
|
||||
Extracts `soundfont_bank` and `soundfont_program` parameters from the Track metadata received in the project's JSON payload.
|
||||
|
||||
### B. MIDI Channel Routing & FluidSynth Rendering
|
||||
|
||||
* **Channel Rules:** If `soundfont_bank == 128` or the track is marked as percussion (`is_percussion`), rigidly assigns `midi_channel = 9` (Channel 10 under the General MIDI standard). Otherwise, assigns free channels from 0 to 8.
|
||||
* Executes `program_select` settings on the FluidSynth Instance targeting the correct channel, bank, and program before feeding the note sequence into the audio rendering buffer.
|
||||
|
||||
### C. Rendering Pianobook (`.dspreset`)
|
||||
|
||||
* If a Track selects a Pianobook instrument source:
|
||||
* Calls `DecentSamplerManager` passing the absolute file path to the `.dspreset` file.
|
||||
* The manager automatically changes the Current Working Directory (CWD) temporarily to the parent folder of the `.dspreset` file, loads the preset into VST3, and subsequently restores the original working directory to prevent "Sample Not Found" errors on relative `.wav` sample files.
|
||||
|
||||
|
||||
|
||||
### D. Rendering VST3 via Pedalboard
|
||||
|
||||
Prior to passing the MIDI note array into the VST3 Plugin, inserts 2 initialization MIDI messages at timestamp $0.0\text{s}$:
|
||||
|
||||
* A `control_change` message (Control 0, Value = bank).
|
||||
* A `program_change` message (Program = program).
|
||||
|
||||
---
|
||||
|
||||
## 6. Verification & Testing Workflow
|
||||
|
||||
* **Catalog API Verification:** Use Postman or a browser to call `GET /api/v1/plugins/soundfonts/catalog`, confirming that the returned payload contains both `full_catalog` and `condensed_catalog`.
|
||||
* **AI Response Verification:** Input the command *"Compose 8 bars of Brass horns and a drum kit"* $\rightarrow$ Inspect the returned JSON from the AI to verify that the Brass track has `program: 56`, `bank: 0` and the Drums track has `program: 0`, `bank: 128`.
|
||||
* **Audio Output Verification:** Export the WAV file and listen to confirm that the Brass horn and Drum sounds are rendered using the correct instrument patches.
|
||||
@@ -0,0 +1,130 @@
|
||||
# ASSESSMENT REPORT & OPTIMIZATION PLAN: LINUX VST3 & SOUNDFONT MAPPING ENGINE
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Review
|
||||
|
||||
Your plan closely aligns with the current codebase status and correctly identifies key bottlenecks (such as missing `libcurl4`, missing `sf2utils`, hardcoded `program_select(0, fid, 0, 0)` calls in `render_engine.py`, and missing schema fields in the AI Tool Schema).
|
||||
|
||||
However, to guarantee stable system operation within the Docker Linux environment and prevent failures during End-to-End execution, the plan requires the 5 critical technical refinements detailed below.
|
||||
|
||||
---
|
||||
|
||||
## 2. 5 Mandatory Technical Refinements
|
||||
|
||||
### 💡 Refinement 1: AI Prompt Context Size Control (Avoiding Token Overflow)
|
||||
|
||||
* **Problem in Previous Plan:** Injecting the entire `soundfont_catalog.json` file into the AI System Instruction. A full SoundFont file (like GeneralUser GS or SGM-V2.01) can contain hundreds to thousands of presets/notes, causing LLM Token Limit overflows, inflating costs, and degrading response latency.
|
||||
* **Solution:**
|
||||
* Implement `get_condensed_catalog_summary()` within `SoundFontInspector` to extract only a condensed catalog (categorized into core instrument groups: Piano, Organ, Guitar, Bass, Strings, Ensemble, Brass, Reed, Pipe, Synth Lead, Synth Pad, Drum Kit).
|
||||
* Inject a maximum of 40–50 of the most common instruments along with their representative bank and program codes into the AI Prompt.
|
||||
|
||||
|
||||
|
||||
### 💡 Refinement 2: MIDI Channel Handling for Percussion Kits (Bank 128 / Percussion)
|
||||
|
||||
* **Problem in Previous Plan:** Defaulting to `channel=0` for all tracks when invoking `program_select(0, fid, bank, prog)`. In General MIDI and SoundFont (`.sf2`) standards, Drum/Percussion sounds (Bank 128) must reside on MIDI Channel 9 (the 10th channel, 0-based index 9).
|
||||
* **Solution:**
|
||||
* In `render_engine.py`, if `soundfont_bank == 128` or `is_percussion == True`, automatically assign that track's MIDI Channel to `channel = 9` for both FluidSynth rendering and `mido` message generation.
|
||||
|
||||
|
||||
|
||||
### 3. Refinement 3: Catalog Refresh on User SoundFont Upload (Cache Invalidation)
|
||||
|
||||
* **Problem in Previous Plan:** The `GET /api/v1/plugins/soundfonts/catalog` endpoint scans the catalog only once or upon application startup. When a user uploads a new `.sf2` file via `POST /api/v1/audio/upload-soundfont`, the AI remains unaware of the newly added file.
|
||||
* **Solution:**
|
||||
* Implement a Cache Invalidation mechanism: Upon successfully saving an uploaded `.sf2` file, automatically invoke `SoundFontInspector.generate_full_catalog()` to update the `soundfont_catalog.json` file.
|
||||
|
||||
|
||||
|
||||
### 💡 Refinement 4: Handling Relative Sample Paths for Pianobook `.dspreset` Files
|
||||
|
||||
* **Problem in Previous Plan:** Pianobook `.dspreset` files contain relative path links pointing to subfolder `samples/*.wav` files. When DecentSampler VST3 loads a `.dspreset` file via `pedalboard`, if the Working Directory is not set to the folder containing the `.dspreset` file, the VST3 engine triggers a "Sample Not Found" error (resulting in silence).
|
||||
* **Solution:**
|
||||
* Before invoking `plugin.load_preset(dspreset_path)`, ensure an absolute path (`os.path.abspath(dspreset_path)`) is passed and temporarily switch the Working Directory or properly configure the Root Sample Directory for DecentSampler.
|
||||
|
||||
|
||||
|
||||
### 💡 Refinement 5: Robust Error Handling in `SoundFontInspector`
|
||||
|
||||
* **Problem in Previous Plan:** If a user uploads a corrupted or malformed `.sf2` file, the `sf2utils` library may throw an exception, crashing the entire Catalog scanning workflow.
|
||||
* **Solution:**
|
||||
* Wrap each `.sf2` file processing block inside a `try...except` block in `SoundFontInspector`. If a file is corrupted, log a warning and skip that specific file instead of interrupting the complete scan process.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Updated Execution Plan
|
||||
|
||||
### Task A: SoundFont Inspection Engine (`sf2utils`)
|
||||
|
||||
* [x] **A1:** Add `sf2utils>=0.9.0` to `requirements.txt`.
|
||||
* [x] **A2:** Create `app/core/soundfont_inspector.py`:
|
||||
* Add `inspect_sf2_file(filepath)` wrapped in a `try...except` block.
|
||||
* Add `generate_full_catalog(output_json_path)` scanning both `/opt/daw_engine/soundfonts` and `app/storage/uploads/soundfonts`.
|
||||
* Add `get_condensed_catalog_summary()` to build a condensed summary for the AI Context Prompt.
|
||||
|
||||
|
||||
* [x] **A3:** Create API Endpoint `GET /api/v1/plugins/soundfonts/catalog` in `app/api/v1/plugins.py`:
|
||||
* Return Full Catalog for Frontend UI and Condensed Catalog for AI Agent.
|
||||
* Integrate cache refresh functionality triggered upon new `.sf2` file uploads.
|
||||
|
||||
|
||||
|
||||
### Task B: DecentSampler + Pianobook Support
|
||||
|
||||
* [x] **B1:** Update `Dockerfile`:
|
||||
* Add `libcurl4` to the `apt-get install` package list.
|
||||
* Pre-create directory structures `/opt/daw_engine/vst3/` and `/opt/daw_engine/samples/pianobook/`.
|
||||
|
||||
|
||||
* [x] **B2:** Create local host directory structure `vst_plugins/` and `samples/pianobook/` (Update `.gitignore`).
|
||||
* [x] **B3:** Update `app/core/vst_engine.py`:
|
||||
* Add `DecentSamplerManager` supporting `.dspreset` loading using absolute paths.
|
||||
|
||||
|
||||
* [x] **B4:** Integrate Pianobook rendering into `app/core/render_engine.py` when a Track selects a Pianobook instrument.
|
||||
|
||||
### Task C: AI Tool Schema & Prompt Injection
|
||||
|
||||
* [x] **C1:** Update `DEFAULT_TOOLS` in `app/static/js/services/aiGateway.js`:
|
||||
* Add 3 properties to the `generate_multitrack_midi` schema: `soundfont_id` (string), `soundfont_bank` (integer), `soundfont_program` (integer).
|
||||
|
||||
|
||||
* [x] **C2:** Inject condensed instrument catalog into System Instruction within `aiGateway.js`.
|
||||
* [x] **C3:** Load Catalog automatically upon Frontend application startup (`app.jsx`).
|
||||
|
||||
### Task D: Server Render — Program Change & Channel Mapping
|
||||
|
||||
* [x] **D1:** Update `render_engine.py`:
|
||||
* Read `soundfont_bank` and `soundfont_program` from Track metadata.
|
||||
* MIDI channel rules: If `soundfont_bank == 128` (Drums), automatically assign `channel = 9` (GM Standard Channel 10). Otherwise, assign channels from 0 through 8.
|
||||
|
||||
|
||||
* [x] **D2:** Update FluidSynth render path:
|
||||
```python
|
||||
midi_channel = 9 if (bank == 128 or track.get("is_percussion")) else target_channel
|
||||
fl.program_select(midi_channel, fid, bank, prog)
|
||||
|
||||
```
|
||||
|
||||
|
||||
* [x] **D3:** Update VST3/Pedalboard render path:
|
||||
* Insert `CONTROL_CHANGE` (CC 0 for Bank) and `PROGRAM_CHANGE` events into the note sequence prior to rendering the audio buffer.
|
||||
|
||||
|
||||
|
||||
### Task E: Client SoundFont Player — Program Change
|
||||
|
||||
* [x] **E1:** Update `app/static/js/services/soundfontPlayer.js`:
|
||||
* Add `programChange(channel, program)` and `controllerChange(channel, controller, value)` methods.
|
||||
|
||||
|
||||
* [x] **E2:** Add `applyAITrackInstrument(trackId, bank, program)` function to dynamically switch instrument sounds in real time when AI generates new Tracks on the UI.
|
||||
|
||||
### Task F: Validation & Testing
|
||||
|
||||
* [x] **F1:** Test Catalog API: `GET /api/v1/plugins/soundfonts/catalog`.
|
||||
* [x] **F2:** Test AI Generation: Input prompt *"Compose 8 bars of Brass horns and a drum kit"* $\rightarrow$ Verify AI returns JSON with `program=56` (Brass) and `bank=128` (Drums).
|
||||
* [x] **F3:** Test Server Render: Export WAV $\rightarrow$ Listen to output audio file to verify correct Brass horn and Drum sound execution.
|
||||
@@ -0,0 +1,128 @@
|
||||
# OPERATION GUIDE & AUDIO PLAYBACK WORKFLOW FOR MIDI TRACKS (SOUNDFONT / VST3)
|
||||
|
||||
This document describes in detail the user interface interaction workflow when using the Synth button and explains the underlying technical architecture required for MIDI Notes on a Track to output audio via a selected SoundFont or VST3 Plugin.
|
||||
|
||||
---
|
||||
|
||||
## 1. User Interface Workflow Description
|
||||
|
||||
### Activating the Instrument Selection Menu
|
||||
|
||||
* On the Track Control Panel (the left-side pane of Track 01), the user clicks the **🎵 Synth: BAN-DI** button (or the orange Synth button below it).
|
||||
* A dropdown selection menu appears directly underneath the button.
|
||||
|
||||
### Instrument Menu Layout
|
||||
|
||||
* **None (Default Synth):** Uses the application's default synthesizer (a simple Oscillator Synth).
|
||||
* **SOUNDFONTS:** Displays a list of SoundFont (`.sf2`) soundbanks loaded into the system (e.g., `SoundFont_DSK_Asia`, `SoundFont_SGM_v2`, `weedsgm3`).
|
||||
* **VST INSTRUMENTS:** Displays a list of native 64-bit Linux VST3 Plugins (e.g., `Vital`, `DecentSampler`, `libSurge XT`).
|
||||
|
||||
### Selection & State Update Operations
|
||||
|
||||
* The user clicks to select an instrument (e.g., selecting `SoundFont_DSK_Asia` or `Vital`).
|
||||
* The menu closes, and the button label updates to reflect the chosen instrument (e.g., **🎵 DSK_Asia** or **🎵 Vital**).
|
||||
* The instrument configuration payload is directly assigned to the Track State object (`session.tracks[0].synth_engine`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Technical Execution Flow for MIDI Note Audio Output (SoundFont / VST3)
|
||||
|
||||
To ensure that the purple MIDI note bars on the Timeline or Piano Roll play back audio accurately using the chosen instrument, the system processes tasks across two primary workflows:
|
||||
|
||||
```text
|
||||
+------------------------------------+
|
||||
| User selects SoundFont / VST3 |
|
||||
+-----------------+------------------+
|
||||
|
|
||||
+--------------------------+--------------------------+
|
||||
| |
|
||||
v v
|
||||
[ 1. Real-time Client Preview ] [ 2. Server-side Offline Export ]
|
||||
(Audio Playback in Browser) (High-Quality WAV Rendering)
|
||||
| |
|
||||
+--------------------+--------------------+ +------------+------------+
|
||||
| | | |
|
||||
v v v v
|
||||
(If SoundFont) (If VST3) (If SoundFont) (If VST3)
|
||||
FluidSynth Wasm / Load Wasm Module / PyFluidSynth C-API Python Pedalboard
|
||||
SoundfontPlayer.js AudioWorklet Synth Dispatches Bank/Program Loads .vst3 binary
|
||||
Dispatches programChange Preview Synth Renders Audio Buffer Renders PCM Buffer
|
||||
| | | |
|
||||
+-----------------+-----------------+ +------------+------------+
|
||||
| |
|
||||
v v
|
||||
AudioContext Destination Audio Export Output File
|
||||
(User Speakers) (Downloaded WAV File)
|
||||
|
||||
```
|
||||
|
||||
### A. Real-time Client Playback (Browser Audio Preview)
|
||||
|
||||
When the user clicks the Play button or clicks a key on the Piano Roll:
|
||||
|
||||
1. **Audio Routing Update:**
|
||||
* The client reads the instrument parameters from `track.synth_engine`.
|
||||
* **If SoundFont (`.sf2`) is selected:** The client dispatches `controllerChange(channel, 0, bank)` and `programChange(channel, program)` configuration calls to the `soundfontPlayer.js` module (running FluidSynth WebAssembly).
|
||||
* **If VST3 Plugin (`Vital`, `DecentSampler`, etc.) is selected:** Because browsers cannot natively run Linux `.vst3`/`.so` binary executables directly, the client uses an equivalent WebAssembly Synth or Preview Synth to output real-time audio with $0\text{ ms}$ latency.
|
||||
|
||||
|
||||
2. **Note Scheduling:**
|
||||
* The Transport driver (`PrecisionAudioScheduler`) scans for MIDI notes located within the moving Playhead range.
|
||||
* Each MIDI note includes: `pitch` (0–127), `start_beat` (start position), `duration_beats` (length), and `velocity` (keypress intensity 0.0–1.0).
|
||||
* The scheduler converts beat timing to absolute time in seconds (`exactAudioTime`) and dispatches audio events:
|
||||
* `noteOn(pitch, velocity, exactAudioTime)`
|
||||
* `noteOff(pitch, exactAudioTime + durationSec)`
|
||||
|
||||
|
||||
* Audio signals generated by the WebAssembly Engine travel through `Track Gain Node` $\rightarrow$ `Track Pan Node` $\rightarrow$ `Master Bus` $\rightarrow$ `AudioContext.destination` (User Speakers).
|
||||
|
||||
|
||||
|
||||
### B. Server-side Offline Render (High-Quality WAV Export)
|
||||
|
||||
When the user exports a track (Bounce Track / Export WAV), the Python Backend on the server receives the project's JSON payload:
|
||||
|
||||
1. **Reading Track Instrument Metadata:**
|
||||
```json
|
||||
{
|
||||
"track_id": "track_01",
|
||||
"synth_engine": {
|
||||
"type": "VST3",
|
||||
"plugin_id": "Vital",
|
||||
"soundfont_bank": 0,
|
||||
"soundfont_program": 0
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
2. **Rendering SoundFont (`.sf2`) Instruments:**
|
||||
* `render_engine.py` initializes a FluidSynth instance.
|
||||
* Calls `fl.program_select(channel, sf_id, bank, program)`.
|
||||
* Feeds the list of MIDI notes directly to FluidSynth to render an Audio Buffer.
|
||||
|
||||
|
||||
3. **Rendering VST3 (`.vst3`) Instruments:**
|
||||
* `vst_engine.py` invokes `pedalboard.VST3Plugin("/opt/daw_engine/vst3/Vital.vst3")`.
|
||||
* If DecentSampler is selected, it loads the corresponding Pianobook sample preset file (`.dspreset`).
|
||||
* Converts all MIDI Notes into an array of `pedalboard.Message` events:
|
||||
* Inserts `control_change` (Bank Select) and `program_change` events at timestamp $0.0\text{ s}$.
|
||||
* Inserts `note_on` and `note_off` events matching the pitch and duration parameters of each note.
|
||||
|
||||
|
||||
* Feeds the MIDI message stream into the VST3 instance to generate a high-fidelity Float32 PCM audio stream.
|
||||
* Mixes down the Track PCM Audio Buffers into the Master Mix and creates the final `.wav` output file.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Instrument Selection Checklist
|
||||
|
||||
To ensure that selecting an instrument via the Synth button produces audio output successfully:
|
||||
|
||||
* [ ] **Track is Unmuted:** Verify that the Mute button `[M]` is not active (orange/red) and that the Solo button `[S]` on other tracks is not muting the current track.
|
||||
* [ ] **MIDI Notes in Valid Key Range:** Some instruments (such as Bass or Horns) operate within constrained pitch boundaries (e.g., C1 to C5). Ensure the notes drawn on the Piano Roll fall within the playable range of the selected SoundFont or VST3 instrument.
|
||||
* [ ] **VST3 / SoundFont Files Ready on Server:** Confirm that the `.vst3` binary files are placed inside `/opt/daw_engine/vst3/` and `.sf2` files are present in `/opt/daw_engine/soundfonts/`.
|
||||
* [ ] **Appropriate Volume / Gain Settings:** Verify that the Track 01 Volume slider is configured to $0\text{ dB}$ to avoid signal clipping or silent playback.
|
||||
@@ -0,0 +1,86 @@
|
||||
Nguyên nhân xuất hiện thông báo lỗi từ SpessaSynth Core:
|
||||
`basic_synthesizer_core.ts:169 No preset found for 0:0:0! Did you forget to add a sound bank?`
|
||||
|
||||
Thông số `0:0:0` trong thông báo đại diện cho `Bank MSB : Bank LSB : Program Number` (cấu hình mặc định ban đầu của kênh MIDI). Lỗi này xảy ra do **3 nguyên nhân chính** sau:
|
||||
|
||||
---
|
||||
|
||||
### 1. Phân tích nguyên nhân kỹ thuật
|
||||
|
||||
1. **Chưa gửi lệnh `programChange` & `controllerChange` sang SpessaSynth khi chọn "Pipa"**:
|
||||
* Khi chọn nhạc cụ "Pipa" trên UI, ứng dụng mới chỉ cập nhật giao diện mà **chưa kích hoạt lệnh đổi tiếng** (`synthInstance.programChange(channel, program)`) sang kênh MIDI tương ứng của SpessaSynth.
|
||||
* Khi gõ phím trên `SE49` (nhận dữ liệu Raw MIDI `[144, 65, 78]`), SpessaSynth vẫn truy xuất bộ tiếng ở vị trí mặc định là `Bank 0 : Program 0` (thường là Grand Piano theo chuẩn General MIDI).
|
||||
* Trong SoundFont `DSK_Asian_DreamZ`, vị trí `0:0:0` không tồn tại hoặc không chứa thông tin nốt, khiến SpessaSynth thông báo không tìm thấy preset.
|
||||
|
||||
|
||||
2. **Sai chỉ số Bank / Program của nhạc cụ "Pipa"**:
|
||||
* Tệp SoundFont `DSK_Asian_DreamZ` là bộ tiếng tùy chỉnh (Non-General MIDI). 8 nhạc cụ bên trong (*Pipa, Pipa Tremolo, Luan, Guzhen, Erhu, Koto, Ban Di, Percussion*) được đánh số `Program` riêng biệt.
|
||||
* Nếu UI gửi `program = 0` nhưng thực tế trong file SoundFont, tiếng Pipa nằm ở `program = 1` hoặc `bank = 1`, SpessaSynth sẽ không tìm thấy preset tương ứng.
|
||||
|
||||
|
||||
3. **Lỗi Header/Mapping khi convert từ `.sf2` sang `.sf3**`:
|
||||
* Một số công cụ nén (như `mscore` CLI) khi convert các SoundFont tùy chỉnh có cấu trúc phi chuẩn có thể làm mất hoặc biến đổi bảng thông tin Preset Header.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### 2. Các bước khắc phục
|
||||
|
||||
#### **Bước 1: Gọi `programChange` & `controllerChange` ngay khi chọn nhạc cụ trên UI**
|
||||
|
||||
Đảm bảo khi người dùng chọn nhạc cụ trên giao diện, hàm chọn Bank/Program được kích hoạt trên kênh MIDI phát tiếng:
|
||||
|
||||
```javascript
|
||||
// Khi người dùng chọn "Pipa" trên Dropdown Synth UI
|
||||
async function onSelectTrackInstrument(trackChannel, sfId, bank, program) {
|
||||
// 1. Nạp file .sf3 vào SpessaSynth (nếu chưa nạp)
|
||||
await soundFontPlayerInstance.loadSoundFont(sfId);
|
||||
|
||||
// 2. BẮT BUỘC: Gửi lệnh đổi Bank (CC 0) và Program Change sang SpessaSynth
|
||||
if (soundFontPlayerInstance.synthInstance) {
|
||||
soundFontPlayerInstance.synthInstance.controllerChange(trackChannel, 0, bank);
|
||||
soundFontPlayerInstance.synthInstance.programChange(trackChannel, program);
|
||||
console.log(`[SonicSF] Switched Channel ${trackChannel} -> Bank: ${bank}, Program: ${program}`);
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### **Bước 2: Kiểm tra chính xác chỉ số Bank & Program của Pipa từ Catalog API**
|
||||
|
||||
Sử dụng API `GET /api/v1/plugins/soundfonts/catalog` (từ mô-đun `SoundFontInspector` đã xây dựng) để tra cứu vị trí chính xác của "Pipa":
|
||||
|
||||
```json
|
||||
"dsk_asian_dreamz": {
|
||||
"soundfont_id": "dsk_asian_dreamz",
|
||||
"instruments": [
|
||||
{ "bank": 0, "program": 0, "name": "Pipa" },
|
||||
{ "bank": 0, "program": 1, "name": "Pipa Tremolo" },
|
||||
{ "bank": 0, "program": 6, "name": "Ban Di" }
|
||||
]
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
*Lưu ý:* Nếu kết quả trả về tiếng Pipa nằm ở `program: 1` hoặc `bank: 1`, hãy truyền đúng thông số này vào hàm `programChange`.
|
||||
|
||||
#### **Bước 3: Kiểm tra danh sách Presets mà SpessaSynth đọc được từ file `.sf3**`
|
||||
|
||||
Để đảm bảo quá trình convert `.sf3` không làm hỏng dữ liệu Preset Header, bạn hãy log danh sách preset sau khi nạp tệp:
|
||||
|
||||
```javascript
|
||||
// Thêm log kiểm tra sau khi addSoundFont vào SpessaSynth
|
||||
try {
|
||||
await this.synthInstance.soundFontManager.addSoundFont(buffer);
|
||||
|
||||
// In danh sách các preset đọc được ra console để kiểm tra
|
||||
const loadedSF = this.synthInstance.soundFontManager.soundFonts[0];
|
||||
console.log("[SonicSF] Loaded Presets in SF3:", loadedSF.presets);
|
||||
} catch (e) {
|
||||
console.error("[SonicSF] Error parsing SF3:", e);
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
* Nếu `loadedSF.presets` rỗng (`[]`), file `.sf3` đã bị hỏng khi nén. Bạn hãy thử nạp lại file `.sf2` gốc chưa nén để đối chiếu.
|
||||
@@ -0,0 +1,322 @@
|
||||
# TECHNICAL SPECIFICATION: CLIENT SOUNDFONT OPTIMIZATION USING SF3 AND SPESSASYNTH
|
||||
|
||||
This document details a two-step technical workflow to upgrade the real-time client audio playback experience:
|
||||
|
||||
1. **Server Asset Conversion:** Converts original `.sf2` files into compressed `.sf3` (Ogg Vorbis) format, reducing file size from $30 - 150\text{ MB}$ down to just $3 - 6\text{ MB}$ ($\sim 85-90\%$ compression).
|
||||
2. **Client Engine Upgrade:** Replaces the oscillator emulation logic inside `soundfontPlayer.js` with the SpessaSynth library (Web Audio API / AudioWorklet Engine), achieving $100\%$ authentic audio rendering relative to the server exporter with initial load times of only $1 - 2\text{ seconds}$.
|
||||
|
||||
---
|
||||
|
||||
## STEP 1: AUTOMATED SF2 TO SF3 ASSET CONVERSION ON SERVER
|
||||
|
||||
### 1.1 Technical Principles of the `.sf3` Format
|
||||
|
||||
* `.sf2` files store raw uncompressed PCM Float/Int audio samples (Raw Uncompressed Audio).
|
||||
* `.sf3` files preserve the complete Header, Preset, and Instrument Mapping structure of SF2, but compress raw WAV sample streams using the Ogg Vorbis compression algorithm.
|
||||
* Human ears cannot distinguish quality differences between `.sf2` and `.sf3` playback, but the reduced footprint ensures exceptionally fast browser downloads.
|
||||
|
||||
### 1.2 Installing Conversion Utilities in Server Docker (`Dockerfile`)
|
||||
|
||||
Append `mscore` (MuseScore CLI) or `sf2pack` packages to the `Dockerfile`:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
RUN apt-get update && apt-get install -y \
|
||||
mscore \
|
||||
vorbis-tools \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
```
|
||||
|
||||
### 1.3 Python Automated SoundFont Converter Module (`app/core/soundfont_converter.py`)
|
||||
|
||||
Creates a Python module to automatically scan `.sf2` files within system/upload directories and generate parallel `.sf3` converted files:
|
||||
|
||||
```python
|
||||
import os
|
||||
import subprocess
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class SoundFontConverter:
|
||||
def __init__(self, target_dirs=None):
|
||||
if target_dirs is None:
|
||||
self.target_dirs = [
|
||||
"/opt/daw_engine/soundfonts",
|
||||
"app/storage/uploads/soundfonts"
|
||||
]
|
||||
else:
|
||||
self.target_dirs = target_dirs
|
||||
|
||||
def convert_sf2_to_sf3(self, sf2_path: str) -> str:
|
||||
"""
|
||||
Converts a single .sf2 file to .sf3 using MuseScore CLI.
|
||||
Returns the path to the converted .sf3 file.
|
||||
"""
|
||||
if not os.path.exists(sf2_path):
|
||||
raise FileNotFoundError(f"Source SF2 file not found: {sf2_path}")
|
||||
|
||||
sf3_path = os.path.splitext(sf2_path)[0] + ".sf3"
|
||||
|
||||
# Check if already converted and up-to-date
|
||||
if os.path.exists(sf3_path) and os.path.getmtime(sf3_path) >= os.path.getmtime(sf2_path):
|
||||
return sf3_path
|
||||
|
||||
try:
|
||||
logger.info(f"Converting '{sf2_path}' -> '{sf3_path}'...")
|
||||
# Command: mscore -o output.sf3 input.sf2
|
||||
cmd = ["mscore", "-o", sf3_path, sf2_path]
|
||||
result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
|
||||
if result.returncode == 0 and os.path.exists(sf3_path):
|
||||
logger.info(f"Successfully created SF3 asset: {sf3_path} ({os.path.getsize(sf3_path) / (1024*1024):.2f} MB)")
|
||||
return sf3_path
|
||||
else:
|
||||
logger.error(f"SF2 to SF3 conversion failed: {result.stderr}")
|
||||
return sf2_path # Fallback to original SF2
|
||||
except Exception as e:
|
||||
logger.error(f"Error executing SF2 conversion: {str(e)}")
|
||||
return sf2_path
|
||||
|
||||
def batch_convert_all(self):
|
||||
"""
|
||||
Scans all target directories and converts any missing .sf3 files.
|
||||
"""
|
||||
for sdir in self.target_dirs:
|
||||
if not os.path.exists(sdir):
|
||||
continue
|
||||
for root, _, files in os.walk(sdir):
|
||||
for file in files:
|
||||
if file.lower().endswith('.sf2'):
|
||||
full_sf2_path = os.path.join(root, file)
|
||||
self.convert_sf2_to_sf3(full_sf2_path)
|
||||
|
||||
```
|
||||
|
||||
### 1.4 API Endpoint Serving `.sf3` Files to Clients (`app/api/v1/plugins.py`)
|
||||
|
||||
Provides a static download route serving optimized `.sf3` assets:
|
||||
|
||||
```python
|
||||
@router.get("/soundfonts/download/{sf_id}")
|
||||
async def download_soundfont_asset(sf_id: str):
|
||||
"""
|
||||
Returns the optimized .sf3 file if available, otherwise falls back to .sf2.
|
||||
"""
|
||||
sf3_path = f"/opt/daw_engine/soundfonts/{sf_id}.sf3"
|
||||
sf2_path = f"/opt/daw_engine/soundfonts/{sf_id}.sf2"
|
||||
|
||||
if os.path.exists(sf3_path):
|
||||
return FileResponse(sf3_path, media_type="application/octet-stream", filename=f"{sf_id}.sf3")
|
||||
elif os.path.exists(sf2_path):
|
||||
return FileResponse(sf2_path, media_type="application/octet-stream", filename=f"{sf_id}.sf2")
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="SoundFont asset not found")
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## STEP 2: UPGRADING CLIENT PLAYER USING SPESSASYNTH
|
||||
|
||||
SpessaSynth (`spessasynth_lib`) is a next-generation JavaScript SoundFont Synthesizer written entirely using the Web Audio API & AudioWorklet. It supports direct loading of `.sf3` files without requiring complex C/Wasm compilation wrappers.
|
||||
|
||||
### 2.1 Integrating the SpessaSynth Library into Frontend
|
||||
|
||||
Add the npm package or embed the ES Module script directly inside `index.html`:
|
||||
|
||||
```html
|
||||
<!-- index.html -->
|
||||
<script type="module">
|
||||
import { Synthesizer } from 'https://cdn.jsdelivr.net/npm/spessasynth_lib@latest/dist/spessasynth_lib.js';
|
||||
window.SpessaSynthClass = Synthesizer;
|
||||
</script>
|
||||
|
||||
```
|
||||
|
||||
### 2.2 Client Storage Optimization (`IndexedDB`)
|
||||
|
||||
Caches downloaded `.sf3` files inside `IndexedDB` so that upon reopening the browser, the application loads audio buffers instantly in $0\text{ms}$ without re-fetching from the server.
|
||||
|
||||
```javascript
|
||||
// app/static/js/services/soundfontStorage.js
|
||||
class SoundFontStorage {
|
||||
constructor() {
|
||||
this.dbName = "DAW_SoundFont_Cache";
|
||||
this.storeName = "sf3_buffers";
|
||||
}
|
||||
|
||||
async openDB() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const request = indexedDB.open(this.dbName, 1);
|
||||
request.onupgradeneeded = (e) => {
|
||||
const db = e.target.result;
|
||||
if (!db.objectStoreNames.contains(this.storeName)) {
|
||||
db.createObjectStore(this.storeName);
|
||||
}
|
||||
};
|
||||
request.onsuccess = () => resolve(request.result);
|
||||
request.onerror = () => reject(request.error);
|
||||
});
|
||||
}
|
||||
|
||||
async getBuffer(sfId) {
|
||||
const db = await this.openDB();
|
||||
return new Promise((resolve) => {
|
||||
const tx = db.transaction(this.storeName, "readonly");
|
||||
const store = tx.objectStore(this.storeName);
|
||||
const req = store.get(sfId);
|
||||
req.onsuccess = () => resolve(req.result || null);
|
||||
req.onerror = () => resolve(null);
|
||||
});
|
||||
}
|
||||
|
||||
async saveBuffer(sfId, arrayBuffer) {
|
||||
const db = await this.openDB();
|
||||
return new Promise((resolve, reject) => {
|
||||
const tx = db.transaction(this.storeName, "readwrite");
|
||||
const store = tx.objectStore(this.storeName);
|
||||
const req = store.put(arrayBuffer, sfId);
|
||||
req.onsuccess = () => resolve(true);
|
||||
req.onerror = () => reject(req.error);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export const sfStorage = new SoundFontStorage();
|
||||
|
||||
```
|
||||
|
||||
### 2.3 Comprehensive Upgrade of `soundfontPlayer.js`
|
||||
|
||||
Replaces oscillator emulation loops with the SpessaSynth Engine:
|
||||
|
||||
```javascript
|
||||
// app/static/js/services/soundfontPlayer.js
|
||||
import { sfStorage } from './soundfontStorage.js';
|
||||
|
||||
class RealSoundFontPlayer {
|
||||
constructor() {
|
||||
this.audioCtx = null;
|
||||
this.synthInstance = null;
|
||||
this.currentSfId = null;
|
||||
this.isInitialized = false;
|
||||
}
|
||||
|
||||
async init(audioContext) {
|
||||
if (this.isInitialized) return;
|
||||
this.audioCtx = audioContext;
|
||||
|
||||
if (window.SpessaSynthClass) {
|
||||
// Initialize SpessaSynth Synthesizer routed to Web Audio Destination
|
||||
this.synthInstance = new window.SpessaSynthClass(this.audioCtx.destination);
|
||||
this.isInitialized = true;
|
||||
console.log("[SonicSF] SpessaSynth Engine Initialized successfully.");
|
||||
} else {
|
||||
console.warn("[SonicSF] SpessaSynth library not loaded. Falling back to basic audio.");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads .sf3 file from IndexedDB Cache or Server API
|
||||
*/
|
||||
async loadSoundFont(sfId = "generaluser_gs") {
|
||||
if (!this.isInitialized) return;
|
||||
if (this.currentSfId === sfId) return;
|
||||
|
||||
console.log(`[SonicSF] Loading SoundFont asset: ${sfId}...`);
|
||||
|
||||
// 1. Try fetching from IndexedDB Cache
|
||||
let buffer = await sfStorage.getBuffer(sfId);
|
||||
|
||||
if (!buffer) {
|
||||
// 2. If missing, download .sf3 asset from Server (~4MB footprint)
|
||||
try {
|
||||
const response = await fetch(`/api/v1/plugins/soundfonts/download/${sfId}`);
|
||||
if (!response.ok) throw new Error("Network download failed");
|
||||
buffer = await response.arrayBuffer();
|
||||
|
||||
// Save to IndexedDB for instant future loads
|
||||
await sfStorage.saveBuffer(sfId, buffer);
|
||||
} catch (err) {
|
||||
console.error(`[SonicSF] Failed to load SoundFont '${sfId}':`, err);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Load .sf3 ArrayBuffer into SpessaSynth Engine
|
||||
try {
|
||||
await this.synthInstance.soundFontManager.addSoundFont(buffer);
|
||||
this.currentSfId = sfId;
|
||||
console.log(`[SonicSF] SoundFont '${sfId}' loaded into Wasm/JS memory.`);
|
||||
} catch (e) {
|
||||
console.error("[SonicSF] Error parsing SF3 buffer in SpessaSynth:", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures MIDI Channel, Bank, Program
|
||||
*/
|
||||
applyAITrackInstrument(channel, bank, program) {
|
||||
if (!this.synthInstance) return;
|
||||
// Bank Select (CC 0)
|
||||
this.synthInstance.controllerChange(channel, 0, bank);
|
||||
// Program Change
|
||||
this.synthInstance.programChange(channel, program);
|
||||
}
|
||||
|
||||
/**
|
||||
* Plays a MIDI note in real time with 100% authentic instrument sound
|
||||
*/
|
||||
playNote(pitch, velocity = 0.8, durationSec = 1.0, channel = 0) {
|
||||
if (!this.synthInstance) return;
|
||||
|
||||
const midiPitch = Math.min(127, Math.max(0, pitch));
|
||||
const midiVelocity = Math.floor(velocity * 127);
|
||||
|
||||
// Note On
|
||||
this.synthInstance.noteOn(channel, midiPitch, midiVelocity);
|
||||
|
||||
// Note Off scheduled by duration
|
||||
setTimeout(() => {
|
||||
this.synthInstance.noteOff(channel, midiPitch);
|
||||
}, durationSec * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
export const soundFontPlayerInstance = new RealSoundFontPlayer();
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UI INTEGRATION WORKFLOW (`app.jsx`)
|
||||
|
||||
1. **Application Startup:**
|
||||
* When the user clicks on the web page or triggers Transport Play, call `soundFontPlayerInstance.init(audioCtx)` and trigger a background fetch for the default General SoundFont (`generaluser_gs.sf3`).
|
||||
|
||||
|
||||
2. **When User Selects Instrument via Synth Button:**
|
||||
* Read `sf_id` from the selected instrument object.
|
||||
* Call `await soundFontPlayerInstance.loadSoundFont(sf_id)`.
|
||||
* Call `soundFontPlayerInstance.applyAITrackInstrument(channel, bank, program)`.
|
||||
|
||||
|
||||
3. **When Playing Piano Roll / Timeline:**
|
||||
* Every emitted MIDI note invokes `soundFontPlayerInstance.playNote(pitch, velocity, durationSec, channel)`.
|
||||
* Audio signals pass through Envelopes, Modulators, and Standard General MIDI Sample Mapping via SpessaSynth $\rightarrow$ outputs $100\%$ authentic instrument audio matching the server WAV export engine.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## POST-OPTIMIZATION PERFORMANCE COMPARISON
|
||||
|
||||
| Metric | Before Optimization (SF2 + Oscillator) | After Optimization (SF3 + SpessaSynth) |
|
||||
| --- | --- | --- |
|
||||
| **Asset Download Size** | $35\text{ MB} - 140\text{ MB}$ (Extremely Heavy) | 🟢 $3.5\text{ MB} - 5.5\text{ MB}$ (Ultra Light) |
|
||||
| **Initial Load Time** | $10 - 25\text{ seconds}$ | ⚡ $1 - 2\text{ seconds}$ |
|
||||
| **Subsequent Load Time** | $10 - 25\text{ seconds}$ | ⚡ $0\text{ seconds}$ (Retrieved from IndexedDB Cache) |
|
||||
| **Preview Fidelity** | 🔴 Crude Emulated Waveform (Oscillator) | 🟢 $100\%$ Authentic SoundFont Rendering |
|
||||
| **Keypress Latency** | $0\text{ms}$ | ⚡ $0\text{ms}$ (Runs on AudioWorklet) |
|
||||
@@ -0,0 +1,144 @@
|
||||
# CLIENT-SIDE EXECUTION FLOW (SF3 + SPESSASYNTH + INDEXEDDB)
|
||||
|
||||
This document describes the step-by-step processing chain that takes place inside the Client Browser, from launching the Web DAW application, downloading and buffering `.sf3` instrument files, and setting up MIDI channels, to outputting real-time audio.
|
||||
|
||||
---
|
||||
|
||||
## 1. SEQUENCE DIAGRAM
|
||||
|
||||
```text
|
||||
[ USER / UI ] [ APP / CLIENT ] [ INDEXEDDB ] [ SERVER API ] [ SPESSASYNTH ENGINE ]
|
||||
| | | | |
|
||||
1. Open Web Page ---------> | Initial AudioCtx | | |
|
||||
| | Init SpessaSynth -----------------------------------------------> | Connect Destination
|
||||
| | Fetch Catalog --------------------------->| Get /catalog |
|
||||
| | | | |
|
||||
2. Select Instrument -----> | Read (sf_id, bank, prog) | |
|
||||
(e.g., Pipa) | Query SF3 Buffer --->| Check Key (sf_id) | |
|
||||
| | | -- (Miss) -------->| Fetch /download/sf_id |
|
||||
| | | | Return .sf3 (~4MB) |
|
||||
| | <--------------------| Save ArrayBuffer --| |
|
||||
| | Load SF3 Memory ------------------------------------------------> | addSoundFont(buffer)
|
||||
| | | | |
|
||||
3. Channel Router --------> | Switch Bank/Program -------------------------------------------> | controllerChange(ch, 0, bank)
|
||||
| | | | | programChange(ch, prog)
|
||||
| | | | |
|
||||
4. Trigger MIDI Key ------> | Raw MIDI Event | | |
|
||||
(or Timeline Play) | (noteOn: pitch, vel) -------------------------------------------> | noteOn(ch, pitch, vel)
|
||||
| | | | | AudioWorklet Synthesis
|
||||
| | <------------------------------------------------------------------ | Audio Out (User Speakers)
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. DETAILED PROCESSING PHASES
|
||||
|
||||
### PHASE 1: BOOTSTRAPPING & ENGINE INIT
|
||||
|
||||
* **Web Audio Context Initialization:** Upon the user's first interaction with the web page (Mouse Click/Keypress), the application initializes the `AudioContext`.
|
||||
* **SpessaSynth Synthesizer Initialization:** The `soundfontPlayer.js` module instantiates `SpessaSynthClass` and connects its output directly to `audioCtx.destination`:
|
||||
```javascript
|
||||
this.synthInstance = new window.SpessaSynthClass(this.audioCtx.destination);
|
||||
|
||||
```
|
||||
|
||||
|
||||
* **Instrument Catalog Load (Catalog Context):** The Frontend dispatches a `GET /api/v1/plugins/soundfonts/catalog` request to load the `condensed_catalog`, which contains lookup tables for `sf_id`, `bank`, and `program`.
|
||||
|
||||
---
|
||||
|
||||
### PHASE 2: `.SF3` ASSET LOADING & CACHING
|
||||
|
||||
Triggered when a user selects an instrument via the Synth UI button (or when the AI Copilot spawns a new Track with a designated instrument, e.g., `dsk_asian_dreamz`):
|
||||
|
||||
* **Query Browser Cache (IndexedDB):** The Client calls `sfStorage.getBuffer(sfId)` to search for the `.sf3` file's `ArrayBuffer` inside the `DAW_SoundFont_Cache` database.
|
||||
* **Handling Cache Hit vs Cache Miss:**
|
||||
* **Cache Hit ($0\text{ms}$):** Retrieves the `ArrayBuffer` directly from the browser's RAM/Storage.
|
||||
* **Cache Miss:**
|
||||
1. Sends a `GET /api/v1/plugins/soundfonts/download/{sf_id}` request to the Server.
|
||||
2. Downloads the compressed, optimized `.sf3` asset (ultra-lightweight size $\sim 3.5 - 5.5\text{ MB}$).
|
||||
3. Invokes `sfStorage.saveBuffer(sfId, arrayBuffer)` to store it inside IndexedDB for subsequent visits.
|
||||
|
||||
|
||||
|
||||
|
||||
* **Load Data into SpessaSynth Wasm/JS Memory:** Passes the `ArrayBuffer` to SpessaSynth Engine's `SoundFontManager`:
|
||||
```javascript
|
||||
await this.synthInstance.soundFontManager.addSoundFont(buffer);
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### PHASE 3: BANK/PROGRAM ROUTING & MIDI CHANNEL SETUP
|
||||
|
||||
This is the most critical phase to resolve `No preset found for 0:0:0` errors.
|
||||
|
||||
* **MIDI Channel Assignment:**
|
||||
* **Melodic Instruments (Piano, Pipa, Strings, Brass, etc.):** Allocated to Channels 0 through 8.
|
||||
* **Percussion / Drum Kits (Bank 128):** Mandatory allocation to Channel 9 (GM Standard Channel 10).
|
||||
|
||||
|
||||
* **Dispatch Bank Select & Program Change to SpessaSynth Engine:** Prior to scheduling any note events, the Client triggers two simultaneous control events:
|
||||
```javascript
|
||||
// 1. Select Bank (Control Change 0)
|
||||
this.synthInstance.controllerChange(channel, 0, bank);
|
||||
|
||||
// 2. Select Program (Program Change)
|
||||
this.synthInstance.programChange(channel, program);
|
||||
|
||||
```
|
||||
|
||||
|
||||
*Example for Pipa (`dsk_asian_dreamz`):* Calls `controllerChange(0, 0, 0)` and `programChange(0, 0)`. SpessaSynth switches Channel 0's state to the Pipa instrument patch.
|
||||
|
||||
---
|
||||
|
||||
### PHASE 4: REALTIME SYNTHESIS & AUDIO OUTPUT
|
||||
|
||||
Triggered when receiving note-control signals (from a Hardware MIDI Keyboard or Timeline Transport Playback):
|
||||
|
||||
* **Scenario A: User plays a Hardware MIDI Keyboard (e.g., Nektar SE49)**
|
||||
1. The browser receives a Raw MIDI Event: Web MIDI API captures message `[144, 65, 78]` (`NoteOn`, `Pitch 65`, `Velocity 78`).
|
||||
2. **Latency Compensation:** Calculates real-time offsets and issues `NoteOn` to SpessaSynth:
|
||||
```javascript
|
||||
const midiPitch = pitch;
|
||||
const midiVelocity = Math.floor(velocity * 127);
|
||||
this.synthInstance.noteOn(channel, midiPitch, midiVelocity);
|
||||
|
||||
```
|
||||
|
||||
|
||||
3. **Key Release:** Triggers a `NoteOff` event:
|
||||
```javascript
|
||||
this.synthInstance.noteOff(channel, midiPitch);
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
* **Scenario B: User triggers Play on Timeline / Piano Roll**
|
||||
1. **Transport Controller & Scheduler (`PrecisionAudioScheduler`):** Scans for MIDI notes located within the moving Playhead window.
|
||||
2. **Note Scheduling:**
|
||||
* Converts beat positions to precise audio timing based on BPM tempo (`exactAudioTime`).
|
||||
* Dispatches `noteOn(channel, pitch, velocity)` at the exact timestamp $T_{\text{start}}$.
|
||||
* Dispatches `noteOff(channel, pitch)` at timestamp $T_{\text{start}} + T_{\text{duration}}$.
|
||||
|
||||
|
||||
|
||||
|
||||
* **Audio Worklet Audio Rendering:** SpessaSynth Engine reads Ogg/WAV sample data inside the `.sf3` asset, applies Envelopes (ADSR), Modulators, and Gain Control parameters on the designated Channel, and pushes PCM audio data directly to user speakers with $0\text{ms}$ latency.
|
||||
|
||||
---
|
||||
|
||||
## 3. 100% RELIABILITY VERIFICATION CHECKLIST
|
||||
|
||||
* [ ] `.sf3` files loaded into the browser open without triggering `Corrupted File` errors.
|
||||
* [ ] The `sfStorage.getBuffer` function successfully stores and retrieves `ArrayBuffer` data from IndexedDB.
|
||||
* [ ] Both `controllerChange(channel, 0, bank)` and `programChange(channel, program)` are invoked immediately upon changing instruments on the UI.
|
||||
* [ ] Percussion/Drum instruments are persistently allocated to Channel 9.
|
||||
* [ ] Console logs confirm: `[SonicSF] Switched Channel X -> Bank: B, Program: P`.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user