feat(studio): HEAL invents what was under the brush with MI-GAN, and keeps the blurred fill for the machines that cannot run it
The dust brush filled its hole with a blur of the hole's own rim — the light beside it, carried in. Right for a speck on a wall, and visibly thin over anything with structure: a stroke across a shirt came back as the average of a shirt. The fill is now a learned one. engine/inpaint.ts runs MI-GAN (andraniksargsyan/migan, the same `migan_pipeline_v2.onnx` inpaint-web ships) through onnxruntime-web on the visitor's own machine: 27MB of weights in public/models, WebGPU where the graph fits the provider, the processor otherwise, one warm-up pass so the first repair is not the one that pays for the kernels. It is handed the masked crop and nothing else — the model's mask polarity is 255 KEEP / 0 HOLE, the opposite of the obvious guess, and the other way round repaints the frame and leaves the hole (measured). Step 6e of the renderer cuts the crop at the resolution the model will actually run at (512 on its longest side — the graph is fully convolutional and would take a whole frame, and ten seconds over one), builds the mask from the brush's own circles, and draws the answer into the fill surface at its own place in the box. The composite, the feathered mask and the untouched frame around it are unchanged, so nothing outside the mask moves — and when the model cannot be had the blurred fill is still built first and still stands. The preview gets this for free: renderPhoto was already async and the preview was already debounced. Node probe: the model + polarity + read/write path run; heal-check.mjs keeps measuring the fallback and now asserts the wiring and the polarity. Browser: engine/inpaint.ts in a real Chromium returns a patch (hole moves, kept pixels within 0.02 of a level), and the built app renders a 42-spot stroke in 4.7s on single-threaded wasm with no fallback warning — 1.15s and the warning when the model is taken away.
This commit is contained in:
@@ -9,6 +9,13 @@
|
||||
// measured here, on synthetic photos whose answer is known, with CanvasKit
|
||||
// running the same two passes exportEngine runs.
|
||||
//
|
||||
// The fill now has a second, better author where one can be had — MI-GAN, over
|
||||
// the mask (engine/inpaint.ts) — and this check deliberately measures the other
|
||||
// one: the blur that stands in when the model cannot be had, which is the case
|
||||
// on a machine without the runtime and the case this file can run offline. What
|
||||
// is asserted about the model is only that the pass is still wired in, and with
|
||||
// which polarity.
|
||||
//
|
||||
// node scripts/heal-check.mjs
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
@@ -48,6 +55,22 @@ assert.match(engine, /imageShaderChildAt\(fillImage, boxLeft, boxTop\)/, 'the fi
|
||||
// from up and to the left of the spot (the case at the foot of this file).
|
||||
assert.match(engine, /drawImageRect\(cutImage, box, onto, /, 'the fill is drawn at the box\'s frame coordinates, not at the surface\'s own corner');
|
||||
|
||||
// The model half is wired into the same pass: the masked crop is asked for, the
|
||||
// answer is written into the fill surface over the box, and the mask it is
|
||||
// handed is built from the brush's own circles. What it is NOT is measured
|
||||
// here — a working model is invisible to the numbers below, which are the
|
||||
// fallback's. The polarity is MI-GAN's, 255 KEEP and 0 HOLE, and it is the one
|
||||
// thing that, got backwards, repaints the frame and leaves the hole (measured in
|
||||
// engine/inpaint.ts; the probe is gone).
|
||||
const inpaint = readFileSync(new URL('../src/engine/inpaint.ts', import.meta.url), 'utf8');
|
||||
assert.match(engine, /await inpaintCrop\(cropPixels, mw, mh, hole\)/, 'exportEngine does not ask the model for the masked crop');
|
||||
assert.match(engine, /patchSurface\.getCanvas\(\)\.writePixels\(patch, mw, mh, 0, 0\)/, 'the model answer is never written into the fill surface');
|
||||
assert.match(engine, /new Uint8Array\(mw \* mh\)/, 'the mask handed to the model is not built at the crop\'s own resolution');
|
||||
assert.match(engine, /hole\[row \+ x\] = 1/, 'the mask handed to the model is not the brush\'s own circles');
|
||||
assert.match(inpaint, /fill\(255\)/, 'the model mask no longer starts at KEEP');
|
||||
assert.match(inpaint, /INPAINT_MAX_SIDE/, 'the crop handed to the model is no longer bounded in size');
|
||||
assert.match(inpaint, /inpaintCropSize/, 'nothing tells the renderer the resolution the model will run at');
|
||||
|
||||
// --- the stored spots --------------------------------------------------------
|
||||
// A recipe written by the older brush carries sx/sy. It still loads: those two
|
||||
// numbers are no longer read, and nothing else about the spot moves.
|
||||
|
||||
Reference in New Issue
Block a user