Opening the export menu imported onnxruntime-web, and importing that runtime is
what starts its thread pool: one Web Worker per thread, each holding a copy of
the 27MB wasm build, held by the page until it dies whether or not an export is
ever asked for. Those workers are the child processes a visitor finds hanging off
the tab (and the tab, and the browser, goes when one of them is killed), and the
memory is what a laptop has none of when the studio is closed and opened again.
The menu now fetches the files by name and lets the browser's cache do the
warming; the runtime is imported by the export that needs it. The threads are
capped at four, and pagehide hands the session back instead of leaving the
renderer to reap it with the page.
The GPU export now runs the same upscaler in half precision. The chip is handed
2.34MB of weights instead of 4.88MB, and where its shaders can multiply in fp16
it does twice the work per pass.
`realesr-fp16.py` is the conversion, run on what `realesr-gpu.py` already
wrote (the PReLU-rewritten model), never instead of it. onnxconverter-common's
`keep_io_types` needed two of its own mistakes put right:
- It rewrites the consumers of the graph input but misses the one that never
goes through the network. This model adds a Resize of the ORIGINAL photo to
the upsampler's output, that Resize reads the graph input directly, and the
runtime refuses a graph whose final Add mixes fp32 and fp16. The consumer is
rewired onto the cast that `keep_io_types` should have sent it through.
- It also half-precisions Resize's `scales` — ONNX defines that input as
float32 whatever the rest of the graph does, and a runtime that opens the
file at all rejects the whole graph: "Type 'tensor(float16)' of input
parameter (/Constant_output_0) of operator (Resize) is invalid", on the GPU
as much as on the processor. The script widens it back and asserts it did.
The tensor the app builds stays float32 and the model's two Cast nodes are its
own edge, so nothing in superRes.ts or App.tsx has to know which copy it got:
205 nodes, 101 fp16 weights, io still float.
`openSession` asks for the model only where the adapter advertises
`shader-f16` — a provider without it emulates the type on the same file at the
same speed, so the smaller download would be the only thing gained. The order
is fp16 on the GPU, fp32 on the GPU, fp32 on the processor, each attempt
falling through on its own failure.
Measured on the rebuilt container (BASE=http://localhost:8090):
- fp16 vs fp32 on a 128x128 tile, same graph: max abs diff 0.0025 (0.65/255),
mean 0.00028, psnr 71.0dB.
- sr-f16-chooser.cjs 4 PASS / 0 FAIL: on a forged adapter advertising
`shader-f16`, the fp16 file is the FIRST model asked for; on one whose device
refuses, the fp32 file is fetched for the processor and the 4K export still
lands (7,555,377 bytes, 19.6s), no console errors.
- superres-test.cjs 32 PASS / 0 FAIL, sr-crop-export.cjs 0 FAIL,
web-smoke.cjs 0 FAIL, sr-model-probe.cjs 0 FAIL.
- npx tsc --noEmit clean.
ponytail: the speed of the fp16 path is NOT measured — this container has no
WebGPU adapter (not even lavapipe/swiftshader, headed through xvfb), so every
export here runs the wasm fallback. sr-model-probe.cjs on a machine with a GPU
is what would show it.
Also worth noting for the next person: in a browser with no working adapter,
the runtime builds the device BEFORE it fetches the model, so no probe in a
GPU-less container can observe which model was chosen — a stub whose device
throws leaves the network silent. The chooser probe forges a device good enough
to be accepted for exactly that reason.
An export larger than the photo came back flat: the model was never shown the
finest detail the photo held. Before it ran, the source was drawn down to
`scale / 4` of its size — floored at half — and only then handed over, on the
reasoning that a four-for-one model reading `target / 4` invents exactly the
destination and a whole photo would waste three quarters of its output. That
holds for a perfect resampler; it is not what this one is. A 2400px photo going
to 4K was fed at 1200px, and detail finer than the feed's own pixel — 1px
stripes, skin, foliage, fabric — was averaged into flat grey before the model
ever saw it. The draw then had that grey to enlarge, and no model can put back
what it was never given.
The feed is now the bitmap itself, read once at its own size: `drawImage(bitmap,
0, 0)`, no intermediate scale, no floor. The model's four-for-one is spent in
the destination draw instead, which reduces to `scale` and keeps what the photo
actually held. That draw also stops defaulting to `low` — it is usually a
reduction by up to four, and `low` would keep one sample in four of what the
model has just drawn.
Measured against the same running stack, a 2400x1800 source exported at 4K with
bands of 1/2/4/8/16px stripes and a patch of per-pixel grain, each band scored
by how it correlates with the pattern the source held at the source's own pixel
pitch (`r` / on-minus-off swing), plus the grain's high-frequency energy:
p1 p2 grain sd secs
before -0.01 / -0.0 0.94 / 204.0 13.9 51.0
after 0.89 / 179.3 0.95 / 214.7 25.3 188.4
hqresize 0.96 / 83.2 0.99 / 125.9 16.1 —
The 1px band went from uncorrelated and flat to 0.89 — the finest detail the
photo has now reaches the file. Grain lands above the plain-resize reference
rather than below it, which is the model enlarging texture instead of a filter
smearing it.
ponytail: the whole photo per tile means 80 tiles for a 2400px source where 20
were enough, so the wasm path (no WebGPU in the test chromium) grew from 51s to
188s for that export. It is the price of the detail and it is paid once per
export, off the critical path; a device with WebGPU, or a smaller source, does
not pay it this way.
Verified on the rebuilt container (BASE=http://localhost:8090):
- sr-detail-probe.cjs, midtone source so the app's tone pipeline cannot clip
the very detail being measured (an earlier all-contrast version of it reported
"grain 0.00" for the model AND for a plain resize — it was measuring the clip)
- superres-test.cjs 32 PASS / 0 FAIL (export sizes, 4K tile seams clean)
- sr-crop-export.cjs 0 FAIL
- npx tsc --noEmit clean.
The WebGPU execution provider has no PReLU kernel. The model is 34 convolutions
with a PReLU after every one of them, so an export that took the GPU path was
split 33 times: each activation came off the chip to be activated on the
processor and went straight back, a 64-channel map in both directions, per tile.
A machine with a good graphics chip was not exporting any faster for having it.
PReLU(x) is exactly Relu(x) - slope * Relu(-x), and Relu, Neg, Mul and Sub the
provider does implement, so scripts/realesr-gpu.py writes the 33 activations out
as those four and drops the slopes nobody reads any more. The model file is the
output of that script, not the file as published.
One 256x256 tile through the model before and after, on a WebGPU session: the
runtime no longer reports nodes left off the preferred provider (it did, once,
before) and the processor path answers bit for bit what it answered before. The
warning itself cannot be switched off from here - env.logLevel is read when the
runtime module initialises, before any of this runs - so the graph was fixed
rather than the lines hidden.
onnxruntime hands a webgpu session whatever adapter the browser picks by
default, which on a laptop with both is the one built into the processor: the
export then waits on the slow half of the machine for no reason. The runtime
reads `env.webgpu.powerPreference` when it builds the webgpu session, so set it
to high-performance; a browser with nothing to honour the preference with
still falls back to the threaded wasm path exactly as before.
The model's own factor is 4 and the export's target is some number of pixels,
and the two were never reconciled: a 2400x1800 photo exporting at 4K was run
through the model at 4x — 9600x7200 of invented detail — and then three
quarters of it were thrown away by the draw that lands the file on 3840. The
arithmetic was the whole wait. Measured on the wasm path, one export: 153.2s.
The photo is now resampled once to `targetLongest / 4` before the model reads
it, so the model still answers at its own 4x and the answer is the size the
export asked for. Same 2400x1800 to 4K: 42.7s, 80 tiles of model for 20. Half
the photo's pixels is the floor — below that the model is no longer enlarging
the picture, it is drawing a new one from memory — and the ceiling is the
photo's own size, so a gain past 4 behaves exactly as it did.
Nothing in the finished file gives the smaller input away: the 6px stripes come
back at full contrast (254.9 vs 254.8), the black-to-white step lands on the
same pixel (x=625 in both) and rises in 1px instead of 3.
The 32MB of runtime and model are also fetched, and one 16x16 tile pushed
through the graph, when the export menu opens rather than after a size is
picked: the visitor waits for the pixels, not for the download.
crop 1:1 2400x1800 to 4K: 155.8s -> 52.6s, crop 3:4: 153.0s -> 54.1s.
The super-resolution export ran single-threaded because the site was not
cross-origin isolated and the runtime had no SharedArrayBuffer to spread a tile
over. nginx now sends COOP and COEP — on the document, and on the script
responses a nested worker fetches, which Chromium checks the same way and blocks
as `coep-frame-resource-needs-coep-header` without them — and the loader asks
for `min(8, hardwareConcurrency)` threads whenever the page is isolated, falling
back to one if the headers ever go missing. A worker script is also why the
landing's QR image needed `crossOrigin`: COEP refuses a cross-origin image that
did not opt in with CORS.
The unpack was the other half. Each tile was clamped a channel at a time and
painted whole, padded ring and all; it now writes straight into the
Uint8ClampedArray, which clamps and rounds on assignment, and skips the ring
rather than drawing it and clipping it away.
640x480 to 4096: 39.0s to 16.2s. 1000x750 to 4096: 89.6s to 30.9s. One 256px
tile through the model: 6.8s to 1.8s. Measured on the wasm path — the test
browser has no GPU adapter — so a WebGPU export, still per-tile inference, keeps
its own times.
A tile's destination rectangle was placed at x0 * scale, and that scale
is rarely whole, so every 256px boundary landed on a fraction of a pixel.
The edge was drawn half covered, stayed transparent, and the JPEG export
flattened that transparency onto black: a dark line down each seam.
Snap both destination edges to whole pixels instead, so neighbouring
tiles share the exact same boundary, and make the destination context
opaque so no partly covered pixel can survive as transparency again.
Measured on a 640px source: the seam at 2K was 46 levels darker than its
neighbours (96 at 4K); it is now within one level of them.
The server still never sees a photo, so the model has to run in the page.
Real-ESRGAN x4v3 ships as a 4.9MB ONNX in public/models and is loaded
lazily on the first export that actually needs it; the wasm runtime is
copied next to CanvasKit at build time and stays lazily fetched, cached
for 30 days. Vite is told onnxruntime-web is external-wasm so no 28MB
asset lands in the bundle.
UNCHANGED keeps the old path and the tier cap; 2K/4K/custom upscale only
when the request is larger than the photo being edited, otherwise they
resize down. Guests keep UNCHANGED and 2K. Tiling is 256px with an 8px
overlap, so memory follows the target size rather than four times it.
3dtours