The blown gate draws a clipped pixel at its own maximum -- one value across
the three channels, so the frame's highlights carry no cast -- but the
per-channel tone curve runs after it and re-tints what the gate had just
made neutral. One cubic a channel, fitted on a grid that has no block left
to fit where the frame ran out (previewMatch drops the fully blown ones),
so at the plateau the three curves agree only at 1.0 and part company
either side of it.
On the ORF this was reported on, the moonlit sky came back 254,255,255 and
253,255,254 -- red under green across a quarter of the frame.
The gate is the develop's own statement that the pixel had no colour of its
own, so it is re-read on the value that leaves the shader. The blown pixels
now measure 254.95,254.94,254.97 (R/G 1.0000) against 253.44,254.94,254.51
(0.9941) before; their two most common triples, 254,255,255 at 149315 and
253,255,254 at 122127, collapse to 255,255,255 at 278099. The bright bands
land at -0.0,-0.0,-0.0 against the embedded preview where they were
-2.1,-0.0,-1.0, and midtones and the lower half are untouched.
The least-squares 3x3 that mapped the develop onto the camera's JPEG won
on luminance, whose variance is ~80x the chroma's, so it paid for its
match by crushing the R-G axis: 11.6 -> 4.6 standard deviations, against
8.3 in the camera's own preview and 12.3 in its JPEG. That is the olive
cast -- the sky and the rice both shifted yellow-green.
The colour chain was never at fault: camera_mul x rgb_cam already agrees
with LibRaw to the digit, and the app's fit is bit-identical to a numpy
reference of the same problem, which is how the 3x3 was cleared.
Fit only the per-channel tone curve now. R-G comes back to 8.6 (900px)
and 10.5 (full), and B/G lands on the preview's 0.841 to three places.
The develop preferred cam_xyz, whose rows are the XYZ of each camera channel and
which nothing normalised: on the ORF this was reported on it left the frame
green and blue (R/G 0.921, B/G 0.886) where the file's own preview sits at
1.013 / 0.841, and saturated reds came back as the dark purple the frame was
reported for — a red pixel's green ran negative through a row that carries
-2.64, so it clipped to 0 while the knee pulled red down with it.
LibRaw hands back dcraw's own rgb_cam, already the camera -> sRGB transform with
rows summing to one, and it is now applied as handed back: R/G 1.020 B/G 0.920,
and the fit against the embedded preview follows to mean 167.5,164.8,138.5
against the preview's 167.0,164.8,138.6 and the camera's own JPEG's 165,162,133.
Dividing rgb_cam by pre_mul — which carries that row normalisation — is what the
frame before this one did instead, and it undoes it.
The cam_xyz chain stays as the fallback for a file with no rgb_cam, with its
rows normalised so a neutral frame opens neutral. invert3x3, unused since the
frame stopped going through the chain, is dropped.
A scan of 36 real files off the local disk took 8957ms — 249ms a frame — and
the page was idle for nearly all of it: a CPU profile over the walk is 61.5%
idle, so what the catalogue was doing was waiting, not working. Of that time the
bytes and the LibRaw preview are 2656ms and the tile 5761ms, while the EXIF read
is 14ms of the lot. One frame at a time spends the disk's latency and the
decoder's thread on nothing, and a JPEG was read whole — 8MB through a JS array
to find a date in its first kilobyte — then decoded at 24MP to be drawn at 512px.
The walk now runs four frames at a time. A frame that is new is read as a 256KB
header slice unless it is a RAW, which LibRaw has to have whole to seek to the
preview inside it; the shutter time comes off that slice. The tile is asked of
the decoder at 512 on the long edge, so a quarter of the pixels of a 24MP frame
are ever allocated, and the aspect comes from the frame's own SOF header for a
JPEG and from an eight-pixel decode for anything else; the canvas is left to
re-encode what comes back. The column still counts a frame the moment the scan
reaches it, so which frames are up is unchanged, and a frame that has not moved
is still dropped on its size and its time before a byte of it is read.
Measured on the same roll (24 JPEG 8.2MB + 12 RAW 22.6MB, two levels deep,
served over loopback with no added delay):
one frame at a time 8957ms 249ms/frame
four lanes 5474ms 152ms/frame
+ the header slice 4793ms 133ms/frame
Six lanes came out worse than four (5259ms) and is not what this does: past a
few the disk and the decoder are the limit.
Verified:
library-check.mjs — 35 steps, all passed. scan-nav-check.mjs and
roll-walk-check.mjs — all passed.
frontend tsc --noEmit clean. Live 8090 on index-CdakqRi8.js matching dist/:
/, /library and /app 200 with 0 console errors.
ponytail: the lanes run on the main thread, and a RAW still reads whole per file
— a few LibRaw opens at once now — so a 5000-frame folder is still lumpy; move
the walk into a worker and a RAW's preview onto a sync handle when that is real.
A frame smaller than 512 is now scaled up to it rather than drawn at its own
size, which is invisible at the 132–220px a tile is painted at; keep the old fit
if a print is ever taken off a tile.
Co-authored-by: PenguinHarness <noreply@penguin.local>
A RAW studio that cannot see a folder is one photo at a time. /library
now takes a folder through Chromium's directory picker, keeps the handle
in IndexedDB so the folder is there on the next visit, and walks it into
a grid: one thumbnail per frame, the frame's own date, and the recipe it
was last graded with. Nothing is uploaded and nothing is read twice —
the RAW itself is opened only when a tile is clicked, at which point the
studio develops it and the recipe comes back on top. The studio files
every change back against the frame, debounced, so reopening a RAW is
not doing the grade again.
Thumbnails come off LibRaw's unpack_thumb for a RAW, which is a seek and
a copy where a develop is a full decode of every pixel, and off
createImageBitmap for anything else. A RAW with no preview inside it
gets a placeholder tile rather than a minute of decoding per file.
node scripts/library-check.mjs
ok both frames indexed as tiles — 2 tiles from P1010256.JPG + P1010256.RW2
ok thumbnail for P1010256.JPG — 40400 bytes, jpeg=true
ok thumbnail for P1010256.RW2 — 39895 bytes, jpeg=true (LibRaw preview)
ok studio developed the frame from its handle
ok the address was handed back — url=/app (no ?lib= left behind)
ok the look was filed back against the frame — baseFilter=none, 19 knobs
ok the tile says the frame is edited
LibRaw's half-size demosaic was on. The Ricoh GR's own DNG (D0004128.DNG)
developed to 3010x2012 while the JPEG written beside it in the same second is
6000x4000, and the Fuji's RAF to 3008x2007 against its own 6000x4000 -- the
quarter was the flag, not the file. With `halfSize: false` the same develop
returns 6020x4024 and it is the sensor's frame on every body tried:
D0004128.DNG 6020x4024 IMGP6916.DNG 6028x4024
DSCF1701.RAF 6016x4014 _DSC0009.ARW 6024x4024
AFXT2721.RAF 6246x4170 Nikon-D850 NEF 6216x4136
_GDN0447.NEF 4284x2844 P1010607.RW2 3472x3472
5G4A9396.CR2 2880x1920
Nine files, 27s to 155s a develop on one core. Checked through the app
itself, not only through LibRaw: photo-dims 6020x4024 on the DNG against
6000x4000 on the JPEG, both err none.
The colour it opens with is now fitted per file to the preview the camera wrote
into it (previewMatch.ts): a 3x3 over a block grid of the develop against the
same grid of that preview, then one cubic a channel for what the 3x3 leaves.
The offline per-body table this replaces (cameraMatch.ts) stopped matching the
moment the path under it changed -- its rows no longer summed to 1 once the
highlight knee landed ahead of it -- and a body with a row opened with a cast
one without did not. The file's own preview does not age.
The white level the gain carries is the frame's own plateau rather than
`maximum` (sensorWhite.ts), a factor of 1.89 to 2.00 out; without it every
frame opened a stop bright and a body that sat lower (X-Trans, 1.892) never
reached the highlight desaturation at all.
The desaturation gate reads the gain-lifted levels as well as the sensor's,
which is the whole of the magenta: on a body whose cam_mul lifts red and blue
(the GR's [2.64, 1, 1.73]) a blown sky crosses the white level at 0.38 of the
raw range in red while green crosses at 1.0, so a gate read on the sensor's
levels alone stayed shut across it. Measured in the app against the camera's
own JPEG, mean dRGB over a 16x16 block grid: +1.20, -5.95, -6.11 with the
sensor's clip alone, +0.21, +0.24, +0.47 with both, mean |dL| 21.5 against
10.3. The same grid on the Fuji comes back balanced (+4.7, +5.0, +3.6) and best
aligned at offset 0,0.
-HL is recovery and +HL is a lift, so they are different moves now: recovery is
the doc's soft knee in linear light over the top half, which is the only term
in the tone shader that is not a shift and the only one that can put detail
back into a blown sky rather than merely darken it.
The four checks pin the develop down where it can only run in a browser:
raw-develop-check, preview-match-check, white-level-check, highlight-knee-check.
An RGBA_F32 image with an sRGB tag comes back off the GPU backend sampled on a
1/255 grid; the same shader on a raster surface returns the floats untouched.
The plane is raw/65535, so the shadows the black level is there to keep sit at
1e-3 and quantise to zero -- a 3010x2012 develop landed 41189 pixels under luma
2 with the dark end speckled blue/yellow, against none on the raster surface.
A half is uploaded as float, so the plane stays exact either way.
Rejects the earlier guess that the render target's colour space was to blame:
gpu+rt-srgb and gpu+img-untagged came back byte-identical to gpu.
scripts/half-check.mjs checks the conversion: the named encodings, and no plane
value in a 14-bit sensor's range moving more than 4.8e-4 relative.
LibRaw is deliberately kept out of white balance and tone here, so a RAW opened
in the studio lands on the neutral demosaic — while the JPEG on the back of the
camera carried the body's own rendering. cameraMatch.ts holds that difference as
one 3x3 per body, fitted offline against the camera's own preview of the same
frame and applied in the develop shader right after the sRGB encode.
Measured on held-out blocks, mean CIEDE2000 against the camera preview:
GR III 5.99 -> 4.28 X100V 8.19 -> 4.22
GR II 11.68 -> 9.73 X100S 7.41 -> 7.27
X-T3 9.03 -> 6.12
The matrix is fitted luma-preserving and the shader holds that exactly, so the
profile moves colour and never exposure: a preview that came out dark stays
dark, by design. What is left over is largely high-frequency (sharpening, noise
reduction, demosaic) — the error keeps falling as the blocks grow.
The fits are weak evidence on their own. Validation on colour charts came out
poor: the daylight chart is an Adobe DNG Converter export that aligns to the
body's own develop at only 0.785 correlation and gets worse with the profile
applied, and the tungsten chart is a different illuminant entirely. The honest
claims are the self-fit numbers above and that the X100S — whose cast was small
to begin with — barely moves.
Verified end to end through the real develop: an unfitted body (Sony ILME-FX30)
develops byte-for-byte identically to before, and reading the matrix back out of
each profiled develop recovers the fitted one.
ponytail: one matrix per body, no tone curve and no 3D LUT (a curve on top
measured 2% better and needs a spline plus array uniforms). The match is applied
to the 8-bit band the develop already produces — give develop 16-bit output if a
profile ever has to grade rather than match.
The column of mask knobs belongs to an armed LINEAR or RADIAL shape, and it
stayed on the stage after the hand had moved on: arm a shape, draw it, click
another chip or another tab, and the strip of mask sliders was still there
belonging to a tool that was no longer in hand. A capture-phase `pointerdown` on
`window`, armed only while `maskTool` is set, now puts the tool down in the same
gesture that reaches for something else. A press on the rail (the tabs) or on
any chip except the shape's own two dismisses; a press inside the column is left
alone, because the column's own chips handle their click themselves, and a press
on the photo is left alone, because dragging on the photo is how the shape is
drawn. It is the dismissal the STRAIGHTEN tool already had, one effect over, so
the tab-switch effect needed no `setMaskTool(null)` of its own.
A RAW's blown highlight came out magenta, and was measured before it was
touched. `example-sony.ARW` through the lab (`rawblow.html`, the same camera
white/black and the same `rgb_cam` the app's develop uses): white 16380, black
512, cam_mul green-normalised to (2.581, 1, 1.553). The pixels the sensor could
not hold — 0.7% of the frame, raw max channel at or past 0.99 of the white level
— average (0.775, 1.416, 1.108) in raw, and per channel 26.3% / 96.0% / 46.7% are
at or over white: green is 1.4x the white level while red is still under it.
The develop shader clamped every channel to 1.0 BEFORE the white-balance gains,
and that one clamp is the whole cast. Green is the channel the gains are
normalised to, so it stopped at 1.0, while red and blue — which need their 2.581
and 1.553 — were already past it and were carried over by the multiply. The
blown area therefore left the matrix at (1.0, 0.53, 0.62) instead of at white:
measured on the develop output, (254.4, 217.0, 242.9) — red and blue 38 above
green, which is magenta. On the stage, pixels with red and blue over 235 and
green under 225 in the same framing: 1191 with the clamp, 33 without it.
The shader now only floors at zero, keeps the channel ratios through the matrix,
and fades whatever ran past white towards white (`mix(rgb / mx, 1, 1 - 1/mx)`,
the desaturate-to-white dcraw uses for the same problem). The blown area comes
out (254.5, 253.9, 246.6): an overflow that stays bright and stops taking a hue,
broken per channel at sd 8.7 / 28.0 / 26.4 today against 5.4 / 3.3 / 17.8 now.
Whole-frame averages move by 0.008/0.278/0.140 of a level — the fade only
touches pixels that were over white, which are the 0.7%.
"cannot be rescued" is the second half of the same fact, and it is now a
measurement rather than a hope: LIGHT's HIGHLIGHT row is a curve over what
develop emitted, and while red and blue were pinned at 255 the curve had nothing
to pull on. The fade leaves a compressed ramp there instead, which is what the
row now pulls. LibRaw's own reconstruction modes are not the answer on this
file: `-H` 1, 2 and 3 hand back byte-identical develop output to `-H` 0
(`pxAt65535` is 0 — the sensor never reached its 65535, only the camera's white
level), so `SETTINGS.highlight` stays 0. What the app cannot do is keep the two
stops above white, because the band still leaves develop as 8-bit JPEG; that
ceiling is named at the point of the fade, for whoever needs RAW highlights
recovered rather than merely correct.
`npm run typecheck` and `npm run build` are clean (bundle `index-BJy_3HCz.js`).
Probes: verify-mask-column (dev server, real photo, arm a shape and draw it,
then reach for another chip and for LIGHT and FX — 8 checks, 8 pass: the column
stands while the shape is armed, survives a press on the photo and on the
shape's own kind chips, and goes on any other chip or tab), rawblow (the real
ARW through the real develop maths in the page, before/after chains side by
side, which is where the magenta and the reconstruction modes were measured),
probe-raw-highlight (the app itself, ARW uploaded, blown pixels counted and the
HIGHLIGHT row driven to both ends).
The studio took JPEG, PNG and HEIC and nothing else, so a photographer's own
negatives never reached it. A RAW now loads the way any other file does —
`isRawName` reads the extension off a 24-entry list, the file goes into OPFS
under one slot (`current_image.raw`, beside `current_image.name`, so a reload
finds it again) and `rawDevelop` runs it through LibRaw-wasm: half size, 16-bit
output, camera white balance and the camera's own 3x3 matrix, in bands of 2M
pixels so a 30MB file never holds a second copy of itself. `example-sony.ARW`
(30.3MB) lands as a 3120x2084 picture, no page error.
DEHAZE joins the FX tab, where Lightroom keeps it: a chip off the same
PARAM_DEFS entry (`dehaze`, 0..10) so nothing new renders chips, and the pass is
the dark channel prior — `atmosphericLight` reads A off a 32x32 draw of the
photo, `DEHAZE_SKSL` takes omega up to 0.95 over a floor of 0.1 — measured at
71.8% of the stage's pixels moved between 0 and 10.
The gradient mask grows the six knobs the phone's has: HIGHLIGHT, SHADOW,
WHITE, BLACK, CLARITY and DEHAZE. The mask's falloff is a smoothstep rather than
a line, and CLARITY/DEHAZE inside a mask get a blurred copy of the photo plus
the air A as a second child of the mask shader — so a mask's clarity is clarity
and not a flat brightness lift. The column shows all nine rulers; CLARITY 9
moves 42.2% of the stage, DEHAZE 9 moves 27.9%.
CLARITY stops reading the whole photo per pixel: the single pass that sampled a
15x15 box 225 times is now the three passes the same math wants — 1x15, then
15x1, then a blend, `orig + (orig - B) * 3.2` — about 30 reads. Both signs work
(77.4% of the stage moves at +10, 79.6% at -10), and the negative branch keeps
its mist as it was.
The pointer reviews a look before it is taken: resting on a PHOTO STYLE chip or
a recipe chip lays that look on the photo while it stays there and gives it back
the moment it leaves — byte-identical, measured on four of them (24.9%, 23.8%,
24.5%, 25.3% of the stage moves on, 0.00% off) — while the recipe, the UNDO
stack and the session stay on the look the click left. A hovered look brings its
colour alone: the masks, the dust spots and the mosaic of the photo being edited
ride along, or a pointer crossing a chip row would rub them off. A PRO sim is
left out, since a hover that showed its look would hand over what the click
gates.
Probes: e2e-raw-verify, e2e-dehaze-mask, e2e-mask-verify, e2e-clarity-verify,
e2e-hover-preview2.
3dtours