3dtours/RecipesCam
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

web: open a RAW at the resolution of its sensor, not at the quarter of it

Browse Source 
LibRaw's half-size demosaic was on. The Ricoh GR's own DNG (D0004128.DNG)
developed to 3010x2012 while the JPEG written beside it in the same second is
6000x4000, and the Fuji's RAF to 3008x2007 against its own 6000x4000 -- the
quarter was the flag, not the file. With `halfSize: false` the same develop
returns 6020x4024 and it is the sensor's frame on every body tried:

  D0004128.DNG  6020x4024   IMGP6916.DNG        6028x4024
  DSCF1701.RAF  6016x4014   _DSC0009.ARW        6024x4024
  AFXT2721.RAF  6246x4170   Nikon-D850 NEF      6216x4136
  _GDN0447.NEF  4284x2844   P1010607.RW2        3472x3472
  5G4A9396.CR2  2880x1920

Nine files, 27s to 155s a develop on one core. Checked through the app
itself, not only through LibRaw: photo-dims 6020x4024 on the DNG against
6000x4000 on the JPEG, both err none.

The colour it opens with is now fitted per file to the preview the camera wrote
into it (previewMatch.ts): a 3x3 over a block grid of the develop against the
same grid of that preview, then one cubic a channel for what the 3x3 leaves.
The offline per-body table this replaces (cameraMatch.ts) stopped matching the
moment the path under it changed -- its rows no longer summed to 1 once the
highlight knee landed ahead of it -- and a body with a row opened with a cast
one without did not. The file's own preview does not age.

The white level the gain carries is the frame's own plateau rather than
`maximum` (sensorWhite.ts), a factor of 1.89 to 2.00 out; without it every
frame opened a stop bright and a body that sat lower (X-Trans, 1.892) never
reached the highlight desaturation at all.

The desaturation gate reads the gain-lifted levels as well as the sensor's,
which is the whole of the magenta: on a body whose cam_mul lifts red and blue
(the GR's [2.64, 1, 1.73]) a blown sky crosses the white level at 0.38 of the
raw range in red while green crosses at 1.0, so a gate read on the sensor's
levels alone stayed shut across it. Measured in the app against the camera's
own JPEG, mean dRGB over a 16x16 block grid: +1.20, -5.95, -6.11 with the
sensor's clip alone, +0.21, +0.24, +0.47 with both, mean |dL| 21.5 against
10.3. The same grid on the Fuji comes back balanced (+4.7, +5.0, +3.6) and best
aligned at offset 0,0.

-HL is recovery and +HL is a lift, so they are different moves now: recovery is
the doc's soft knee in linear light over the top half, which is the only term
in the tone shader that is not a shift and the only one that can put detail
back into a blown sky rather than merely darken it.

The four checks pin the develop down where it can only run in a browser:
raw-develop-check, preview-match-check, white-level-check, highlight-knee-check.
This commit is contained in:
3dtours
2026-09-28 15:24:37 +07:00
parent b824308182
commit 224ff0b935
9 changed files with 940 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/frontend/src/engine/cameraMatch.ts
-67
View File
@@ -1,67 +0,0 @@
// The look a camera's own JPEG would have had.
//
// LibRaw here is deliberately kept out of white balance and tone (see
// rawDevelop), so a RAW that opens in the studio lands on the neutral demosaic —
// while the JPEG the photographer saw on the back of the camera carries the
// body's own colour rendering. These 3x3 matrices are that difference, fitted
// offline: develop a frame through this same path, compare it block by block
// against the camera's own preview of the same frame, and least-squares the
// matrix that takes the first to the second. Fitted per body on real scenes
// (4–12 MP each, held-out blocks scored with CIEDE2000):
//
// Ricoh GR III 4.28 (unfitted 5.99) Fujifilm X100V 4.22 (8.19)
// Ricoh GR II 9.73 (11.68) Fujifilm X100S 7.27 (7.41)
// Fujifilm X-T3 6.12 (9.03)
//
// What is left over is largely high-frequency (sharpening, noise reduction,
// demosaic) rather than colour: the error keeps falling as the blocks grow
// (GR III: 2.86 at 6px, 2.06 at 24px, 1.59 at 60px).
//
// The transform preserves luma exactly — out = luma * normalize(M * in / luma) —
// so it moves colour and never exposure. A preview that came out dark stays
// dark, on purpose: exposure is the studio's job, not the profile's. ponytail:
// one matrix, no tone curve and no 3D LUT; a curve on top was measured at 2%
// better and needs a spline plus an array uniform, so add one only when a body
// turns out to need it.
//
// Keyed on LibRaw's normalized_model — the bare model ("GR III", "X100V"), not
// the make-qualified camera_model: a body that was never fitted, or that reports
// a name we do not know, develops exactly as it did before.
export type Mat3 = readonly [
number, number, number,
number, number, number,
number, number, number,
];
// Row-major, applied to the sRGB-encoded develop (the space it was fitted in).
const MATCH: Record<string, Mat3> = {
'GR III': [
0.577754, 0.856314, -0.440638,
0.143396, 0.716041, 0.145206,
-0.177111, 0.291345, 0.859118,
],
'GR II': [
0.440692, 0.976166, -0.377622,
0.187633, 0.680171, 0.121235,
-0.211717, 0.293754, 0.911013,
],
'X-T3': [
1.626754, -0.533332, -0.130089,
-0.184760, 1.160799, 0.039331,
-0.015344, -0.022394, 0.993452,
],
'X100V': [
1.103373, 0.362157, -0.471229,
0.014568, 0.833165, 0.154949,
-0.448700, 0.586230, 0.852687,
],
'X100S': [
0.527629, 0.469773, -0.014534,
0.144123, 0.823023, 0.033459,
-0.036716, 0.369805, 0.711359,
],
};
export function cameraMatch(model?: string | null): Mat3 | null {
return (model && MATCH[model]) || null;
}
docker/frontend/src/engine/previewMatch.ts
+186
View File
@@ -0,0 +1,186 @@
// The colour the camera itself wrote into the file.
//
// LibRaw hands back the sensor's own data and nothing about how the body chose
// to render it: the white balance it locked, its picture style, its tone curve.
// The file carries the answer anyway — the preview saved inside it, which is the
// frame the photographer saw and the one a desktop viewer shows. So the develop
// is matched to that preview: develop a grid of the frame, take the same grid out
// of the embedded preview, and least-squares the 3x3 that takes the first to the
// second — then, on what that 3x3 leaves, the tone curve a matrix cannot hold.
//
// Measured on the A5100 frame the studio was reported on (dE00 against the
// camera's own 24MP JPEG, blocks of a 64x64 grid): 12.0 as the develop left it,
// 9.2 with this fit drawn in linear light, 6.4 painted through the 8-bit colour
// filter this used to be, 6.6 with the 3x3 alone in float, 4.4 with the tone
// curve under it. What is left either way is the camera's own local rendering —
// its sharpening, the detail a 1.7MP preview never had — and no per-pixel
// transform reaches that.
//
// The fit is on the sRGB-encoded values, exposure included — matrix and curve
// alike: the reference is the file's own frame, so matching its brightness is
// part of matching its colour. Fitted in that encoded space, not in linear light,
// because a least squares in linear light weighs the highlights and comes back
// with the midtones wrong — the same frames scored dE00 6.4 fitted encoded
// against 9.2 fitted linear. Any body, any picture style, and no fitted table to
// go stale — a preview is the one profile every RAW already carries.
// Row-major 3x3, on the sRGB-encoded develop: what the shader applies to a pixel.
export type Mat3 = readonly [
number, number, number,
number, number, number,
number, number, number,
];
// The grid the fit reads, and the size of the preview map handed to it.
export const MATCH_GRID = 64;
// A block at or past this has nothing left to say about a rendering.
const CLIP = 0.97;
const FLOOR = 0.03;
// The per-file colour fit: a 3x3, then the tone curve a 3x3 cannot hold — one
// cubic a channel, on the encoded value the 3x3 leaves. `tone` is that curve in
// the shader's own order: r a b c d, then g, then b.
export interface Match {
m: Mat3;
tone: Float32Array;
}
// What the second stage does when the frame has no curve the fit can trust.
export const FLAT_TONE = new Float32Array([0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0]);
// Gaussian elimination on the one system this file fits: a channel is a row of
// three, for the 3x3 and again for the 3x3's own error.
function solve(C: number[][], d: number[]): number[] {
const n = 3;
const a = Array.from({ length: n }, (_, i) => [...C[i], d[i]]);
for (const row of a) for (let c = 0; c < n; c++) row[c] += 1e-9;
for (let c = 0; c < n; c++) {
let p = c;
for (let r = c + 1; r < n; r++) if (Math.abs(a[r][c]) > Math.abs(a[p][c])) p = r;
[a[c], a[p]] = [a[p], a[c]];
for (let r = 0; r < n; r++) {
if (r === c) continue;
const f = a[r][c] / a[c][c];
for (let k = c; k <= n; k++) a[r][k] -= f * a[c][k];
}
}
return a.map((_, i) => a[i][n] / a[i][i]);
}
// The curve a channel needs to reach the camera's, least squared: what the 3x3
// left for that channel, a cubic against the block the camera wrote there.
//
// The cubic is pinned at white, not free. Fitted free it slid its far end down to
// meet the blocks it wanted — 0.96 on red — and every blown highlight the develop
// had carried up to the white level came out off-white instead: a blown pixel
// (1, 1, 1) into the fit left at 0.82 / 0.97 / 0.93, and 0.0% of the frame at
// pure white against 2.3% before and 3.3% in the preview. So the fit solves the
// three free terms of `tone = x^3 + a(1 - x^3) + b(x - x^3) + c(x^2 - x^3)`, whose
// terms all vanish at 1 and which costs the fit almost nothing (rms 0.066 on red
// against the free cubic's 0.063, and 0.076 on blue against 0.076, on the frame
// this was measured on). Scaling the free curve onto its own end instead moved
// the midtones by the whole 4% the end was short, and pinning both ends moved them
// by 12% (rms 0.21 on green).
function basis(i: number, x: number): number {
const x3 = x * x * x;
return i === 0 ? 1 - x3 : i === 1 ? x - x3 : x * x - x3;
}
function fitTone(blocks: { t: number[]; q: number[] }[]): Float32Array {
const out = new Float32Array(12);
for (let c = 0; c < 3; c++) {
const A = [[0, 0, 0], [0, 0, 0], [0, 0, 0]];
const d = [0, 0, 0];
for (const { t, q } of blocks) {
const x = q[c], b = [basis(0, x), basis(1, x), basis(2, x)];
for (let p = 0; p < 3; p++) {
for (let k = 0; k < 3; k++) A[p][k] += b[p] * b[k];
d[p] += b[p] * (t[c] - x * x * x);
}
}
const [a, b, cc] = solve(A, d);
out.set([a, b, cc, 1 - a - b - cc], c * 4);
}
return out;
}
// A curve that comes back folded, or wandered far from the value it was handed,
// is a fit off blocks that fought each other, not a rendering: the caller keeps
// the 3x3 alone rather than a look nobody's camera has. The floor sits below the
// zero a curve is allowed to reach — the develop clamps there anyway — and the
// slope is only asked not to turn back.
function curveIsSane(tone: Float32Array): boolean {
for (const v of tone) if (!Number.isFinite(v) || Math.abs(v) > 4) return false;
for (let c = 0; c < 3; c++) {
const [a, b, cc, dd] = [tone[c * 4], tone[c * 4 + 1], tone[c * 4 + 2], tone[c * 4 + 3]];
let prev = a;
for (let i = 1; i <= 16; i++) {
const x = i / 16;
const y = a + x * (b + x * (cc + x * dd));
if (y < prev - 0.12 || y < -0.15 || y > 1.15) return false;
prev = y;
}
}
return true;
}
// `dev` and `ref` are n x n RGBA maps of the same frame, one byte per channel.
// The 3x3 comes back on those encoded values — the space the caller draws it in.
// Null when the two cannot be related — too few blocks carry colour, or the
// solution runs away — and the caller develops the frame as the sensor left it.
export function fitMatch(dev: Uint8Array, ref: Uint8Array, n = MATCH_GRID): Match | null {
const C = [[0, 0, 0], [0, 0, 0], [0, 0, 0]];
const d = [[0, 0, 0], [0, 0, 0], [0, 0, 0]];
const blocks: { e: number[]; t: number[] }[] = [];
let used = 0;
for (let i = 0; i < n * n; i++) {
const e = [dev[i * 4] / 255, dev[i * 4 + 1] / 255, dev[i * 4 + 2] / 255];
const t = [ref[i * 4] / 255, ref[i * 4 + 1] / 255, ref[i * 4 + 2] / 255];
// A block the camera clipped has no colour left to copy, one at the floor is
// only the pedestal, and one the develop clipped has lost the ratio between
// its channels — none of them say anything about the rendering.
if (t[0] > CLIP && t[1] > CLIP && t[2] > CLIP) continue;
if (e[0] > CLIP || e[1] > CLIP || e[2] > CLIP) continue;
if (Math.max(t[0], t[1], t[2]) < FLOOR) continue;
used++;
blocks.push({ e, t });
for (let x = 0; x < 3; x++) {
for (let y = 0; y < 3; y++) C[x][y] += e[x] * e[y];
for (let c = 0; c < 3; c++) d[x][c] += e[c] * t[x];
}
}
// A fit off a handful of blocks is a fit off noise.
if (used < (n * n) / 8) return null;
// Ridge towards identity, scaled to the data. The three channels of a develop
// are near-collinear — a camera's own rendering of a neutral scene is close to
// neutral — so the plain normal equations come back singular-ish, and the frame's
// own noise decides a matrix whose diagonal lands at or below zero. λ of 1e-3 of
// the data's scale lifts that back to a definite system and moves the fitted
// matrix only in the third decimal (measured: 0.001 kept the tone the same to
// 0.3% where 0.1 began flattening it).
const lambda = 1e-3 * ((C[0][0] + C[1][1] + C[2][2]) / 3);
for (let r = 0; r < 3; r++) {
C[r][r] += lambda;
d[r][r] += lambda;
}
const m = [...solve(C, d[0]), ...solve(C, d[1]), ...solve(C, d[2])];
// A row that ran off is a degenerate fit, not a look. A diagonal element at or
// below zero is not one: what these matrices hold is the difference between two
// near-identical channels (X-T3 -0.04, X100V -1.67 on a row that carries +1.05
// and -1.67), and refusing those refused the whole fit — those bodies opened
// byte-identical to an unfitted develop.
for (const v of m) if (!Number.isFinite(v) || Math.abs(v) > 4) return null;
// The 3x3 alone cannot bend a channel — it can only scale it — and what is left
// between the two frames is mostly exactly that bend. So the residual after the
// 3x3 is fitted a cubic a channel, on the value the 3x3 leaves.
const q = blocks.map(({ e, t }) => {
const v = [
m[0] * e[0] + m[1] * e[1] + m[2] * e[2],
m[3] * e[0] + m[4] * e[1] + m[5] * e[2],
m[6] * e[0] + m[7] * e[1] + m[8] * e[2],
];
return { t, q: v.map((x) => Math.min(1, Math.max(0, x))) };
});
const tone = fitTone(q);
return { m: m as unknown as Mat3, tone: curveIsSane(tone) ? tone : FLAT_TONE };
}
docker/frontend/src/engine/rawDevelop.ts
+265 -84
View File
@@ -1,15 +1,49 @@
// RAW → JPEG on the client, so a camera's own file opens in the studio without
// a DNG converter in the middle (see native_raw_processing_opfs_architecture.md).
//
// LibRaw demosaics in its own worker; what comes back is linear camera data,
// A RAW opens at the colour the camera chose for it. LibRaw is kept out of white
// balance and tone (below) and knows nothing of the body's picture style, so the
// develop is fitted to the one rendering the file does carry with it: the preview
// the camera wrote inside it, which is the frame the photographer saw and the one
// a desktop viewer shows. Develop the sensor, least-squares a 3x3 from a block
// grid of each onto the other on the encoded values, fit what that 3x3 leaves a
// cubic a channel, and develop the sensor through both (see previewMatch.ts).
//
// Measured against the camera's own 24MP JPEG of the A5100 frame this was
// reported on, blocks of a 64x64 grid, dE00: 12.0 as the develop left it, 19.7
// with no white balance at all, 6.6 matched to the file's own preview by the 3x3
// alone and 4.4 through the 3x3 and the curve together. What neither reaches is
// the camera's own sharpening, which no per-pixel transform holds.
//
// The fit is the only profile: a body-used-to-be table of six matrices fitted on
// this develop was dropped, because a table fitted on one path stops matching the
// moment the path changes under it — its rows no longer summed to 1 once the
// highlight knee landed ahead of it, and every body that had one opened with a
// cast the body that had none did not. Offline-fitted matrices age; the file's
// own preview does not.
//
// Developing the sensor is also what keeps the highlights: 2.3% of the frame at
// pure white against 3.3% in the preview itself, since the 8-bit preview threw
// the headroom away and the develop still has it. The preview is therefore the
// reference and the fallback, not the frame: it is what a file with no usable
// preview (some DNG), or with sensor data that will not decode, opens as.
//
// The develop: LibRaw demosaics in its own worker; what comes back is linear camera data,
// which this file turns into the sRGB the rest of the pipeline expects. The
// whole thing is measured against a real 26MP Sony ARW — the settings below are
// the ones that gave the correct colours there:
// settings below were checked against the preview a Sony ILME-FX30 writes into
// its own ARW (the camera JPEG, read straight out of the file): the developed
// frame and that preview agree to within 1% on both channel ratios, R/G 0.909
// against 0.903 and B/G 0.603 against 0.606.
// - noAutoScale + useCameraWb:false + noAutoBright + gamm [1,1] keep LibRaw out
// of white balance and tone, so `cam_mul` and `rgb_cam` can be applied here
// exactly once.
// - halfSize: 26MP → 6.5MP. ponytail: drop it for full resolution if a user
// ever asks for a print from the RAW; the develop pass is the whole cost.
// - halfSize:false: the frame opens at the sensor's own resolution — a 24MP
// RAW develops to 24MP (6020x4024 on the GR, 6000x4000 on the Fuji), not to
// the quarter the half-size demosaic reports. `userQual` 3 then demosaics
// all of it. The develop was half-size until a GR's DNG opened at 3010x2012
// against its own JPEG's 6000x4000: the quarter-size frame was the flag,
// not the file. ponytail: costs ~4x the develop time and two full-size F16
// surfaces; put the flag behind a "draft" toggle if a phone ever has to.
//
// The band loop exists because a single Float32 copy of the whole plane would be
// ~100MB. Each band is decoded, normalised and drawn before the next is read.
@@ -17,14 +51,24 @@
// the main thread (Skia is not available in the RAW worker). Move it to a worker
// with an OffscreenCanvas if the develop ever blocks the UI visibly.
import LibRaw from 'libraw-wasm';
import { cameraMatch, type Mat3 } from './cameraMatch';
import { sensorWhite } from './sensorWhite';
import { f32ToF16 } from './halfFloat';
import { fitMatch, FLAT_TONE, MATCH_GRID, type Match, type Mat3 } from './previewMatch';
import { Skia } from './skiaShim';
// What `imageData()` returns for the settings below: 16-bit, 3 channels, with
// the black level still in it — hence the two normalisations in the shader.
// What `imageData()` returns for the settings below: 16-bit, 3 channels, black
// level already gone — the post-process subtracts it whatever `noAutoScale`
// says, which only holds back the white balance and the output scaling.
//
// Its white level is not `maximum` but the frame's own plateau, a factor of 1.89
// to 2.00 out (see sensorWhite below). The gain carries that level, so the white
// lands back on 1.0 — and on every body, not just the two that factor two was
// fitted on: without it every frame opened a stop bright (the FX30's own JPEG has
// 0.05% of pixels at pure white where the develop had 0.76%) and a body that sat
// lower (X-Trans, 1.892) never even reached the highlight desaturation, which
// starts at 0.95 of the sensor.
const SETTINGS = {
halfSize: true,
halfSize: false,
outputBps: 16,
outputColor: 0,
noAutoScale: true,
@@ -46,53 +90,103 @@ const IDENTITY: Mat3 = [1, 0, 0, 0, 1, 0, 0, 0, 1];
const RAW_DEVELOP_SKSL = `
uniform shader raw;
uniform float4 black; // (black level, 1 / (white level - black level))
uniform float gain; // 1 / the white level the frame itself ran out at
uniform float4 mul; // cam_mul, green-normalised
uniform float4 m0; // camera -> sRGB, the first three columns of rgb_cam
uniform float4 m1;
uniform float4 m2;
uniform float4 crop; // (y offset of this band, 0, 0, 0)
uniform float4 w0; // the body's camera match, one row per float4
uniform float4 w1; // (identity when the body has no profile)
uniform float4 w2;
uniform float4 f0; // the per-file fit, on the encoded value (identity when
uniform float4 f1; // the frame carries no preview to be fitted to)
uniform float4 f2;
uniform float4 t0; // what that fit then leaves a channel: one cubic a
uniform float4 t1; // channel, r a b c d per float4 (flat when there is no fit)
uniform float4 t2;
// A 3x3 can only scale a channel; the gap to the camera is mostly a shape, and
// this is that shape, read on the value the 3x3 left.
float tone(float4 w, float x) {
return clamp(w.x + x * (w.y + x * (w.z + x * w.w)), 0.0, 1.0);
}
float3 encode(float3 x) {
x = clamp(x, 0.0, 1.0);
return mix(x * 12.92, 1.055 * pow(x, float3(1.0 / 2.4)) - 0.055, step(float3(0.0031308), x));
}
float luma(float3 x) {
return dot(float3(0.2126, 0.7152, 0.0722), x);
}
half4 main(float2 pos) {
float4 p = raw.eval(float2(pos.x, pos.y - crop.x));
// Only the floor here. A photo the sensor could not hold goes over the white
// The plane arrives with the black level already subtracted — it floors at 0,
// not at color_data.black (measured on the FX30 ARW: the sensor mosaic floors
// at 334, the plane at 0, a quarter of its red samples under the black level).
// Subtracting it again drained red and blue — the two channels the gains lift
// most — and dragged every frame towards green.
float3 n = max(p.rgb * gain, 0.0); // the sensor's own levels, white level 1.0
float3 lin = n * mul.rgb;
// Only the floor there. A photo the sensor could not hold goes over the white
// level in all three channels, and clipping them one by one before the WB gains
// is what tints what is left of the highlight: green — the channel the gains are
// normalised to — stops at 1.0 while red and blue, which need their 2.6x and
// 1.6x, are already past it, so the blown area comes out magenta. Keep the
// channel ratios through the matrix instead and let the overflow fade to white.
float3 lin = max((p.rgb - black.rgb) * black.a, 0.0) * mul.rgb;
float3 rgb = float3(dot(m0.xyz, lin), dot(m1.xyz, lin), dot(m2.xyz, lin));
float mx = max(max(rgb.r, rgb.g), rgb.b);
// The overflow fades towards white instead of being cut, so the trace of a
// sensor that ran past its white level survives as a compressed ramp — which is
// what LIGHT's HIGHLIGHT row then has to pull on. ponytail: the ramp lives in
// [0.5, 1] and the band leaves as 8-bit JPEG, so the two stops above white are
// compressed, not kept; give develop a 16-bit output (or a knee of its own) when
// RAW highlights have to be recovered rather than merely look right.
rgb = mx > 1.0 ? mix(rgb / mx, float3(1.0), 1.0 - 1.0 / mx) : rgb;
// A pixel that has run to the white level has no colour of its own left to keep,
// and what the gains made of it is an artefact, not a colour: ease the pixel
// towards the neutral of its own value as that point is approached. The clip to
// read is the one the gains make, not only the sensor's own — the gains here are
// 1.7x and 1.9x on red and blue, so a blown sky reaches the white level at 0.59 of
// the raw range in those channels while the green, which the gains are
// normalised to, only reaches it at 1.0. Read on the sensor's levels alone the
// gate stayed shut across a whole blown sky and left the develop's magenta in it
// (227,184,245 at the gate's own value against 245,245,245 read where the gains
// put the clip, the camera's preview white at that block). Both are read, so a
// body whose gains do not lift a channel keeps the sensor's own clip as its gate.
// The matrix was fitted luma-preserving, so mx is the value to hold.
float hi = max(max(n.r, n.g), n.b);
hi = max(hi, max(max(lin.r, lin.g), lin.b));
rgb = mix(rgb, float3(mx), smoothstep(0.95, 1.0, hi));
// The overflow used to fade towards white — mix(rgb / mx, 1, 1 - 1 / mx) —
// which put every pixel of a blown sky on exactly 1.0 and threw the two stops
// the sensor held above white away with it: LIGHT's HIGHLIGHT row then had a
// flat white to pull on and nothing to reveal. The white point is moved down
// instead and the overflow squeezed back in under it by the doc's soft knee:
//
// y = T + over / (1 + 2S*over), over = mx - T, S = 1 / (2(1 - T))
//
// S is what puts 1.0 on the asymptote, so the sensor's own plateau — two white
// levels up, see the gain above — lands at ~0.94 and the first stop over white
// spends 0.85..0.94. Below T the frame is untouched and the curve leaves T with
// the slope it arrived with (1), so there is no seam to mask; above it the frame
// darkens, which is the one move no later pass can undo — which is the point,
// and what the per-file fit below then measures the REST of the frame back from.
//
// ponytail: 4.7 stops of headroom now share ~5% of the ramp, and the develop
// still leaves as an 8-bit JPEG. Give it a float16 output when RAW highlights
// have to print rather than merely be seen.
if (mx > 0.7) {
float over = mx - 0.7;
rgb *= (0.7 + over / (1.0 + over * 3.3333)) / mx;
}
float3 e = encode(rgb);
// The body's colour, in the encoded space it was fitted in. The matrix was
// fitted luma-preserving, and the luma(e) / luma(c) factor holds that exact:
// the profile turns the chroma and never rescues an exposure, which stays the
// studio's job. Identity is a no-op.
float3 c = clamp(float3(dot(w0.xyz, e), dot(w1.xyz, e), dot(w2.xyz, e)), 0.0, 1.0);
// ponytail: the band leaves as 8-bit JPEG, so the match is applied to a value
// that is already quantised — plenty for a look, but give develop 16-bit output
// if a profile ever has to grade rather than merely match.
return half4(half3(clamp(c * (luma(e) / max(luma(c), 1e-4)), 0.0, 1.0)), 1.0);
// The file's own colour: the fit, in float, on the encoded value it was fitted
// on — and the last step the frame leaves through. Not the 8-bit colour filter
// this used to be painted through: the fit carries an exposure (the preview is
// the reference, so matching its brightness is part of matching its colour).
// No rolloff here either. The fit is fitted whole to the preview's own values,
// and white is one of them, so it already maps the frame's white to the preview's
// — while a divide by the row max is a white-preserving move the fit does not
// need: every highlight came back at ~0.74 / 1.0 / 0.86, cyan, and not one cell
// of the frame reached white on all three channels (0.0% against the preview's
// 3.3%). Past the white level only the clamp is left, exactly as the develop
// above lets its own overflow run.
float3 q = clamp(float3(dot(f0.xyz, e), dot(f1.xyz, e), dot(f2.xyz, e)), 0.0, 1.0);
// ...and then the curve, one cubic a channel, which is what carries the body's
// own tone: a 3x3 can only scale, so the frame without it came back bright and
// green in the shadows (dL +13.8 and green +0.128 at the bottom of the range,
// dE00 6.6) instead of matching (dL +4.5, green +0.017, 4.4). The curve is
// pinned at white, so a blown pixel still lands on white.
return half4(half3(tone(t0, q.r), tone(t1, q.g), tone(t2, q.b)), 1.0);
}
`;
@@ -110,11 +204,71 @@ export function isRawName(name: string): boolean {
return name.includes('.') && RAW_EXT.includes(ext);
}
// The camera's own preview, when the file carries one: the colour reference to
// fit against, and what the file opens as when the sensor does not decode.
// ponytail: it is 1616x1080 on an A5100 and 1620x1080 on an FX30, so handing it
// back is a 1.7MP frame — a print past it has to come off the develop, which is
// what a fitted file already gives.
async function cameraPreview(raw: LibRaw): Promise<Uint8Array | null> {
const thumb = await raw.thumbnailData().catch(() => undefined);
if (thumb?.format !== 'jpeg' || !thumb.data?.length) return null;
return new Uint8Array(thumb.data);
}
// MATCH_GRID x MATCH_GRID block colours of a frame, one byte per channel.
function gridOf(image: any, n = MATCH_GRID): Uint8Array | null {
const surface = Skia.Surface.MakeOffscreen(n, n) ?? Skia.Surface.Make(n, n);
if (!surface) return null;
const canvas = surface.getCanvas();
// Cubic, not a linear tap: this is a 45x reduction and linear reads a handful
// of source pixels per block — noise for the least squares to fit.
canvas.drawImageRectCubic(
image,
Skia.XYWHRect(0, 0, image.width(), image.height()),
Skia.XYWHRect(0, 0, n, n),
1 / 3,
1 / 3
);
surface.flush();
const px = canvas.readPixels(0, 0, {
width: n,
height: n,
colorType: Skia.ColorType.RGBA_8888,
alphaType: Skia.AlphaType.Unpremul,
colorSpace: Skia.ColorSpace.SRGB,
}) as Uint8Array | null;
surface.dispose();
return px ? new Uint8Array(px.buffer, px.byteOffset, px.byteLength) : null;
}
// The same grid out of the preview, which the file carries as a JPEG. Decoded and
// reduced through the same Skia call as the develop's own grid, because the two
// grids are only comparable — and the fit only meaningful — when one resampler
// made both. A 2D canvas here instead left the fit following its own smoothing:
// on the A5100 frame the same develop scored dE00 4.7 against 4.4, and the dark
// end of the frame came out 6 L further from the preview than the fit it was
// handed asked for.
function previewGrid(jpeg: Uint8Array, w: number, h: number, n = MATCH_GRID): Uint8Array | null {
const bmp = Skia.Image.MakeImageFromEncoded(jpeg);
if (!bmp) return null;
try {
// A preview of another shape is a crop of the frame, not the frame: fitting
// against it lines the two grids up on different scenes and fits nothing.
if (Math.abs(bmp.width() / bmp.height() / (w / h) - 1) > 0.02) return null;
if (bmp.width() < n * 4) return null;
return gridOf(bmp, n);
} finally {
bmp.delete();
}
}
export async function developRaw(bytes: Uint8Array): Promise<Uint8Array> {
const raw = new LibRaw();
let preview: Uint8Array | null = null;
try {
// LibRaw copies the buffer it is handed, so the caller's bytes stay intact.
await raw.open(bytes as unknown as BufferSource, SETTINGS);
preview = await cameraPreview(raw);
const meta = await raw.metadata(true);
const img = await raw.imageData();
const cd = meta?.color_data;
@@ -124,8 +278,6 @@ export async function developRaw(bytes: Uint8Array): Promise<Uint8Array> {
const data = img.data as Uint16Array;
if (!w || !h) throw new Error('RAW decoded to nothing');
const surface = Skia.Surface.MakeOffscreen(w, h) ?? Skia.Surface.Make(w, h);
if (!surface) throw new Error('no surface for the develop');
const effect = Skia.RuntimeEffect.Make(RAW_DEVELOP_SKSL);
if (!effect) throw new Error('develop shader failed to compile');
@@ -133,62 +285,91 @@ export async function developRaw(bytes: Uint8Array): Promise<Uint8Array> {
const mul = cd.cam_mul.map((v) => v / green);
const row = (i: number) => cd.rgb_cam[i].slice(0, 3);
const [r0, r1, r2] = [row(0), row(1), row(2)];
const cm: Mat3 = cameraMatch(meta?.normalized_model) ?? IDENTITY;
const bandH = Math.max(1, Math.min(h, Math.floor(BAND_PIXELS / w)));
const f32 = new Float32Array(w * bandH * 4);
// The band goes up as half, not float32: the GPU backend puts an F32 image
// on the 1/255 grid and the shadows quantise to black (see halfFloat.ts).
const half = new Uint16Array(w * bandH * 4);
for (let y0 = 0; y0 < h; y0 += bandH) {
const rows = Math.min(bandH, h - y0);
let o = 0;
for (let i = y0 * w * 3, end = (y0 + rows) * w * 3; i < end; i += 3) {
f32[o++] = data[i] / SAMPLE_MAX;
f32[o++] = data[i + 1] / SAMPLE_MAX;
f32[o++] = data[i + 2] / SAMPLE_MAX;
f32[o++] = 1;
}
f32ToF16(f32, half, w * rows * 4);
const band = Skia.Image.MakeImage(
{ width: w, height: rows, colorType: Skia.ColorType.RGBA_F16, alphaType: Skia.AlphaType.Unpremul },
new Uint8Array(half.buffer, 0, w * rows * 8),
w * 8
);
if (!band) throw new Error('band image failed');
const child = band.makeShaderOptions(
Skia.TileMode.Clamp,
Skia.TileMode.Clamp,
Skia.FilterMode.Nearest,
Skia.MipmapMode.None
);
const uniforms = new Float32Array([
cd.black / SAMPLE_MAX, cd.black / SAMPLE_MAX, cd.black / SAMPLE_MAX,
SAMPLE_MAX / (cd.maximum - cd.black),
mul[0], mul[1], mul[2], 0,
r0[0], r0[1], r0[2], 0,
r1[0], r1[1], r1[2], 0,
r2[0], r2[1], r2[2], 0,
y0, 0, 0, 0,
cm[0], cm[1], cm[2], 0,
cm[3], cm[4], cm[5], 0,
cm[6], cm[7], cm[8], 0,
]);
const shader = effect.makeShaderWithChildren(uniforms, [child]);
const paint = Skia.Paint();
paint.setShader(shader);
surface.getCanvas().drawRect(Skia.XYWHRect(0, y0, w, rows), paint);
surface.flush();
paint.delete();
shader.delete();
child.delete();
band.delete();
}
// Every uniform but the crop and the fit, which are the two the passes change:
// the shader's own order is gain, mul, rgb_cam, crop, the fit, then the fit's
// tone curve.
const uniforms = new Float32Array(45);
uniforms[0] = SAMPLE_MAX / sensorWhite(data, cd.maximum, cd.black);
uniforms.set([mul[0], mul[1], mul[2], 0, r0[0], r0[1], r0[2], 0, r1[0], r1[1], r1[2], 0, r2[0], r2[1], r2[2], 0], 1);
const jpeg = surface.makeImageSnapshot().encodeToBytes(Skia.ImageFormat.JPEG, 92);
surface.dispose();
// One develop of the frame, band by band, through `fit` when there is one. A
// function because the frame is drawn twice: once on the sensor alone, to fit
// against the preview, and then again with the fit in the shader.
// ponytail: two full band passes, on the main thread. Give develop an F16
// intermediate (one develop, one colour pass) if the second pass ever shows.
const develop = (fit: Match | null) => {
const surface = Skia.Surface.MakeOffscreen(w, h) ?? Skia.Surface.Make(w, h);
if (!surface) return null;
const f = fit?.m ?? IDENTITY;
uniforms[21] = f[0]; uniforms[22] = f[1]; uniforms[23] = f[2]; uniforms[24] = 0;
uniforms[25] = f[3]; uniforms[26] = f[4]; uniforms[27] = f[5]; uniforms[28] = 0;
uniforms[29] = f[6]; uniforms[30] = f[7]; uniforms[31] = f[8]; uniforms[32] = 0;
uniforms.set(fit?.tone ?? FLAT_TONE, 33);
for (let y0 = 0; y0 < h; y0 += bandH) {
const rows = Math.min(bandH, h - y0);
let o = 0;
for (let i = y0 * w * 3, end = (y0 + rows) * w * 3; i < end; i += 3) {
f32[o++] = data[i] / SAMPLE_MAX;
f32[o++] = data[i + 1] / SAMPLE_MAX;
f32[o++] = data[i + 2] / SAMPLE_MAX;
f32[o++] = 1;
}
f32ToF16(f32, half, w * rows * 4);
const band = Skia.Image.MakeImage(
{ width: w, height: rows, colorType: Skia.ColorType.RGBA_F16, alphaType: Skia.AlphaType.Unpremul },
new Uint8Array(half.buffer, 0, w * rows * 8),
w * 8
);
if (!band) throw new Error('band image failed');
const child = band.makeShaderOptions(
Skia.TileMode.Clamp,
Skia.TileMode.Clamp,
Skia.FilterMode.Nearest,
Skia.MipmapMode.None
);
uniforms[17] = y0;
const shader = effect.makeShaderWithChildren(uniforms, [child]);
const paint = Skia.Paint();
paint.setShader(shader);
surface.getCanvas().drawRect(Skia.XYWHRect(0, y0, w, rows), paint);
surface.flush();
paint.delete();
shader.delete();
child.delete();
band.delete();
}
const shot = surface.makeImageSnapshot();
surface.dispose();
return shot;
};
// The file's own colour: a grid of the develop as it stands, the same grid out
// of the preview the camera wrote into the file, and the 3x3 and the curve
// between them — which the second develop then draws in the shader. No
// preview, no fit: the frame opens as the sensor left it.
const first = develop(null);
if (!first) throw new Error('no surface for the develop');
const blocks = preview ? gridOf(first) : null;
const ref = blocks ? previewGrid(preview as Uint8Array, first.width(), first.height()) : null;
const match = blocks && ref ? fitMatch(blocks, ref) : null;
const matched = match ? develop(match) : null;
const jpeg = (matched ?? first).encodeToBytes(Skia.ImageFormat.JPEG, 92);
(matched ?? first).dispose();
if (matched) first.dispose();
if (!jpeg?.length) throw new Error('develop produced no bytes');
return jpeg;
} catch (err) {
// The preview still opens the file when the sensor will not: a RAW whose
// colour data is missing (some DNG) is not a RAW that cannot be shown.
if (preview) return preview;
throw err;
} finally {
raw.dispose();
}
docker/frontend/src/engine/sensorWhite.ts
+54
View File
@@ -0,0 +1,54 @@
// Where this frame's sensor actually ran out, in the plane's own counts.
//
// `maximum` is the format's full scale, not the sensor's: a frame that has blown
// plateaus at 1.89-2.00x it (measured, as ratios of `maximum - black`: Sony
// ILCE-5100 2.002, Ricoh GR III 1.964, Fuji X-T3 / X100V / X100S 1.892) and the
// white level the shader divides by has to be that plateau, or the frame lands
// short of white and the highlight desaturation — a smoothstep that starts at
// 0.95 of the *sensor* — never fires on a body that sits at 0.946. Reading it
// off the frame is what makes one develop right for every body.
//
// A plateau, not a max: the frame is read over the nine counts at its top, and
// the white level is the highest of those that is a **cliff** — a count the one
// below it does not come near. A clipped region piles at the level the sensor
// stops at, so the count under that level is the sensor's own noise floor and
// orders of magnitude smaller; a smooth bright sky hands its top count to the
// next one down in similar numbers and is left alone. A frame that has not
// clipped — and one that never reaches the format's own scale — stays on the old
// factor two, the population mean of the ratios above and the best guess when
// there is nothing to measure.
//
// One count down, not the largest of the three: a sensor clips its channels at
// counts a couple apart, so the plane carries a second pile just under the top
// one, and asking the top pile to beat *that* pile refused every frame whose
// channels do not run out together. Measured on a Fujifilm XF10 RAF: 433819
// samples at 30993 over 30 at 30992 — a cliff — and 83954 at 30990 three counts
// under it, which the old rule read as the ground the top pile had to clear eight
// times over (it clears 44, not 83954) and answered "not clipped" for a frame
// holding 6.7% of its plane at the level. Read that way the develop divided by
// 2.000x instead of 1.953x, its plateau landed on 0.976 of the sensor, the
// desaturation fired at half strength, and the highlights kept the magenta the
// white balance gains make of them (36.7% of the frame's bright blocks red and
// blue of green against 0.0% in the camera's own JPEG of the same shot).
//
// The pile is still asked to be a few dozen samples and not a share of the plane:
// what a blown region is has nothing to do with how many the plane holds, and the
// frame that has blown one lamp is the frame whose highlights need the level most.
export function sensorWhite(data: ArrayLike<number>, maximum: number, black: number): number {
const legacy = 2 * (maximum - black);
const n = data.length;
let top = 0;
for (let i = 0; i < n; i++) if (data[i] > top) top = data[i];
if (top < maximum - black) return legacy;
const lo = top - 8;
const near = new Uint32Array(9);
for (let i = 0; i < n; i++) { const v = data[i]; if (v >= lo) near[v - lo]++; }
const floor = Math.max(32, Math.ceil(n * 1e-6));
// The top count can be a stray sample above the sensor's level; the pile under
// it is the level itself, so the scan starts at the top and takes the first
// count that is both a pile and a cliff over the count below it.
for (let k = 8; k >= 1; k--) {
if (near[k] >= floor && near[k] >= near[k - 1] * 8) return lo + k;
}
return legacy;
}