web: open a RAW at the resolution of its sensor, not at the quarter of it
LibRaw's half-size demosaic was on. The Ricoh GR's own DNG (D0004128.DNG) developed to 3010x2012 while the JPEG written beside it in the same second is 6000x4000, and the Fuji's RAF to 3008x2007 against its own 6000x4000 -- the quarter was the flag, not the file. With `halfSize: false` the same develop returns 6020x4024 and it is the sensor's frame on every body tried: D0004128.DNG 6020x4024 IMGP6916.DNG 6028x4024 DSCF1701.RAF 6016x4014 _DSC0009.ARW 6024x4024 AFXT2721.RAF 6246x4170 Nikon-D850 NEF 6216x4136 _GDN0447.NEF 4284x2844 P1010607.RW2 3472x3472 5G4A9396.CR2 2880x1920 Nine files, 27s to 155s a develop on one core. Checked through the app itself, not only through LibRaw: photo-dims 6020x4024 on the DNG against 6000x4000 on the JPEG, both err none. The colour it opens with is now fitted per file to the preview the camera wrote into it (previewMatch.ts): a 3x3 over a block grid of the develop against the same grid of that preview, then one cubic a channel for what the 3x3 leaves. The offline per-body table this replaces (cameraMatch.ts) stopped matching the moment the path under it changed -- its rows no longer summed to 1 once the highlight knee landed ahead of it -- and a body with a row opened with a cast one without did not. The file's own preview does not age. The white level the gain carries is the frame's own plateau rather than `maximum` (sensorWhite.ts), a factor of 1.89 to 2.00 out; without it every frame opened a stop bright and a body that sat lower (X-Trans, 1.892) never reached the highlight desaturation at all. The desaturation gate reads the gain-lifted levels as well as the sensor's, which is the whole of the magenta: on a body whose cam_mul lifts red and blue (the GR's [2.64, 1, 1.73]) a blown sky crosses the white level at 0.38 of the raw range in red while green crosses at 1.0, so a gate read on the sensor's levels alone stayed shut across it. Measured in the app against the camera's own JPEG, mean dRGB over a 16x16 block grid: +1.20, -5.95, -6.11 with the sensor's clip alone, +0.21, +0.24, +0.47 with both, mean |dL| 21.5 against 10.3. The same grid on the Fuji comes back balanced (+4.7, +5.0, +3.6) and best aligned at offset 0,0. -HL is recovery and +HL is a lift, so they are different moves now: recovery is the doc's soft knee in linear light over the top half, which is the only term in the tone shader that is not a shift and the only one that can put detail back into a blown sky rather than merely darken it. The four checks pin the develop down where it can only run in a browser: raw-develop-check, preview-match-check, white-level-check, highlight-knee-check.
This commit is contained in:
@@ -1,15 +1,49 @@
|
||||
// RAW → JPEG on the client, so a camera's own file opens in the studio without
|
||||
// a DNG converter in the middle (see native_raw_processing_opfs_architecture.md).
|
||||
//
|
||||
// LibRaw demosaics in its own worker; what comes back is linear camera data,
|
||||
// A RAW opens at the colour the camera chose for it. LibRaw is kept out of white
|
||||
// balance and tone (below) and knows nothing of the body's picture style, so the
|
||||
// develop is fitted to the one rendering the file does carry with it: the preview
|
||||
// the camera wrote inside it, which is the frame the photographer saw and the one
|
||||
// a desktop viewer shows. Develop the sensor, least-squares a 3x3 from a block
|
||||
// grid of each onto the other on the encoded values, fit what that 3x3 leaves a
|
||||
// cubic a channel, and develop the sensor through both (see previewMatch.ts).
|
||||
//
|
||||
// Measured against the camera's own 24MP JPEG of the A5100 frame this was
|
||||
// reported on, blocks of a 64x64 grid, dE00: 12.0 as the develop left it, 19.7
|
||||
// with no white balance at all, 6.6 matched to the file's own preview by the 3x3
|
||||
// alone and 4.4 through the 3x3 and the curve together. What neither reaches is
|
||||
// the camera's own sharpening, which no per-pixel transform holds.
|
||||
//
|
||||
// The fit is the only profile: a body-used-to-be table of six matrices fitted on
|
||||
// this develop was dropped, because a table fitted on one path stops matching the
|
||||
// moment the path changes under it — its rows no longer summed to 1 once the
|
||||
// highlight knee landed ahead of it, and every body that had one opened with a
|
||||
// cast the body that had none did not. Offline-fitted matrices age; the file's
|
||||
// own preview does not.
|
||||
//
|
||||
// Developing the sensor is also what keeps the highlights: 2.3% of the frame at
|
||||
// pure white against 3.3% in the preview itself, since the 8-bit preview threw
|
||||
// the headroom away and the develop still has it. The preview is therefore the
|
||||
// reference and the fallback, not the frame: it is what a file with no usable
|
||||
// preview (some DNG), or with sensor data that will not decode, opens as.
|
||||
//
|
||||
// The develop: LibRaw demosaics in its own worker; what comes back is linear camera data,
|
||||
// which this file turns into the sRGB the rest of the pipeline expects. The
|
||||
// whole thing is measured against a real 26MP Sony ARW — the settings below are
|
||||
// the ones that gave the correct colours there:
|
||||
// settings below were checked against the preview a Sony ILME-FX30 writes into
|
||||
// its own ARW (the camera JPEG, read straight out of the file): the developed
|
||||
// frame and that preview agree to within 1% on both channel ratios, R/G 0.909
|
||||
// against 0.903 and B/G 0.603 against 0.606.
|
||||
// - noAutoScale + useCameraWb:false + noAutoBright + gamm [1,1] keep LibRaw out
|
||||
// of white balance and tone, so `cam_mul` and `rgb_cam` can be applied here
|
||||
// exactly once.
|
||||
// - halfSize: 26MP → 6.5MP. ponytail: drop it for full resolution if a user
|
||||
// ever asks for a print from the RAW; the develop pass is the whole cost.
|
||||
// - halfSize:false: the frame opens at the sensor's own resolution — a 24MP
|
||||
// RAW develops to 24MP (6020x4024 on the GR, 6000x4000 on the Fuji), not to
|
||||
// the quarter the half-size demosaic reports. `userQual` 3 then demosaics
|
||||
// all of it. The develop was half-size until a GR's DNG opened at 3010x2012
|
||||
// against its own JPEG's 6000x4000: the quarter-size frame was the flag,
|
||||
// not the file. ponytail: costs ~4x the develop time and two full-size F16
|
||||
// surfaces; put the flag behind a "draft" toggle if a phone ever has to.
|
||||
//
|
||||
// The band loop exists because a single Float32 copy of the whole plane would be
|
||||
// ~100MB. Each band is decoded, normalised and drawn before the next is read.
|
||||
@@ -17,14 +51,24 @@
|
||||
// the main thread (Skia is not available in the RAW worker). Move it to a worker
|
||||
// with an OffscreenCanvas if the develop ever blocks the UI visibly.
|
||||
import LibRaw from 'libraw-wasm';
|
||||
import { cameraMatch, type Mat3 } from './cameraMatch';
|
||||
import { sensorWhite } from './sensorWhite';
|
||||
import { f32ToF16 } from './halfFloat';
|
||||
import { fitMatch, FLAT_TONE, MATCH_GRID, type Match, type Mat3 } from './previewMatch';
|
||||
import { Skia } from './skiaShim';
|
||||
|
||||
// What `imageData()` returns for the settings below: 16-bit, 3 channels, with
|
||||
// the black level still in it — hence the two normalisations in the shader.
|
||||
// What `imageData()` returns for the settings below: 16-bit, 3 channels, black
|
||||
// level already gone — the post-process subtracts it whatever `noAutoScale`
|
||||
// says, which only holds back the white balance and the output scaling.
|
||||
//
|
||||
// Its white level is not `maximum` but the frame's own plateau, a factor of 1.89
|
||||
// to 2.00 out (see sensorWhite below). The gain carries that level, so the white
|
||||
// lands back on 1.0 — and on every body, not just the two that factor two was
|
||||
// fitted on: without it every frame opened a stop bright (the FX30's own JPEG has
|
||||
// 0.05% of pixels at pure white where the develop had 0.76%) and a body that sat
|
||||
// lower (X-Trans, 1.892) never even reached the highlight desaturation, which
|
||||
// starts at 0.95 of the sensor.
|
||||
const SETTINGS = {
|
||||
halfSize: true,
|
||||
halfSize: false,
|
||||
outputBps: 16,
|
||||
outputColor: 0,
|
||||
noAutoScale: true,
|
||||
@@ -46,53 +90,103 @@ const IDENTITY: Mat3 = [1, 0, 0, 0, 1, 0, 0, 0, 1];
|
||||
|
||||
const RAW_DEVELOP_SKSL = `
|
||||
uniform shader raw;
|
||||
uniform float4 black; // (black level, 1 / (white level - black level))
|
||||
uniform float gain; // 1 / the white level the frame itself ran out at
|
||||
uniform float4 mul; // cam_mul, green-normalised
|
||||
uniform float4 m0; // camera -> sRGB, the first three columns of rgb_cam
|
||||
uniform float4 m1;
|
||||
uniform float4 m2;
|
||||
uniform float4 crop; // (y offset of this band, 0, 0, 0)
|
||||
uniform float4 w0; // the body's camera match, one row per float4
|
||||
uniform float4 w1; // (identity when the body has no profile)
|
||||
uniform float4 w2;
|
||||
uniform float4 f0; // the per-file fit, on the encoded value (identity when
|
||||
uniform float4 f1; // the frame carries no preview to be fitted to)
|
||||
uniform float4 f2;
|
||||
uniform float4 t0; // what that fit then leaves a channel: one cubic a
|
||||
uniform float4 t1; // channel, r a b c d per float4 (flat when there is no fit)
|
||||
uniform float4 t2;
|
||||
|
||||
// A 3x3 can only scale a channel; the gap to the camera is mostly a shape, and
|
||||
// this is that shape, read on the value the 3x3 left.
|
||||
float tone(float4 w, float x) {
|
||||
return clamp(w.x + x * (w.y + x * (w.z + x * w.w)), 0.0, 1.0);
|
||||
}
|
||||
|
||||
float3 encode(float3 x) {
|
||||
x = clamp(x, 0.0, 1.0);
|
||||
return mix(x * 12.92, 1.055 * pow(x, float3(1.0 / 2.4)) - 0.055, step(float3(0.0031308), x));
|
||||
}
|
||||
|
||||
float luma(float3 x) {
|
||||
return dot(float3(0.2126, 0.7152, 0.0722), x);
|
||||
}
|
||||
|
||||
half4 main(float2 pos) {
|
||||
float4 p = raw.eval(float2(pos.x, pos.y - crop.x));
|
||||
// Only the floor here. A photo the sensor could not hold goes over the white
|
||||
// The plane arrives with the black level already subtracted — it floors at 0,
|
||||
// not at color_data.black (measured on the FX30 ARW: the sensor mosaic floors
|
||||
// at 334, the plane at 0, a quarter of its red samples under the black level).
|
||||
// Subtracting it again drained red and blue — the two channels the gains lift
|
||||
// most — and dragged every frame towards green.
|
||||
float3 n = max(p.rgb * gain, 0.0); // the sensor's own levels, white level 1.0
|
||||
float3 lin = n * mul.rgb;
|
||||
// Only the floor there. A photo the sensor could not hold goes over the white
|
||||
// level in all three channels, and clipping them one by one before the WB gains
|
||||
// is what tints what is left of the highlight: green — the channel the gains are
|
||||
// normalised to — stops at 1.0 while red and blue, which need their 2.6x and
|
||||
// 1.6x, are already past it, so the blown area comes out magenta. Keep the
|
||||
// channel ratios through the matrix instead and let the overflow fade to white.
|
||||
float3 lin = max((p.rgb - black.rgb) * black.a, 0.0) * mul.rgb;
|
||||
float3 rgb = float3(dot(m0.xyz, lin), dot(m1.xyz, lin), dot(m2.xyz, lin));
|
||||
float mx = max(max(rgb.r, rgb.g), rgb.b);
|
||||
// The overflow fades towards white instead of being cut, so the trace of a
|
||||
// sensor that ran past its white level survives as a compressed ramp — which is
|
||||
// what LIGHT's HIGHLIGHT row then has to pull on. ponytail: the ramp lives in
|
||||
// [0.5, 1] and the band leaves as 8-bit JPEG, so the two stops above white are
|
||||
// compressed, not kept; give develop a 16-bit output (or a knee of its own) when
|
||||
// RAW highlights have to be recovered rather than merely look right.
|
||||
rgb = mx > 1.0 ? mix(rgb / mx, float3(1.0), 1.0 - 1.0 / mx) : rgb;
|
||||
// A pixel that has run to the white level has no colour of its own left to keep,
|
||||
// and what the gains made of it is an artefact, not a colour: ease the pixel
|
||||
// towards the neutral of its own value as that point is approached. The clip to
|
||||
// read is the one the gains make, not only the sensor's own — the gains here are
|
||||
// 1.7x and 1.9x on red and blue, so a blown sky reaches the white level at 0.59 of
|
||||
// the raw range in those channels while the green, which the gains are
|
||||
// normalised to, only reaches it at 1.0. Read on the sensor's levels alone the
|
||||
// gate stayed shut across a whole blown sky and left the develop's magenta in it
|
||||
// (227,184,245 at the gate's own value against 245,245,245 read where the gains
|
||||
// put the clip, the camera's preview white at that block). Both are read, so a
|
||||
// body whose gains do not lift a channel keeps the sensor's own clip as its gate.
|
||||
// The matrix was fitted luma-preserving, so mx is the value to hold.
|
||||
float hi = max(max(n.r, n.g), n.b);
|
||||
hi = max(hi, max(max(lin.r, lin.g), lin.b));
|
||||
rgb = mix(rgb, float3(mx), smoothstep(0.95, 1.0, hi));
|
||||
// The overflow used to fade towards white — mix(rgb / mx, 1, 1 - 1 / mx) —
|
||||
// which put every pixel of a blown sky on exactly 1.0 and threw the two stops
|
||||
// the sensor held above white away with it: LIGHT's HIGHLIGHT row then had a
|
||||
// flat white to pull on and nothing to reveal. The white point is moved down
|
||||
// instead and the overflow squeezed back in under it by the doc's soft knee:
|
||||
//
|
||||
// y = T + over / (1 + 2S*over), over = mx - T, S = 1 / (2(1 - T))
|
||||
//
|
||||
// S is what puts 1.0 on the asymptote, so the sensor's own plateau — two white
|
||||
// levels up, see the gain above — lands at ~0.94 and the first stop over white
|
||||
// spends 0.85..0.94. Below T the frame is untouched and the curve leaves T with
|
||||
// the slope it arrived with (1), so there is no seam to mask; above it the frame
|
||||
// darkens, which is the one move no later pass can undo — which is the point,
|
||||
// and what the per-file fit below then measures the REST of the frame back from.
|
||||
//
|
||||
// ponytail: 4.7 stops of headroom now share ~5% of the ramp, and the develop
|
||||
// still leaves as an 8-bit JPEG. Give it a float16 output when RAW highlights
|
||||
// have to print rather than merely be seen.
|
||||
if (mx > 0.7) {
|
||||
float over = mx - 0.7;
|
||||
rgb *= (0.7 + over / (1.0 + over * 3.3333)) / mx;
|
||||
}
|
||||
float3 e = encode(rgb);
|
||||
// The body's colour, in the encoded space it was fitted in. The matrix was
|
||||
// fitted luma-preserving, and the luma(e) / luma(c) factor holds that exact:
|
||||
// the profile turns the chroma and never rescues an exposure, which stays the
|
||||
// studio's job. Identity is a no-op.
|
||||
float3 c = clamp(float3(dot(w0.xyz, e), dot(w1.xyz, e), dot(w2.xyz, e)), 0.0, 1.0);
|
||||
// ponytail: the band leaves as 8-bit JPEG, so the match is applied to a value
|
||||
// that is already quantised — plenty for a look, but give develop 16-bit output
|
||||
// if a profile ever has to grade rather than merely match.
|
||||
return half4(half3(clamp(c * (luma(e) / max(luma(c), 1e-4)), 0.0, 1.0)), 1.0);
|
||||
// The file's own colour: the fit, in float, on the encoded value it was fitted
|
||||
// on — and the last step the frame leaves through. Not the 8-bit colour filter
|
||||
// this used to be painted through: the fit carries an exposure (the preview is
|
||||
// the reference, so matching its brightness is part of matching its colour).
|
||||
// No rolloff here either. The fit is fitted whole to the preview's own values,
|
||||
// and white is one of them, so it already maps the frame's white to the preview's
|
||||
// — while a divide by the row max is a white-preserving move the fit does not
|
||||
// need: every highlight came back at ~0.74 / 1.0 / 0.86, cyan, and not one cell
|
||||
// of the frame reached white on all three channels (0.0% against the preview's
|
||||
// 3.3%). Past the white level only the clamp is left, exactly as the develop
|
||||
// above lets its own overflow run.
|
||||
float3 q = clamp(float3(dot(f0.xyz, e), dot(f1.xyz, e), dot(f2.xyz, e)), 0.0, 1.0);
|
||||
// ...and then the curve, one cubic a channel, which is what carries the body's
|
||||
// own tone: a 3x3 can only scale, so the frame without it came back bright and
|
||||
// green in the shadows (dL +13.8 and green +0.128 at the bottom of the range,
|
||||
// dE00 6.6) instead of matching (dL +4.5, green +0.017, 4.4). The curve is
|
||||
// pinned at white, so a blown pixel still lands on white.
|
||||
return half4(half3(tone(t0, q.r), tone(t1, q.g), tone(t2, q.b)), 1.0);
|
||||
}
|
||||
`;
|
||||
|
||||
@@ -110,11 +204,71 @@ export function isRawName(name: string): boolean {
|
||||
return name.includes('.') && RAW_EXT.includes(ext);
|
||||
}
|
||||
|
||||
// The camera's own preview, when the file carries one: the colour reference to
|
||||
// fit against, and what the file opens as when the sensor does not decode.
|
||||
// ponytail: it is 1616x1080 on an A5100 and 1620x1080 on an FX30, so handing it
|
||||
// back is a 1.7MP frame — a print past it has to come off the develop, which is
|
||||
// what a fitted file already gives.
|
||||
async function cameraPreview(raw: LibRaw): Promise<Uint8Array | null> {
|
||||
const thumb = await raw.thumbnailData().catch(() => undefined);
|
||||
if (thumb?.format !== 'jpeg' || !thumb.data?.length) return null;
|
||||
return new Uint8Array(thumb.data);
|
||||
}
|
||||
|
||||
// MATCH_GRID x MATCH_GRID block colours of a frame, one byte per channel.
|
||||
function gridOf(image: any, n = MATCH_GRID): Uint8Array | null {
|
||||
const surface = Skia.Surface.MakeOffscreen(n, n) ?? Skia.Surface.Make(n, n);
|
||||
if (!surface) return null;
|
||||
const canvas = surface.getCanvas();
|
||||
// Cubic, not a linear tap: this is a 45x reduction and linear reads a handful
|
||||
// of source pixels per block — noise for the least squares to fit.
|
||||
canvas.drawImageRectCubic(
|
||||
image,
|
||||
Skia.XYWHRect(0, 0, image.width(), image.height()),
|
||||
Skia.XYWHRect(0, 0, n, n),
|
||||
1 / 3,
|
||||
1 / 3
|
||||
);
|
||||
surface.flush();
|
||||
const px = canvas.readPixels(0, 0, {
|
||||
width: n,
|
||||
height: n,
|
||||
colorType: Skia.ColorType.RGBA_8888,
|
||||
alphaType: Skia.AlphaType.Unpremul,
|
||||
colorSpace: Skia.ColorSpace.SRGB,
|
||||
}) as Uint8Array | null;
|
||||
surface.dispose();
|
||||
return px ? new Uint8Array(px.buffer, px.byteOffset, px.byteLength) : null;
|
||||
}
|
||||
|
||||
// The same grid out of the preview, which the file carries as a JPEG. Decoded and
|
||||
// reduced through the same Skia call as the develop's own grid, because the two
|
||||
// grids are only comparable — and the fit only meaningful — when one resampler
|
||||
// made both. A 2D canvas here instead left the fit following its own smoothing:
|
||||
// on the A5100 frame the same develop scored dE00 4.7 against 4.4, and the dark
|
||||
// end of the frame came out 6 L further from the preview than the fit it was
|
||||
// handed asked for.
|
||||
function previewGrid(jpeg: Uint8Array, w: number, h: number, n = MATCH_GRID): Uint8Array | null {
|
||||
const bmp = Skia.Image.MakeImageFromEncoded(jpeg);
|
||||
if (!bmp) return null;
|
||||
try {
|
||||
// A preview of another shape is a crop of the frame, not the frame: fitting
|
||||
// against it lines the two grids up on different scenes and fits nothing.
|
||||
if (Math.abs(bmp.width() / bmp.height() / (w / h) - 1) > 0.02) return null;
|
||||
if (bmp.width() < n * 4) return null;
|
||||
return gridOf(bmp, n);
|
||||
} finally {
|
||||
bmp.delete();
|
||||
}
|
||||
}
|
||||
|
||||
export async function developRaw(bytes: Uint8Array): Promise<Uint8Array> {
|
||||
const raw = new LibRaw();
|
||||
let preview: Uint8Array | null = null;
|
||||
try {
|
||||
// LibRaw copies the buffer it is handed, so the caller's bytes stay intact.
|
||||
await raw.open(bytes as unknown as BufferSource, SETTINGS);
|
||||
preview = await cameraPreview(raw);
|
||||
const meta = await raw.metadata(true);
|
||||
const img = await raw.imageData();
|
||||
const cd = meta?.color_data;
|
||||
@@ -124,8 +278,6 @@ export async function developRaw(bytes: Uint8Array): Promise<Uint8Array> {
|
||||
const data = img.data as Uint16Array;
|
||||
if (!w || !h) throw new Error('RAW decoded to nothing');
|
||||
|
||||
const surface = Skia.Surface.MakeOffscreen(w, h) ?? Skia.Surface.Make(w, h);
|
||||
if (!surface) throw new Error('no surface for the develop');
|
||||
const effect = Skia.RuntimeEffect.Make(RAW_DEVELOP_SKSL);
|
||||
if (!effect) throw new Error('develop shader failed to compile');
|
||||
|
||||
@@ -133,62 +285,91 @@ export async function developRaw(bytes: Uint8Array): Promise<Uint8Array> {
|
||||
const mul = cd.cam_mul.map((v) => v / green);
|
||||
const row = (i: number) => cd.rgb_cam[i].slice(0, 3);
|
||||
const [r0, r1, r2] = [row(0), row(1), row(2)];
|
||||
const cm: Mat3 = cameraMatch(meta?.normalized_model) ?? IDENTITY;
|
||||
|
||||
const bandH = Math.max(1, Math.min(h, Math.floor(BAND_PIXELS / w)));
|
||||
const f32 = new Float32Array(w * bandH * 4);
|
||||
// The band goes up as half, not float32: the GPU backend puts an F32 image
|
||||
// on the 1/255 grid and the shadows quantise to black (see halfFloat.ts).
|
||||
const half = new Uint16Array(w * bandH * 4);
|
||||
for (let y0 = 0; y0 < h; y0 += bandH) {
|
||||
const rows = Math.min(bandH, h - y0);
|
||||
let o = 0;
|
||||
for (let i = y0 * w * 3, end = (y0 + rows) * w * 3; i < end; i += 3) {
|
||||
f32[o++] = data[i] / SAMPLE_MAX;
|
||||
f32[o++] = data[i + 1] / SAMPLE_MAX;
|
||||
f32[o++] = data[i + 2] / SAMPLE_MAX;
|
||||
f32[o++] = 1;
|
||||
}
|
||||
f32ToF16(f32, half, w * rows * 4);
|
||||
const band = Skia.Image.MakeImage(
|
||||
{ width: w, height: rows, colorType: Skia.ColorType.RGBA_F16, alphaType: Skia.AlphaType.Unpremul },
|
||||
new Uint8Array(half.buffer, 0, w * rows * 8),
|
||||
w * 8
|
||||
);
|
||||
if (!band) throw new Error('band image failed');
|
||||
const child = band.makeShaderOptions(
|
||||
Skia.TileMode.Clamp,
|
||||
Skia.TileMode.Clamp,
|
||||
Skia.FilterMode.Nearest,
|
||||
Skia.MipmapMode.None
|
||||
);
|
||||
const uniforms = new Float32Array([
|
||||
cd.black / SAMPLE_MAX, cd.black / SAMPLE_MAX, cd.black / SAMPLE_MAX,
|
||||
SAMPLE_MAX / (cd.maximum - cd.black),
|
||||
mul[0], mul[1], mul[2], 0,
|
||||
r0[0], r0[1], r0[2], 0,
|
||||
r1[0], r1[1], r1[2], 0,
|
||||
r2[0], r2[1], r2[2], 0,
|
||||
y0, 0, 0, 0,
|
||||
cm[0], cm[1], cm[2], 0,
|
||||
cm[3], cm[4], cm[5], 0,
|
||||
cm[6], cm[7], cm[8], 0,
|
||||
]);
|
||||
const shader = effect.makeShaderWithChildren(uniforms, [child]);
|
||||
const paint = Skia.Paint();
|
||||
paint.setShader(shader);
|
||||
surface.getCanvas().drawRect(Skia.XYWHRect(0, y0, w, rows), paint);
|
||||
surface.flush();
|
||||
paint.delete();
|
||||
shader.delete();
|
||||
child.delete();
|
||||
band.delete();
|
||||
}
|
||||
// Every uniform but the crop and the fit, which are the two the passes change:
|
||||
// the shader's own order is gain, mul, rgb_cam, crop, the fit, then the fit's
|
||||
// tone curve.
|
||||
const uniforms = new Float32Array(45);
|
||||
uniforms[0] = SAMPLE_MAX / sensorWhite(data, cd.maximum, cd.black);
|
||||
uniforms.set([mul[0], mul[1], mul[2], 0, r0[0], r0[1], r0[2], 0, r1[0], r1[1], r1[2], 0, r2[0], r2[1], r2[2], 0], 1);
|
||||
|
||||
const jpeg = surface.makeImageSnapshot().encodeToBytes(Skia.ImageFormat.JPEG, 92);
|
||||
surface.dispose();
|
||||
// One develop of the frame, band by band, through `fit` when there is one. A
|
||||
// function because the frame is drawn twice: once on the sensor alone, to fit
|
||||
// against the preview, and then again with the fit in the shader.
|
||||
// ponytail: two full band passes, on the main thread. Give develop an F16
|
||||
// intermediate (one develop, one colour pass) if the second pass ever shows.
|
||||
const develop = (fit: Match | null) => {
|
||||
const surface = Skia.Surface.MakeOffscreen(w, h) ?? Skia.Surface.Make(w, h);
|
||||
if (!surface) return null;
|
||||
const f = fit?.m ?? IDENTITY;
|
||||
uniforms[21] = f[0]; uniforms[22] = f[1]; uniforms[23] = f[2]; uniforms[24] = 0;
|
||||
uniforms[25] = f[3]; uniforms[26] = f[4]; uniforms[27] = f[5]; uniforms[28] = 0;
|
||||
uniforms[29] = f[6]; uniforms[30] = f[7]; uniforms[31] = f[8]; uniforms[32] = 0;
|
||||
uniforms.set(fit?.tone ?? FLAT_TONE, 33);
|
||||
for (let y0 = 0; y0 < h; y0 += bandH) {
|
||||
const rows = Math.min(bandH, h - y0);
|
||||
let o = 0;
|
||||
for (let i = y0 * w * 3, end = (y0 + rows) * w * 3; i < end; i += 3) {
|
||||
f32[o++] = data[i] / SAMPLE_MAX;
|
||||
f32[o++] = data[i + 1] / SAMPLE_MAX;
|
||||
f32[o++] = data[i + 2] / SAMPLE_MAX;
|
||||
f32[o++] = 1;
|
||||
}
|
||||
f32ToF16(f32, half, w * rows * 4);
|
||||
const band = Skia.Image.MakeImage(
|
||||
{ width: w, height: rows, colorType: Skia.ColorType.RGBA_F16, alphaType: Skia.AlphaType.Unpremul },
|
||||
new Uint8Array(half.buffer, 0, w * rows * 8),
|
||||
w * 8
|
||||
);
|
||||
if (!band) throw new Error('band image failed');
|
||||
const child = band.makeShaderOptions(
|
||||
Skia.TileMode.Clamp,
|
||||
Skia.TileMode.Clamp,
|
||||
Skia.FilterMode.Nearest,
|
||||
Skia.MipmapMode.None
|
||||
);
|
||||
uniforms[17] = y0;
|
||||
const shader = effect.makeShaderWithChildren(uniforms, [child]);
|
||||
const paint = Skia.Paint();
|
||||
paint.setShader(shader);
|
||||
surface.getCanvas().drawRect(Skia.XYWHRect(0, y0, w, rows), paint);
|
||||
surface.flush();
|
||||
paint.delete();
|
||||
shader.delete();
|
||||
child.delete();
|
||||
band.delete();
|
||||
}
|
||||
|
||||
const shot = surface.makeImageSnapshot();
|
||||
surface.dispose();
|
||||
return shot;
|
||||
};
|
||||
|
||||
// The file's own colour: a grid of the develop as it stands, the same grid out
|
||||
// of the preview the camera wrote into the file, and the 3x3 and the curve
|
||||
// between them — which the second develop then draws in the shader. No
|
||||
// preview, no fit: the frame opens as the sensor left it.
|
||||
const first = develop(null);
|
||||
if (!first) throw new Error('no surface for the develop');
|
||||
const blocks = preview ? gridOf(first) : null;
|
||||
const ref = blocks ? previewGrid(preview as Uint8Array, first.width(), first.height()) : null;
|
||||
const match = blocks && ref ? fitMatch(blocks, ref) : null;
|
||||
const matched = match ? develop(match) : null;
|
||||
const jpeg = (matched ?? first).encodeToBytes(Skia.ImageFormat.JPEG, 92);
|
||||
(matched ?? first).dispose();
|
||||
if (matched) first.dispose();
|
||||
if (!jpeg?.length) throw new Error('develop produced no bytes');
|
||||
return jpeg;
|
||||
} catch (err) {
|
||||
// The preview still opens the file when the sensor will not: a RAW whose
|
||||
// colour data is missing (some DNG) is not a RAW that cannot be shown.
|
||||
if (preview) return preview;
|
||||
throw err;
|
||||
} finally {
|
||||
raw.dispose();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user