web: shoot through the live camera

The one path where the look is chosen before the picture exists: OPEN CAMERA
grades the camera's own feed with the recipe in force, many times a second, and
the shutter hands the studio the sensor's still under that same recipe. Preview
and file differ in resolution only — the still is `takePhoto`'s own frame, not a
copy of the small preview video, with `grabFrame` and a 2d copy of the element
behind it for the browsers that ship no ImageCapture.

The renderer gains two inputs for it: `sourceImage`, a picture the caller
already decoded (re-encoding the camera's frame to JPEG only to decode it again
would cost more than the whole render), and `drawTo`, which paints the finished
picture instead of encoding it. One render is in flight at a time; a frame that
arrives during one is dropped, so a slow device shows a lower frame rate rather
than a queue of moments that have passed.

The view flashed black on a phone. Setting width/height on a canvas resets its
bitmap: measured on the preview, a resize leaves mean 0 until the next render
lands, which on this box is 0.5s and on a phone more. The buffer was sized from
every incoming frame, and a capture that renegotiates its resolution — which
Chromium does when the page is too slow to consume its frames, and this pipeline
runs ~2 fps at 720p under software GL — strobed black/picture at every switch.
The buffer is now sized on the first frame and after that only when the frame's
aspect changes: a same-aspect frame is scaled into it. Swapping a 1280x720
stream for a 640x360 one mid-view now leaves the buffer at 1280x720 with no
black frame, and 640x360 renders at 6-13 fps instead of 2.

The frames are read from a <video>, which is now IN the document (1px, behind
the black backdrop) rather than detached: Safari draws blank frames from a
detached video, which is the same black-between-pictures. It leaves the document
with the view, and the tracks are stopped, so the camera light goes out.

Probes: cam-smoke (feed painted, resolution, frame rate, a monochrome sim
reaching the live frames, shutter into the studio, close, console clean),
cam-renegotiate (no resize, no blank frame, status line on the frames),
cam-close-flip (flip returns a picture; video gone on close).
This commit is contained in:
2026-09-26 08:26:28 +07:00
parent fd2d9935c1
commit cb1839e36b
9 changed files with 576 additions and 13 deletions
+18 -2
View File
@@ -96,13 +96,22 @@ export interface RenderOptions {
}
export interface RenderInput {
source: Uint8Array;
source?: Uint8Array;
recipe: Recipe;
frameId: FrameId;
useGeotag?: boolean;
gpsInfo?: GPSInfo | null;
options?: RenderOptions;
assets: RenderAssets;
// A picture the caller already decoded, instead of `source`'s bytes. The live
// camera has a new one every frame: re-encoding it to JPEG only for this to
// decode it again would cost more than the whole render.
sourceImage?: any;
// Paint the finished picture instead of encoding it, and hand it over here.
// The live view blits it and lets it go; the bytes of a JPEG would be thrown
// away thirty times a second. Called synchronously, with an image that is
// already dead once this returns.
drawTo?: (image: any) => void;
}
export interface RenderResult {
@@ -370,7 +379,7 @@ export async function renderPhoto(input: RenderInput): Promise<RenderResult | nu
return obj as T;
};
try {
const skImage0 = Skia.Image.MakeImageFromEncoded(input.source);
const skImage0 = input.sourceImage ?? (input.source ? Skia.Image.MakeImageFromEncoded(input.source) : null);
if (!skImage0) {
console.error('Failed to parse source image');
return null;
@@ -1056,6 +1065,13 @@ export async function renderPhoto(input: RenderInput): Promise<RenderResult | nu
}
}
// 10. The live view stops here: it wants the picture, not a file, and it
// says so with `drawTo` — no encode, no DPI patch, no bytes.
if (input.drawTo) {
input.drawTo(resultImage);
return { bytes: new Uint8Array(), width: resultImage.width(), height: resultImage.height() };
}
let bytes = resultImage.encodeToBytes(Skia.ImageFormat.JPEG, 95);
if (!bytes || bytes.length === 0) {
console.error('Failed to encode image to JPEG');