web: a scan watches the catalogue on a clock and keeps its place in a file, not in five megabytes of local storage

LIBRARY read the whole catalogue back on every batch the scan wrote: `reload()`
is a `getAll` of the `photos` store, every row and every thumbnail in it, and on
a catalogue of 150,000 frames one read was measured at 2036ms, twenty of them at
315s with the worst at 58s, the heap behind them going from 44MB to 89MB — beside
a LibRaw open of a quarter of a gigabyte and four lanes of RAW bytes. On a folder
of RAW that is the crash, and on any long roll it is the strip taking the roll's
own time to move.

The read-back now runs on a clock: at most once every five seconds of a scan, and
once more when the reading ends, which is when the last of it is due. A batch
lands every second or so, and reading it back costs every frame in the catalogue
— so a scan that read it back each time spent the roll doing nothing else. The
strip still fills while the scan runs; it fills in steps.

The walk's own position went to the origin private file system. It is one path
per frame the walk has found and not read — about twenty-eight characters each,
measured over a folder of three thousand, which is the 85,626-character write —
and local storage on this origin takes 5,000,000 characters and then throws,
measured the same way. That is a hundred and seventy-five thousand frames: past
it the first write of a long roll fails, the position is nowhere, and the next
visit walks the whole roll from the top. That is the reading that goes back to
the start, and OPFS takes the same text as a file. It goes down at most once
every two seconds, for the same reason the catalogue is read back on a clock:
the string is a few megabytes of paths.

The folder is walked again for being raised no more than once every thirty
seconds. A hundred thousand frames is a hundred thousand stats on the disk, and
this screen is raised every time the reader comes back from the studio.

And the column no longer says there is no folder before it has looked. The note
waited on `folders` alone, which is empty for as long as the screen is coming up:
on the catalogue of 150,000 it said so for the whole of the load and showed the
folder after. It now waits on `loaded`, and says nothing until there is something
to say.

Checked on the running bundle: a 3000-frame scan reads the catalogue back 13
times where the code read it back once per batch, writes the position 21 times,
and leaves the peak heap at 116MB with no long tasks. A reading stopped at 483 of
4000 leaves `walk/ROLL.json` at 101,107 characters naming 3,550 unread frames,
with no local storage key beside it, and the visit after a reload carries on at
585 rather than at the top. LIBRARY on 150,000 frames shows no "No folder yet" at
any point and settles on "150000 photos". The wall, the strip, the grid, the deep
link and the phone's recipes all pass their checks.

docker/frontend/src/Library.tsx

@@ -108,6 +108,14 @@ interface Shot {
 // the read in flight rather than the object under it.
 const shots = new Map<string, Promise<Shot | null>>();
 
+// How often a running scan is allowed to have its catalogue read back, and how
+// often the tab may walk the folder again just for being raised. Both are the
+// same trade — the reading is watched, not driven — and both are seconds for the
+// same reason: the roll is the long thing, and neither the strip nor a folder
+// that has not moved is worth a frame of it.
+const CATALOGUE_MS = 5000;
+const CATCHUP_MS = 30000;
+
 
 
 export function Library() {
@@ -308,6 +316,14 @@ export function Library() {
     // walked into something new.
     let wrote = -1;
     let drew = '';
+    // And when it was last read back. A batch lands every second or so on a
+    // folder of JPEGs, and reading a catalogue of a hundred thousand frames is a
+    // second and a half of the page held still and sixty megabytes of rows thrown
+    // away — a scan that read it back every batch spent the roll doing nothing
+    // else, and a folder of RAW, with a quarter of a gigabyte of LibRaw open
+    // beside it, went past what a page is given and took the tab with it. On a
+    // clock the strip still fills while the scan runs.
+    let read = 0;
     const watching = () => {
       bump();
       const session = scanSession();
@@ -334,8 +350,10 @@ export function Library() {
       // may have no such count: the frames it has stored are then read back on a
       // frame, as they were before there was one.
       const written = progress.written ?? progress.done;
-      if (written !== wrote) {
+      const now = Date.now();
+      if (written !== wrote && now - read >= CATALOGUE_MS) {
         wrote = written;
+        read = now;
         void reload();
       }
     };
@@ -436,6 +454,11 @@ export function Library() {
   // frames that landed in the folder since. No note: the ring on the row and the
   // progress line are the report.
   const rootKey = node ? node.split('/')[0] : null;
+  // When the folder was last walked for being raised. It stands outside the
+  // effect because the effect is rebuilt for the folder the reader moves to, and
+  // the walk it guards is of the folder on the disk, where nothing has moved
+  // either way.
+  const synced = useRef(0);
   useEffect(() => {
     const root = folders.find((f) => f.name === rootKey);
     if (!loaded || !root || !ready.has(root.name)) return;
@@ -444,6 +467,13 @@ export function Library() {
     // keeps this one from reading the same roll a second time.
     const catchUp = () => {
       if (scanSession()) return;
+      // A folder of a hundred thousand frames is a hundred thousand stats on the
+      // disk, and this screen is raised every time the reader comes back from the
+      // studio — so the walk waits for the folder to have had a moment to change,
+      // which is a second of a roll the reader was not going to notice and not
+      // the several it took to walk it again.
+      if (Date.now() - synced.current < CATCHUP_MS) return;
+      synced.current = Date.now();
       void scanBusy().then((busy) => {
         if (!busy && !scanSession()) void scan(root, true);
       });
@@ -902,7 +932,10 @@ export function Library() {
               ) : null}
             </div>
 
-            {folders.length === 0 ? <p className="hint adm-note">{t('lib.noFolders')}</p> : null}
+            {/* The folders are on their way in for as long as the catalogue behind
+                them is being read back, and a screen that says there is no folder
+                before it has looked is a screen that says it every visit. */}
+            {loaded && folders.length === 0 ? <p className="hint adm-note">{t('lib.noFolders')}</p> : null}
 
             {folders.length > 0 ? (
               <div className="adm-col">

docker/frontend/src/engine/library.ts

@@ -34,6 +34,10 @@ const BATCH = 50;
 // on the next frame is time the roll does not get back. Past a few, the disk and
 // the decoder are the limit and the page has less room to breathe.
 const LANES = 4;
+// How often the walk's position is written down, in the same seconds as the
+// catalogue is read back on: the position is the whole of what the walk has found
+// and not read, and writing it is a string of a few megabytes on a long roll.
+const WALK_MS = 2000;
 
 // One LibRaw open at a time, whoever asks for it. A RAW is opened inside a worker
 // the library builds with a quarter of a gigabyte of linear memory of its own
@@ -210,7 +214,7 @@ export async function removeFolder(name: string): Promise<void> {
   });
   // The reading's position goes with the folder: a folder that is gone is not
   // one to carry on reading, and a folder picked again starts at the top.
-  clearWalk(name);
+  await clearWalk(name);
 }
 
 // A handle kept in IndexedDB comes back without its permission: the browser
@@ -361,14 +365,33 @@ export interface ScanProgress {
 // the catalogue only holds the frames that were read — so the reading would
 // start again at the top of the roll and walk every folder a second time.
 //
-// Local storage holds exactly that position, and holds it for the whole origin,
-// which is the point: the tab that reloads carries on from the frame it stopped
-// at, and so does the next context to open the app — the installed one beside the
-// browser, where session storage, being the tab's own, handed it nothing and the
-// walk started at the top of the roll a second time, every folder listed again
-// and every frame it already held read again only to be skipped. The key goes
-// when the reading finishes.
-const WALK_KEY = (folder: string) => `recipescam.library.walk.${folder}`;
+// The position is held for the whole origin, which is the point: the tab that
+// reloads carries on from the frame it stopped at, and so does the next context
+// to open the app — the installed one beside the browser, where session storage,
+// being the tab's own, handed it nothing and the walk started at the top of the
+// roll a second time, every folder listed again and every frame it already held
+// read again only to be skipped. The file goes when the reading finishes.
+//
+// It is a file in the origin private file system and not a key in local storage,
+// which is where it was. The position is one path per frame the walk has found
+// and not read — about twenty-eight characters each, measured over a folder of
+// three thousand — and local storage on this origin takes 5,000,000 characters
+// and then throws, also measured. That is a hundred and seventy-five thousand
+// frames: past it the first write of a long roll fails, the position is nowhere,
+// and the next visit walks the whole roll from the top — the reader's own
+// complaint, and one a roll of RAW reaches in a morning. OPFS takes the same text
+// as a file, has room for a catalogue's worth of them, and is the origin's the
+// same way. An unwritable position is still a reading that starts at the top:
+// private mode lands there, as it always did.
+const WALK_DIR = 'walk';
+const WALK_FILE = (folder: string) => `${encodeURIComponent(folder)}.json`;
+
+async function walkFile(folder: string, create: boolean): Promise<FileSystemFileHandle | null> {
+  const root = await navigator.storage?.getDirectory?.();
+  if (!root) return null;
+  const dir = await root.getDirectoryHandle(WALK_DIR, { create });
+  return dir.getFileHandle(WALK_FILE(folder), { create });
+}
 
 interface WalkSaved {
   // The folders the walk is through, and the ones it has not read yet, in the
@@ -385,7 +408,7 @@ interface WalkSaved {
   progress: ScanProgress;
 }
 
-function loadWalk(folder: string): WalkSaved | null {
+async function loadWalk(folder: string): Promise<WalkSaved | null> {
   // One reading at a time: a peer that has announced this folder is walking it
   // right now, and what it has written down is a step of a walk still in
   // progress. Taken up here, two readings would write the same roll down at
@@ -393,9 +416,9 @@ function loadWalk(folder: string): WalkSaved | null {
   // come up; now that the position is the origin's, it can.
   if (peer?.folder === folder) return null;
   try {
-    const raw = localStorage.getItem(WALK_KEY(folder));
-    if (!raw) return null;
-    const saved = JSON.parse(raw) as WalkSaved;
+    const fh = await walkFile(folder, false);
+    if (!fh) return null;
+    const saved = JSON.parse(await (await fh.getFile()).text()) as WalkSaved;
     return Array.isArray(saved?.walked) &&
       Array.isArray(saved?.pending) &&
       Array.isArray(saved?.frames) &&
@@ -409,27 +432,33 @@ function loadWalk(folder: string): WalkSaved | null {
 }
 
 // A position that cannot be written down is a reading that starts from the top,
-// which is what this did before — private mode and a full quota both land here.
-function saveWalk(
+// which is what this did before — private mode lands here, as it always did.
+async function saveWalk(
   folder: string,
   walked: string[],
   pending: string[],
   frames: string[],
   restored: number,
   progress: ScanProgress
-): void {
+): Promise<void> {
   try {
-    localStorage.setItem(WALK_KEY(folder), JSON.stringify({ walked, pending, frames, restored, progress } satisfies WalkSaved));
+    const fh = await walkFile(folder, true);
+    if (!fh) return;
+    const w = await fh.createWritable();
+    await w.write(JSON.stringify({ walked, pending, frames, restored, progress } satisfies WalkSaved));
+    await w.close();
   } catch {
     // Nothing to write down: the reading starts from the top, as it did before.
   }
 }
 
-function clearWalk(folder: string): void {
+async function clearWalk(folder: string): Promise<void> {
   try {
-    localStorage.removeItem(WALK_KEY(folder));
+    const root = await navigator.storage?.getDirectory?.();
+    const dir = await root?.getDirectoryHandle(WALK_DIR, { create: false });
+    await dir?.removeEntry(WALK_FILE(folder));
   } catch {
-    // Nothing to clear.
+    // Absent is the desired state.
   }
 }
 
@@ -468,7 +497,7 @@ export async function scanFolder(
   jump?: () => string | null
 ): Promise<ScanProgress> {
   const known = new Map((await listPhotos(folder.name)).map((p) => [p.id, p]));
-  const saved = loadWalk(folder.name);
+  const saved = await loadWalk(folder.name);
   const tree: Walk = { root: folder.handle, pending: [], walked: new Set(saved?.walked ?? []) };
   // A queue comes back as paths, so the folders are asked for again; a reading
   // with nothing written down starts at the picked folder.
@@ -528,9 +557,19 @@ export async function scanFolder(
   // frames the written-down count has already counted.
   let inHand: string[] = [];
   const pathOf = (row: LibraryPhoto) => (row.dir ? `${row.dir}/${row.name}` : row.name);
-  const write = () => {
+  // The position goes down on a clock, not on every batch. The whole of it is
+  // written each time — a copy of every path the walk has found and not read —
+  // and on a roll of a hundred thousand that is a few megabytes turned into a
+  // string per batch, which is the page held still for the whole roll. A position
+  // a step behind is a frame or two read again on the next visit, and a frame read
+  // again is skipped on its size and its time.
+  let wroteAt = 0;
+  const write = async () => {
+    const at = Date.now();
+    if (at - wroteAt < WALK_MS) return;
+    wroteAt = at;
     const held = [...inHand, ...batch.map(pathOf)];
-    saveWalk(
+    await saveWalk(
       folder.name,
       [...tree.walked],
       tree.pending.map((p) => p.rel),
@@ -557,7 +596,7 @@ export async function scanFolder(
       tx.onerror = () => reject(tx.error);
     });
     progress.written += rows.length;
-    write();
+    await write();
   };
   // One frame, end to end: its bytes, its tile, its shutter time, and the row the
   // catalogue keeps. A frame that has not moved is dropped on its size and its
@@ -696,7 +735,7 @@ export async function scanFolder(
       // reads the pass's frames again, and a frame already in the catalogue is
       // skipped anyway — where the other order would leave the last pass's
       // frames out of the catalogue until the next visit.
-      write();
+      await write();
       onProgress?.(progress);
     }
     await drain();
@@ -704,7 +743,7 @@ export async function scanFolder(
   await flush();
   // A reading that came to its end has no position worth keeping: the next one
   // walks the roll from the top and skips what has not moved.
-  if (!stop) clearWalk(folder.name);
+  if (!stop) await clearWalk(folder.name);
   return progress;
 }