Show4DSTEM Storyboard

Show4DSTEM Storyboard#

Use with Storyboard.

MacBook support is a first-class Show4DSTEM target, not an afterthought. For large real 4D-STEM data, agents should explicitly test Apple Silicon raw-Metal/MPS loading and detector-binned U8 browse workflows, then separately test browser/WebGPU exported HTML and any CUDA/Torch workstation path relevant to the release.

Stories#

S4D-01: Open 4D-STEM Data Quickly#

User story: As a 4D-STEM user opening a scan, I want a useful virtual image and diffraction preview in about a second for normal preview sizes so I can start inspecting data immediately.

Primary widgets: Show4DSTEM.

Data to use: real or tutorial 4D-STEM stack; include full and binned preview variants when available.

Acceptance checks:

  • Load from Jupyter and exported HTML when supported.

  • Measure first visible paint, scan shape, diffraction shape, dtype, native bytes, binning/downsampling, and WebGPU availability.

  • Verify virtual image and diffraction panels render with labels, scale bars, and correct units.

  • Verify frontend-only pan/zoom/detector interactions do not make the kernel busy unless backend recomputation is expected.

S4D-02: Relate Scan Position To Diffraction Pattern#

User story: As a scientist inspecting 4D-STEM, I want scan-position movement in the virtual image to update the diffraction panel immediately.

Primary widgets: Show4DSTEM.

Data to use: real 4D-STEM scan with recognizable diffraction variation.

Acceptance checks:

  • Click and drag scan position across the virtual image.

  • Verify diffraction updates at the current scan position and labels/readouts stay synchronized.

  • Use keyboard or slider navigation if available.

  • Record FPS or latency for scan-position movement.

S4D-03: Tune Virtual Detectors#

User story: As a 4D-STEM user, I want BF, ABF, ADF, HAADF, and custom detector controls to update the virtual image interactively.

Primary widgets: Show4DSTEM.

Data to use: real 4D-STEM data with a visible central disk.

Acceptance checks:

  • Switch detector presets and verify virtual image changes.

  • Drag detector masks/rings and verify the virtual image updates without visible lag.

  • Verify detector radius/angle labels are scientifically meaningful.

  • Compare at least one detector result against a Python/reference computation for correctness-sensitive changes.

S4D-04: Inspect Diffraction Details#

User story: As a diffraction user, I want to pan, zoom, change contrast, and inspect diffraction features without losing the linked scan context.

Primary widgets: Show4DSTEM.

Data to use: real diffraction stack with visible Bragg/disk features.

Acceptance checks:

  • Pan and zoom the diffraction panel.

  • Change diffraction contrast, colormap, log/linear scale, and smoothing.

  • Verify detector overlays remain aligned during pan/zoom/resize.

  • Verify colorbar/histogram controls are readable on dark and light displays.

S4D-05: Use WebGPU And Fallback Paths Correctly#

User story: As a user on different hardware, I want the right compute path for the surface I am using: MPS/raw Metal or CUDA/Torch for live Python-backed work, WebGPU for browser/offline interaction, and a clear fallback when acceleration is not available.

Primary widgets: Show4DSTEM.

Data to use: WebGPU-capable Mac/browser, MacBook MPS load path, CUDA/Torch workstation path when available, plus a fallback browser or disabled WebGPU environment when possible.

Acceptance checks:

  • Record WebGPU adapter availability in the report.

  • Record Python backend and data loader path: CUDA/Torch, raw Metal/MPS, Torch-MPS, CPU, or browser/WebGPU.

  • Verify accelerated detector/virtual-image updates when WebGPU is available.

  • Verify MacBook live-Jupyter browsing can use MPS/raw Metal loading and computation for first-pass review.

  • Verify backend=”web” exported/offline pages use browser WebGPU and do not need Python, Torch, or MPS after export.

  • Verify fallback path is usable and clearly communicated when WebGPU is not available.

  • Do not claim WebGPU performance from CPU fallback.

  • Do not claim MPS/raw-Metal performance from a Torch-MPS or CPU path.

S4D-06: Save, Export, And Reopen 4D-STEM Views#

User story: As a notebook or sharing user, I want a compact two-panel saved preview and shareable export that preserve the scientific context and make the export precision obvious.

Primary widgets: Show4DSTEM.

Data to use: real or tutorial 4D-STEM notebook.

Acceptance checks:

  • Press Cmd+S and reload/reopen the notebook.

  • Verify the static two-panel fallback is visible.

  • Open Export and verify the menu follows the same vocabulary as the other storyboards: HTML uint8 for compact browse export, HTML full or HTML uint16 for count-preserving export, with approximate size when known.

  • Export compact uint8 HTML and reopen it.

  • Export full/uint16 HTML or folder HTML when supported and reopen it.

  • Drive scan position, diffraction pan/zoom, detector controls, and contrast in the exported page.

  • Check lightweight save state for heavy-buffer leaks.

  • Verify export status clears after completion/cancel in the same way as Show2D and Show3D.

S4D-07: Use 4D-STEM On A Phone Or Narrow View#

User story: As a user checking a 4D-STEM result on a phone or narrow screen, I want virtual image, diffraction, and detector controls to remain reachable.

Primary widgets: Show4DSTEM.

Data to use: compact real or tutorial 4D-STEM export.

Acceptance checks:

  • Test a narrow mobile viewport.

  • Verify panels stack or resize intentionally and labels remain readable.

  • Test touch-style scan-position movement, diffraction pan/zoom, detector control, and menu access.

  • For iPhone-specific claims, serve the page to physical iPhone Safari.

S4D-08: Export U8 And Full Data With Honest Reducers#

User story: As a 4D-STEM user sharing data, I want compact U8 HTML for quick browser inspection and full/count-preserving export when quantitative detector counts matter, so collaborators know when a view is browse-quality and when it can be used for quantitative checks.

Primary widgets: Show4DSTEM.

Data to use: real or real-derived 4D-STEM data with detector counts above 255 and a smaller count-limited control dataset where U8 should be nearly lossless.

Acceptance checks:

  • Export encoding="uint8" with downsample=1 and with detector downsample values such as 2, 4, and 8; verify the UI and status text identify it as compact/browse U8 data.

  • Verify U8 detector downsample uses the documented reducer, currently mean/average, so detector blocks do not immediately clip and wash out the bright-field disk.

  • Export encoding="full" or uint16 where supported and verify detector counts are preserved.

  • When full/uint16 export is downsampled, verify the reducer is scientifically explicit. Prefer sum for count-preserving detector binning when the exported dtype can hold the result; use mean only when the goal is browse/display stability and label it that way.

  • Compare at least one compact U8 export and one full/uint16 export against a Python reference for detector pixel values, virtual BF/ADF sums, and a custom mask.

  • Confirm exported HTML opens without a Python kernel and that scan-position movement, detector masks, diffraction contrast, and virtual images remain interactive.

S4D-09: Match The Shared Viewer GUI#

User story: As a user moving between Show2D, Show3D, and Show4DSTEM, I want the GUI layout and labels to feel consistent so I do not have to relearn export, contrast, scale, reset, copy, and panel controls for each viewer.

Primary widgets: Show4DSTEM, with Show2D and Show3D as visual references.

Data to use: one Show4DSTEM export plus one Show2D and one Show3D reference page using comparable colors, labels, scale bars, and top toolbar actions.

Acceptance checks:

  • Compare top toolbar order, compact switch/menu styling, export button labels, reset/copy placement, histogram/color controls, scale bars, and status text against Show2D and Show3D.

  • Verify the two-panel 4D-STEM layout keeps the virtual image and diffraction panel visually balanced on desktop, notebook, and narrow viewports.

  • Verify labels and readouts use the same row/column convention and units as the other storyboards.

  • Verify Export GUI choices match the Python API terms: mode, encoding, and downsample rather than older ambiguous names.

  • Drive the same user path in live Jupyter and exported HTML; document any GUI difference that is intentional.

S4D-10: Stress Export And WebGPU Reopen#

User story: As a user sharing large 4D-STEM screening results, I want export to finish in a practical time and the reopened artifact to stay responsive, so large browser-shareable views do not become dead files.

Primary widgets: Show4DSTEM.

Data to use: the largest real or real-derived 4D-STEM dataset available on the HPC/workstation backend for routine testing, plus a smaller deterministic dataset for reference parity.

Acceptance checks:

  • Measure export time, exported file/folder size, first paint after reopen, and WebGPU adapter availability.

  • Reopen U8 HTML and full/folder HTML where supported; verify no Python kernel is required for the expected interactions.

  • Drag scan position, detector ring, detector mask, diffraction pan/zoom, and contrast controls; record FPS or latency.

  • Verify folder export clearly fails or explains what is missing if the companion data folder is moved.

  • Add timings, reducer choice, dtype, downsample, browser, and backend host to the signoff report.

S4D-11: Use MacBook MPS For Live Loading And U8 Export#

User story: As a MacBook user opening large 4D-STEM data, I want first-pass browsing to use the fast Apple Silicon path, usually detector-binned U8, so the viewer opens quickly without exhausting unified memory.

Primary widgets: Show4DSTEM.

Data to use: a real 4D-STEM master file on a MacBook or a MacBook-connected Jupyter server, plus a smaller deterministic fixture for export parity.

Acceptance checks:

  • Load with load(path, backend="mps", det_bin=4 or 8, dtype="u8") and construct Show4DSTEM from that result.

  • Record load time, first paint, detector bin, dtype, resident memory, and whether the path is raw Metal/MPS, Torch-MPS, or CPU.

  • Export compact HTML with encoding="uint8" and reopen it in the browser.

  • Verify reopened HTML uses browser/WebGPU for interaction when available, not the Python MPS backend.

  • Compare one virtual detector and one diffraction frame against a Python reference at the same binned/U8 precision, and separately document any expected clipping from U8 browse data.

  • Repeat with encoding="full" or uint16 when the data size allows, and verify count-preserving expectations separately from the U8 browse path.

S4D-12: Explain Raw Metal MPS Versus Torch-MPS#

User story: As a developer or power user debugging MacBook performance, I want the report to say whether the viewer used raw Metal/MPS kernels or Torch-MPS, because those paths have different memory behavior and performance risks.

Primary widgets: Show4DSTEM.

Data to use: one MacBook MPS dataset large enough to expose memory pressure, plus a tiny deterministic comparison dataset.

Acceptance checks:

  • State why the selected path is raw Metal/MPS or Torch-MPS for the test.

  • For the raw Metal/MPS path, verify loading and detector binning avoid materializing an unnecessary full CPU copy.

  • For any Torch-MPS path, record tensor dtype, device, peak memory, and whether the operation falls back to CPU for unsupported kernels.

  • Verify the same scientific operation is compared against a CPU/Python reference: detector bin, BF/ADF virtual image, diffraction frame, and ROI summed/mean diffraction when relevant.

  • Document in the signoff whether the raw Metal path is used because it offers tighter control over chunking, dtype, and memory than generic Torch-MPS for this workflow.

S4D-13: Keep GPU Memory Lifecycle Outside The Viewer UI#

User story: As a user running many heavy 4D-STEM notebooks, I want GPU memory to be released by backend/session lifecycle controls rather than by a scientific viewer button, so the viewer stays focused on inspecting data and does not hide ownership of GPU resources.

Primary widgets: Show4DSTEM, plus backend loader/session tooling.

Data to use: repeated open/close of a large MPS or CUDA-backed 4D-STEM dataset.

Acceptance checks:

  • Verify closing/deleting a widget view does not imply the backend data object or GPU allocation is freed unless the owning Python object/session is also released.

  • Verify the documented cleanup path is backend/session level: delete or replace the loaded data object, clear references, stop/restart the kernel, or use the backend-specific cache cleanup utility when one exists.

  • Verify notebook save/reopen does not persist GPU buffers or export payloads when save_state=False.

  • Verify exported HTML has no live Python GPU allocation and therefore does not need a “free GPU” control.

  • If a future GUI exposes memory status, verify it reports backend ownership and links to cleanup instructions instead of pretending the viewer alone can free all GPU memory.

S4D-14: Watch A Live 4D-STEM Acquisition Folder In Place#

User story: As a microscopist collecting 4D-STEM, I want one already mounted Jupyter Show4DSTEM to discover every newly completed acquisition, expose it exactly once without rebuilding or silently changing precision, and remain interactive while incomplete detector files finish writing.

Primary widgets: Show4DSTEM.from_folder(...). Test the CUDA/CPU Dataset5dstem path and the public backend="mps" path separately because their paging and memory lifecycles differ. Standalone HTML is a snapshot and does not continue watching a filesystem.

Data to use: A temporary watched folder and at least three genuine 4D-STEM acquisition groups. Begin with one ready *_master.h5. Introduce a second master before one linked detector-data file exists, then complete it; add a third compatible master atomically. Record master/chunk paths, scan and detector shape, source dtype, requested det_bin and dtype, native bytes, backend host, selected GPUs, and widget commit. Tiny generated HDF5 is a CI lifecycle control only and does not establish real-workflow signoff.

Acceptance checks:

  • Mount Show4DSTEM.from_folder(folder, watch=True, watch_interval=..., ...) before the later masters arrive. Capture the Python identity, widget model ID, browser container, Dataset/page, panel order, stars, hidden panels, detector ROI, scan cursor, zoom, and playback state.

  • Keep one compact accessible watch badge near the folder/title area in stable DOM for both CUDA/CPU and MPS. Require green-dot Watching only while the actual watcher worker is alive. Enter Updating while discovery is active and keep it through real master/chunk validation and append. An idle poll may briefly show Updating but must return to Watching without decode, transfer, or repaint. If an arrival has a tile on the visible page, remain Updating until the new tile’s current-generation, raw-backed virtual image is authoritatively painted; an older cached preview may remain visible while it refreshes but does not satisfy this transition. Use amber Waiting for file completion while a master/chunk is incomplete or not yet stable, red Watch error with corrective detail for a bad contract, paint failure, or worker failure, and gray Stopped or Not watching after stop/close or when liveness cannot be established. watch=False has no badge. A restored notebook model, static fallback, or standalone snapshot must never restore green Watching without a live worker. Capture assertions and screenshots for every state; never infer state from color alone or leave a false green badge after exit.

  • A missing, corrupt, unopenable, or still-changing linked data file must not append or load. Once the master and every required link are readable and stable, append it exactly once in deterministic acquisition order. A known master rewrite or removal must not duplicate or silently delete the active scientific view.

  • Measure two visible stages separately: filesystem-ready to Dataset label, count, page control, or reserved placeholder paint; then selecting/requesting the new dataset to first virtual-image and diffraction paint. Do not call a Python trait update alone “append-to-paint.”

  • On CUDA/CPU, append each master as a cold lazy Dataset5dstem slot. Do not eagerly load every arrival, clear unrelated reduced pages, exceed page_budget, or silently change shape, dtype, detector bin, or scan bin. Recompute fit and placement safely after each append; cross-check the paging and generation rules in S4D-17 and S4D-18.

  • On multi-GPU CUDA, record per-card budget and residency, prove every selected GPU receives work, serialize work within one device, and permit concurrent waves only across independent devices. Hidden panels remain released and are excluded from compare recompute.

  • Repeat through Show4DSTEM.from_folder(..., backend="mps") on Apple Silicon. Identify the lazy MacBook/raw-Metal path explicitly, verify append ordering and memory, and report unsupported dtype or page-budget options as limitations rather than claiming CUDA Dataset5dstem behavior.

  • Run a small CPU fallback for lifecycle correctness, labeling it as non-performance evidence. On the same genuine source, separately verify a count-preserving full path such as det_bin=1, dtype="u16" and an explicit browse/downsample path such as det_bin=4; binned success is not proof of full-resolution support.

  • After append, verify compare_dp_mode="selected" follows the clicked new dataset and compare_dp_mode="average" matches a CPU reference over the current visible ready page, excluding hidden and incomplete datasets. Compare virtual images and diffraction patterns at two or more scan positions.

  • Drive the live path in real JupyterLab through the in-app browser while files arrive. Capture before/after screenshots, console errors, Debug UI FPS and folder/page/cache/memory counters, detector drag, scan movement, diffraction pan/zoom, page flip/playback, and both latency stages.

  • Verify stop_folder_watch() is idempotent and restartable. close() or free() must join watcher, page, preload, and cache workers; a file arriving after cleanup must not mutate the widget.

S4D-15: Sign Off Real Heavy 4D-STEM Performance#

User story: As a microscopist deciding whether Show4DSTEM is ready for a real acquisition session, I want one report that proves the viewer loads a large master quickly, chunks memory safely, appends new masters, and stays interactive in the browser.

Primary widgets: Show4DSTEM with an NVIDIA/CUDA backend when available, plus standalone exported HTML. Use the lazy MPS multi-dataset handle only for MacBook fallback checks.

Data to use: local real *_master.h5 files from a lab workstation or HPC-backed acquisition folder. Do not commit these files or their generated HTML reports to GitHub.

Acceptance checks:

  • Run PYTHONPATH=src:. python scripts/widget_show4dstem_heavy_signoff.py --backend cuda with an explicit local real-data root or QUANTEM_WIDGET_4DSTEM_ROOTS.

  • Verify first-master load time, widget build time, backend shape, dtype, device, resident memory, and GPU memory before/after are in show4dstem-heavy-signoff-report.json.

  • Verify at least one additional ready master is measured through the current backend’s append strategy: CUDA records eager stack-growth/reload timing; MPS records lazy live append timing.

  • After multiple masters are loaded, drive the Dataset/frame slider end to end and record flip latency/FPS. A report that only measures first load does not prove the real browsing workflow.

  • Verify standalone HTML export records explicit uint8/uint16 and detector-bin settings, output size, and export time.

  • Drive the exported HTML in Chromium and verify WebGPU/browser information, Dataset/frame flip FPS, virtual-detector drag FPS, scan-position drag FPS, recompute latency, and wheel-zoom FPS are recorded.

  • Treat --skip-browser as backend/export debugging only, not performance signoff.

  • For NVIDIA no-bin stress, run a separate capacity probe with 30-40 ready masters and --det-bin 1. Passing means either the data fit and browser flip-around is measured, or the report fails clearly with the maximum loaded master count, allocation error, and GPU cleanup evidence. Do not call a 30-40 file no-bin workflow supported just because a smaller stack is smooth.

S4D-16: Screen Many 4D-STEM Datasets In Multiple Mode#

User story: As a microscopist reviewing a session with many related 4D-STEM acquisitions, I want Show4DSTEM to show many virtual images at once while sharing the diffraction ROI and scan cursor, so I can quickly decide which datasets are useful, hide bad ones, star good ones, and preserve that curation for later notebook cells or shared HTML.

Primary widgets: Show4DSTEM in view_mode="multiple" with 5D data or a lazy multi-dataset handle.

Data to use: 8-14 binned real or real-derived 4D-STEM datasets for routine browser smoke; 30-40 ready masters on CUDA or MPS for heavy signoff when the backend and memory budget allow it.

Acceptance checks:

  • Construct Show4DSTEM(..., view_mode="multiple", compare_cols=...) from multiple datasets and verify the multiple grid renders all ready panels without materializing an unsafe full stack on MPS.

  • Verify desktop compare_cols is a maximum column count and the phone or narrow viewport caps the grid at two columns with readable tiles.

  • Verify compare_panel_gap_px=0 removes horizontal and vertical gutters between multiple panels, and nonzero values intentionally restore spacing.

  • Scroll over multiple tiles and the single-panel diffraction/virtual-image canvases; verify wheel input zooms the image instead of scrolling the page, and zoom-out behaves symmetrically.

  • Toggle compare_dp_mode between "average" and "selected"; verify the diffraction panel either averages visible multiple panels or follows the clicked dataset.

  • Star at least one useful dataset and hide at least one rejected dataset from the GUI; verify the visible panel count, labels, and selected dataset remain coherent.

  • Reorder panels by drag or click-then-click reorder mode; verify dynamic order changes before/after release and the saved order is still visible after reset/show-all actions.

  • Round-trip compare_panel_order, compare_hidden_panels, compare_starred_panels, and compare_dp_mode through state_dict() / load_state_dict() and through exported HTML.

  • Drive the same workflow in a physical phone browser when making iPhone/Safari claims; Chromium mobile emulation is only a pre-check.

  • Record dataset count, ready count, backend, dtype, detector bin, grid column count on desktop/mobile, FPS, and whether the artifact is live Jupyter or standalone HTML.

S4D-17: Page A Folder Safely On One CUDA GPU#

User story: As a scientist with one CUDA GPU, I want to open a folder whose complete 4D-STEM series exceeds usable VRAM, see the first useful page quickly, and browse every dataset at the requested resolution without calculating a manual memory limit or encountering a raw CUDA failure.

Primary widgets: Show4DSTEM.from_folder(...) in multiple mode.

Data to use: enough compatible real masters to exceed one selected GPU’s safe raw-residency budget after the requested detector bin and dtype. Include a deterministic small fixture with a forced two-frame budget for CI.

Acceptance checks:

  • Open with gpus=[0] and page_budget="auto". Verify discovery and the first useful page succeed whenever one processed master fits.

  • Keep the visible page size independent from the raw residency window. A page may contain more panels than fit as raw tensors; the backend must process it in bounded waves without silently changing dtype, detector bin, or shape.

  • Leave decoder/reduction headroom equal to at least one largest processed master plus bounded workspace before declaring the complete series resident.

  • Verify foreground page loading cancels or safely waits for full-series preload and cache warming. Concurrent decoders must not touch the same CUDA device.

  • Drive page 1, page 2, the last page, and page 1 again. Verify raw residency remains bounded, evicted folder masters reload correctly, and a warmed reduced page returns without reloading raw data.

  • Record click-to-first-panel, click-to-complete-page, warm-return latency, resident bytes, evictions, reloads, cache hits, and GPU memory before/after.

  • Treat CUDA illegal-address, host-register, OOM, stale-panel, or stuck worker errors as failures. Verify close() leaves no page/preload/cache worker.

S4D-18: Pool Multiple GPUs And Stream Pages Progressively#

User story: As a scientist with multiple CUDA GPUs, I want every selected GPU to contribute its safe capacity and compute throughput while folders larger than the combined working set remain paged. When I flip pages, stable panel slots should appear immediately and fill progressively as datasets become ready, rather than waiting for the slowest panel before showing anything.

Primary widgets: Show4DSTEM.from_folder(...) in multiple mode with gpus=[...] or gpus="all".

Data to use: real compatible masters on two or more CUDA GPUs, including equal cards, intentionally unequal free-memory budgets, and—when available—a folder distributed across independent physical disks. Use deterministic fake budgets and delayed loaders for CI cancellation/progress tests.

Acceptance checks:

  • Use only the explicitly selected process-visible GPUs. Compute a safe budget for each card and place cold masters according to available capacity; spare memory on a larger/freer card must not be stranded by fixed round-robin assignment. Every report must record CUDA_VISIBLE_DEVICES and map each logical index to the physical GPU UUID, PCI bus ID, model name, total memory, and free memory at the start of the run.

  • Load one independent master allocation per participating GPU in each progressive wave. Different GPUs may load/compute concurrently, but work on one GPU remains serialized and independently reclaimable by LRU eviction.

  • Keep the page grid stable from the click: reserve every expected slot, show a quiet loading state, and replace each slot as its float32 virtual image arrives without resizing or remounting the rest of the grid.

  • Give every page request a generation. Rapid page 1 -> 2 -> 3 changes cancel obsolete work after its current safe wave, and late page-1/page-2 results must never overwrite page 3.

  • Prioritize the visible page and selected diffraction source, then prefetch the next and previous pages for the current detector preset. Full-series preload and other detector-preset warming run only when foreground work is idle.

  • Preserve reduced virtual-image pages in the bounded host cache independently of raw GPU residency. Verify a warm return does not require the old raw page to remain on a GPU.

  • Record placeholder acknowledgement, first-panel, half-page, complete-page, and warm-return latency; per-GPU budget/resident bytes; cache hits/misses; evictions/reloads; stale-result drops; and browser paint/FPS evidence.

  • Compare one-GPU and multi-GPU cold-page timing with the same data. Verify all selected GPUs receive work and that adding a useful GPU does not reduce safe capacity or make first-panel latency worse without an explained I/O limit.

  • Repeat after a live folder append and after external memory pressure changes. Verify placement is recalculated safely and existing warmed pages remain valid.

S4D-19: Reopen A Folder With Persistent Scientific Previews#

User story: As a scientist returning to a large 4D-STEM folder, I want the BF/ABF/ADF/HAADF images I already computed to appear immediately while the authoritative raw data loads in the background. I need the viewer to say when I am seeing a cached preview and when fresh raw-backed interaction is ready, so a second open is useful instead of showing black panels for another cold decode.

Primary widgets: Show4DSTEM.from_folder(...) in multiple mode, first on one NVIDIA CUDA GPU and then on multiple selected CUDA GPUs. The CPU path is a lifecycle control; MPS support is a follow-up and must not be inferred from the CUDA signoff.

Data to use: the same real 82-or-more-master folder used for cold progressive paging, with linked detector chunks where present. Retain enough standard virtual images to exceed one visible page, reopen from a new widget instance, then change, replace, remove, and append individual masters/chunks. Use a tiny multi-master fixture for deterministic CI hit, miss, corruption, eviction, and cancellation cases.

Acceptance checks:

  • Expose preview_cache="auto", preview_cache_dir=None, preview_cache_max_bytes=4 << 30, and rebuild_preview_cache=False on Show4DSTEM.from_folder(...). False disables persistent reads and writes; the automatic mode uses the user cache, and an explicit directory supports a chosen local SSD. A shared or network filesystem is unverified until its atomic-rename and multi-process writer/rebuild/clear behavior pass explicitly; process-local locking alone is not a shared-cache guarantee. Keep this disk budget independent from compare_cache_max_bytes host memory and page_budget raw CUDA residency.

  • Persist only reduced float32 BF, ABF, ADF, and HAADF virtual images. Never persist raw 4D detector tensors, diffraction patterns, arbitrary detector masks, CUDA allocations, credentials, or private source data outside the configured cache directory. warm_cache=True may fill the standard presets proactively; normal use writes a standard preset after computing it.

  • Store previews per master, not per display page. Hiding, starring, reordering, changing page size, or moving a master to another page must reuse that master’s valid preview without duplicating it.

  • Validate each entry against a versioned processing key and the current source fingerprint: canonical master identity; master size, nanosecond mtime/ctime, device, and inode; the ordered identities, sizes, nanosecond mtimes/ctimes, devices, and inodes of every linked detector chunk; processed scan/detector shape; source/load dtype; detector bin; scan override; center and preset radii/mask geometry; and the cache schema/compute version. A change to one master or chunk invalidates only that master’s presets. An unchanged append must not invalidate existing entries.

  • A missing, unreadable, incomplete, or changing required chunk is not a valid cache hit. A corrupt, truncated, incompatible, or partially written cache artifact becomes a counted miss and is rebuilt safely; it must not break folder discovery or paint unverified pixels.

  • Publish cache files atomically and make concurrent readers safe. Enforce preview_cache_max_bytes with deterministic whole-entry eviction while no writer can leave a manifest pointing at an incomplete payload. Cache lookup must not decode raw 4D data or allocate CUDA memory.

  • On a matching second open, reserve the normal stable grid slots and paint each cached panel as soon as it is read. Show a quiet, explicit state such as Cached preview · loading raw data; never label cached pixels Fresh or show an empty black panel where a valid preview is available.

  • Keep startup accounting honest: this CUDA-first phase still validates and loads one raw master synchronously to establish detector shape, calibration, and the selected diffraction pattern. Report API-call-to-model-ready separately from model-ready-to-cached-paint, and do not claim a cache-only mount. A future metadata/calibration bootstrap may remove that final raw dependency without weakening provenance checks.

  • Continue raw loading and reduction through the normal capacity-aware CUDA scheduler. Replace the cached pixels in the same panel slot when the current generation’s fresh result arrives, without remounting the grid, changing contrast unexpectedly, or flashing black. Once raw data is ready, detector changes and diffraction inspection use the authoritative requested dtype and resolution.

  • When a persistent preview is shown for a newly arrived master on the visible page, keep the folder-watch badge at Updating while the cached pixels stay useful. Return to Watching only after the corresponding current-generation raw-backed tile has reached authoritative browser paint; otherwise transition to the truthful amber waiting or red corrective-error state.

  • Support partial hits. Paint cached panels first, show honest per-page progress, and schedule raw work only as needed for misses and authoritative refresh. Rapid page 1 -> 2 -> 3 changes must cancel obsolete refresh work after a safe wave; late cached or fresh results must never overwrite page 3.

  • If refresh fails after a valid preview painted, keep the preview visible and mark it Cached preview · refresh failed with a corrective error. Do not silently relabel stale pixels as fresh, and do not discard a useful preview merely to return to a black placeholder.

  • Expose a read-only preview_cache_info property with enabled state, path, byte limit/current bytes, entry count, hits, misses, invalidations, evictions, and errors. clear_preview_cache() deletes this widget’s persistent preview namespace and resets its accounting without clearing ShowFolder thumbnails or pretending to free raw CUDA memory. rebuild_preview_cache=True ignores old entries for the new run and repopulates them safely.

  • Measure browser paint, not only Python traits. Record click-to-cached-first panel, click-to-cached-visible-page, click-to-fresh-first panel, click-to-fresh-visible-page, click-to-complete-page, and neighbor-prefetch completion separately, plus cache lookup/read/write bytes and time, hit/miss counts, raw decode/reduction time, per-GPU residency, FPS, and console errors. The browser probe exposes the receipt and after-paint proxy fields under window.__quantemShow4DSTEMPerf.comparePage as firstCachedPanelReceiptAtMs, firstFreshPanelReceiptAtMs, firstCachedPanelPaintAtMs, firstFreshPanelPaintAtMs, cachedVisiblePaintAtMs, and freshVisiblePaintAtMs; keep receipt and double-animation-frame after-paint proxy evidence labeled separately.

  • Compare cold first open, matching second open, partial-hit reopen, forced rebuild, and disabled-cache runs on one selected NVIDIA GPU. Cached-first paint must be materially faster than the approximately one-second progressive cold first panel. Keep the measured approximately 11.27-second visible-page completion separate from the approximately 22.91-second worker/neighbor- prefetch-idle time; cached previews must expose useful panels without waiting for either. On the reference host, over five fresh-widget page opens, require median click-to-cached-first <= 500 ms and <= 50% of the matched cold median, plus median cached-visible-page <= 2 s and <= 25% of matched cold visible-page time. Report p95 as evidence rather than hiding a slow outlier.

  • Define storage conditions for every timing run. Distinguish a fresh Python/widget process from an OS/filesystem-page-cache-cold run, and record source/cache filesystem and locality, storage device class, HDF5 compression, linked-chunk count, bytes read, and achieved throughput. Never attribute an I/O-limited result to GPU scaling without that evidence.

  • Persistent lookup must not serialize the raw refresh. On the same host and source, median click-to-fresh-visible-page and complete-page time may regress by at most 10% versus preview_cache=False; otherwise record the cache I/O contention as a failed performance gate. Neighbor prefetch must start only after the visible foreground request reaches its ready state.

  • Repeat on two or more selected NVIDIA GPUs. Persistent hits must remain backend-independent, while misses and refreshes use all eligible cards under S4D-18’s per-device serialization and generation rules. Adding a GPU must not duplicate disk entries, corrupt the cache, exceed either memory budget, or make cached-first paint wait for the slowest raw wave.

  • Verify close() joins cache readers/writers and CUDA page workers. Reopen in a fresh Python process to prove persistence, then clear the cache and prove the next open is a true miss. Leave cache artifacts and real-data reports outside git unless deliberately promoted into a maintainer fixture or runbook.

S4D-20: Prove Folder Paging And Cache Behavior Overnight#

User story: As a scientist leaving a large acquisition folder open overnight, I want automatic paging, cached previews, live arrivals, and fresh raw-backed replacement to remain truthful and responsive on either one or two NVIDIA GPUs, so an ended runner or a green badge cannot hide a stalled, memory-leaking, or stale scientific view.

Primary widgets: Show4DSTEM.from_folder(...) in a real JupyterLab session and its browser frontend. This story is the endurance composition of S4D-14, S4D-17, S4D-18, and S4D-19; those stories remain the canonical source for watch-state, scheduler, generation, and cache correctness rather than being repeated here.

Data to use: One compatible real acquisition series large enough to exceed the safe raw-residency budget of one selected GPU, with linked detector chunks when present. Use a staged watched-folder view of the real files for arrival tests so the source acquisition is never rewritten. Add a separate real full-detector control using det_bin=1 and a count-preserving dtype, with at least seven masters and enough masters to exceed the selected raw budget when the source permits. Synthetic data is a CI control only.

Acceptance checks:

  • Run an explicit, serial matrix so the topologies do not contend: one selected physical NVIDIA GPU, then the same workflow on two selected physical NVIDIA GPUs. Give each topology at least four clock hours and 100 completed canonical navigation cycles, for at least eight hours total. A canonical cycle requests page 1, page 2, the last page, and page 1 again; performs a rapid page 1 -> 2 -> 3 cancellation check; and exercises selected/average diffraction, hide, star, scan movement, detector movement, diffraction zoom, and pan. A partial or restarted cycle does not count as completed.

  • Hold source, ready-master set, page size, page_budget="auto", detector bin, dtype, detector preset, cache budget, and page sequence constant for the matched one-GPU/two-GPU comparison. Run cold cache-disabled, cache-populating, and matching cache-enabled phases in fresh Python processes. Reopen the matching cache at least five times per topology, and include the partial-hit, changed-master/chunk, corrupt-entry, forced-rebuild, disabled-cache, and clear cases from S4D-19 without corrupting source data or the canonical cache copy.

  • During both topologies, introduce at least one real master with a required chunk withheld, then make the complete acquisition visible atomically. Prove it remains waiting while incomplete, appears exactly once when ready, keeps a valid cached preview visible when one exists, and reaches green Watching only after a current-generation fresh tile receives the browser paint acknowledgement required by S4D-14 and S4D-19. A Python trait publication, cached paint, or backend worker completion is not fresh-paint proof.

  • Exercise the viewer in actual JupyterLab through a controlled browser for each topology, not only through Python traits or an exported snapshot. Capture the stable loading slots, cached-first paint, fresh replacement in the same slot, every watch-badge state, page cancellation, and representative scientific interactions. Preserve timestamped screenshots, browser console output, model and kernel errors, and the receipt-versus-after-paint fields from window.__quantemShow4DSTEMPerf.comparePage.

  • Record a topology and provenance snapshot before every process: host, UTC start time, widget commit and dirty-diff identity, Python/Torch/CUDA/driver versions, CUDA_VISIBLE_DEVICES, and each logical index’s physical UUID, PCI bus ID, model, total memory, and free memory. Record source/cache canonical paths, filesystem and storage device, locality, compression, linked-chunk count, source fingerprint, ready-master count, shapes, dtype, detector bin, and cache schema/compute version. Repeat GPU memory and filesystem snapshots after each phase and at cleanup.

  • Apply the cache latency and no-more-than-10-percent fresh-refresh regression gates in S4D-19 to each topology, including median and p95 over the required fresh-process reopens. For the matched GPU comparison, require two-GPU median fresh-first latency to be no more than 110% of one-GPU latency and median fresh-visible and complete-page latency to be no slower than one GPU. Prove both GPUs receive work. If storage/decode saturation prevents scaling, retain the measurements and mark the scaling gate limited or failed; do not convert an explanation into a pass.

  • Divide each topology’s completed endurance cycles into first and last quartiles. Require no more than 20% regression in p95 fresh-visible and complete-page latency, no monotonic resident host/GPU/cache growth after warm-up, zero stale-generation paints, zero unhandled browser/kernel errors, zero CUDA illegal-address/OOM errors, and zero stuck page, preload, watch, or cache workers. Report cache hits/misses/invalidations/evictions, raw evictions/reloads, stale drops, bytes read/written, throughput, per-GPU residency, and memory high-water marks even when the gate fails.

  • Write an atomic run manifest after every phase and completed cycle. Include a monotonically advancing checkpoint, active topology/phase, cache namespace, owned process IDs, last successful page generation, and artifact paths. Emit a timestamped heartbeat at least every five minutes with progress, worker liveness, GPU memory/utilization, and the most recent error. A watchdog treats three missed heartbeats or 15 minutes without forward progress as a failure, captures stacks/logs/GPU state, stops only owned processes, and resumes from the last complete checkpoint in a fresh process.

  • Prove resume behavior once with a controlled child-process interruption. The resumed run must preserve valid cache entries, avoid duplicate live arrivals and stale page paint, and distinguish pre-interruption, resumed, and fully continuous time in the report. Automatic restart attempts are bounded and visible; exhausting them fails the run instead of leaving it indefinitely running or declaring success because the launcher exited.

  • Run the separate real no-bin leg on one GPU and then two GPUs with automatic paging and the same provenance/cleanup capture. At minimum, browse first, second, last, and warm-return pages and compare one virtual image and one diffraction result with a CPU reference. If one processed master cannot fit, or the available series cannot exceed the selected raw budget, record that exact capacity boundary as an unmet gate rather than inferring support from a detector-binned run.

  • On normal completion, interruption, and failure, call the public cleanup path and prove the watcher, page, preload, cache, notebook, browser, tunnel, and watchdog processes owned by the run are gone. Record final GPU memory against baseline and preserve corrective evidence for any residual allocation. Never delete the real source; clear or corrupt only a run-owned cache/staging copy.

  • Produce one durable top-level index.html plus machine-readable JSON summary, atomic manifest, phase/cycle timing table, GPU samples, source/cache provenance, browser screenshots and console log, kernel/worker logs, cache inventory, parity results, failure/restart timeline, and exact commands. Keep artifacts outside git, publish the exact report path and review URL, and mark every gate pass, fail, limited, skipped, or unavailable. A run is complete only when the report is readable and all required artifacts are present; an old report, an ended task, or a missing heartbeat is not current signoff.