# ShowFolder Storyboard

ShowFolder is the folder-level entry point for electron microscopy sessions. It
should help a user look before loading: choose a folder, scan the files, inspect
calibrated thumbnails and metadata, select the useful files, and save a compact
selection without embedding large raw data in the notebook.

This file is the canonical protocol for ShowFolder behavior, testing, and agent
signoff. Keep scratch notebooks, screenshots, and phone-test pages outside the
repository unless a release explicitly needs them as committed examples.

## SF-1: Choose a Folder

**User story**: A microscopist opens a notebook and wants to point ShowFolder at
a session folder without writing boilerplate path code.

**Primary widgets**: ShowFolder folder chooser or fallback path input.

**Data to use**: A small local microscopy-like folder with at least two image
files and one nested subfolder. Use real or real-derived files when available.

**Acceptance checks**:

- `ShowFolder()` renders a visible folder selection UI in JupyterLab.
- If `ipyfilechooser` is installed, the chooser can select a directory and run
  the folder browser.
- If `ipyfilechooser` is not installed, the fallback path field clearly shows
  the selected path and the `Open folder` button runs the same browser.
- Invalid paths show a useful error with the path that failed.
- The selected folder is remembered by key for the next widget creation.

## SF-2: Survey Real Session Files

**User story**: A user opens a real microscopy session and needs fast thumbnails,
file names, calibrated scale bars, and enough metadata to decide what to load.

**Primary widgets**: ShowFolder and image gallery rows.

**Data to use**: A real or real-derived session folder containing calibrated 2D
images. Include EDS files only to confirm ShowFolder inventories them without
opening the EDS viewer.

**Acceptance checks**:

- `ShowFolder(path, thumb=256, group_by="fov")` scans the folder and shows
  image thumbnails without loading full-resolution arrays into notebook state.
- Scale bars use calibrated sampling when available.
- File names, shapes, and errors are readable.
- EDS files are listed in the inventory when present, but ShowFolder does not
  create ShowEDS launchers or EDS selection controls.
- `*_master.h5` files are summarized in the 4D-STEM master QC table with
  status, scan shape, detector shape, dtype, chunk count, and a corrective next
  step. This is a header-only check; it must not decompress detector frames or
  allocate GPU memory.
- The UI remains usable when the folder contains many files.

## SF-3: Cache and Rerun

**User story**: The first scan may spend time reading files, but the second scan
should reuse thumbnails and metadata when files have not changed.

**Primary widgets**: ShowFolder cache.

**Data to use**: A folder with multiple image files and a temporary cache
directory.

**Acceptance checks**:

- First run reports cache misses and writes `manifest.json` plus
  `thumbnails.npz`.
- Second run reports cache hits and does not reread unchanged image files.
- Editing, replacing, or touching a file invalidates only that file's cached
  entry.
- `cache=False` runs without writing cache files.
- `cache="folder"` writes under `.quantem/showfolder-cache`.
- `clear_cache()` removes the active manifest and thumbnail bundle.

## SF-4: Watch a Live Folder

**User story**: A user is collecting or generating microscopy files while a
notebook is open and wants ShowFolder to show new results without rerunning the
cell or losing their current selection. When every new image matters, the user
also wants one normal control to keep both the all-image Show2D gallery and
Show3D stack visibly synchronized.

**Primary widgets**: ShowFolder watcher, cache status banner, image gallery,
and the ShowFolder-managed Show2D/Show3D preview handoffs. Direct
``Show2D.from_folder(...)``, ``Show3D.from_folder(...)``, and
``Show4DSTEM.from_folder(...)`` are separate full-resolution scientific-data
watchers covered by S2D-18, S3D-17, and S4D-14.

**Data to use**: A real or real-derived folder where files can be copied in and
removed during the test. Prefer lab data from an HPC/workstation or microscope
backend; keep the raw files outside the widget repository.

**Acceptance checks**:

- `ShowFolder(path).watch(interval=...)` inserts a visible accessible status
  line. Green ``Watching`` means the actual ShowFolder worker is alive;
  ``Updating`` appears only after a real arrival or removal is detected, not on
  idle polls. Use amber ``Waiting for file completion`` for an incomplete file,
  red ``Watch error`` with corrective detail for an unreadable source or worker
  failure, and gray ``Stopped`` or ``Not watching`` after stop/close or when
  liveness cannot be established.
- A direct Show2D, Show3D, or Show4DSTEM snapshot handoff created with
  ``watch=False`` has no watch badge. Restored notebook state, a static fallback,
  exported HTML, or another non-live ShowFolder snapshot must never show green
  ``Watching`` without an actually alive worker.
- Adding a file updates the already displayed browser in place.
- The status line reports cached versus newly read files.
- Existing stars/hidden panels are preserved when those files still exist.
- Removing a file removes it from the browser, selection, and next cache
  manifest so stale files are not retained in memory or cache state.
- New `*_master.h5` 4D-STEM files trigger the same watch path and refresh the
  active lazy Show4DSTEM handoff without preloading every master. This
  ShowFolder handoff may rebuild its owned viewer; it does not replace the
  same-widget guarantee required of direct ``Show4DSTEM.from_folder(...)``.
- When the all-image viewers are open (`open_show2d(all_images=True)` or
  `open_show3d(all_images=True)`), new readable image files append to the same
  Show2D/Show3D widget instance on the next poll.
- `Open All Both` and ``open_both(all_images=True)`` activate both all-image
  viewers through public paths. With both visible, background ``watch()`` and a
  genuine EMD arrival update the same two widget model IDs without private
  ``_active_selected_modes`` or ``_selected_*`` assignments.
- The browser test clicks the real control, copies an EMD after watching starts,
  and verifies both canvases repaint. Calling ``watch_once()`` and inspecting
  Python counts remains useful automation, but it is not the full user story.
- `watch_once()` performs one deterministic poll for tests and automation.
- `stop_watch()` stops the background watcher before kernel shutdown.

## SF-5: Select and Save

**User story**: A user reviews a session, marks useful files, and saves the
selection so analysis code can use the exact file list later.

**Primary widgets**: ShowFolder selection controls.

**Data to use**: A mixed folder with at least three images and one file that
should not be selected.

**Acceptance checks**:

- Users can select individual image files and folders from the rendered browser.
- `selected()`, `paths()`, `selected_paths()`, and `selected_folders()` return
  stable path information.
- `save(path)` writes a compact JSON selection.
- `load(path)` restores the selected files without rerunning expensive image
  loading.
- Saved selections use file paths and metadata, not large image buffers.

## SF-6: Open Selected Images Immediately

**User story**: A microscopist stars interesting images while browsing a folder
and wants to inspect them immediately without remembering a second API or
rewriting notebook boilerplate.

**Primary widgets**: ShowFolder selection panel, Show2D, Show3D.

**Data to use**: A real or real-derived microscopy folder with at least three
calibrated image files. Prefer files from an HPC/workstation or lab backend; do
not commit those raw files to the widget repository.

**Acceptance checks**:

- Starred images appear in the selection summary in acquisition order.
- `Open Show2D` renders a compact Show2D gallery below ShowFolder.
- `Open Show3D` renders the same starred images as a frame stack below
  ShowFolder.
- `Open Both` renders the starred images in both viewers, and `Open All Both`
  renders every readable image in both viewers through one normal user action.
- `Open All Show2D` and `Open All Show3D` render every readable image in the
  folder and live-append new files while the watcher is running.
- ``open_both(all_images=False)`` and ``open_both(all_images=True)`` expose the
  same selected/all behavior for notebook authors without requiring private
  mode manipulation.
- The explicit Python methods `show_selected()` and `show_selected_stack()`
  return the same widgets for notebook authors who prefer code.
- Scale bars and labels are preserved when all selected previews share a
  compatible calibration.
- The embedded viewers use cached thumbnail/preview data for immediate
  inspection and do not silently embed full-resolution raw acquisitions.
- ShowFolder keeps its maintained thumbnail cache as numeric arrays
  (`thumbnails.npz`) because selected Show2D/Show3D handoff needs array data.
  WebP thumbnails are only for human review pages and dashboards.
- ShowFolder owns watching for viewers opened from its cached preview and
  selection workflow. Direct ``Show2D.from_folder(...)`` and
  ``Show3D.from_folder(...)`` own independent full-resolution watchers and do
  not use ShowFolder thumbnails.

## SF-7: Notebook and Documentation Behavior

**User story**: A user can trust the docs and notebooks without accidentally
creating giant notebook files.

**Primary widgets**: ShowFolder, documentation examples.

**Data to use**: Documentation sample data and one real local session folder.

**Acceptance checks**:

- API docs explain the folder chooser, fallback path input, cache modes, and
  selection save/load workflow.
- Tutorial notebooks do not embed huge raw file buffers in saved widget state.
- Documentation examples use real or real-derived microscopy data whenever
  possible.
- Generated GitHub-preview notebooks are copied outside `docs/tutorials/`
  before static conversion.
- Agent signoff says whether browser testing used JupyterLab, exported HTML, or
  a static report.
- API docs point folder browsing to `ShowFolder`, not a separate `survey()`
  function.

## SF-8: Export a Shareable Folder Browser

**User story**: A user finishes browsing a folder and wants to send a lightweight
interactive browser to a collaborator without asking them to run a live kernel.

**Primary widgets**: ShowFolder, nested Show2D rows, HTML export.

**Data to use**: A real or real-derived microscopy folder with multiple grouped
rows and calibrated scale bars.

**Acceptance checks**:

- `ShowFolder(path).export_html("session-showfolder.html")` writes a standalone
  HTML file.
- Opening the exported file restores the nested Show2D rows, not only the outer
  container.
- The exported header identifies the folder, file count, thumbnail size, and
  cache status.
- The exported browser keeps thumbnails and interaction state compact; it does
  not embed full raw acquisition arrays.
- Exported HTML uses the same public export convention as Show2D and Show3D,
  with `export_html(path=None, title=None)` returning the written path.

## SF-9: Prebuild for High-Throughput Sessions

**User story**: A user has a large session on a workstation or SSD-backed
scratch path and wants the folder browser to open quickly during live analysis.

**Primary widgets**: `prebuild_showfolder_cache`, ShowFolder cache.

**Data to use**: A real or real-derived high-throughput session folder with many
microscopy files. Prefer 4K-class files when available; use smaller public data
only when the heavy data is unavailable.

**Acceptance checks**:

- `prebuild_showfolder_cache(path, cache_dir=...)` builds the same cache that
  `ShowFolder(path, cache_dir=...)` later reuses.
- A warm ShowFolder run reports cache hits and avoids re-reading unchanged image
  files.
- Cache validation uses relative path, file size, and modification time so
  edited or replaced files are refreshed.
- The notebook can point ShowFolder at a fast cache directory without moving the
  raw microscope data.
- The signoff reports cold and warm timing, file count, cache path, and whether
  the data was real, real-derived, or synthetic.

## Required Test Commands

Run these before committing ShowFolder changes:

```bash
PYTHONPATH=src pytest -q tests/test_showfolder.py tests/test_showfolder_core.py
PYTHONPATH=src:. python scripts/widget_showfolder_live_smoke.py --artifact-dir /tmp/quantem-widget-showfolder-live
```

The live smoke report must include WebP thumbnail previews and 4D-STEM master
QC rows. Those report images are visual evidence for humans; they are not the
cache used by ShowFolder widgets.

When a change affects docs or examples, also run the docs build used by the
current branch and open the rendered ShowFolder page in a browser.

## Signoff Report

Use this short report in commit notes, release notes, or agent handoff:

```text
ShowFolder signoff:
- Story IDs driven:
- Backend host:
- Browser/frontend:
- Folder path and file count:
- Cache mode/result:
- Selection save/load result:
- Tests run:
- Not verified:
```