Live folder watching

Live folder watching#

Open in Colab

During a microscope session, results land in a folder one file at a time. The viewers can watch that folder and grow in place - keep one widget open at the scope and let new acquisitions appear, instead of re-running a notebook after every file.

Where this earns its keep:

  • Survey acquisition - Show2D.from_folder adds a panel per new HAADF as it lands, so you pick the next field of view from what you already have.

  • In-situ / time series - Show3D.from_folder appends frames to a scrubbable stack while the experiment runs.

  • 4D-STEM sessions - Show4DSTEM.from_folder appends each ready *_master.h5 to the dataset slider, deferring files that cannot yet be read.

Tip

Run this exact notebook with the Colab badge above, or View or download this notebook on GitHub. For finished results, use HTML and file export to export interactive HTML or share a trusted notebook with widget state. A standalone HTML export is a snapshot; filesystem watching requires a live Python kernel.

from quantem.widget.datasets import showfolder_gold

folder = showfolder_gold(verbose=False)  # real gold HAADF session

Watch survey images with Show2D#

Show2D.from_folder reads every readable image at full resolution and starts background polling (watch=True is the default, watch_interval=1.0 seconds). Files added to the folder later become new panels in this same widget; partially-written files are retried until stable. Here it opens the real session’s overview images:

import pathlib
import shutil
import tempfile

from quantem.widget import Show2D

session = pathlib.Path(tempfile.mkdtemp(prefix="live_session_"))
for p in sorted(folder.glob("*overview 0[12]*")):
    shutil.copy(p, session / p.name)

viewer = Show2D.from_folder(session)
viewer

With the kernel running, drop another file into the folder and the widget adds the panel by itself within a poll interval:

shutil.copy(next(folder.glob("*overview 03*")), session)  # a new panel appears

On a live session you would point it at the acquisition folder and leave it open:

viewer = Show2D.from_folder("/data/session", pattern="*.emd")   # keeps growing
viewer = Show2D.from_folder("/data/session", watch=False)       # one-shot read

Watch a growing stack with Show3D#

For same-size frames (in-situ series, tilt series, a denoiser writing results), Show3D.from_folder plays the folder as one stack and appends new frames in place:

from quantem.widget import Show3D

stack = Show3D.from_folder("/data/growth_run", pattern="frame_*.tif")

Watch a 4D-STEM session with Show4DSTEM#

For live scope folders, construct the watcher from the folder itself. Each readable *_master.h5 joins the same Dataset slider as a lazy slot; selecting the new dataset performs the scientific load. Masters whose required linked data are not readable yet remain retryable:

from quantem.widget import Show4DSTEM

viewer = Show4DSTEM.from_folder(
    "/data/session",
    pattern="*_master.h5",
    watch_interval=2.0,
    det_bin=4,
)
viewer

See Show4DSTEM live scope folders for CUDA, CPU, Apple Silicon, paging, and precision choices.

From the command line#

The CLI writes a live ShowFolder notebook for image folders and a lazy Show4DSTEM notebook for master folders. Use the direct Python APIs above when the full-resolution Show2D/Show3D watcher itself is the workflow under test:

quantem show2d ./frames/ --watch     # ShowFolder + live all-image Show2D preview
quantem show3d ./frames/ --watch     # ShowFolder + live all-image Show3D preview
quantem show4dstem ./masters/        # lazy live-master Show4DSTEM