load

load#

Reads compressed 4D-STEM data straight onto the GPU (CUDA / Apple Metal) or CPU and returns a LoadResult you hand to Show4DSTEM. Public import:

from quantem.widget import load

Reference#

quantem.widget.io.hdf5.load(filepath, *args, dtype: str | None = None, gpus=None, stack: bool = True, max_concurrent=None, scan_region=None, **kwargs)#

Load 4D-STEM data — one master, or many.

  • load(master) → one LoadResult.

  • load(master, scan_region=(r0, r1, c0, c1)) → one cropped LoadResult without loading the full scan first.

  • load([masters]) → the masters stacked into one 5D dataset (the series/viewer case).

  • load([masters], gpus=[0, 1]) (or stack=False) → a list of separate LoadResult, read in parallel across disks and placed across GPUs — the joint-reconstruction path (gpus: None current device / int all-that-GPU / list per-master round-robin). Decode is serial (concurrent in-process CUDA decode corrupts the device); reads overlap across disks so bandwidth adds.

Also recommends / applies the smallest lossless browse dtype: dtype=None prints the recommendation; dtype='u8' clips@255 + casts; dtype='auto' picks uint8 only if lossless.

Tip

det_bin=2 (or 4) bins the detector on load to cut memory and speed first paint; pass a list of file paths to stack several datasets behind a single “Dataset” slider.

Backend (CUDA / Apple Silicon / CPU)#

load detects the device automatically: an NVIDIA box loads onto CUDA, a Mac loads onto Apple Metal (MPS), and anything else falls back to CPU. No flag is required - the same call works everywhere:

from quantem.widget import load, Show4DSTEM

data = load("scan_master.h5")   # CUDA on a workstation, MPS on a MacBook
Show4DSTEM(data)

On a MacBook the read uses a zero-copy Metal path, so a laptop can browse 4D-STEM data that does not fit in RAM if you bin at load: det_bin reduces the detector on the way in with integer-sum binning, so the full multi-gigabyte stack never has to materialize. A typical laptop session:

data = load("scan_master.h5", det_bin=8)   # MPS, detector binned 8x -> small
Show4DSTEM(data)

MPS loads include a preflight memory guard. Before allocating Metal buffers, the loader estimates the output footprint from HDF5 metadata and compares it to the Mac’s recommended Metal working set. No-bin loads that exceed that budget fail early with a specific det_bin recommendation instead of risking a frozen laptop:

data = load("scan_master.h5", backend="mps", det_bin=4)

Bypass the check only when you have a clean memory budget and need the exact no-bin stack:

data = load("scan_master.h5", backend="mps", skip_mps_memory_check=True)

For fastest browse workflows, combine detector binning with compact dtype:

data = load("scan_master.h5", backend="mps", det_bin=8, dtype="u8")
Show4DSTEM(data)

That path is intended for screening, layout, and export-to-browser review. Use uint16/full precision when detector counts are part of the scientific claim and the GPU memory budget is clean.

The same is one shell command - see the CLI: quantem show4dstem scan_master.h5 --bin 8.

Multi-File / 5D Browsing#

Use one public entry point for a time-series or tilt-series stack:

from quantem.widget import load, Show4DSTEM

masters = [
    "/data/session/file_001_master.h5",
    "/data/session/file_002_master.h5",
    "/data/session/file_003_master.h5",
]

r = load(masters, det_bin=4, dtype="u8")
w = Show4DSTEM(r)

The returned data has shape (n_files, scan_row, scan_col, det_row, det_col). Show4DSTEM labels the extra axis as Dataset and uses the source filenames as slider labels when they are available.

dtype="u8" is a browse contract, not a reconstruction contract. It routes to the direct uint8 output path before stacking or sharding, so the loader avoids materializing a full uint16 stack first. Counts above 255 clip to 255; use dtype="u16" or omit dtype when detector counts are part of the scientific claim.

For larger series or solver workflows that should stay sharded across GPUs, use:

parts = load(masters, det_bin=2, gpus=[0, 1], stack=False)

For browse-scale multi-GPU sessions, use the same public dtype vocabulary:

parts = load(masters, det_bin=1, dtype="u8", devices=[0, 1])

The sharded result keeps one stack per GPU and records the file-to-device map in the metadata. It is disk-aware: when masters are split across independent NVMe mounts, the loader interleaves files by physical disk before assigning them to GPUs. When all files live on one disk, sharding still increases GPU capacity, but cold load remains disk-bound.

For an explicit logical series object:

series = load(
    masters,
    det_bin=4,
    dtype="u8",
    series_type="time",
    series=[0.0, 1.0, 2.0],
    units=["s", "pixels", "pixels", "mrad", "mrad"],
)

Memory rule of thumb for Sample-scale 512x512x192x192 data:

mode

approximate resident size per file

no bin, uint16

18-20 GiB

det_bin=2, uint16

4.5-5 GiB

det_bin=4, uint16

1.1-1.3 GiB

det_bin=4, dtype="u8"

about half of uint16

Use det_bin=4, dtype="u8" for first-pass browsing. Use uint16/no-bin only when exact diffraction intensities matter and the GPU memory budget is clean.

Scan-region loading for ROI workflows#

Use load(..., scan_region=...) when a reconstruction or denoise workflow needs only a rectangular scan patch, not the full scan plane. This is different from loading the full frame and slicing afterward: the loader reads only the selected HDF5 detector-frame chunks, decompresses them on CUDA, and returns a local patch.

from quantem.widget import load

patch = load(
    "scan_master.h5",
    scan_region=(160, 293, 234, 367),  # row_start, row_stop, col_start, col_stop
).data

print(patch.shape)
# (133, 133, 192, 192)

The returned LoadResult.data shape is (region_rows, region_cols, detector_rows, detector_cols). Metadata records both the original scan grid and the loaded patch:

result = load("scan_master.h5", scan_region=(160, 293, 234, 367))
print(result.metadata["full_scan_shape"])  # e.g. (512, 512)
print(result.metadata["scan_region"])

For drift-corrected time-series work, compute the source scan box from the shared specimen ROI plus a small halo, load that patch, then apply your existing subpixel sampler in local patch coordinates. Do not save the sampled patch as a new raw acquisition; drift is scan-position metadata, and detector counts stay physically unchanged.

Measured on a native-detector Pari 5D-STEM ROI loader timing check (10 x 128 x 128 x 192 x 192, CUDA, no detector binning):

Path

Loader wall time

Max loaded CuPy buffer

load() full frame, then crop

9.66 s

18.0 GiB

load(..., scan_region=...) patch, then crop

2.44 s

1.215 GiB

The patch path is CUDA-only today and targets chunked 4D-STEM masters with one detector frame per HDF5 chunk. Use load() for full-field browsing and for Apple Metal/MPS until the region loader is ported there.

Region-loader compatibility reference#

load_scan_region() remains available for existing code, but new examples should use load(..., scan_region=...).

quantem.widget.io.hdf5.load_scan_region(filepath: str, scan_region: tuple[int, int, int, int] | list[int], *, backend: str = 'cuda', scan_shape: tuple[int, int] | None = None, det_bin: int = 1, apply_mask: bool = True, verbose: bool = True, auto_narrow: bool = True, output_dtype: type | dtype | None = None) → LoadResult#

Load only a rectangular scan region from a raw HDF5 master.

Parameters:

  • filepath – Dectris/Arina master HDF5 file.

  • scan_region – (row_start, row_stop, col_start, col_stop) in the full scan grid.

  • scan_shape – Full acquisition scan shape. When omitted, it is derived from metadata.

  • det_bin – Optional detector binning factor. The scan region is never binned.

Returns:

data is a backend array with shape (region_rows, region_cols, det_rows, det_cols). Metadata keeps the full acquisition grid in full_scan_shape and the loaded patch in scan_region.

Return type:

LoadResult