Save and share your data

Save and share your data#

QEM stores measurements and their scientific metadata together. It provides a common Python workflow across supported detector inputs, so a saved acquisition can be reopened without the original vendor files. It is an experimental open format; use the current source installation and record its Git revision.

Convert one HDF5 acquisition#

Keep the ARINA master beside all its detector files, then run:

quantem-gpu convert scan_master.h5 --out ./qem

This writes qem/scan.qem with QEM checksums and disk synchronization, and keeps the originals. Existing destinations are never overwritten. A folder can replace scan_master.h5 to convert a collection. Copies larger than the original HDF5 acquisition are reported and not written.

Full source comparison and source-file SHA-256 hashing are off by default. Request either check independently when needed, or both for validation:

quantem-gpu convert scan_master.h5 --out ./qem --verify --source-hashes

No session file is required. Supply any missing calibration directly:

quantem-gpu convert scan_master.h5 --out ./qem \
  --voltage-kv 300 --semiangle-mrad 30 \
  --scan-sampling-a 0.5 --detector-sampling-mrad 0.8

Use your measured values, not these examples. Scan sampling is in angstrom per pixel; detector sampling is in mrad per pixel. One sampling value applies to both axes; two values, such as --scan-sampling-a 0.5 0.6, specify row, column. Values must be positive and finite.

Explicit options override dataset.yaml or dataset.yml, which override the acquisition metadata. Only one of these session files may be present. The converter records the supplied values and their provenance inside QEM and prints them with their units. Missing calibration is reported as unknown, not guessed. The conversion reference describes the session schema and source metadata retained.

Repair detector-mask pixels#

Conversion is unfiltered by default. To repair only pixels identified by the detector’s stored mask:

quantem-gpu convert scan_master.h5 --out ./qem --hot-pixel-correction median

Median correction uses valid neighbors in a 3 × 3 detector neighborhood, excludes other flagged pixels, and rounds fractional medians down. With no valid neighbors the replacement is zero. This is not whole-image smoothing or automatic outlier detection. zero replaces flagged pixels with zero; none requests no correction.

The output reports the number and (row, column) coordinates of repaired pixels, their original flags, and verification of the replacements when requested. QEM records this processing history. Its encoding is lossless after the declared correction; the corrected values are not unchanged raw measurements. Reopening does not apply the correction again. Reconstruction settings such as defocus guesses, iteration counts, or an assumed scan-detector rotation belong in a separate recipe.

Without correction, the original dtype and every measurement bit are preserved, including flagged uint32 markers. Median and zero correction are opt-in and supported for uint8/uint16/uint32 acquisitions.

The same converter from Python#

from quantem.gpu.io.convert import convert

result = convert(
    "scan_master.h5", "scan.qem",
    voltage_kV=300, semiangle_mrad=30,
    scan_sampling_A=0.5, detector_sampling_mrad=0.8,
    hot_pixel_correction="median",
)
result.verified

The last expression returns None when full comparison was not requested. Pass verify=True to compare every saved value, and source_hashes=True to record SHA-256 fingerprints of every original file. A successful requested comparison returns True. Both options default to False. Inspect result.skipped or result.larger when no file was written. The CLI calls this same function; no dataset-specific conversion script is needed. Use --dry for a GPU payload-size estimate without writing, and --backend cuda or --backend mps only when selecting a backend deliberately.

If an editable source environment has not installed the console entry point, use python -m quantem.gpu.cli convert ... with the same arguments.

Save a loaded acquisition#

from quantem.gpu import io

data = io.load("gold_master.h5")
io.save("gold.qem", data)

Keep data open while using it, then call data.close(). Saving creates a new file and refuses to overwrite an existing destination. No codec configuration is needed: supported GPU acquisitions use ANS storage, and saving an encoded acquisition retains that storage without expanding the full array.

What is in the file?#

Content	Why it matters
Detector measurements, shape, and dtype	Reopen the stored values and their array geometry
Sampling and units	Interpret scan and detector coordinates physically
Microscope fields retained by the reader	Keep voltage, angles, and other available acquisition context
Source metadata and processing history	Know where values came from and which corrections were applied
Integrity checks	Detect damaged or incomplete stored data

Lossless storage preserves the values being saved. Loading defaults to hot_pixel_correction="median": each flagged detector pixel holds the median of its valid neighbors, and the record says so; every other value keeps its original bits and dtype. With hot_pixel_correction="none" the original measurements and detector flags, uint32 flag markers included, survive encoding and saving, and scientific detector products exclude invalid pixels using the separate validity mask. Saving cannot undo acquisition-time processing or an explicitly requested replacement. Missing calibration stays unknown. Retaining vendor tags does not mean that every tag has been interpreted or that every proprietary object is archived.

QEM is not an HDF5 container. Open it with QuantEM rather than h5py.

Reopen and inspect#

from quantem.gpu import detector

saved = io.load("gold.qem")
bf = detector.bf(saved)
saved.shape, saved.dtype, saved.sampling, saved.units
saved.metadata

The same indexing and detector calls work after reopening. saved[10, 12] returns a GPU Torch tensor; bf is a reduced NumPy image. Close saved when finished. Read sampling together with its unit; do not assume an unlabelled number is in angstrom, nm, or mrad.

The Python GPU path retains signed and unsigned 8/16/32/64-bit integers and float16/32/64 bits, including signed zero and NaN payloads. Storage support does not imply identical calculation precision on every GPU. Integer detector sums use checked 64-bit accumulation; a sum that cannot fit raises instead of wrapping. Torch MPS does not support float64 tensors. For exact float64 reads on Metal:

pattern = saved.read(scan_region=(10, 11, 12, 13), output="native")
try:
    original_pattern = pattern.get()  # one exact pattern, copied to host
finally:
    pattern.release()  # Metal result ownership ends here

CUDA native reads normally return CuPy arrays. FP8 uses an owned byte wrapper with dtype, shape, get(), to_torch(), and release(), because CuPy has no corresponding numeric dtype. Default reads return typed Torch tensors on either accelerator. See original FP8 input. Metal display and SSB products round to float32 on the GPU without modifying the saved float64 measurements.

Review calibration once#

data.calibration shows voltage, convergence angle, scan sampling and detector sampling, with units and Recorded / Overridden / Missing status. Set a value by name; a single number for a sampling applies to both (row, col) axes. The notebook summary and io.report(path) expose these quantities without decoding counts. Unknown calibration does not prevent raw inspection or saving.

data.calibration
data.calibration.semiangle_mrad = 30
data.calibration.scan_sampling_A = (0.4, 0.4)
io.save("calibrated.qem", data)

Use measured values for your acquisition. Values you set are kept beside the recorded metadata and survive save, move, reopen and resave. Nothing saves automatically. Recreate an existing reconstruction session after editing its calibration. SSB needs voltage and scan sampling, plus detector sampling or a calibrated convergence angle from which the disk radius can determine detector sampling. It cannot determine an absolute angular scale from the disk radius alone.

For a header-only table without a GPU, use io.report("gold.qem"). For a stored-file integrity check:

python -m quantem.gpu.formats.qem.validation gold.qem

Integrity checks detect storage errors; they do not establish the physical accuracy of a calibration or the correctness of the original measurements.

Continue to DPC and SSB#

The main Gold workflow puts loading, DPC and SSB on one page. Saving as QEM is optional; those operations also accept the original loaded acquisition.

The advanced Gold notebook demonstrates saving the calibration, reopening the file, and showing a selected diffraction pattern before saving and after reopening. It then inspects the mean pattern, fitted aberrations and model probe, and compares native and 4× phase.

Which files can I convert?#

Use io.load(source_path) followed by io.save("copy.qem", data). Supported layouts include qualified HDF5, NCEM EMD, DM3/DM4, NumPy arrays, and specified EMPAD float exports. Support depends on dtype and geometry; see the I/O input table before converting a new source. DM3/DM4 need the dm extra. Raw EMPAD2 sensor words are different from calibrated float exports. Python MPS/CUDA support does not imply that every native application build can open the same geometry or codec.