QEM container and scientific metadata

QEM container and scientific metadata#

This is the implementation specification for container version 1. For ordinary Python use, follow Save and share your data.

Read the scientific contract, binary envelope, and metadata requirements when building a reader or writer. The codec layouts define payload bytes; metadata mapping defines vendor-field handling; conformance checks verify independent implementations. Specification revisions below describe additional requirements; they are not user-selectable compression settings.

QEM specification 0.0.4#

Specifies the flat calibration keys and the retained metadata block. The stored scientific_metadata is unchanged: its schema string stays quantem.scientific-metadata/2 and every existing schema-2 file reads with the same calibration.

Flat calibration keys#

The flat keys are the reading and writing interface of the format. A reader returns them beside the data (data.metadata in Python, NativeANSSnapshot.metadata in Swift); a writer accepts them for a new copy. Each key has one unit, stated in its name, and one fixed place in scientific_metadata, which remains the stored authority.

Flat key	Unit	Type	Stored at
`voltage_kV`	kV	number	`electron_microscope["electron_source/accelerating_voltage"]`
`semiangle_mrad`	mrad	number	`electron_microscope["illumination_system/semi_convergence_angle"]`
`scan_sampling_A`	angstrom per scan position	`[row, col]`	`axes[0].sampling`, `axes[1].sampling` (and the two `scan_controller/regular_scan/pixel_size_*` quantities)
`detector_sampling_mrad`	mrad per detector pixel	`[row, col]`	`axes[2].sampling`, `axes[3].sampling` with unit `mrad`
`detector_sampling_inv_A`	1/angstrom per detector pixel, cycles	`[row, col]`	`axes[2].sampling`, `axes[3].sampling` with unit `1/angstrom`
`dwell_time_us`	us	number	`electron_microscope["scan_controller/regular_scan/dwell_time"]`
`camera_length_mm`	mm	number	`electron_microscope["imaging_system/camera_length"]`

A pair is always [row, col], never one number for both axes. “Per pixel” means per stored pixel of the dataset the key describes.
Exactly one of the two detector keys is present, and neither when the detector is uncalibrated. The flat layer does not convert between them: that needs the voltage. A writer given both refuses the copy.
An absent key means the quantity is unknown.
Reading: every flat key is derived from scientific_metadata, the recorded quantity replaced by its calibration_overrides entry when one is saved. A reader never takes calibration from the retained metadata block. Files written before this revision hold copies there (scan_sampling_A, voltage_kV, detector_sampling with detector_sampling_unit); they stay readable and the copies are not read. One of those values still says something no other field does: a retained origin without units is in the unit the detector step was supplied in, and detector_sampling_unit names it (rad, 1/nm). A reader that returns origin scales it by the factor that unit has to the stored one (1000 for rad, 0.1 for 1/nm), the factor applied to the step when it was recorded.
Writing: every flat key is accepted for a quantity the source did not record. The unit-less pair detector_sampling with detector_sampling_unit is refused with a message that names the two detector keys.
For an HDF5 master the dwell time is count_time when the master holds one, else frame_time, whatever the detector. A loader reports the same value the saved copy records.

The scan-to-detector rotation and whether it is flipped are not stored in a .qem file. They are flat keys of result and configuration records (scan_detector_rotation_deg, scan_detector_rotation_flipped).

Retained `metadata` block#

The header’s metadata block keeps the provenance a reader needs and nothing that scientific_metadata already states.

Key	Meaning
`source_path`, `original_source_path`	path the data was loaded from; the path before conversion
`source_files[]` (`name`, `bytes`, `sha256`), `source_master_file` (`name`, `encoding`, `data`)	every source file of a converted acquisition; the master file itself, zlib then base64
`source_dtype`, `dtype`, `exact_float_narrowing`, `file_counts_exact`, `working_counts_exact`	evidence for a declared dtype change
`hot_pixel_correction`	what was done to flagged detector pixels, see below
`flagged_markers_stored_as_zero`	legacy count of uint32 flag markers stored as 0; new raw-preserving writers leave markers intact

Writers do not store:

the flat calibration keys (they are derived on reading);
second copies of source_metadata, source_documents, source_metadata_coverage, sample and calibration_overrides, which a new copy supplies as inputs and which are stored once, in scientific_metadata;
what a loader knows about its host and the resident it built (shapes, byte counts, backend, device, binning, timings, the raw entry/... source paths).

Other keys a caller adds are kept unchanged. Readers apply the same rule to older files, so a reopened copy never returns a stale stored value for one of these keys.

source_dtype, file_counts_exact and working_counts_exact describe the original acquisition: the dtype it was recorded in, and whether the stored counts are its counts. They are not statements about a later load. A reader returns them as the header states them and adds none where the header has none, so a copy of a file stores the same values as the file. (A file whose uint32 source was stored as uint16 keeps source_dtype: "uint32"; a file whose flagged pixels were replaced keeps file_counts_exact: false.) For the scaled codec the stored values are codes, never counts: its readers report file_counts_exact: false whatever the block holds.

hot_pixel_correction is a record with schema ("quantem.hot-pixel-correction/3"), method (median, zero, none), largest_window_px, source, pixel_count, coordinates_row_col, source_flags, applied, stage and backend. A median fill takes the valid pixels of the 3 x 3 window around a flagged pixel, or of the smallest larger window (5 x 5, 7 x 7, …) that holds one; largest_window_px is the largest side used. An even number of neighbors averages the two middle values, rounded half to even. A record with another schema, or none, is read as absent; processing still declares a replacement.

Schema 1 is refused#

Readers accept quantem.scientific-metadata/2 only. A file with schema 1 is refused with a message that names the schema; export it again from its source. No reader converts schema-1 units.

QEM specification 0.0.3#

Adds an optional sample group to scientific_metadata: the specimen as a person declared it (for example a session’s dataset.yaml at conversion). The schema string stays quantem.scientific-metadata/2; readers that do not know the group keep it unchanged on re-save and ignore it otherwise. Nothing derived and nothing defaulted is written into it.

"sample": {
  "provenance": "dataset.yaml",
  "evidence": "dataset.yaml sha256:..., files[38]",
  "id": "film-STO-01",
  "name": "BaTiO3 film on SrTiO3",
  "geometry": "cross-section",
  "growth_direction": [0, 0, 1],
  "orientation_relationship": "(001)[100] film || (001)[100] STO",
  "components": {
    "film": {
      "role": "film",
      "chemical_formula": "BaTiO3",
      "zone_axis": [0, 0, 1],
      "cif": {"document": "BaTiO3.cif.json", "sha256": "..."},
      "thickness_estimates": [
        {"method": "ptychography_multislice", "value": 410, "unit": "angstrom", "uncertainty": 40,
         "region": {"rows": [320, 512], "cols": [128, 384]}, "reference": "...", "date": "2026-09-26", "preferred": true}
      ]
    }
  },
  "components_in_view": ["film"]
}

provenance and evidence cover the whole group: who declared it and in which file.
geometry is cross-section (component thicknesses side by side) or plan-view (stacked along the beam).
zone_axis and growth_direction are integer [u, v, w] in the CIF’s cell (or hexagonal [u, v, t, w]).
role is film, substrate, support or particle. Component labels are free text.
cif names a JSON document in source_documents[] holding the CIF text ({"cif": "..."}) and its sha256.
A thickness is always an estimate: thickness_estimates lists each with method (diffraction_ridge, pacbed_fit, ssb_depth, ptychography_multislice, cross_section, nominal), value in angstrom, optional uncertainty or range, region (a component label, {rows, cols} or {point} in scan positions, or absent for the whole scan), reference, date, and at most one preferred.
Python validates the group (validate_sample); the Swift reader keeps it unchanged and does not interpret it yet.

QEM specification 0.0.2#

Changes from 0.0.1. There is no compatibility period: 0.0.2 readers refuse the 0.0.1 forms named here, and the file must be exported again from its source.

Calibration quantities are named by array axis: scan_controller/regular_scan/pixel_size_row and pixel_size_column, imaging_system/reciprocal_pixel_size_row and reciprocal_pixel_size_column. The 0.0.1 names ending in _y and _x are refused in electron_microscope and in calibration_overrides. Source files keep their own x and y names inside source_metadata; a reader maps source y to row and source x to column.
processing is required. It lists every operation between the source measurements and the stored measurements. Each record has an operation name and a boolean changes_measurements.
Two operations are defined in addition to lossless_storage: exact_integer_narrowing (changes_measurements: false, with source_dtype and stored_dtype) and flagged_pixel_replacement (changes_measurements: true, with method and pixel_count).

Validate a file#

quantem-gpu validate acquisition.qem another.qem

The command needs no GPU. For each file it checks the envelope, the header and every body checksum, the codec layout, and the metadata rules of this specification, then prints one JSON report. The report lists processing and, separately, measurements_changed_by, so a reader sees at once whether the stored counts differ from the source. A file whose retained reader record shows replaced detector pixels or a narrower stored dtype, without the matching processing record, fails validation. The exit status is nonzero when any file fails. tests/data/qem-v2/invalid-metadata.json lists header mutations that every conforming reader must refuse; the Python and Swift readers share them.

Microscopy-friendly metadata (since 0.0.1)#

Writers use quantem.scientific-metadata/2. The binary container and measurement codecs remain unchanged. Since 0.0.4 readers accept schema 2 only and refuse schema 1 rather than reinterpret units.

Schema 2 stores normalized quantities and calibration overrides in these units:

Quantity	JSON unit	Typical display
Scan-axis sampling	`angstrom`	Å
Reciprocal-length detector sampling	`1/angstrom`	Å⁻¹
Angular detector sampling and convergence semi-angle	`mrad`	mrad
Accelerating voltage	`kV`	kV
Beam energy, when explicitly supplied	`keV`	keV
Scan dwell time	`us`	µs
Camera length	`mm`	mm
Sample thickness estimate (0.0.3)	`angstrom`	nm

For example, {"value": 0.42, "unit": "angstrom"} means 0.42 Å per scan step. Applications may display another convenient unit, but must read the unit, not infer it from a field name. Exposure, dwell and frame interval are not synonyms; angular and reciprocal-length sampling are not interchangeable. Original source_metadata, evidence and provenance are preserved. Codec-private restoration metadata and internal calculation APIs keep their existing units; the public scientific metadata is the authoritative human-readable unit layer. Measurement bytes remain exact. Calibration conversion uses floating-point arithmetic without display-style rounding; physical equivalence is tested to a relative tolerance of 1e-14. For identified ARINA sources, count_time takes precedence over frame_time when recording the scan timing. The original fields remain available separately; generic NXmx photon energy is not inferred to be an electron accelerating voltage.

Open an acquisition, preserve its scientific measurements and calibration, save one portable file, then restore the same measurements without the source folder. The extension names quantitative electron microscopy, not dimensionality or a compression algorithm. This is a project format, not a claim of an industry standard or formal NeXus compliance.

See the field-by-field metadata map and the portable references and validation workflow.

Calibration ownership and independent readers#

For schema 2, scientific_metadata is the sole calibration authority. Recorded axis sampling owns the scan/detector step; duplicate microscope step fields must agree within 1e-14 relative tolerance. Explicit calibration_overrides take precedence for calculations, never for rewriting the recorded values. Private restoration fields are derived conveniences, not a second authority. Missing public calibration means unknown; a reader must not revive a stale private value. Reciprocal lengths use cycles per distance, not radians per distance: there is no implicit factor of 2π. Angular sampling in mrad is a distinct quantity. Known calibrated quantities require nonempty provenance. Source coverage is explicit; unknown vendor fields may be retained but must not be guessed.

The reference CPU path is explicit (backend="cpu"), not a silent replacement for an accelerated backend. It preserves exact measurements and is intended for portability, small examples and independent verification, not speed claims.

Scientific contract#

No implicit crop, binning, masking, background subtraction, integer narrowing or float quantization is permitted. Integer counts and floating-point bit patterns must round-trip exactly. Current Python GPU writers preserve the source dtype through the typed bit-lane profile; no range audit or narrowing is needed. Older files may declare the following processing operations. They are permitted only when declared in processing. A writer may store uint32 source counts as uint16 after checking every stored value, refusing the file when any count exceeds 65535 (exact_integer_narrowing). Masked pixels are not an exception to this range check. The record requires verified exact-count evidence from the reader, not just a difference between source and stored dtypes. A writer may replace detector pixels flagged in the source pixel mask, for example with the median of their valid neighbors (flagged_pixel_replacement); all other pixels stay exact, and the record states that measurements changed. A detector pixel marked invalid in the header valid mask is excluded from analysis, but its stored source value must still round-trip exactly unless replacement is explicitly declared in processing. The mask never permits an undeclared change to stored values. Axes are ordered, named and sized; row precedes column. Each calibrated axis has an explicit value, unit and provenance. An absent quantity is unknown, not zero. Detector sensor pitch is not specimen scan step.

The metadata vocabulary follows the microscope companion files used with ARINA: electron_source/accelerating_voltage, illumination_system/semi_convergence_angle, scan_controller/regular_scan/pixel_size_row and pixel_size_column, scan_controller/regular_scan/dwell_time, and imaging_system/camera_length, all below electron_microscope. Source x maps to column and y to row. Other detectors map to these scientific quantities without changing their recorded detector identity. Angular and reciprocal-length sampling are distinct.

A converter that walks the complete source master declares source_metadata_coverage: "exhaustive". It stores every field under its HDF5 path and every attribute as path@attribute, so a unit sits beside its value (.../count_time and .../count_time@units). Arrays longer than 64 values are named with their dtype and shape, and travel complete in metadata.source_master_file, the master file itself as zlib then base64. metadata.source_files records the name, size and SHA-256 of every source file.

Keep interpreted quantities separate from source_metadata, which retains the original reader-provided fields and units. source_metadata_coverage must say whether this is a reader-retained subset or an exhaustive archive. A bounded display collector must never claim complete original-file preservation. Recorded calibration and explicit overrides are separate; overrides carry provenance. Compression alone never claims that a correction was applied.

Binary envelope#

All integer fields are unsigned little-endian. The first 56 bytes contain:

Offset	Bytes	Meaning
0	8	ASCII `QEMDATA1`
8	8	UTF-8 JSON header length
16	8	Body start, exactly 56 + header length
24	32	SHA-256 of the JSON header

The JSON header is at most 16 MiB. It declares container = "quantem.qem", container_version = 1, codec, and scientific_metadata. Codec versions are independent of the container version. Unknown versions or codecs fail with an actionable error; they must never be guessed from the filename. Body checksums cover consecutive 64 MiB blocks including alignment padding. Complete byte coverage and bounds must be validated before GPU consumption.

The first implemented codec is runtime-column-rans-spatial-v2, retaining the existing exact uint8/uint16 stream arrays and spatial indexes. Its existing profile, version, interval, shape, dtype, valid, chunks, bytes, sha256, and metadata fields retain their meanings. The scientific metadata schema is dimension-independent; this first codec remains 4D-only. Naming the container generically does not imply that every dimensionality or dtype already has an implemented decoder.

Implementation status#

Publishing a destination is atomic and must not overwrite an existing file.

Native Swift and Python share the envelope and stream codec. Hardware parity, source-format qualification, floating-point codecs and actual application UI coverage must be reported separately from metadata-schema support. WebGPU and arbitrary 3D/5D codecs are not implied by this contract.

Native EMPAD floating-point codec#

float32-bit-lanes-rans-v1 preserves float32 bit patterns in two encoded uint16 lanes per detector pixel, including signed zero, NaN payloads, and infinities. Detector geometry is stored explicitly, not inferred as 128×128. The former empad-xor-row-packed-v1 acquisition codec is retired: re-export original data. See the normative codec specification.

empad stores the original format identity, reader-retained microscope metadata, scan calibration and supplier correction statement. An explicitly selected dark reference saves its detector-shaped mean float32 plane and calibration identity in the checksummed header. The original sample values remain encoded unchanged; the restored plane is subtracted once for display and products. Changing this saved recipe currently requires reopening the original and exporting another copy. Encoded EMPAD2 sensor words still require matching gain/dark calibration and are not accepted as float32 merely because an XML label says float32.

Supported entry points#

Python: io.save("copy.qem", encoded_resident, backend="mps"), then io.load("copy.qem", backend="mps"). The shared integer codec also has a CUDA reader, but CUDA execution must be qualified independently. Explicit backend="cpu" reads both integer and EMPAD float32 codecs as original dense measurements, and writes supported NumPy arrays. Python GPU float32 ANS-QEM reading is implemented; physical backend qualification remains separate. See the Python workflow and normative codec specification. Normalized quantities are available as acquisition.metadata["scientific_metadata"].
Native Swift: NativeNPYSource(url:) followed by MetalRuntimeANSResidentSource.load(array:device:) and saveSnapshot(to:) converts C-order little-endian uint8/uint16 NumPy .npy arrays. The four axes must be (scan_row, scan_column, detector_row, detector_column). NativeCountArray lets DM4 and NumPy reuse the same bounded Metal encoder. The NumPy header is retained, but microscope sampling is unknown, not guessed. Floating-point, signed, Fortran-order, zipped and non-4D arrays are rejected with corrective guidance. This native entry point does not imply automatic .npy dispatch in the Python API. For native float32 NumPy and .qem, use NativeEMPADSource.open followed by MetalEMPADResidentSource.load and saveQEM(to:). Read actual dimensions from source.detectorShape; do not use the static EMPAD sensor dimensions for an arbitrary array. Application dispatch and rendering need their own integration tests. Run bash scripts/check_npy_qem_roundtrip.sh counts.npy on Metal to verify every diffraction pattern and three detector masks against the input counts, plus metadata preservation. Add --reject=unsupported.npy for negative cases.
Shared Metal conversion: MetalQEMExporter.save(_:to:device:) accepts validated count-array, indexed HDF5, or EMPAD readers. It owns bounded conversion and atomic output; the calling app owns scheduling, progress presentation, and explicit dark-reference/already-corrected choices. It has no dependency on app preferences, dialogs, or a particular viewer.
Regional scaled-uint16 results (for example merged tilts) use the scaled-uint16-column-rans-v1 codec: native Swift MetalPackedSource.saveQEM(to:scientificMetadata:) and MetalPackedSource.loadQEM(url:device:) keep the resident codes and calibration unchanged. Python io.load(path) reopens them on Apple GPUs as encoded float32 data (backend="cpu" is a dense NumPy reference) and quantem-gpu validate checks them. Python CUDA/MPS also support the typed bit-lane profile for real integer and floating arrays through 64 bits. That profile is not yet implemented by native Swift or WebGPU. Unsupported exports fail rather than writing another container under a .qem filename.

scientific_metadata.calibration_overrides stores explicit user edits separately from recorded quantities and axis sampling. Each entry uses the same microscope path with value, unit, provenance, and nonempty evidence. provenance is "user_override" for a value set by hand, or "bright_field_disk_fit" for a detector sampling data.calibration.fit_detector_sampling() fitted on the bright-field disk (detector sampling only). That fit also keeps its disk in the retained metadata block as bright_field_disk: schema "quantem.bright-field-disk/1", method, bright_field_center_px (row, col) and bright_field_edge_radius_px, in detector pixels. A record of another schema is ignored. In schema 2, scan sampling uses angstrom; beam voltage uses kV; dwell time uses us; camera length uses mm; semi-angle uses mrad. Detector row/column sampling uses a shared unit of mrad or 1/angstrom. Both members of each sampling pair are required. Invalid or incomplete calibration is rejected, not guessed.

The native calibrationOverrides API retains its existing calculation units (m, V, s, m, mrad, and detector mrad/1/nm/1/Å). Conversion happens only at the scientific-metadata boundary. Native exporters accept calibrationOverrides. Omitting it preserves existing saved edits; supplying a complete dictionary replaces them in the new copy; supplying [:] clears them. NativeQEMCalibration validates and reads these quantities. Native readers expose the serialized dictionary under qem_calibration_overrides; applications apply it ahead of recorded calibration. Only .qem carries calibration overrides; no other destination is accepted. Python readers expose the effective calibration as the flat keys of 0.0.4 in acquisition metadata while retaining the complete scientific metadata, including original values and every override. CUDA execution still requires independent hardware qualification.

Reader-retained source metadata is preserved, not a full archive of every HDF5 object or vendor binary tag. Do not describe this as complete original-file preservation. This populates the existing optional field in container version 1; the binary envelope and codec are unchanged.

Verification gates#

Original -> saved -> reopened: exact shape, dtype, bad-pixel mask and retained source metadata; calibrated axes and microscope quantities unchanged.
Real independent source frames and detector reductions, not merely two reads through the same decoder. Distinguish sampled checks from exhaustive checks.
Moved-file reopen without relying on original filesystem paths.
Explicit rejection of truncated files, unknown codecs and altered checksums.
Performance measured separately: save, checksum/read, resident restoration, first native presentation and interaction; include device and cache state.

Run bash scripts/check_qem_roundtrip.sh original new-copy.qem for a release-mode native check. The test compares seven selected DP frames, calibration and all reader-retained fields. Integer BF/ADF/DF images are compared in full before and after saving, with sampled independent host reductions; EMPAD reductions are checked against host float64 sums. A generated fixture also checks saved dark subtraction, negative values and rejection of corrupted headers/bodies. scripts/check_qem_collection.sh visits supported acquisition entry points in a local testing collection, removes only its own temporary copies, and reports rejections separately in its output. It never changes the original acquisitions.

Run bash scripts/check_qem_calibration_roundtrip.sh counts-uint16.npy new-copy.qem to compare every original DP count, assert microscopy-friendly saved units, restore overrides without local preferences, and check resident-save preservation and an explicitly uncalibrated copy of the original. The command also creates new-copy.preserved.qem and new-copy.reset.qem; all destinations must be new.