QEM metadata mapping and completeness

This is the implemented container-v1, scientific-metadata-schema-2 contract, documented as QEM specification 0.0.4, not a claim that every vendor field is understood. The format specification owns the binary envelope. The format is independent of the compression algorithm and app version.

Status vocabulary

• Preserved: reader-retained source fields are carried without replacing them.

• Normalized: a known quantity is mapped into a common path with explicit units.

• Missing: no valid recorded quantity exists; omit it, never invent a zero.

• Unsupported: the current reader cannot interpret or export that representation.

Normalization and preservation are complementary. source_metadata_coverage is reader-retained, not exhaustive. Keep original acquisitions for archival use.

Field-by-field map

Paths below are relative to scientific_metadata. Microscope paths are relative to its electron_microscope object. Source y means row and x means column.

Quantity	Common destination	Input recognized by native reader	Saved unit/status
Scan/detector shape	axes[].name, axes[].size	Validated source dimensions; DM4 axes reversed into scan row/column, detector row/column	Preserved, four named axes
Measurement representation	header dtype, shape	Python CUDA/MPS: real integer and floating dtypes through 64 bits; native applications support their declared codec set	Preserved; no implicit narrowing
Scan row/column step	axes[0:2].sampling, scan_controller/regular_scan/pixel_size_row and pixel_size_column	ARINA/NCEM scan calibration; DM4 ImageData.Calibrations.Dimension.*.Scale/Units; EMPAD reader calibration	Normalized to angstrom; unknown omitted
Detector row/column step	axes[2:4].sampling	Reader detector calibration; DM4 reciprocal axes (1/nm, 1/Å, 1/µm), read as detector_sampling_inv_A	Normalized to mrad or 1/angstrom; angular and reciprocal-length units remain distinct
Beam voltage	electron_source/accelerating_voltage	NCEM same path (V/kV); ARINA entry/instrument/detector/incident_energy (eV/keV); DM4 ImageTags.Microscope Info.Voltage (V)	Normalized to kV; existing electron-energy conversion policy unchanged
Convergence semi-angle	illumination_system/semi_convergence_angle	NCEM same path with rad/mrad	Normalized to mrad; not inferred from detector pitch
Dwell time	scan_controller/regular_scan/dwell_time	NCEM same path with s/ms/us/µs/μs; ARINA count_time, else frame_time	Normalized to us
Camera length	imaging_system/camera_length	NCEM same path with m/cm/mm	Normalized to mm
Angular detector sampling	imaging_system/reciprocal_pixel_size_row and reciprocal_pixel_size_column	NCEM reciprocal_pixel_size_y (row) and reciprocal_pixel_size_x (column) with rad/mrad	Normalized to mrad; named by array axis, not a conversion to reciprocal length
Detector identity	source_metadata.camera_model, camera_id	DM4 ImageTags.Acquisition.Device.Source Model/Source ID; equivalent retained camera fields	Preserved; never replace K3 identity with ARINA
Acquisition processing	source_metadata.acquisition_processing	DM4 ImageTags.Acquisition.Parameters.High Level.Processing	Preserved text, not assumed to mean background-corrected
Acquisition date	header metadata.acquisition_date	DM4 ImageTags.SI.Acquisition.Date; reader-provided date	Preserved when present; not a required normalized microscope field
Vendor fields	source_metadata	Fields retained by the source reader; DM4 names prefixed dm4.	Preserved subset, not every vendor object
Original XML/JSON documents	source_documents[]	Native EMPAD companion XML and explicitly attached UTF-8 XML/JSON	Complete text, filename, media type and SHA-256; separate from interpreted quantities
User calibration	calibration_overrides	Explicit scan/detector sampling, voltage, semi-angle, dwell, camera-length edits	Normalized values, units, user_override provenance and evidence; original fields stay separate
Lossless storage history	processing	Exporter operation	lossless_storage, changes_measurements=false
Dark/background recipe	header empad	Explicit native EMPAD dark reference and supplier correction evidence	Saved recipe/plane/identity; packed sample remains unchanged; subtraction applied once
Bad-pixel validity	header valid for integer codec	Reader detector-validity mask	Preserved independently of raw counts
Specimen (0.0.3)	sample (id, name, geometry, growth_direction, orientation_relationship)	Session dataset.yaml specimen: at conversion	Declared; provenance dataset.yaml with its sha256 as evidence; never derived
Specimen components	sample/components/<label> (role, chemical_formula, zone_axis, cif)	specimen.components.<label>; the CIF file as a JSON document in source_documents[]	Declared; lattice and space group read from the CIF, not stored
Thickness estimates	sample/components/<label>/thickness_estimates[]	files.<n>.thickness.<label>[] (value_nm becomes value in angstrom)	Declared estimates with method and region; automatic readings stay in result files until a person records them
Components in view	sample/components_in_view	files.<n>.components_in_view	Written only when declared

Flat calibration keys

A reader returns the calibration as flat keys and a writer accepts them for a new copy. The mapping is fixed; the format specification gives the rules.

Flat key	Unit, type	Common destination
voltage_kV	kV, number	electron_source/accelerating_voltage
semiangle_mrad	mrad, number	illumination_system/semi_convergence_angle
scan_sampling_A	angstrom, [row, col]	axes[0:2].sampling, scan_controller/regular_scan/pixel_size_row and pixel_size_column
detector_sampling_mrad	mrad, [row, col]	axes[2:4].sampling when its unit is mrad
detector_sampling_inv_A	1/angstrom, [row, col]	axes[2:4].sampling when its unit is 1/angstrom
dwell_time_us	us, number	scan_controller/regular_scan/dwell_time
camera_length_mm	mm, number	imaging_system/camera_length

Exactly one detector key is present, neither when the detector is uncalibrated. A saved calibration override replaces the recorded value in the flat key and leaves the recorded quantity in place.

Only finite positive calibration quantities with recognized units are promoted. Missing metadata is not fabricated. A supported field may still be missing in a particular acquisition. Known metadata in Python and Swift uses the same unit contract, but runtime/source-reader qualification remains separate.

Source-specific boundaries

Source	What is supported	What must not be claimed
K3 DM4	Native uint8/uint16 4D counts, calibrated reciprocal-nm detector axes, retained selected-image tags	Arbitrary DM4 images/dtypes, every vendor object, inferred scan calibration from sensor pitch
ARINA/NCEM HDF5	Reader-qualified counts and companion microscope vocabulary	Archiving every HDF5 object, every acquisition variant, or every private tag
EMPAD	Native float32 codec; explicit Python CPU reference encode/decode, Python CUDA and MPS readers of the float32 codec, and float-export XML/RAW import	Treating encoded EMPAD2 words as corrected float32
NumPy	Native little-endian C-order 4D uint8/uint16, retained NumPy header	Microscope metadata that was not supplied; signed/float/Fortran-order export through this count codec
Derived/scaled results	Regional scaled uint16 codes with their complete precision report (scaled-uint16-column-rans-v1, see QEM codecs)	Scaled codes read as detector counts; arbitrary 3D/5D or reconstruction export as QEM

SSB result images/calibration remain separate JSON/NumPy result artifacts; they are not silently embedded as an acquisition’s recorded microscope calibration.

Original documents and reviewed edits

source_documents preserves the original UTF-8 text, including unknown fields, comments and whitespace. Each entry contains filename, mediaType, content and sha256; the digest covers the UTF-8 bytes of content. Readers validate the digest before accepting a document. The current limits are 16 documents and 4 MiB total document content. XML document types and external entities are not accepted. JSON must contain a top-level object.

Three distinct records must not be conflated:

• source_documents: original document text, available after the source file is removed or the acquisition moves to another machine.

• electron_microscope and axes: supported, normalized recorded quantities.

• calibration_overrides: explicit reviewed edits, without replacing the original recorded quantities or document text.

The native NativeMetadataImport.read API previews recognized EMPAD XML or scientific-metadata JSON quantities before a client applies them. Unknown vendor XML/JSON is preserved without guessed units. Saving an attachment alone must not change calibration. Contents can include names and paths; inspect them before sharing a QEM file. Document preservation does not imply exhaustive DM4/HDF5 tag capture, nor does it establish CUDA support for the float32 measurement codec.

Source map

• Swift: NativeDM4Source, NativeMicroscopeMetadata, NativeQEMMetadata, NativeQEMCalibration, NativeQEMFile, MetalQEMExporter under native/swift/Sources/.

• Python: formats/qem/metadata.py, io/qem.py and source-specific readers.

• The reference and validation workflow distinguishes byte integrity, field preservation, exact decoded counts and hardware coverage.