Reproducing resident count analysis

Reproducing resident count analysis#

Use the installed quantem.gpu package for scientific computation. Keep exact study settings, private source locations, accepted results and figure rendering in the study workspace. A manuscript does not need to be public for its code dependency to be reusable.

For ordinary scientific work, use io.load, array indexing and detector.bf / detector.adf as shown in the README. This guide covers advanced native and exact-integer integration. Its entry points are io.load, detector.prepare, session.frame and session.masked_sum_exact. There is no separate paper API, codec registry or common codec base class. Preserve the implementations until evidence justifies combining them.

Select the existing path#

Input and purpose	Entry point	Retained implementation	Boundary
Ordinary complete uint8/uint16 H5 into an indexed resident	`io.load(path, backend="cuda", representation="encoded", apply_mask=False)`	Runtime encoded streams in `resident/cuda/counts.py`; H5 loading in `io/encoded.py`	Save portable `.qem` with `io.save`; native format/backend acceptance remains separately gated
Portable QuantEM/ANS file	`io.load(path, backend="cuda")`	Container reader in `io/qem.py`, CUDA decoder in `resident/cuda/counts.py`, Metal decoder in `resident/mps/counts.py`	Different format/model from historical storage experiments; MPS also supports its documented subset
Older prepared packed source or prepared series folder	Re-export the original acquisition as `.qem`	None; these readers were removed	`io.load` does not open them
Original HDF5 into native Metal integer packing	Native Swift `MetalOriginalHDF5Packing` workflow	Swift/Metal sources in the same package repository	Separate native lifecycle; not an automatic Python H5-to-packed conversion

The encoded selector names the representation family. The format/profile selects the compatible decoder. rANS and tANS streams are not interchangeable.

See native Metal original-HDF5 packing for its entry points. Existing experimental storage codecs remain distinct from the portable file writer. Do not substitute the public writer into a historical compression comparison and assume that it reproduces its archive bytes.

Run through the public API#

For general H5 inputs, load each complete acquisition explicitly and prepare the returned list. Supply the study’s ordered source list; acquisition order is part of reproducibility.

from contextlib import ExitStack

import numpy as np

from quantem.gpu import detector, io

paths = ["acquisition-0.h5", "acquisition-1.h5"]
with ExitStack() as owners:
    acquisitions = [
        owners.enter_context(io.load(
            path, backend="cuda", representation="encoded", apply_mask=False,
            dtype="native", device=0,
        ))
        for path in paths
    ]
    session = detector.prepare(acquisitions)
    mask = np.ones(session.detector_shape, dtype=bool)
    # Host-returning exact outputs are convenient for independent comparisons.
    images = session.masked_sum_exact(mask)
    patterns = session.frame(0)
    np.savez("detector-products.npz", mask=mask, images=images, patterns=patterns)
    del session

An all-true mask still respects the stored detector-validity policy. Raw point patterns retain original counts. Record the validity mask alongside the requested mask when constructing independent reference sums. Neither example bins, crops or narrows the source. Weighted masks, mean DP and other operations have separate support boundaries; a successful binary detector query does not qualify them.

Keep sources and sessions alive while native consumers use them. Complete consumer work before closing loaded owners. Native outputs avoid the host copy; NumPy output is appropriate for reference comparisons and saved results. Do not report NumPy-save time as GPU kernel time or query time as display FPS.

Pin code and preserve evidence#

Install a recorded release or exact source revision in the study environment. Record git rev-parse HEAD and git status --short for a source install, plus python -m pip freeze. A dirty checkout requires an archived patch and any untracked runtime inputs; a commit identifier alone does not describe it. Package version alone is insufficient when multiple revisions share a version.

Keep a small study-owned record with:

Ordered dataset identifiers, complete input checksums (including H5 external shards or prepared packs), calibration, dtype, shape and validity policy.
Package revision/environment and, where supplied, session.backend_metadata including implementation digest and query ABI.
loaded.metadata["resident_profile"] when supplied, representation, logical bytes and reported resident bytes. Missing metadata means unreported, not zero.
Exact detector masks, scan indices, output arrays and their checksums, plus independent reference comparisons. For integer sums use exact comparison; float32 workflows need their documented bit-preservation and reduction rules.
Hardware, driver, timing boundary, repetitions and memory-accounting scope. Resident payload/index accounting is not full application peak memory.

Store frozen products separately from figure code. Figure rendering reads the accepted result bundle; it must not silently rerun loading or scientific computation. Changing a codec, model, layout or query policy creates a new study result, not an updated label on an old measurement.

The scientific owner package owns reusable algorithms and their tests. QuantEM.GPU owns reusable accelerator infrastructure and its tests. The private study owns manuscript text, exact publication protocols, failed investigations and figure layout. Large inputs and outputs belong in durable data storage. Do not copy private manuscript sources, raw data or workstation credentials into the software repository, or copy production kernels back into the paper workspace.

Verify the boundary#

The host tests exercise reported representation, native/NumPy products and portable-file reference round trips:

python -m pytest -q tests/detector tests/contracts/io/test_ans_public_api.py

With an available CUDA device, run the existing general-source workflows:

python -m pytest -q tests/hardware/cuda/test_streamed_h5.py \
    tests/hardware/cuda/test_native_series.py

Host tests with device stand-ins verify Python routing and output semantics; they do not execute CUDA kernels. Full-series performance and exact reference arrays remain separate study evidence. A refactor that changes scientific code must repeat the affected real-data parity endpoint before reusing its claim.

Revisit implementation overlap later#

Python/MPS and Swift retain separate Metal ANS reader sources. Runtime indexed rANS and paired tANS remain independent implementations with different input contracts. Future consolidation should start with matching format fixtures and unchanged native outputs, not with a shared class hierarchy. Keep old evidence pinned while comparing any replacement through the same public scientific entry points.