Experimental lossless resident ANS

Experimental lossless resident ANS#

ANS means asymmetric numeral systems. Supported original acquisitions now default to ANS on CUDA/MPS through io.load(path). This page retains experimental browser/archive integrations; their qualification is distinct from that default. See the current IO guide for supported formats and the acceptance gate for runtime coverage. Source integration does not mean these changes have been released on PyPI.

Compatibility safeguards#

The loader detects encoded sources. Explicit reference inputs retain their separate contract. Encoding writes a new container atomically and refuses to overwrite an existing file; original acquisitions remain intact. Source ownership and cancellation are covered by regression tests. These are specific safeguards, not a guarantee that every experimental hardware path is qualified. The performance and validation limits below remain part of the feature.

One implementation owner#

quantem.gpu owns the count encoder, validated container reader, CUDA decoders, WebGPU decoders, detector reductions, pattern gathering, and shared GPU display math. Show4DSTEM owns browser permission prompts, progressive panel display, selection, gestures, and scheduling. Its build copies the package’s TypeScript sources into the ignored js/.generated/engine directory and bundles them; those generated files are not an independently maintained codec.

Input

Package implementation

Client route

K3 saved copies, quantem.qem

CUDA and Metal encoding, checksummed reopening, spatial queries

Python CUDA/MPS and native Live4DSTEM macOS

Saved .qem copies

io.save, io.load, io.qem, CUDA count decoder, native reader

CUDA/MPS resident queries; native Live4DSTEM macOS

Retained detector-rANS manifest

WebGPU rans.ts

Show4DSTEM WebGPU export

These are explicitly identified formats, not interchangeable byte streams. The canonical writer is shared; compatibility readers retain older experiments without rewriting source data. Full-series conversion of every retained archive to the canonical container is not a completed acceptance gate.

Encode and query canonical counts#

from quantem.gpu import detector, io

# resident: an encoded acquisition from io.load, native uint8/uint16 counts,
# (scan_row, scan_col, detector_row, detector_col)
io.save("acquisition.qem", resident)
with io.load("acquisition.qem", backend="cuda", device=0) as loaded:
    session = detector.prepare(loaded)
    pattern = session.frame(0, output="native")
    image = session.masked_sum_exact(binary_detector_mask, output="native")

device=0 means the first CUDA-visible device; select the intended physical GPU before launching Python. Saving copies the resident’s encoded bytes without re-encoding; it is not a claim of real-time full-acquisition compression. CUDA residency retains encoded buffers and produces device results without constructing a full dense acquisition. See the container contract for dtype, checksums, invalid pixels, conversion, and lifetime rules.

Compatible ANS acquisitions can be retained independently on MPS and queried as one detector session without dense stacking:

from quantem.gpu import detector, io

loaded = io.load(paths, backend="mps", stack=False)
session = detector.prepare(loaded)
patterns = session.frame(scan_row * scan_columns + scan_column)
images = session.masked_sums_exact(masks)  # (mask, acquisition, scan_row, scan_column)

All files must declare the same complete scan and detector geometry. The series adapter runs each query on each acquisition in turn and stacks the results along the acquisition axis, so every value equals the result of preparing that acquisition alone; it returns only the requested point patterns or detector products. Each Dataset4dstemGPU owner remains caller-owned and must be closed after the session is finished. This is the supported multi-file ANS API, not a claim that arbitrary HDF5 folders are encoded as ANS automatically.

Show4DSTEM WebGPU#

The widget package supplies the browser export, using the same package reader:

from quantem.widget.show4dstem_webgpu_export import export_show4dstem_rans_viewer

html = export_show4dstem_rans_viewer(
    ["acquisition.qem", "next-acquisition.qem"],
    "ans-viewer",
    frame_labels=["First", "Second"],
)

Open the generated launcher/viewer and use Open QEM files to grant access to the linked .qem files in ans-viewer/rans. Export preserves native shape and dtype, links the .qem files, and does not decode or duplicate the full acquisition. Keep the source files available.

The canonical-file workflow above is the browser export integration.

Raw hardware sentinel counts are preserved. The established detector validity mask is applied only to displayed scientific products. Integer native patterns and detector sums require exact reference equality. Count means use explicitly rounded float32 division; native sums remain exact for the validated series.

Qualification and limits#

Releasing the resident owner or closing the viewer releases its encoded source and display buffers. Failed/cancelled admissions must release owned allocations. Do not delete original acquisitions, canonical containers, or retained archive payloads as part of branch or benchmark cleanup.

Maintainer checks#

  • Build Show4DSTEM with QUANTEM_GPU_SRC pointing to this package’s src tree.

  • Run count-ANS integer round trips and browser-export tests for native dtypes.

  • Run tests/webgpu/display-readback-lifecycle.ts for ownership failures.

  • On an identified real adapter, run native source parity and tests/webgpu/resident-pattern-mean-parity.ts.

  • Drive selected and averaged scan drags with the button held, then verify the endpoint against original counts. Separately check detector drags, contrast, zoom, copy, progressive loading, cancellation, and device loss.

  • Count fresh scientific output, not animation callbacks or repeated paints.