Integrating count encodings and Fourier storage

Integrating count encodings and Fourier storage#

This is the integration boundary for ongoing lossless-memory work, not a claim that every research encoding is available through every backend. Keep existing workflows usable while qualifying one new implementation at a time. The representation API describes current calls; backend readiness retains the scientific and physical-device gates.

Four responsibilities, not four public loaders#

Layer	Owner	Retained information
Acquisition	`io`	Original count identity, geometry, dtype, calibration, exclusions
Count representation	`resident/<runtime>`	Dense values or an authenticated lossless payload and all decoding tables
Detector query index	`detector/<runtime>`	Optional exact summaries, checkpoints, or spatial indexes bound to one resident source
Fourier representation	`ssb/<runtime>` and native SSB products	Final Fourier coefficients and all descriptors needed to consume them

authenticated source -> io.load -> dense or ANS-encoded counts
                                     |                  |
                              detector products      SSB preparation
                                                        |
                                                final Fourier fields
                                                        |
                                            aberration updates and search

dense, encoded, and paired describe the count layouts, with no legacy selector aliases. Public CUDA/MPS acquisition loading requires ANS (encoded or the explicit paired ANS profile); it rejects a dense override. The resident_profile metadata names the encoding profile within a layout. File layout and compression are separate: io.save(..., format="quantem", compression="ans") uses the standalone envelope; format="arina" retains the HDF5 layout. Decompression is detected during loading, not selected by a second argument. Do not add load_rans, load_tans, or a backend-specific scientific workflow. Encoding profiles belong in authenticated source metadata and private dispatch. Reject an unknown profile before dispatch; do not guess a codec from a compression ratio or filename. ANS coding probabilities do not approximate measured counts: the decoded integers must remain exact.

Fourier storage is a separate SSB choice. An advanced fourier_representation option is a design proposal, not a shipped argument. Do not overload the count representation argument to select it. The existing source-native load behavior and dense Fourier defaults remain unchanged during extraction. Compact counts do not imply equally compact Fourier coefficients.

Current integration lanes#

Component	Current public boundary	Research lane	Next acceptance gate
Direct bit-packed counts	Native Metal and native Vulkan readers; no Python or WebGPU reader	Supported packed profiles	Repeat exact counts, exclusions, corruption, lifetime, and products.
Existing uint16/LZ4 packed counts	Documented native Metal path; no Python or WebGPU reader	Retained alongside bit-packing	Preserve authentication and distinguish resident bytes from full-file staging peak.
Standalone exact count-ANS envelope	`io.save` from a CUDA/MPS resident or the explicit CPU reference; `io.load` as ANS on Python MPS/CUDA, dense on CPU	New `quantem.gpu.count-ans.v1` integration	Physical CUDA, native file reader, GPU saving/reverse conversions, complete real data, and peak-memory qualification.
Range ANS (rANS) counts	Not a general public `io.load` profile	CUDA and native Metal research consumers	Remove fixed geometry and research paths; freeze a portable stream/table contract and independent complete-count parity.
Table ANS (tANS) counts	Not a general public `io.load` profile	CUDA producer and consumer research	Verify inverse transitions, literal values, malformed streams, bounds, and exact products before packaging.
Exact detector indexes	Backend-specific facilities, not a uniform guarantee	Moving-mask, checkpoint, and spatial-sum candidates	Account for all dependencies; compare changed-mask outputs with the direct exact reference.
Compressed stationary final Fourier fields	No public cross-backend compressed-Fourier option	Opt-in native Swift/Metal research implementation	Extract the qualified consumer and lifecycle; preserve coefficient bits, object/loss parity, invalidation, and packing peak.

Python MPS and native Swift/Metal are distinct consumer paths even when both execute Metal kernels. A native test does not qualify the Python API. CUDA, WebGPU, and Vulkan do not inherit compressed-Fourier support from Metal tests. A bounded decode test is not proof of full-dataset residency or display speed.

Extraction sequence#

Preserve first. Record branch, full commit, staged/unstaged binary patches, untracked-file hashes, dependencies, and retained run manifests. Keep two verified copies before reconciling distinct research trees. A Git commit alone does not preserve an untracked decoder or codebook.
Freeze one profile. Capture byte order, model/table identities, source and working dtypes, ordering, literals, bounds, calibration, exclusions, and invalidation. Retain the NumPy/Torch reference before the accelerated port. Profile versioning belongs in metadata, not method names.
Extract the smallest implementation. The resident layer owns count decoding, detector owns indexes, and SSB owns final Fourier storage. Native clients use package products. No copied app kernels or runtime imports from research directories.
Test both representations. Exact integer equality includes rare maximum values and excluded raw values. Packing final Fourier fields must recover their original IEEE words; downstream arithmetic retains its frozen full-precision comparison. Checkpoint/FFT regeneration is not directly consumable stationary final Fourier storage.
Qualify one backend. Test complete real acquisitions, changed parameters, teardown/cancellation, authentication failures, and representation switching. Separate cold source, prepared creation/reopen, first usable, exact complete, and resident updates. Preserve failures and outliers.
Hand off a pin. Deliver a clean revision, public contract, fixture hashes, commands, results, limitations, and memory accounting. Consumers select application policy and perform their own headed acceptance.

Do not merge whole experimental directories or delete slower baselines because a newer layout exists. Keep rejected experiments outside production imports, with their reason and reproducible retained records.

Memory accounting#

Report source logical count bytes, working logical count bytes, and resident count bytes separately. Resident counts include payloads, offsets, codebooks, literal exceptions, and alignment. Separately report optional indexes, final Fourier payload/descriptors, outputs, and scratch.

Measure peak accelerator allocation and peak process RSS alongside steady resident bytes. Packing can retain both old and new Fourier buffers until verification succeeds. Count shared codebooks once per physical owner, not once per dataset and not zero times. On Apple hardware, logical Metal buffer lengths, driver allocation, RSS, compressed memory, and swap differ; do not add overlapping unified-memory counters. No fixed compression factor is guaranteed for arbitrary data. Admission uses actual payload and dependency sizes and fails explicitly if the exact plan cannot fit.

Reproduce the readiness checks#

Use the existing registries instead of maintaining another support matrix:

python scripts/backend_status.py check
python scripts/backend_status.py summary
python scripts/backend_status.py json --backend cuda
python scripts/run_tests.py tests/contracts/io tests/parity/test_resident_integer_contract.py -q

IO import checks cover both standalone use and coexistence with an installed quantem parent. Imports performed by that parent are attributed separately. CuPy installation does not guarantee a CUDA device; unavailable hardware checks skip explicitly and cannot count as passes.

Small physical Apple checks:

python scripts/run_tests.py tests/hardware/mps/test_resident_integer_contract.py -q -rs

On an available, owned CUDA device:

python scripts/run_tests.py tests/hardware/cuda/test_resident_integer_contract.py -q -rs

These are bounded regressions, not cold-load or full-acquisition performance measurements. Use the profiling runbooks for those gates. Add qualified numbers to the existing benchmark registry only when source, fixture, hardware, sample count, timing boundary, and memory definitions are complete.