# Paired-count resident layout

An opt-in second resident layout for native count acquisitions, next to the
byte-rANS `StreamedCounts` default. Every original count is retained exactly;
the layout changes bytes and query kernels only. Nothing in the default load,
dense or ANS-file paths changes when this layout is not requested.

## Operation pipeline

1. Native `uint8`/`uint16` counts arrive as complete 512-scan blocks
   (`PairedCounts.append`). `io.load(..., representation="paired")` produces
   them from original bitshuffle+LZ4 HDF5: reader threads read whole shards
   with direct I/O into page-locked staging and parse the LZ4 block headers
   there; one producer stream copies, LZ4-decodes and bitshuffles into a ring
   of rolling frame buffers; the consumer stream encodes and indexes each
   filled buffer and returns it through an event. Shard reads run ahead across
   acquisitions, so a series is bounded by the drive, not by the GPU.
2. Each detector pixel's 512-scan stream is coded as consecutive count pairs
   with a 1024-state tANS table from 32 Poisson pair models; streams that are
   empty, constant, sparse (few nonzero counts) or incompressible keep exact
   alternative forms. Stream offsets are grouped (one 32-bit base plus 32
   relative 16-bit offsets).
3. Exact per-scan sums over radial-angular pixel groups (64-pixel leaves,
   16-leaf roots) are bit-packed as the interaction index.
4. `detector.prepare` selects `PairedSeriesCompute`. A detector mask (or its
   difference from the previous mask) is decomposed into index fields plus
   residual pixels; the residual streams are decoded by one thread per stream
   (a 64-bit bit reservoir refilled four bytes at a time from a prefetched
   window, once per three coded pairs) with a warp-wide packed reduction per 32
   scans. Outputs are exact integer virtual images and native diffraction
   patterns, as for the default layout. `masked_sum(..., block_stride=k)` runs
   the plan, index and residual kernels over every k-th 512-scan block only
   (`stride` argument; work items per chunk shrink to `ceil(blocks / k)`), so a
   viewer can preview a moving mask on all acquisitions at about 1/k of the cost;
   the baseline for incremental masks is kept per stride.
5. `PairedCounts.save` writes the resident arrays once; `PairedCounts.load`
   (or `io.load` on the file) reopens them with direct I/O and no decode.
6. Reconstruction consumers read native count blocks back from the resident
   form: `PairedCounts.decode_blocks(first, scans)` decodes whole 512-scan
   blocks of one chunk, and `PairedFeed(sources, amplitude=True)` iterates the
   blocks of a series on a prefetch stream (`depth` buffers ahead, events in
   both directions) so a joint time-series ptychography update never touches a
   dense copy of the data.

Axes are `(scan_row, scan_col, detector_row, detector_col)`; equations use
$I[R_r,R_c,k_r,k_c]$ with $\mathbf R=(R_r,R_c)$ and $\mathbf k=(k_r,k_c)$.

## Contract

| item | value |
| --- | --- |
| query ABI | `paired-polar-counts-v1` |
| input | complete multiples of 512 scans, native `uint8`/`uint16`, detector up to 65,535 pixels per group of 32 streams; chunks of any block count may mix within one series (the residual work list is a one-dimensional grid) |
| H5 loader | `uint16` bitshuffle+LZ4 shards with one frame per chunk (chunk tables and LZ4 block headers are parsed from the staged image, so the drive only sees sequential direct reads); a partial final LZ4 block must hold a multiple of 8 values; one path or a list, every acquisition returned as its own source; `backend="cuda"`, `dtype="native"`, `apply_mask=False`, no selection or binning options |
| virtual image | exact `uint32` sums, `uint64` when a full-detector sum could exceed `2**32` |
| diffraction pattern | native dtype, invalid pixels reported as zero |
| malformed stream | `ValueError` from the query; the result is not returned |
| saved form | `QGPUPAIR` magic, JSON header (`quantem-paired-resident-v1`) plus 4096-aligned arrays; `io.inspect` reports `source_kind="paired"` with `resident_bytes`; `io.load` detects it; reopen refuses another ABI |
| application streaming | `PairedLoader.stream(paths, admit=...)` yields `(path, Dataset4dstemGPU)` per acquisition while later files are still being read |
| queued queries | `session.masked_sum(..., output="native", out=..., wait=False)` and `session.frame(..., wait=False)` return once the kernels are queued; `session.finish()` waits for the oldest and raises for a malformed stream; a failed decode invalidates the incremental plan and taints queued deltas built on it |

Coding parameters (32 models, 1024 states, 2-byte stream header, sparse mode
preferred unless the paired stream saves at least two bytes) are fixed by the
ABI string; any change needs a new ABI.

## Source map

| piece | location |
| --- | --- |
| kernels: tables, encode, compact, decode, decode_range, frame, plan, residual, polar fields, index pack/sum, offsets, planner weights | `src/quantem/gpu/resident/cuda/kernels/paired.cu` |
| `PairedCounts`, polar layout, saved form, `decode_blocks`, `PairedFeed` | `src/quantem/gpu/resident/cuda/paired.py` |
| `PairedSeriesCompute` and its polar planner | `src/quantem/gpu/detector/cuda/paired_series.py` |
| planner hooks in the streamed base | `src/quantem/gpu/detector/cuda/streamed_series.py` (`_plan`, `_cost`) |
| dispatch | `src/quantem/gpu/detector/session.py` |
| H5 streaming loader, saved-form reopen, `io.load` results | `src/quantem/gpu/io/paired.py` (`PairedLoader`) |
| representation selector and magic detection | `src/quantem/gpu/io/representation.py`, `src/quantem/gpu/io/load.py` |
| tests | `tests/contracts/io/test_paired_counts.py`, `tests/contracts/io/test_paired_loading.py` |

## Verification

The contract tests build synthetic sources with a wide literal row, a
saturated row and an invalid pixel at 17x17, 19x19 and 257x257 detectors,
compare every mask and frame with direct sums (including `uint64` sums above
`2**32`), reject a reserved header bit and a shortened stream extent, and
reopen a saved form byte-identically, and iterate two sources through
`PairedFeed` checking every block and amplitude against the raw counts. The
loading tests write four-shard
Arina-style masters with `save_compressed_arina_h5`, stream one and two of them
through `io.load(representation="paired")`, compare every count, a detector
mask and a frame with direct sums, and reopen a saved form through `io.load`
byte-identically. Set `QUANTEM_GPU_PAIRED_REFERENCE` to a JSON file (`path`,
`poses`, `reference_VI_sha256`) to check frozen full-array virtual-image
digests on a native source through the public loader; on 2026-09-09 all six
digests of one native 512x512x192x192 acquisition matched.

Physical-device evidence (private study, 2026-09-09; one RTX PRO 6000
Blackwell, 69 complete 512x512x192x192 uint16 acquisitions resident): the
paired layout decoded arbitrary-center detector updates at 12.5 ms per batch of
69 full virtual images versus 23.9 ms for the byte-rANS layout on the same
counts, with 414 frozen full-array digests exact; resident bytes per source
1.293 GiB versus 1.307 GiB. Loading from the saved form reached the measured
drive ceiling. The grouped-refill decoder (2026-09-09, twelve sources, 356
recorded centre poses) executes 48 instead of 67 warp instructions per coded
pair and takes 0.72 instead of 1.00 ms per update, byte-identical
(`docs/performance/data/paired-decoder-2026-09-09.json`); on a viewer's
device the residual decoder is 88 percent of query time and the plan kernel
10 percent.

Package loader, single run on the same device with the default
`PairedLoader` (69 complete 512x512x192x192 uint16 acquisitions offered from
one PCIe 4.0 NVMe drive whose direct-read ceiling is 5.5 GB/s; a memory-budget
`admit` callback stopped the series at 66):

| measurement | acquisitions | value |
| --- | --- | --- |
| series load, original HDF5 to resident, wall (28.8 s once chunk tables are parsed from the staged image) | 66 | 29.3 s |
| resident-ready wall per acquisition inside the pipeline, median | 66 | 1.50 s |
| encode per acquisition, median | 66 | 0.197 s |
| index per acquisition, median | 66 | 0.120 s |
| resident bytes, sum | 66 | 93.97 GB |
| device bytes in use after load (pool cache and staging included) | 66 | 101.6 GB |
| `detector.prepare` over all sources | 66 | 38 ms |
| first bright-field virtual image, all sources (first source digest exact) | 66 | 8.2 ms |
| first diffraction pattern, all sources | 66 | 1.9 ms |
| save one source as the paired resident form (write plus fsync) | 1 | 1.39 s |
| reopen that form through `io.load`, byte-identical | 1 | 0.53 s |

Shards are read with direct I/O, so the series time is bounded by the drive
(about 145 GB of original chunks in 29.3 s). The complete atomic record with
boundaries, cache states and the admission rule is
{download}`paired-loading-2026-09-09.json <../performance/data/paired-loading-2026-09-09.json>`.
The last acquisitions of a full device fit only with a smaller loader
(`PairedLoader(rolling_scans=512, rings=2)`); that tail policy belongs to the
application.

Complete series through the package only (private owner script over
`PairedLoader`, `PairedCounts.load` and `detector.prepare`; 69 acquisitions of
512x512x192x192 uint16 on one device, single run):

| measurement | value |
| --- | --- |
| original HDF5 to resident, 64 with the default loader then 5 with `PairedLoader(rolling_scans=512, rings=2)` (32.0 s once chunk tables are parsed from the staged image) | 38.7 s |
| resident bytes, sum (index 6.7 GiB of it) | 90.3 GiB |
| peak device bytes in use during loading | 94.4 GiB |
| `detector.prepare` | 53 ms |
| frozen full-array virtual-image digests, six masks per source | 414 exact |
| headless center gestures, 64 fresh masks, wall p50 / p95 | 14.1 / 16.3 ms |
| headless scan gestures (native diffraction pattern), wall p50 | 1.3 ms |
| save every source as the paired form, 39 on one drive and 30 on another | 192 s |
| reopen every saved form, one reader per drive, to 69 resident | 9.6 s |

Record:
{download}`paired-capacity-2026-09-09.json <../performance/data/paired-capacity-2026-09-09.json>`.
Reopening carves every array of a file from one device allocation; with one
allocation per chunk array the driver's allocation granularity cost about 7 GB
across the series and only 66 files fit.

Feeding a reconstruction from the resident form (three 512x512x192x192 sources,
`PairedFeed(depth=2)`, consumer idle, same device shared with a desktop):

| measurement | per acquisition |
| --- | --- |
| native counts, 512-scan blocks | 76 ms |
| native counts, 8192-scan blocks | 90 ms |
| counts plus float32 sqrt amplitude, 512-scan blocks | 119 ms |
| counts plus float32 sqrt amplitude, 8192-scan blocks | 133 ms |

That is 127 G counts/s decoded into native frames, below the 48 ms per
262,144-position fused ptychography iteration only by a factor of about two, so
one decode per iteration hides behind the update when the two overlap on
separate streams. Record:
{download}`paired-feed-2026-09-09.json <../performance/data/paired-feed-2026-09-09.json>`.

**Device tested**: RTX PRO 6000 Blackwell (CUDA). **Date tested**: 2026-09-09.