# Bounded MPS HDF5 loading, 2026-10-23

A full 2048 × 2048 scan with a 192 × 192 uint16 detector now loads into
lossless count-ANS residency in 16.26–16.54 seconds on the tested Apple M5 Max.
The earlier repeated measurements were 36.11–38.33 seconds; a fresh baseline
control repeated in 34.80 seconds. The conservative matched-control improvement
is 2.12×. This measures the Python MPS API, not native Swift application paint.

## Workload and changes

Base revision: `d2db0bdb4c9859230d3e9ba2f6b23fcfae8eb0f1`, with local MPS
loader changes. Physical device: Apple M5 Max, 128 GB unified memory.
Private experiment: `20261023-mps-h5-pipeline`.

The public `io.load` call uses MPS, `dtype="native"`, no hot-pixel correction,
no crop, and no binning. All 4,194,304 frames are ingested: 288 GiB logical
uint16 counts. The unchanged lossless representation occupies 38,332,415,954
bytes, including its spatial index. Native reads still return uint16 counts.

The previous implementation prefetched compressed bytes but decoded and
processed each batch synchronously. The candidate prepares and decodes one
batch ahead while encoding and indexing the preceding batch. It reuses two
bounded decoded buffers and one encoder scratch buffer. Each GPU command
completes before its buffers are recycled; cancellation and failure drain the
pending batch and free staging. No numerical kernel, codec interval, detector
support, precision, or scientific correction policy changed.

Profiling the baseline found about 10.8 seconds per load inside the ANS encode
command wait, versus about 2.4 seconds of reported GPU execution. Reusing its
large scratch buffer removed much of that allocation/residency overhead.
Decoded-buffer reuse provided a further improvement. GPU timestamps do not
measure occupancy, and overlapping stage wall times must not be summed.

## Timings

The existing `benchmark_hdf5_load.py` runner measures public load through backend
synchronization. Validation and complete-resident hashing are outside that
interval. Residents are released between repetitions. OS file caches are
uncontrolled; none of these measurements is labelled cold-disk.

| 2048² case | First source encounter (s) | Subsequent full loads (s) |
|---|---:|---:|
| Original verified baseline | 45.43 | 38.33, 36.11 |
| Fresh baseline control | 39.55 | 34.80 |
| Final candidate | 20.88 | 16.26, 16.54 |

The final repeated median is 16.40 s, with only two repeated observations;
this is not a long-run tail-latency qualification. Baseline controls bracket
the exploratory pipeline and buffer-reuse probes before final validation.

| Scan / detector | Acquisitions | Earlier repeated range (s) | Candidate repeated range (s) |
|---|---:|---:|---:|
| 256² / 192² | 1 | 0.505–0.511 | 0.250–0.259 |
| 512² / 96² | 2 | 0.728–0.747 | 0.386–0.392 |
| 512² / 192² | 5 | 1.912–2.009 | 0.824–0.892 |
| 1024² / 192² | 2 | 8.722–9.374 | 3.278–3.374 |
| 2048² / 192² | 1 | 36.110–38.327 | 16.261–16.536 |

All 33 candidate loads passed their checks. Sampled peak Metal allocation was
40.31–40.40 GB for the final large case versus 40.31–40.32 GB for the fresh
baseline control. The 20 ms sampler can miss brief peaks; the implementation
permits one additional 576 MiB decoded staging buffer. Metal allocation after
release returned to 9,109,504 bytes. System swap was already present and OS
memory pressure was not experimentally controlled.

## Correctness and limits

Both baseline controls and all three final large loads produced the identical
SHA-256 over every encoded chunk and spatial-index buffer:
`a9e04de2c7c737c3cb3357fdfdcf33e4eb834b5f826cf07f52f64025b41e2dfb`.
This establishes complete resident equivalence to the frozen baseline, rather
than a separately decoded full-volume file oracle. Independent HDF5 comparisons
also checked the first, middle and last frame of every external data shard:
252 exact uint16 frames per large load. All source sizes and modification times,
and master/metadata hashes, were unchanged.

Focused hardware, metadata, benchmark and repository checks: **64 passed**.
The strict acquisition runner reported **145 passed, 6 failed, 23 skipped**.
All six failures reproduced on an isolated export of the unchanged base:
one uint64 SSB dtype-dispatch failure and five FP8 hot-pixel-policy failures.
The two full public collections were unavailable. The broad acceptance gate
therefore remains incomplete; these findings are not a release qualification.
CUDA and native Swift loading are separate paths and are not certified by the
MPS optimization. No release was performed.

The retained evidence includes source-file hashes, an AST fingerprint of the
MPS implementation, full timing samples, memory measurements, failure controls,
and disposable-test cleanup receipts. The retired benchmark registry is not
used; this dated record preserves the earlier results above.

## ANS QEM reopening on the internal SSD

The same full 2048 × 2048 × 192 × 192 acquisition was converted with the
canonical converter on MPS and reopened from the internal APFS SSD. The first
QEM load during conversion verification took **5.077 s**. Three subsequent
public loads took **5.618, 4.128, and 3.687 s** (median **4.128 s**), with the
resident released between runs. Timing includes integrity checking and backend
synchronization, but excludes source-value verification and reconstruction.
These are post-write/repeated observations with uncontrolled OS caches, not
cold-disk performance. Existing system swap and other desktop applications were
present; this is not an isolated tail-latency qualification.

The QEM file is **38,332,680,910 bytes** (35.70 GiB), versus 54,595,035,757 bytes
for the original HDF5 acquisition. Codec: `runtime-column-rans-spatial-v2`.
The converter uses exact uint8 integer packing for these uint16 source counts;
public native reads remain uint16. This is lossless value preservation, with
no crop, binning, clipping, or hot-pixel correction. Exhaustive independent
source comparison checked **154,618,822,656 values with zero differences**.
The three reloads also passed shape, dtype, detector-binning, and source/working
geometry checks. Recorded calibration is carried; missing convergence angle
remains unknown. No SSB result is claimed.

One-time conversion took **780.246 s** overall: HDF5 loading/encoding took
27.636 s, QEM writing including file flush/fsync took 28.759 s, and exhaustive
verification took 698.946 s (including its 5.077 s QEM load). The remaining
24.904 s includes source hashing and metadata/publication overhead. The long
verification time is not incurred by ordinary QEM reopening. Earlier 16.3–16.5 s
HDF5 loads were measured under different runtime/cache conditions; do not use
these observations as a strictly matched speedup ratio.

No QEM implementation was changed for this measurement. Source fingerprints,
raw timings, verification, and cleanup receipts are retained under private
experiment `20261023-mps-h5-pipeline`, QEM follow-up. An external-volume write
attempt was interrupted before any accepted load measurement, then removed.
All internal and external numerical benchmark scratch was removed after use.