Native Metal HDF5 Loader Postmortem

Date: 2026-08-13

Hardware: Apple M5 Max, 128 GB unified memory

Evidence: 512x512x192x192, source-native uint16, three external HDF5 shards, no scan crop, detector binning, or precision reduction

What was slow

The fused Bitshuffle/LZ4 decoder was not the main problem. The application called otherwise reasonable kernels in a wasteful full-volume topology:

1. Decode the uint16 source into a clipped uint8 volume while auditing the source count range.

2. Discover that millions of valid counts exceed 255, discard the complete 9.66 GB uint8 volume, and decode the same compressed HDF5 data again into exact uint16.

3. Materialize a complete 19.327 GB scan-major uint16 volume, then perform a separate serialized 19.327 GB scan-major-to-detector-major transpose.

4. Wait for batch command buffers inside the submission loop, preventing HDF5 decode, detector-product work, and layout conversion from overlapping.

The first pass was scientifically cautious but operationally wrong. A full speculative conversion is not a dtype probe. On the center tilt, the exact audit found a maximum count of 1833 and 17,420,309 unmasked values above 255, so the speculative uint8 result could never be retained.

This is the central lesson: profile the whole pass graph and data lifetime before micro-optimizing an individual kernel. A fast kernel called twice over 9.66 billion detector values is still a slow loader.

Retained topology

The native application now chooses the source dtype before allocation and performs one exact uint16 decode. Each bounded batch:

1. maps compressed HDF5 bytes;

2. decodes exact counts into a scratch buffer;

3. computes the exact count audit, BF/ABF/DF products, and mean diffraction;

4. signals a shared Metal event; and

5. transposes a tiled batch directly into its final detector-major offset on a second command queue.

Eight 16,384-frame scratch slots keep the two queues busy without a complete scan-major intermediate. The transpose moves 32x32 tiles with 16x16 threadgroups. Count auditing aggregates in threadgroup atomics and performs only bounded global atomic updates instead of one global update per detector value.

GPU work summed across concurrent queues can exceed wall time. That is expected and must not be reported as a regression without first checking queue overlap.

Result and parity gate

Measurement	Before	Retained path
One full tilt	2.486-2.960 s	0.944 s median, 0.928-0.950 s
Resident result	19.327 GB exact uint16	19.327 GB exact uint16
Estimated one-load peak	about 38.65 GB	about 28.99 GB
Full-volume decodes	two for a uint16 source	one
Full scan-major temporary	yes	no

The final five-run timing uses fresh processes, one at a time. Total executable launch averaged about 1.20 seconds, including roughly 0.25 seconds of SwiftUI/catalog startup outside the measured loader.

Parity is exact on the retained path:

• source dtype remains uint16;

• center-tilt maximum count is 1833;

• center-tilt unmasked count above 255 is 17,420,309;

• center-tilt selected-frame hash is ce268593ec0ac969 before and after;

• all seven tilt hashes and count audits are unchanged; and

• BF/ABF/DF use exact integer sums.

Mandatory prevention checklist

Before editing a CUDA, MPS/Metal, or WebGPU kernel:

1. Draw the production pass graph from file or resident input to the user-visible output. Include fallback and audit branches.

2. List every full-volume allocation, read, write, conversion, transpose, and readback with its byte count. Treat every extra 10-20 GB pass as a primary suspect.

3. Select dtype and scientific evidence policy before allocating the output. Never run a complete lossy/compact conversion merely to discover whether it can be retained. Fuse an exact audit into the native pass or reuse a trusted, invalidated-with-the-source audit cache.

4. Time process startup/catalog/index preparation separately from mapping, decode, products, layout, finalization, and total user wall time.

5. Search for synchronization inside submission loops. Prefer bounded pipelining and explicit dependencies when independent stages can overlap.

6. Compute both resident and peak transient memory. A topology that needs two full raw volumes is not acceptable merely because each kernel is fast.

7. Freeze parity before tuning: dtype, count range, corrected-frame hashes, product sums, frame order, and bad-pixel behavior.

8. Benchmark fresh and warm states separately, and run fresh-process trials sequentially. Concurrent index helpers can contend for the same cache and invalidate timing.

9. Optimize individual kernels only after the topology, synchronization, and memory-traffic audit is complete.

Bounded experiments

• Increasing the streaming batch to 32,768 frames reduced available pipeline depth and regressed wall time to 1.022-1.066 s; retain 16,384.

• The useful improvement came from removing redundant passes and overlapping stages. Smaller launch-shape changes were secondary.

• A compact uint8 browse path remains valid only for a native uint8 source or an explicit, already-trusted count-range contract. It must never silently replace exact uint16 microscope evidence.