Benchmark methodology

Benchmark methodology#

Performance is reported only after scientific parity. A faster result that changes source coverage, detector sampling, precision, mask, objective, or output meaning is a different experiment.

The continuous profiling plan defines which checks belong on every pull request, which require specified physical hardware, and how a qualified measurement is promoted to the dashboard. Its machine-readable platform/module schedule is benchmarks/profile_matrix.json.

Required labels#

Every benchmark records:

repository path, branch, HEAD, and dirty-diff hash;
operating system, driver/runtime, device model, and dependency versions;
source identity, byte hash, compressed bytes, decoded bytes, shape, and dtype;
scan/detector region, scan/detector bin, mask, accumulation/output dtype;
exact command, environment variables, seed, and cache-preparation state;
output hashes and parity metric; and
process memory, accelerator memory, total-card occupancy, pressure, and swap.

Source and cache states#

Report these separately:

Label	Meaning
Cold source	First encounter after a documented storage-cache reset or reboot; process and kernel caches are also identified
Controlled uncached source pages	A platform-specific page-reuse control such as macOS `F_NOCACHE` is applied to every declared source descriptor; source audit, index, process, and destination state are still reported separately
Warm source	Raw source reopened with operating-system page cache or storage cache available
Warm process	Same process and reusable allocations/kernel compilation retained
Saved-result reopen	A derived cache or persisted resident payload is opened; not a raw-source load

If true cache eviction was not controlled, use first process or first observed source rather than cold.

Controlled source-page IO is not automatically arbitrary-source cold. For example, a run may use F_NOCACHE on an immutable source while reusing a sealed value-range audit, or it may build a new QH5 index while the source itself is already qualified. Name each retained state instead of compressing them into one “cold” label.

End-to-end stages#

User-facing load time is wall clock from the initiating action to the first complete usable scientific product. Profile at least:

file open and metadata/discovery;
index lookup or construction;
read-span planning and host allocation;
storage read and header parsing;
decode/decompression;
dtype conversion, masking, and bin decision;
allocation and resource-pressure admission;
detector/scan reduction and layout conversion;
GPU transfer where memory is not unified;
first complete BF/ADF/DF/CoM/DPC product;
cache write/finalization; and
total wall time.

Overlapped GPU intervals are not summed and presented as wall time. On unified memory, page-in and GPU access may be inseparable; report that fact instead of inventing an upload stage.

Statistics#

Use a smoke run before any matrix. For accepted configurations, report enough repetitions to provide p50, p95, and maximum wall time. Preserve the run-level records so initialization outliers and cache effects remain visible.

Kernel microbenchmarks are useful for diagnosis but are not user-facing load time. End-to-end application evidence remains required.

Timing regressions are evaluated only within an exact comparison key: protocol, module/platform cell, source identity and plan, cache state, timing boundary, device/runtime, precision, and source revision. A new key begins in report-only mode. Do not introduce a blocking percentage threshold until at least five accepted sessions establish the variance of that exact configuration.

Memory#

Compressed file size is not a fit estimate. Admission includes decoded output, decoder scratch, reduction/layout buffers, products, allocator reserve, cache population, and concurrent-service baseline.

Every load row separates:

source, requested, working, accumulation, and resident/output dtype;
resident payload bytes calculated from the recorded output shape and dtype;
planner-estimated peak and its included/excluded allocations;
measured process RSS/footprint and host peak;
measured accelerator allocated/reserved peak and total-device occupancy; and
memory pressure and swap where the platform exposes them.

Do not call a payload size “peak memory.” Do not call uint8 lossless unless a complete source-identity-bound audit records the corrected maximum and zero values above 255. A saturating uint8 run records its saturation count and is labeled browse-only.

On Apple unified memory, record four separate layers:

logical resident payload, calculated exactly from output shape and dtype;
Metal-driver allocation, sampled during the timed interval and again after output release;
process RSS/footprint, which does not necessarily include every direct Metal allocation; and
whole-system pressure and swap before, during, and after the run.

torch.mps.current_allocated_memory() covers Torch-managed allocations. It may remain zero for buffers created directly through Metal/PyObjC; in that case it must not be presented as total accelerator memory. Record torch.mps.driver_allocated_memory() or an equivalent Metal counter alongside RSS, and name the counter precisely.

A repeated-load protocol must explicitly release every caller-owned direct Metal output before the next repetition. Clearing a framework cache or deleting the Python wrapper is not proof that a newBuffer... allocation was released. Record the Metal-driver allocation after output release and fail the run if it grows across repetitions without an intentional cache explanation.

On CUDA, record process allocated/reserved VRAM and total-card occupancy before, during, and after the run. On WebGPU, browser-process RSS is a useful host signal but is not a complete GPU-device allocation measurement; an 8 GB gate also requires whole-system pressure/swap and the physical device run.

CUDA compressed-resident memory#

For lossless packed and ANS sources, record these quantities separately:

Memory kind	Meaning
Used source payload	Encoded bytes actually occupied; exclude unused capacity
Resident metadata	Offsets, modes, bit widths, model tables, validity and checksums retained on device
Interaction indexes	Exact summaries or checkpoints needed by the measured query path
Logical resident layout	Sum of used source, retained metadata and indexes; state table sharing and padding assumptions
Allocator live high-water	Maximum simultaneously live bytes tracked by the named allocator during the measured interval
Allocator reserved high-water	Maximum reservation, including reusable free blocks; separate pools by owner and lifetime
Incremental process/device peak	Extra actual allocation above a stable, recorded baseline, including memory outside the allocator
Whole-device sampled peak	All occupants plus this operation; identify sampling cadence and gaps
Host peak	Process RSS and pinned staging, with their counters and overlap stated

Peak is the maximum simultaneous footprint, not the sum of stage peaks. Do not add live allocations to the reservation containing them. Do not add active model bytes to scratch that already contains their allocated capacity. If independent-pool maxima occur at different times, their sum is an upper bound rather than a measured simultaneous peak. Persistent source growth, fragmentation and retained allocator blocks can all affect the final admission requirement.

A benchmark that reserves a common arena for every codec measures the common arena. It cannot infer a per-codec peak difference from those identical reservations. Record the difference as unmeasured until standalone ownership and allocations are profiled. Similarly, summing used block sizes estimates a resident layout; streaming every block through small scratch does not measure the peak of keeping the whole source resident.

For ANS-to-packed conversion with independent output ownership, retain and count both source and destination at the overlap peak. In-place conversion, source release and replacement are separate ownership contracts. Avoiding a dense intermediate does not eliminate the two encoded representations’ overlap.

Measure the entire source-to-ready interval and its components: compressed input staging, native decode, encoding, index construction, first exact product and output handoff. Repeat load/release cycles, synchronize completed device work before interpreting a release, and record post-release live/reserved baselines. A smaller payload does not by itself establish a lower load peak or support for a particular GPU capacity.

See the CUDA count-codec investigation for an example that explicitly keeps calculated resident storage separate from measured shared scratch and unmeasured full-load peak. Missing peak counters stay null in its downloadable evidence; they are not zeros or inferred fit guarantees.

Minimum-device memory gates#

Minimum-device support is a complete-pipeline claim, not a payload comparison:

CUDA floor: 6 GiB of dedicated VRAM. Count process allocation and reserve, decoder/reduction scratch, products, staging, and other card occupants.
WebGPU floor: 8 GB of total physical laptop RAM. Count the operating system, browser, JavaScript heap, staging, GPU buffers, presentation, memory pressure, and swap.
Apple native/MPS floor: 8 GB of unified RAM when that row is claimed. Record the same whole-process and system-pressure signals as WebGPU.

Gate vocabulary is fail-closed:

Status	Meaning
✓	Complete headed or native physical-device run at or below the floor, with parity and peak-memory evidence
Pending	Payload is a plausible candidate, but complete physical peak or parity evidence is missing
No	Payload alone exceeds the floor, or the complete run exceeds it
Test	A capped larger device or software adapter passed as a pre-check; physical floor signoff is still missing

Do not convert Pending or Test to ✓ from a larger device, a calculated payload, a kernel-only microbenchmark, or a prepared-source reopen.

Interaction sidecars and scientific resolution#

A detector-binned interaction sidecar is a distinct scientific sampling plan, even when it is built automatically after a native-detector load. Its speed may be reported only with the sidecar detector bin and output meaning. It cannot be used as parity evidence for native-detector CoM, DPC, iDPC, a diffraction pattern, or another resolution-sensitive product.

If the public API promises native resolution, parity must exercise the full-resolution reducer. If a client chooses the interaction sidecar, metadata and UI provenance must identify its detector bin; the application must not present the result as native resolution.

Acceptance#

An optimization is retained only when:

strict parity passes on the unchanged fixture and parameters;
the intended physical device shows a reproducible wall-time or memory win;
tails and responsiveness do not regress;
provenance remains complete; and
the implementation does not create a backend-specific public API.

A faster sidecar, prepared index, saved result, cropped scan, or detector-binned representation never inherits the acceptance state of the native source plan.

Rejected experiments stay in the optimization ledger. Every scheduled or diagnostic run also keeps a machine-readable manifest and a terminal row in the RUNS.md registry of the private evidence archive.