# Testing and evidence

Run the smallest gate that can disprove the change first, then widen coverage.

## Portable Python

```bash
python -m pip install -e ".[dev,movie]"
npm ci
PYTHONPATH=src python -m pytest -q          # interactive tier, about a minute on one GPU
PYTHONPATH=src python -m pytest -q -m ""    # everything, including the slow SSB fits (about 12 minutes)
```

Plain `pytest` skips tests marked `slow` (SSB fits and long hardware checks,
each several seconds or more). Mark a new test `@pytest.mark.slow` when it
takes more than about 3 s. CI and every release run the full set.

`npm ci` installs the pinned JavaScript development tools from
`package-lock.json` (esbuild, TypeScript, jsfive, WebGPU types) into
`node_modules/`. The WebGPU contract tests bundle the TypeScript sources with
them and run the bundles in Node.js 22 or newer; they are skipped only when
Node.js is not installed.

Portable CI proves imports, public contracts, CPU references, source packaging,
and tests that do not require a physical accelerator. Backend skips are
reported; they are not parity evidence.

Validate the profiling schedule and retained run registry in the same portable
environment:

```bash
python scripts/check_profile_registry.py
python scripts/benchmark_registry.py validate
```

This checks that every parity capability has exactly one cell for CPU reference,
CUDA, Python MPS, native Swift/Metal, and WebGPU; unsupported cells stay
fail-closed; and terminal experiment manifests retain full revisions, input and
output hashes, timestamps, and a human registry row. It also checks every exact
benchmark gate, evidence import, runbook command, generated coverage table, and
measured-row timing/parity requirement.

## Choosing a reproducible performance gate

Do not begin from a command copied out of an old experiment. Ask the registry
for the open rows on the backend or computer you own:

```bash
python scripts/benchmark_registry.py next --platform "Python MPS"
python scripts/benchmark_registry.py next --computer "MacBook Air (M2, 8 GB)"
python scripts/benchmark_registry.py show io.mps.apple-m5-max-128gb.bin2.cold-original
python scripts/benchmark_registry.py command io.mps.apple-m5-max-128gb.bin2.cold-original
```

The command view prints the preflight, required environment, repository-owned
entry point, required artifacts, and promotion boundary. If it says
`parity preflight`, the command cannot produce a physical timing claim. If a
performance harness is missing, the coverage row stays partial or pending until
one is added.

The standard reusable entry points are:

| Operation | Entry point | Primary output |
|---|---|---|
| Python CUDA/MPS/CPU HDF5 load | `scripts/benchmark_hdf5_load.py` | Run-level p50/p95/max, full-volume hash/dtype/shape parity, logical resident bytes, sampled accelerator/process peak, and release snapshots |
| CUDA/MPS screening | `scripts/benchmark_screening.py` | Separate build and prepared-reopen distributions, product hashes, stages, cache bytes, and memory |
| Native Swift/Metal indexed load | `metal-4dstem-indexed-load-benchmark` | Exact resident volume/products, stage timing, hashes, and Metal/process memory |
| Hardware-WebGPU HDF5 load | `scripts/benchmark_webgpu_h5_browser.py` | Browser-local profile, checksums, run timing, and browser telemetry |
| Python MPS SSB | `scripts/benchmark_ssb_mps_scaling.py` | Reconstruction, parity-pair, and complete-fit JSONL |
| Native Swift/Metal SSB | `metal-ssb-benchmark` | Reconstruction, phase/loss, calibration, and cache-policy report |

Each physical run still needs a manifest and `RUNS.md` row in the private
evidence archive before launch; raw logs stay there, never in this repository,
and the dated summary (question, setup, numbers, conclusion) goes under `docs/`. The harness JSON is evidence, not permission to promote a dashboard
number without review.

## Native Swift and Metal

```bash
xcrun swift-format lint --strict --recursive \
  Package.swift \
  native/swift/Sources \
  native/swift/Tests \
  native/swift/Benchmarks
swift test
swift test -c release --filter MetalSSBKernelsTests
```

Opt-in real-source tests retain their fixture hashes and environment variables.
A simulator or compile-only result does not replace physical Metal execution.
The package-owned SSB benchmark accepts an exact metadata file, plane-major BF
source, independent phase reference, repetition count, cache policy, fit-trial
count, and fit-repetition count:

```bash
swift run -c release metal-ssb-benchmark \
  METADATA_JSON FULL_BF_U8 REFERENCE_PHASE_F32 7 full 200 3
```

Record preparation separately from warm reconstruction, exact loss, and the
complete fit. A finite cache budget is a different resource-policy row, not a
replacement for the complete-cache timing.

## Documentation

```bash
python -m pip install -r docs/requirements.txt
jupyter-book build docs
```

The documentation build is hardware-independent and never downloads private
data. It publishes retained evidence generated by qualified hardware runs.

## Backend parity

Follow [Cross-backend parity](../performance/parity.md). Re-run each unexpected
hardware mismatch twice on the same source/environment before deciding whether
it is deterministic drift or nondeterminism.

Never edit a frozen golden, expected hash, reference value, or tolerance merely
to make a new implementation pass.

Hardware timing cadence and promotion rules are defined in
[Continuous profiling](../performance/continuous-profiling.md). A PR smoke test
never becomes a physical-device performance claim.