Native Direct Ptycho preparation, 2026-11-05#
The complete five-second native-app target is not yet established. These measurements concern Swift/Metal; they do not qualify the Python MPS or CUDA implementations.
Workload and reference#
Apple M5 Max, 128 GiB unified memory, AC power. Full 512 × 512 scan positions and 192 × 192 detector pixels, original integer counts, 200 search trials, both estimated-rotation branches, fifth-order refinement and the existing half-band consistency check. Source calibration, BF selection, precision, search settings and fallback rules remain fixed within each comparison.
The private experiment ID is live4dstem-rc2-optimization-20261105. Its frozen
backend control is 99bf9026, based on 373725d8 with the preceding native
integration changes preserved. Raw source identities, command receipts,
patches, arrays and app screenshots remain in that private archive.
Verified preparation improvements#
Fuse the final FFT transpose with the Hermitian-cache write, retaining exactly the original Fourier values.
Fuse the integer conversion with the first 512-point FFT stage. Threadgroup staging preserves the original arithmetic for uint8, uint16 and uint32. A register-only alternative failed exact parity and was rejected.
Gather cached spectra in larger batches and traverse the existing power buffer contiguously. Each frequency retains its original Double summation order. Partly cached and fully streamed cases retain bounded batches.
Decode QEM columns with contiguous stores through a bounded shared tile. Specialize native-width output to avoid unnecessary row-padding indexing. Padded scans use the same exact count decoder with the original layout.
Matched warm Fourier preparation with the first fusion already enabled fell from 0.59–0.63 s to 0.54–0.56 s across three acquisitions. Model preparation fell from approximately 0.26–0.33 s to 0.15–0.24 s. These are isolated stages, not total interaction times.
For two QEM acquisitions with 9,081 and 9,110 selected detector columns, interleaved column-decoder measurements gave:
Selected-column GPU decoding |
Original |
Coalesced stores |
Native-width |
Parallel chunks |
|---|---|---|---|---|
Acquisition A |
1.88–1.90 s |
1.45 s |
1.25 s |
0.331 s |
Acquisition B |
1.89–1.91 s |
1.47 s |
1.28 s |
0.333 s |
All returned column hashes match the frozen original. The reusable runner is
metal-qem-detector-update-benchmark --columns acquisition.qem pixels.json 3;
the JSON pixel list contains detector indices in the requested order.
It hashes all returned counts outside synchronized GPU timing and checks
original diffraction samples independently. The parallel-chunk comparison uses
three repetitions per condition, interleaved serial/parallel/serial. Independent
chunks write disjoint positions of the existing bounded 32-plane output buffer;
the ANS stream order and integer counts are unchanged. An argument table adds
32 bytes per chunk (1 KiB for these acquisitions, bounded at 4 KiB), included in
resident accounting. The Metal allocator reports the same allocation totals;
this is not a claim of zero extra metadata bytes or a 16 GiB working set.
Synthetic tests additionally cover partial restart blocks, multiple chunks starting inside a scan row, flagged pixels, repeated indices, scan padding, and zero, constant, sparse, entropy and literal count streams.
Correctness and acceptance boundaries#
QEM stored moments include flagged counts, whereas ordinary selected-column
reads mask them. includeFlaggedPixels: true explicitly retrieves original
counts for exact moment replacement; the default still masks them. Neither
the source nor its validity mask is changed. Native app rotation now uses the
original flagged values and verifies the remaining independent valid total.
Full 200-trial, fifth-order benchmark results on three acquisitions match the frozen fitted parameters, branch decisions, objectives and consistency checks exactly. Independent count tests and native direct/Phase/Complex image comparisons are separate gates. Format-specific BF estimation differences must be recorded; matched comparisons use explicit session-only geometry, without modifying source calibration.
Small-batch and eight-candidate layouts, paired Double normalization, additional zero-support culling and several decoder layouts preserved values but failed to establish a speed improvement; they are archived, not retained as production tuning choices. GPU and host timings drift under normal desktop conditions. Report first observations separately from repeated timings, and do not call a warm minimum a cold result.
The first cleaned default passed 17 focused backend tests and seven native reconstruction journeys plus three native binding-driven slider journeys. The registered final candidate passed 15 uninterrupted native journeys (three repetitions for five acquisition/format combinations), with exact initial/final fits, exact initial-method images and positive first/final drawable receipts. The first repetition of each combination also checked all three reconstruction methods. Two injected memory-pressure recovery journeys passed.
Matched serial/parallel/serial native Fourier preparation was 1.801/0.639/1.828 s for acquisition A and 1.920/0.688/1.920 s for acquisition B. All fits and method images matched. This supports a roughly 2.8-fold preparation improvement; it is not a whole-pipeline speedup. Lazy compilation now builds only the requested column layout and the separately requested totals pipeline.
The final three-repeat native measurements are:
Acquisition/source |
Run to checked image, median (range) |
Launch to checked image, median (range) |
|---|---|---|
A / HDF5 |
7.85 (5.11–9.18) s |
10.27 (7.84–11.81) s |
A / QEM |
7.09 (4.93–8.94) s |
8.62 (7.42–10.33) s |
B / HDF5 |
11.26 (6.61–13.35) s |
14.06 (9.26–15.90) s |
B / QEM |
11.17 (9.18–13.75) s |
12.57 (10.79–15.15) s |
C / HDF5 |
8.96 (5.22–9.37) s |
11.31 (7.79–11.83) s |
Native test hooks request Run and Refine in separate immediately consecutive actions. Their action gap and consistency checks are included; screenshots occur after timed work. The source page-cache state is unspecified. The macOS thermal state moved from nominal to fair during the run, coincident with increased fit time. GPU clocks were not measured, so this does not establish the sole cause of timing drift. Fourier preparation remained comparatively stable. These are native-window tests, not physical mouse or trackpad acceptance.
Acquisitions A and B retain the original standard-range fallback and untrusted higher-order flag; C retains the wide-range result. Exact parity includes these decisions. A completed check is not the same as trusted physical coefficients.
The full Swift backend suite passed 311 tests with 31 fixture-dependent skips and zero failures. Frozen CUDA image references and native full-count audits are separate acceptance gates; their completion is recorded in the private archive. Loading, first image, checked fifth-order result, cache interactions and positive drawable presentation must be reported separately. An active-stage sum that excludes loading or the Run-to-Refine action gap is not open-to-final latency. No release, cross-device speed claim or 16 GiB memory guarantee follows from these measurements.