uint32 flagged-pixel markers and the gold_512 convert size - 2026-10-08

Question

quantem-gpu convert gold_004_master.h5 wrote nothing: the public gold_512 master (512 x 512 scan, 192 x 192 detector, uint32, 4 flagged pixels) failed with “counts above 65535”. After that was fixed, the dry run reported the copy would be larger than the source (4.88 GB to 6.05 GB). Is that expected?

Cause of the refusal

convert loads with hot_pixel_correction="none" so that every stored value is kept. Arina writes 0xFFFFFFFF at every flagged pixel of a uint32 file, and the CUDA (io/encoded.py) and Metal (narrow_u32_u16 in bslz4.msl) narrowing refused any value above 65535, flagged or not. The unmasked maximum is 174.

Change

When uncorrected uint32 counts are stored as uint16, a value above 65535 at a flagged pixel is stored as 0 and counted. Every valid count and every other flagged value is the file’s; a valid count above 65535 is still refused. The pixel mask is saved with the copy and reads exclude flagged pixels, so no measurement changes. Metadata records flagged_markers_stored_as_zero, detector_mask_policy="flagged-markers-stored-as-zero" and file_counts_exact=False; the scientific processing list declares flagged_marker_zeroing (changes_measurements=False), and validate_declared_processing refuses a copy that omits it. convert’s verification checks the mapping value by value and that the count matches the record.

Size of the gold copy

Measured on GPU 0 (RTX PRO 6000 Blackwell) with 8192 sampled scans (every 32nd block of 512 scans):

Quantity	bits per value
HDF5 bitshuffle/LZ4 source (4.88 GB)	4.04
Encoded resident (6.05 GB): payload	4.87
Encoded resident: spatial index	0.14
Per-detector-pixel order-0 entropy	2.58
Entropy of one table for all pixels	4.04

Data property	Value
Zero fraction	0.44
Mean count	16.3
Values >= 32	21 % of all values
Detector pixels with more than 1 % of values >= 32	8168 of 36864

The integer codec (resident/cuda/kernels/streamed.cu) codes symbols 0 to 31 with one of 64 fixed rANS tables chosen per pixel and 512-scan interval; a count of 32 or more is an escape symbol followed by two raw bytes. The gold bright-field disk holds counts up to 233, so 21 % of values cost at least 16 bits: 3.4 bits per value from escapes alone. The copy is therefore larger than the source, and convert keeps the HDF5 file. This is expected for dense, high-count bright-field data; the size rule was not changed.

Not changed

• The size rule (keep HDF5 when the copy is not smaller).

• The Metal kernel change is not run on Apple hardware in this session; run QEM_TEST_BACKEND=mps pytest tests/hardware/test_uint32_flagged_master.py tests/hardware/test_qem_collection.py on a Mac.