Data Transfer Storyboard

Data Transfer Storyboard#

Data transfer prepares large microscopy sessions for fast loading by copying or splitting files across target folders and disks. It is intentionally not a viewer widget right now. The supported interfaces are the quantem data-transfer CLI and the Python utilities under quantem.widget.io.

Keep this workflow separate from ShowFolder, Show2D, Show3D, and Show4DSTEM. Those widgets inspect scientific data after it is already placed. Data transfer only plans, verifies, copies, resumes, and records where files live.

DT-1: Plan a Multi-Disk Session#

User story: A microscopist has 60-80 time-resolved 4D-STEM masters for future ptychography or browsing and wants to split the session across fast NVMe disks before loading it into one or more NVIDIA GPUs.

Primary tools: quantem data-transfer plan, plan_data_transfer, write_data_transfer_manifest.

Acceptance checks:

  • Discovery finds every *_master.h5 without loading detector frames or using GPU memory.

  • Each master stays grouped with its matching *_data_*.h5 sidecars.

  • The plan reports logical ID, source path, target path, source disk, target disk, file count, and bytes.

  • The default strategy balances by acquisition size, not by file count.

  • Planning writes a JSON manifest by default.

  • Dry-run planning never creates target files.

  • Real-data signoff uses private workstation data; raw files are never added to the repository.

DT-2: Copy Safely#

User story: A user reviewed a plan and wants to copy files without corrupting existing targets or partially written outputs.

Primary tools: quantem data-transfer copy, copy_data_transfer, filter_data_transfer_plan.

Acceptance checks:

  • Copy defaults to dry-run unless the CLI receives --execute.

  • Copy writes to *.partial files first, then atomically replaces the final path.

  • Existing targets with matching verification are skipped as already present.

  • Existing targets with different content fail with a corrective error unless overwrite is explicit.

  • Size verification is the fast default; SHA-256 is available when stronger verification is worth the extra reads.

  • Result rows report planned, copied, or exists per physical file.

DT-3: Resume and Verify#

User story: A long transfer is interrupted by a notebook restart, network hiccup, or disk quota problem. The user wants to resume without guessing which files are safe.

Primary tools: quantem data-transfer inspect, read_data_transfer_manifest, inspect_data_transfer, summarize_data_transfer.

Acceptance checks:

  • Manifest reload is enough to inspect state; discovery does not need to rerun.

  • State reports not-started, partial, exists, mismatch, or missing-source.

  • Stale *.partial files are visible and retryable.

  • A filtered plan can copy only pending or partial files.

  • Inspect output is concise enough to paste into a notebook, issue, or lab log.

DT-4: Append New Completed Masters#

User story: A microscope or processing pipeline writes new masters while the session is being prepared. The user wants to append complete groups without changing existing target assignments.

Primary tools: quantem data-transfer update, update_data_transfer_plan, then manifest write/inspect/copy.

Acceptance checks:

  • Newly discovered masters are ignored until readiness checks pass.

  • Existing logical IDs keep their target assignments.

  • New groups use the original target strategy.

  • Repeated updates do not duplicate skipped or not-ready groups.

  • Removed or failed files do not become valid acquisitions silently.

  • The CLI rewrites the same manifest only after producing a valid updated plan.

DT-5: Open Viewers After Transfer#

User story: After files are copied, the user wants to browse them with ShowFolder or load ready target masters with Show4DSTEM without remembering the exact target paths.

Primary tools: quantem data-transfer masters, quantem data-transfer show4dstem, target_masters, ShowFolder(target_folder), and Show4DSTEM(load(...)).

Acceptance checks:

  • Viewer opening is a separate explicit step, not a data-transfer control.

  • Transfer timing and viewer timing are measured separately.

  • Show4DSTEM still relies on its own lazy paging and GPU memory controls.

  • Data-transfer code does not import or construct viewer widgets.

  • target_masters(plan) returns only complete target master groups by default.

  • quantem data-transfer masters --all-masters can print planned target paths before copy completion for lab-log review.

  • quantem data-transfer show4dstem --gpus 0,1 --dtype u8 --bin 1 writes a notebook whose code explicitly uses load(masters, devices=[0, 1], dtype="u8", det_bin=1) and Show4DSTEM(...).

  • The generated notebook reloads the manifest and prints warnings so saved outputs record whether the handoff was complete and multi-disk.

DT-6: Guard Disk-to-GPU Assumptions#

User story: A user expects two GPUs to make browsing fast. They need the tooling to say when a transfer layout still feeds both GPUs from one disk.

Primary tools: data_transfer_load_warnings, quantem data-transfer inspect, quantem data-transfer show4dstem.

Acceptance checks:

  • Warnings appear when not all acquisitions are complete.

  • Warnings appear when multiple target roots resolve to one physical disk.

  • Warnings appear when multiple GPUs are requested but ready targets do not span multiple physical disks.

  • Warnings are advisory and do not block masters --all-masters or dry-run inspection.

  • Real-data signoff records source disk, target disks, requested GPUs, browse dtype, detector binning, cold-load time, warm-load time, and GPU memory before and after.

Required Test Commands#

Run these before committing data-transfer changes:

PYTHONPATH=src pytest -q tests/test_io_data_transfer.py
PYTHONPATH=src pytest -q tests/test_cli.py::test_data_transfer_cli_plan_inspect_copy_update_and_show4dstem
PYTHONPATH=src pytest -q tests/test_hdf5_disk_scheduling.py

For real workstation signoff, run the CLI against private MJGOAT/NVIDIA data and record source disks, target disks, bytes, elapsed copy time, and verification mode. Keep those raw files and generated benchmark artifacts out of Git.