4d — Import

Purpose

The entry point of the Prepare workflow. It reads a raw 4D-STEM dataset from an HDF5 file, optionally applies flat-field (gain) and problematic/dead-pixel correction, always applies a hot-pixel threshold, and writes the result to a new, downstream-compatible HDF5 working file (fd_stack) in the 4d stem_dataset v2.0 format. The output is BLOSC2-compressed and is the file every later script operates on.

Parameters

Algorithm

1. Re-registers the HDF5 BLOSC/BLOSC2 filter plugins against the actual libhdf5 (avoids a known HDF5 1.14 crash) before any compressed I/O.
2. Resolves the input file, auto-derives the fd_stack name if empty, and captures source metadata to carry forward (root and header attributes, everything under positions, and auxiliary detector datasets such as flatfield_mask, deadpixel_mask, vacuumProbe).
3. Locates the diffraction patterns (trying /data/dpatterns, /entry/data/data, …) and loads them.
4. Hot-pixel thresholding: if the threshold > 0, all voxels above it are set to 0 and the count reported.
5. Optional correction (only when enabled): loads the gain reference (float32) and bad-pixel mask (bool); if their shapes disagree the mask is dropped. Correction runs on GPU via applyFlatfieldCorrection_GPU_batched, falling back to CPU on error; 4D stacks are reshaped to (N, qy, qx) for the call.
6. Writes the output HDF5 in stem_dataset v2.0 form.

Results

Images: none.

Data written: a new fd_stack HDF5 file — root attrs (file_type='stem_dataset', format_version, time_created, comments); header group (with gain_corrected); data/dpatterns (corrected/cleaned stack, BLOSC2 lz4, per-pattern chunks) plus carried-over auxiliary datasets and masks; and the positions group if present. The results file records fd_stack and registers the input, output, gain reference, and mask images.

Purpose

A diagnostic tool. It recursively walks the entire structure of both the 4d-style working file (fd_stack) and the original raw file, printing every group and dataset with its shape, dtype, compression/filter details, and all attributes. It optionally computes basic grey-value statistics and writes the structural metadata to the results file. It opens the files read-only and modifies nothing.

Parameters

Algorithm

1. Recursively explores the file: logs all attributes of each group; for each dataset prints shape, dtype and decoded compression, mapping HDF5 filter IDs (gzip, shuffle, lz4, blosc, blosc2 = id 32026) to names and options.
2. If statistics are enabled, computes numpy min/max/mean/std per dataset; the values from /data/dpatterns become the reported global statistics.
3. For any dataset with three or more dimensions, derives ProbePositions, detectorX, detectorY.
4. Runs on fd_stack first, then the original raw file.

Results

Images: none.

Data written: to the results file only — ProbePositions/detectorX/detectorY, a full set of attr_<path>_<name> lines capturing every attribute, statistics if enabled, and image registrations for the two files. The HDF5 files are not modified.

Purpose

Computes the core statistics and reference images for the working file and, crucially, auto-detects the diffraction-disk geometry. It reads the full pattern stack, computes min/max/mean/std, produces average / clipped-average / standard-deviation diffraction maps, locates the central bright-field disk to derive the Ronchigram centre and suggested BF/DF detector radii, and writes all of this back to the file and to the GUI. Running it first gives good detector-geometry defaults for scripts 70 and 75.

Parameters

Algorithm

1. Opens fd_stack read-write and locates the pattern dataset (dpatterns_blosc2 / dpatterns / …).
2. Resolves the shape (4D → scan-y, scan-x, detector-y, detector-x; 3D → positions, detector-y, detector-x), warns if the dataset exceeds ~8 GB, and loads it.
3. Computes min/max/mean/std over the whole array.
4. Computes the average diffraction pattern (mean over all positions), the standard-deviation pattern, and a contrast-clipped average.
5. Runs find_disk_center on the average pattern to locate the central beam; from the disk centre and radius it derives RonchiCenterOffsetX/Y (centre − half-size) and radii DF_Inner = 1.1·r, DF_Outer ≈ 2·r (at least DF_Inner+20), BF_Outer = 0.9·r.
6. Builds a log-scaled 256-bin grey-value histogram.

Results

Images: fd_average_map, fd_average_clipped_map, fd_std_map (each as .mrc + .png) and fd_grey_value_histogram.png; plus an interactive disk overlay if showplot=y.

Computed values written: into the fd_stack header — the statistics, centerX/centerY (the Ronchigram offsets), and DF_Inner/DF_Outer/BF_Outer; the same values and image registrations go to the results file / GUI.

Purpose

Attaches the probe scan pattern to the working file. It reads a scan pattern from a STAR file (system-wide, project-wide, or local), optionally prepends dummy positions, and writes the positions into fd_stack as /positions/units (which SHARP and downstream tools read), along with scan-mode and step-size metadata.

Parameters

Algorithm

1. Selects the STAR file per starfile_choice.
2. Reads the scan pattern, obtaining the position list plus ScanMode, step size, position count, and snake offset.
3. Optionally plots the pattern (honouring the plot count and Y direction).
4. If a dummy count is set, prepends that many copies of the first position.
5. Writes the positions into fd_stack as /positions/units (in [Y, X] order), replacing any existing dataset.
6. Writes ScanMode and step_size_angstroms as header attributes.

Results

Images: an interactive scan-pattern plot only, if enabled (not saved to disk).

Data written: into fd_stack the /positions/units dataset and header attributes ScanMode and step_size_angstroms; the results file records the scan mode and step size and registers the file and STAR file.

Purpose

Forms virtual bright-field (BF) and dark-field (DF) STEM images from the 4D dataset. Using the Ronchigram centre and BF/DF detector radii, it integrates each diffraction pattern over a central disk (BF) and an annulus (DF), then maps those per-probe intensities onto a real-space image using the scan positions. It requires the scan positions written by script 50.

Parameters

Algorithm

1. Verifies BLOSC2 with a write/read self-test.
2. Loads the scan positions from fd_stack (aborts if none); flips Y if requested.
3. Locates and loads the pattern dataset (reshaping 4D to (N, qy, qx)) and computes statistics.
4. Builds the absolute Ronchigram centre and two masks: a DF annulus between DF_Inner and DF_Outer, and a BF disk out to DF_Inner − 10.
5. Virtual-detector integration: for every probe position, sums pattern × mask to get its DF and BF intensity.
6. Real-space mapping: builds a DF_size × DF_size grid over the scan-coordinate range, estimates the probe spacing with a KD-tree, and interpolates the DF/BF values with a piecewise-cubic (Clough–Tocher) interpolator, filling points outside the convex hull with the nearest value. A multiprocessing nearest-position fallback is used if SciPy is unavailable.
7. Normalises each map for MRC and writes 400×400 PNG previews.

Results

Images: fd_DF_image and fd_BF_image (each .mrc + .png); an interactive mask display if enabled.

Data written: into fd_stack the datasets /data/df_map and /data/bf_map (float, gzip) plus header attributes (statistics, centre offsets, DF_Inner/DF_Outer, dataset info); the same values and the four image files are recorded to the results file.

Purpose

Estimates the detector/probe point-spread function (PSF) from isolated diffraction spots in the dark-field region. It searches each diffraction pattern for isolated intensity peaks lying within the DF annulus, extracts small patches centred on those peaks, and averages them into a single PSF image — characterising how a point diffraction event is spread on the detector (useful for downstream deconvolution/reconstruction). It requires the scan positions written by script 50.

Parameters

The GUI parameter list is the same as Get BF and DF image. The parameters that actually drive the PSF are RonchiCenterOffsetX/Y, DF_Inner and DF_Outer (they define the annular region searched for peaks); scan_y_down flips the Y scan coordinates; fd_stack is the input/output file. DF_size, BF_Outer and mask_showplot are parsed but not used in the PSF path. The statistics, ProbePositions and chunks are recomputed outputs.

Algorithm

1. Loads the scan positions from fd_stack (aborts if absent); flips Y if requested.
2. Locates and loads the pattern dataset (reshaping 4D to (N, qy, qx)) and computes statistics.
3. Builds the DF annular mask between DF_Inner and DF_Outer around the centre.
4. Peak search + patch extraction over all patterns: on GPU (PyTorch, CUDA/MPS) it masks each pattern, thresholds at 0.1× its masked maximum, does non-maximum suppression (minimum peak distance 8), and cuts 16×16 patches around surviving peaks, with half-precision and OOM-safe batching (gated by the FD_GPU_PEAKS environment variable); the CPU fallback uses skimage's peak_local_max with the same settings in a thread pool.
5. Averages all 16×16 patches into the final PSF.

Results

Images: psf_darkfield.mrc (or .npy fallback) and psf_darkfield.png (normalised, upscaled to 400×400). If no peaks are found, no PSF is written.

Data written: into the fd_stack header the statistics, centre offsets, DF_Inner/DF_Outer and dataset info (the PSF image itself is saved only as MRC/PNG, not into the HDF5); the results file records the statistics and registers the PSF files.