4d — SHARP

Purpose

The first calibration step. It builds a position-averaged CBED (PACBED) from the raw diffraction stack, detects the bright-field (BF) disk, and derives the reciprocal/real-space calibration every downstream step needs — detector cover angle (DCA), reconstructed pixel size, BF-disk diameter, electron wavelength, and the Ronchigram centre offset. Optionally it computes a per-pattern centre-of-mass "descan" so beam wander is removed before calibration.

Parameters

Algorithm

1. Sums up to the first 10 000 diffraction patterns into a single PACBED image.
2. Detects the BF disk: threshold at half-max, trace the longest contour, take its centre of mass and mean radius → pacbed_bf_disk.
3. DCA = CSA·(detector half-width / BF-disk radius); wavelength from KV; reconstructed pixel size = λ / (2·DCA).
4. Records the BF-disk centre minus the geometric detector centre as RonchiCenterOffsetX/Y.
5. Optional descan: compute each pattern's centre-of-mass offset, store it, re-centre the frames, and re-measure DCA / pixel size / disk on the aligned PACBED.

Results

Image: pacbed.png — a sample frame and the PACBED with the detected centre and BF disk overlaid (three log-scaled panels including the descan offset scatter when descan is on).

Data written: results/GUI values pacbed_DCA, pacbed_PIX_itr, pacbed_bf_disk, pacbed_wavelength, RonchiCenterOffsetX/Y; and (descan mode) the HDF5 dataset data/descan (per-pattern offsets).

Purpose

Splits a multi-pass acquisition into its individual scan frames by detecting the scan fly-back (X-direction reversal) and tags every scan position with an integer frame index, so drift correction and reconstruction can treat each pass over the field of view as a separate temporal layer. It uses the scan positions only — no diffraction data is read.

Parameters

Algorithm

1. Loads positions/units and histograms position-to-position steps to find the typical in-layer step.
2. Marks frame boundaries where the X step changes sign (within a frame, X is monotonic).
3. Filters out spurious intra-frame reversals (a "sandwich" test against the typical frame length), then labels each run of positions with an incrementing frame index.

Results

Image: frame_id.png — one panel per detected frame, positions colour-coded.

Data written: the HDF5 dataset mask/frame_id (per-position integer frame index). No results-file values.

Purpose

Converts the scan step from arbitrary units to Ångström by matching an imaged Quantifoil/carbon hole to its known diameter. It forms a bright-field STEM image, detects the hole edge, fits a circle, and derives the units→Å scale used for correct real-space scan sampling everywhere downstream. It depends on the BF-disk geometry and Ronchi offsets from step 10.

Parameters

Algorithm

1. For every position, sums the pixels inside the BF disk (radius 0.45·pacbed_bf_disk, centred on the Ronchi offset) → a BF-STEM intensity (ice-filled hole bright, carbon dark).
2. Thresholds at scan_percentile, keeps bright points adjacent to a dark neighbour, and places edge points at the bright→dark midpoint.
3. RANSAC-fits a circle to the edge points → hole centre and diameter (in units).
4. units2ang = hole_size·1e4 / diameter; scales all positions to Å.

Results

Image: ice_hole_calibration.png — the BF map with the fitted circle, centre, diameter, and a set-vs-measured annotation.

Data written: HDF5 positions/angs (positions in Å), data/bfvalues, and a hole attribute; results/GUI scan_hole and units2ang.

Purpose

Estimates a single defocus value at the hole centre by fitting a CTFFIND4-style Ronchigram CTF (via the bundled ctf_lmu.py) to the raw patterns of a small central region. It provides a robust starting defocus for ptychography and is the first quantitative defocus output once the geometry is calibrated.

Parameters

Algorithm

1. Selects scan positions within defocus_ctf_fraction·(hole radius) of the hole centre; their raw patterns feed the fit.
2. Optionally descans each pattern (using the shared data/descan when present) so the BF disk is centred.
3. Computes a Gaussian-windowed incoherent power spectrum and fits a CTFFIND4-style Ronchigram CTF to the mean spectrum for a global defocus (plus per-cluster fits for spread).

Results

Image: defocus_ctf.png — the ROI map, the log Ronchigram with the fitted defocus, and a radial profile with the CTF model overlaid.

Data written: the results/GUI value defocus_ctf (Å). No HDF5 writes.

Purpose

Defocus need not be computed over the whole field of view. This lightweight step lays out square patches on a ring around the hole and tags each scan position with the patch it belongs to, so the next step (parallax) measures defocus only inside patches and interpolates between them.

Parameters

Algorithm

1. Places the patch centres at equal angular spacing on a circle of radius defocus_ring·(hole radius) around the hole centre.
2. Each patch is a square of side defocus_patch_size; every position inside a patch is tagged with its 1-based patch index.

Results

Image: defocus_patches.png — the BF map with the ring and the numbered patch squares.

Data written: the HDF5 dataset mask/patches/parallax (per-position patch index).

Purpose

Measures the local defocus inside each patch using py4DSTEM tilt-corrected bright-field (parallax) reconstruction, then fits a tilted plane through the per-patch values and evaluates it at every scan position, giving a smooth per-position defocus map. Its output data/defocus/parallax is consumed by the ptychography step.

Parameters

Algorithm

1. Resamples each patch's irregular scan onto a regular square grid (each cell takes its nearest scan point's full pattern — never blended, so the sub-pixel disk shift is preserved).
2. Runs py4DSTEM parallax on each patch: virtual tilt-corrected BF images, coarse-to-fine cross-correlation alignment, then an aberration fit for the defocus (C1).
3. Fits a plane d = a·x + b·y + c through the patch defoci (constant mean for 1–2 patches) and evaluates it at every scan position.

Results

Images: defocus_map.png (per-patch field and the per-position plane fit), convergence_table.png, and per-patch diagnostics patch_<id>_*.png when metadata is saved.

Data written: HDF5 data/defocus/parallax (per-position defocus, Å, with plane_a/b/c attributes); parallax.json; per-patch defocus and convergence-score values to the results file.

Purpose

Corrects specimen/stage drift before reconstruction. It reassembles the diffraction patterns into per-layer bright-field "shadow" montages, treats the temporal layers as a movie, and aligns them with MotionCor3 — exactly as dose-fractionated cryo-EM movies are aligned. The per-frame drift is interpolated onto every pattern by acquisition time and applied to the scan positions, iterating to shrink the residual drift.

Parameters

Method: drift_method (global = one rotation + conversion factor from shadow sharpness; local = per-position conversion-factor map via KMeans + RBF; CTF = per-position defocus map from a Ronchigram-CTF fit — local/CTF also write a defocus map), drift_clusters, drift_ctf_device, scan_hole_size.

Shadow montage & search: shadow_defocus, shadow_res1/shadow_res2 (sharpness band), the rotation search (shadow_rotation, shadow_rotation_shortcut, shadow_min/max_rotation, shadow_number_rotation), the conversion-factor search (shadow_cf, shadow_cf_shortcut, shadow_cf_range, shadow_number_cf), shadow_njobs, and the search crop (shadow_crop_doit, shadow_crop_nm, shadow_crop).

MotionCor3: mc3_niter (outer refinement passes), mc3_patch, mc3_bft, mc3_Iter, mc3_Tol, mc3_FmRef, mc3_fmdose, mc3_cs, mc3_ampcont, mc3_gpu, mc3_dir, mc3_lib_path. Reads mask/frame_id and the calibration values from steps 10/20.

Algorithm

1. Builds a bright-field image per pattern (BF-disk mask, in-disk mean subtracted).
2. Refines the scan-to-image rotation and conversion factor by the chosen method (global sharpness sweep, or local/CTF per-position maps).
3. Splits the patterns into per-frame layers, builds a Hann-tapered shadow montage centred on the hole for each layer, and stacks them into a movie.
4. Runs MotionCor3 on the movie; interpolates each frame's shift onto every pattern by acquisition time and applies it to the positions.
5. Repeats for mc3_niter passes until the drift converges.

Results

Images: drift_map.png (drift trajectory + per-pattern applied-drift quiver), and the montages shadow_original, aligned_sum, shadow_updated (as PNG and MRC), plus a defocus-map overlay for local/CTF.

Data written: HDF5 positions/shadow (drift-corrected positions, Å) and, for local/CTF, data/defocus/cf or data/defocus/ctf (per-position defocus map).

Purpose

Lays out the regions of interest (ROIs) for the reconstruction, tiling them in concentric shells around the hole centre, and tags every scan position with its 1-based ROI index so the reconstruction step pulls one ROI at a time. It also estimates the mean applied electron dose, returned as pty_dose.

Parameters

Algorithm

1. Estimates the dose: total electrons (sum of all patterns / CPE) divided by the convex-hull area of the scan positions, scaled by DDC → mean dose in e/Ų.
2. Places a square or hexagonal lattice of ROIs (auto-sized to the FOV or manual), rotated by pty_roi_rot.
3. Tags each position with the ROI it falls in (per shape: radius, apothem, or Chebyshev distance).

Results

Image: ptycho_roi.png — the BF map with the hole circle, the numbered ROI outlines, and positions coloured by ROI.

Data written: HDF5 mask/patches/ptycho (per-position ROI index); results/GUI value pty_dose.

Purpose

The main ptychographic reconstruction. It reconstructs one ROI's complex object (amplitude + phase) with py4DSTEM in three recursive "levels," each using progressively fewer scan layers (so less dose/damage), then fuses the levels in Fourier space — the high-dose level supplies robust low frequencies, the low-dose levels supply undamaged high frequencies.

Parameters

ROI & geometry: pty_ROI_id (which ROI), pty_positions_source (units/angs/shadow), pty_defocus_mode (manual/cf/ctf/parallax — averages the chosen defocus map over the ROI) with fallback pty_dF, and the detector-orientation controls pty_detector_orientation, pty_detector_rotation_try, pty_detector_rotation_set.

Per-level lists (comma-separated, one value per level): sharp_lvl (names), sharp_lvl_lrs (scan-layer counts), sharp_lvl_iter, sharp_lvl_batch, sharp_lvl_step (learning rate), sharp_probe_cor (probe-aberration order, level 0 only), positions_max_update and positions_total_distance (per-position drift limits).

Reconstruction & filtering: pty_mode (complex/potential), pty_pure (pure phase), pty_noncoherent_modes (incoherent probe modes; ≥2 → mixed-state), sharp_bcg (high-pass cut-off Å); ripple filter ripple_filter/ripple_width; damage model sharp_dmg_mode (auto dose-weighting vs manual cutoff), sharp_dmg_cut (soft/hard), sharp_lvl_dmg, sharp_lvl_ord; fusion sharp_fusion_mode, sharp_fusion_doseweight; display upsampling pty_upsample_mode/pty_upsample_factor. Reads pixel size, dose, the defocus map, positions, and mask/patches/ptycho from earlier steps.

Algorithm

1. Selects the ROI, resolves the defocus (manual, or ROI-mean of the chosen defocus map), and derives per-level scan-layer counts and per-level dose.
2. Pre-processes: reorients patterns, removes descan (per-pattern centre-of-mass + plane fit), builds the initial probe (mixed-state modes if requested) and object canvas.
3. Level 0 uses all points, fits probe aberrations and refines positions → lvl0_pty.h5.
4. Levels 1 and 2 use subsampled layers, each seeded from a damage-limited fusion of the previous levels → lvl1_pty.h5, lvl2_pty.h5.
5. Between levels the outputs are fused in Fourier space: auto uses a dose-weighted (Grant & Grigorieff critical-dose) Wiener merge; manual uses a hard tanh ring fusion. A ripple band-stop and damage filter are applied per level.

Results

Images: reconstruction.png (per-level and running-fusion amplitude/phase), final_reconstruction.png, publication_figure.png/.pdf (hero phase, amplitude, diffractogram with info-limit ring, radial-PSD resolution estimate), and reconstruction_error.png.

Data written: per-level HDF5 files lvl0_pty.h5, lvl1_pty.h5, lvl2_pty.h5 (each holding the object, cropped object, object FFT, probe, sampling, and error curve). All images are registered to the Results panel.

Purpose

Reconstructs one ROI with Torchslice, a PyTorch/GPU single-slice ptychography engine — the SGD-optimizer counterpart of script 90. It is recursive and multi-level: level 0 uses the full point set with learnable probe aberrations and per-position drift, and later levels use fewer scan layers, each seeded by a dose-weighted Fourier fusion of the prior levels to limit carried-forward damage.

Parameters

ROI & geometry: pty_ROI_id, pty_positions_source (units/angs/shadow), pty_defocus_mode (auto = ROI-mean of the parallax defocus map, or manual) with fallback pty_dF, and pty_dose (from step 80).

Per-level lists: sharp_lvl (labels), sharp_lvl_iter (SGD iterations), sharp_lvl_lrs (scan-layer counts), sharp_lvl_dmg (damage cutoff / fusion band edges Å), sharp_lvl_lrbatch (minibatch size), and per-level learning rates sharp_lvl_lrnet (object), sharp_lvl_lrgrid (positions), sharp_lvl_lrprobe (probe).

Torchslice solver: pty_torchslice_loss (loss function), the optimizers pty_torchslice_netoptim/_gridoptim/_probeoptim (object/positions/probe), and pty_torchslice_cut (hard zeros frequencies past the optimal dose in the dose weighting).

Algorithm

1. Selects the ROI and resolves the defocus; derives per-level scan-layer counts and per-level dose.
2. Builds the microscope/simulation model, an initial probe with the defocus aberration, and a per-pattern descan/tilt model (Zernike fit of the centre-of-mass).
3. For each level, runs a Torchslice SGD reconstruction (chosen loss and optimizers, per-position drift) over that level's point count, then fuses all levels so far by dose-weighting each frequency (Grant & Grigorieff critical dose) to seed the next level.
4. The final fusion combines the three levels' dose-weighted objects with tanh band masks (edges from sharp_lvl_dmg), √dose magnitude weighting, phase preserved.

Results

Images: reconstruction_torchslice.png (a 4×3 grid — levels 0/1/2 and the fused result × amplitude / phase / log power spectrum) and doseweights.png (the per-level dose-weight curves vs spatial frequency).

Data written: per-level MRC files reconstruction_lvl{0,1,2}_{amp,phase}.mrc and the fused reconstruction_fused_amp.mrc / reconstruction_fused_phase.mrc; all images registered to the Results panel.