4d — SHARP

Purpose

The first calibration step. It builds a position-averaged CBED (PACBED) from the raw diffraction stack, detects the bright-field (BF) disk, and derives the reciprocal/real-space calibration every downstream step needs — detector cover angle (DCA), reconstructed pixel size, BF-disk diameter, electron wavelength, and the Ronchigram centre offset. Optionally it computes a per-pattern centre-of-mass "descan" so beam wander is removed before calibration.

Parameters

Algorithm

1. Sums up to the first 10 000 diffraction patterns into a single PACBED image.
2. Detects the BF disk: threshold at half-max, trace the longest contour, take its centre of mass and mean radius → pacbed_bf_disk.
3. DCA = CSA·(detector half-width / BF-disk radius); wavelength from KV; reconstructed pixel size = λ / (2·DCA).
4. Records the BF-disk centre minus the geometric detector centre as RonchiCenterOffsetX/Y.
5. Optional descan: compute each pattern's centre-of-mass offset, store it, re-centre the frames, and re-measure DCA / pixel size / disk on the aligned PACBED.

Results

Image: pacbed.png — a sample frame and the PACBED with the detected centre and BF disk overlaid (three log-scaled panels including the descan offset scatter when descan is on).

Data written: results/GUI values pacbed_DCA, pacbed_PIX_itr, pacbed_bf_disk, pacbed_wavelength, RonchiCenterOffsetX/Y; and (descan mode) the HDF5 dataset data/descan (per-pattern offsets).

Purpose

Splits a multi-pass acquisition into its individual scan frames by detecting the scan fly-back (X-direction reversal) and tags every scan position with an integer frame index, so drift correction and reconstruction can treat each pass over the field of view as a separate temporal layer. It uses the scan positions only, no diffraction data is read.

Parameters

Algorithm

1. Loads positions/units and histograms position-to-position steps to find the typical in-layer step.
2. Marks frame boundaries where the X step changes sign (within a frame, X is monotonic).
3. Filters out spurious intra-frame reversals (a "sandwich" test against the typical frame length), then labels each run of positions with an incrementing frame index.

Results

Image: frame_id.png — one panel per detected frame, positions colour-coded.

Data written: the HDF5 dataset mask/frame_id (per-position integer frame index). No results-file values.

Purpose

Converts the scan step from arbitrary units to Ångström by matching an imaged Quantifoil/carbon hole to its known diameter. It forms a bright-field STEM image, detects the hole edge, locates the circle with a Hough transform, and derives the units→Å scale used for correct real-space scan sampling everywhere downstream. It depends on the BF-disk geometry and Ronchi offsets from step 10.

Parameters

Algorithm

1. For every position, sums the pixels inside the BF disk (radius 0.45·pacbed_bf_disk, centred on the Ronchi offset) → a BF-STEM intensity (ice-filled hole bright, carbon dark).
2. Bins the scattered BF values onto a regular square raster (equal bin width on both axes), fills unsampled gaps of the scan pattern with the nearest sampled value so the sparse scan leaves no artificial edges, and normalises to [0, 1] between the 1st and 99th percentile.
3. Extracts the hole rim with a Canny edge detector, discarding edges outside the scanned support (the closed and filled footprint of the sampled bins) so the border of the scan field cannot be mistaken for the hole.
4. Runs a Hough circle transform over radii from 0.15 to 0.95 of the raster's short side: every edge pixel votes for the circles on which it could lie, and the strongest peak identifies the hole. The vote is global, so a partial or interrupted rim still determines the circle, and isolated edges from ice contamination or the corners of a hexagonal scan pattern do not accumulate enough votes to displace it.
5. Converts the circle from raster pixels back to scanning units, giving the measured hole diameter in scanning units. The known physical diameter is the scan_hole_size parameter, entered in micrometres and converted to Ångström. Their ratio is:
   units2ang = scan_hole_size·1e4 / measured_diameter
   that is, the number of Ångström one scanning unit corresponds to. Every scan position is multiplied by it to give positions/angs.

If detection fails: no result is produced and no substitute method is attempted. A scan in which the Hough transform finds no circle does not contain a hole edge this method can measure, and the scan step has to be calibrated by other means outside Sharp. Since units2ang scales every distance used downstream (defocus mapping, drift correction and reconstruction alike) an unreliable substitute would be worse than no result at all.

Results

Images: ice_hole_calibration.png — the BF map with the fitted circle, centre, diameter, and a set-vs-measured annotation. ice_hole_detection.png — a detection diagnostic showing the BF raster and the Canny edge map, with the circle as voted by the Hough transform (orange, dotted) and the refined circle used for the calibration (green, dashed). Consult this figure first if a fit appears incorrect, as it shows whether the hole rim was detected as an edge at all.

Data written: HDF5 positions/angs (positions in Å), data/bfvalues, and a hole attribute; results/GUI scan_hole and units2ang.

Purpose

Estimates a single defocus value at the hole centre by fitting a CTFFIND4-style Ronchigram CTF (via the bundled ctf_lmu.py) to the raw patterns of a small central region. It provides a robust starting defocus for ptychography and is the first quantitative defocus output once the geometry is calibrated.

Parameters

Algorithm

1. Selects scan positions within defocus_ctf_fraction·(hole radius) of the hole centre; their raw patterns feed the fit.
2. Optionally descans each pattern (using the shared data/descan when present) so the BF disk is centred.
3. Computes a Gaussian-windowed incoherent power spectrum and fits a CTFFIND4-style Ronchigram CTF to the mean spectrum for a global defocus (plus per-cluster fits for spread).

Results

Image: defocus_ctf.png — the ROI map, the log Ronchigram with the fitted defocus, and a radial profile with the CTF model overlaid.

Data written: the results/GUI value defocus_ctf (Å). No HDF5 writes.

Purpose

Defocus need not be computed over the whole field of view. This step decides where it is computed: it lays out square patches and tags every scan position with the patch it falls in, so the measuring steps work only inside patches and the map is interpolated between them. Patches can be placed automatically on a ring around the hole, picked by hand on the BF scan, or replayed from a stored list.

Parameters

The picker window

Seeded with defocus_patch_list if it already holds a layout, otherwise with the automatic ring — so re-running continues from the last layout instead of starting over.

The toolbar's zoom and pan work as usual and are ignored by the picker, so navigating never leaves a stray patch behind. Closing the window with nothing selected falls back to the automatic ring rather than writing an empty mask.

Algorithm

1. Builds the ring layout: patch centres at equal angular spacing on a circle of radius defocus_ring·(hole radius) around the hole centre, each a square of side defocus_patch_size.
2. Depending on the mode, uses that layout, opens it in the picker for editing, or replaces it with defocus_patch_list.
3. Tags every scan position with its 1-based patch index (0 outside all patches), and checks the final layout for overlaps.

Results

Image: defocus_patches.png — the BF map with the ring and the numbered patch squares.

Data written: the HDF5 dataset mask/patches/parallax (per-position patch index) and the final layout back into defocus_patch_list.

Purpose

Measures the local defocus inside each patch using py4DSTEM tilt-corrected bright-field (parallax) reconstruction, and scores how far each measurement can be trusted. A plane is also fitted here through all the patches at once; when some of them are bad, refit it with step 45 instead of accepting this one.

Parameters

Algorithm

1. Resamples each patch's irregular scan onto a regular square grid (each cell takes its nearest scan point's full pattern — never blended, so the sub-pixel disk shift is preserved). Cells with no scan point within two scan steps are masked out: they carry the nearest edge pattern copied outward, which against a straight edge shows up as stripes across the reconstruction. That happens wherever the patch has a hole, most often because another patch overlapped it.
2. Runs py4DSTEM parallax on each patch: virtual tilt-corrected BF images, coarse-to-fine cross-correlation alignment, then an aberration fit for the defocus (C1).
3. Scores the measurement (below) and fits a plane d = a·x + b·y + c through the patch defoci (constant mean for 1–2 patches), evaluated at every scan position.

The quality score

C1 is read straight off the measured BF shift field, so the score asks whether that shift field is worth believing: score = 85 × (signal × fit) + 15 × trace, out of 100.

signal and fit multiply because they are conjunctive — the shifts have to be real and follow the model. Adding them lets either carry the score alone, and both real failure modes do exactly that: r² is scale-free, so a shift field a tenth of a pixel across still fits a defocus pattern beautifully while measuring nothing, and a field of large noise has the magnitude but no structure.

Results

Images: defocus_map_update.png (per-patch field, labelled with the patch number and its C1, plus the per-position plane fit), convergence_table_update.png (per-patch C1, rotation, shift RMS, r², trace drop and score), and five diagnostics per patch in parallax_update_diagnostics/ when metadata is saved — probe size, average BF, convergence curve, shift field and CTF fit, each linked from the Results panel.

Data written: HDF5 data/defocus/parallax (per-position defocus, Å, with plane_a/b/c attributes) and parallax_update.json, which holds the per-patch numbers and is what steps 41 and 45 read.

Purpose

Step 25 fits one CTF at the middle of the hole. This one fits the same CTFFIND4-style Ronchigram model separately inside every patch the picker laid out — the same patches parallax measures — so the two methods can be compared patch by patch.

They are worth comparing because they fail in unrelated ways. Parallax needs localised contrast for its cross-correlation and returns a near-zero defocus on a featureless patch while its error trace still looks converged. The CTF reads Thon rings out of the power spectrum and does not care whether the patch has features, but it does need enough patterns and enough ring contrast. A patch where the two disagree is a patch to look at; a patch one of them measures badly is often measured perfectly by the other.

Parameters

The quality score

Two terms, score = 60 × cc + 40 × consistency, out of 100.

cc is the fit's own correlation coefficient, ramped from a measured noise floor. The floor is not a guess: fitting patterns with the ring structure destroyed — Poisson noise on the PACBED, and pixel-shuffled patterns — returns cc around 0.08–0.15 together with a meaningless C1, while fits that agree with each other and with step 25 sit at 0.32–0.46. cc also climbs with the number of patterns (about 0.18 at 50 patterns against 0.44 at 1500 on the same patch), so a thin patch scores low for a real reason.

consistency is the coefficient of variation of the sub-cluster defoci. Unlike the parallax score these two are added, not multiplied: a patch with a good cc but scattered sub-clusters may simply have a real defocus gradient across it, which is worth knowing rather than a disqualification.

Results

Images: defocus_ctfpicker.png — the patches coloured by CTF defocus, and CTF plotted against parallax with a 1:1 line, where anything off the diagonal means one of the two failed. defocus_ctfpicker_table.png — per-patch defocus, cc, sub-cluster spread, score, and the difference from parallax, flagged when it exceeds 20%.

Data written: defocus_ctfpicker.json (read by step 45) and defocus_ctf_patch_mean back to 4d. Nothing is written to the HDF5 stack.

Purpose

Steps 40 and 41 measure the patches; this one decides which of those measurements the defocus map is built from. It refits the plane through a chosen subset, taking each patch's value from parallax, from the CTF, or from whichever scored higher — and it does so without re-running either measurement. The diffraction patterns are never read, so it runs in seconds and needs neither py4DSTEM nor a GPU.

This matters because one bad patch drags the whole plane. On a real dataset where one patch of six had failed in parallax, the plane fitted through all six had an RMSE of 4046 Å and swung the per-position defocus from −28188 Å to +2718 Å — through focus, which is nonsense for a field the CTF says is flat. Taking that one patch from the CTF instead brought the RMSE to 283 Å.

Parameters

Algorithm

1. Recomputes each patch's bounding box and centre from the patch mask and the scan positions, exactly as the parallax step derives them, so the centres fitted through are the same ones.
2. Resolves the source for every patch. The CTF value carries no sign — ring spacing is the same either side of focus — so its magnitude is given the sign of the parallax consensus, and with no parallax at all to consult it is taken as underfocus, which is what step 40 writes.
3. Least-squares fits d = a·x + b·y + c through the selected patches and evaluates it at every scan position. Fewer than three patches cannot define a plane, so their mean is used as a flat one instead.
4. Compares every patch against that plane, the excluded ones included, so how far out the dropped ones really were is visible.

Results

Images: defocus_plane.png — the measured patches (P or C marking the source, dropped ones dashed in red) and the refitted per-position map. defocus_plane_residuals.png — every patch against the plane, judged both against the fit's own RMSE and as a fraction of the defocus, so a uniformly bad fit cannot look uniformly fine.

Data written: HDF5 data/defocus/parallax, overwriting what step 40 left there — downstream steps read that dataset, so the refit has to land in the same place. The plane coefficients, the patch selection, the per-patch sources and the RMSE are stored on it as attributes, so it is possible to tell from the file alone which fit is currently in there and what produced it.

Purpose

Corrects specimen/stage drift before reconstruction. It reassembles the diffraction patterns into per-layer bright-field "shadow" montages, treats the temporal layers as a movie, and aligns them with MotionCor3 software. The per-frame drift is interpolated onto every pattern by acquisition time and applied to the scan positions, iterating to shrink the residual drift.

Parameters

Method: drift_method (global = one rotation + conversion factor from shadow sharpness; interactive = the same two numbers fitted from alignments you make by hand; local = per-position conversion-factor map via KMeans + RBF; CTF = per-position defocus map from a Ronchigram-CTF fit — local/CTF also write a defocus map), drift_clusters, drift_ctf_device. The hole diameter is not asked for again here: step 20 scales the positions so the fitted circle measures exactly the diameter entered there, so the third component of scan_hole already is that diameter (in Å) and the step reads it from there. Re-entering it could only contradict the calibration the positions carry.

Shadow montage & search: shadow_defocus, shadow_res1/shadow_res2 (sharpness band), the rotation search (shadow_rotation, shadow_rotation_shortcut, shadow_min/max_rotation, shadow_number_rotation), the conversion-factor search (shadow_cf, shadow_cf_shortcut, shadow_cf_range, shadow_number_cf), shadow_njobs (global parallel-worker count, default 1 - each worker holds its own montage and FFT, so raising it multiplies peak memory for a sub-linear speedup), the optional search crop (shadow_crop_doit, shadow_crop_nm), shadow_crop, shadow_invert, and the fixed output sampling (shadow_pixel_doit, shadow_pixel_size).

Interactive alignment (drift_method = interactive): two windows open - interactive - pattern for the pair alignment and its fit, interactive - scaling for the montage preview and the sliders. They share one state (a fit reaches the sliders at once) and every key works in either. The pattern window shows two neighbouring bright-field disks in the montage's own grey, the neighbour laid over the reference at 50% on a black ground, placed at the separation drift_interactive_cf0 implies. Slide the neighbour with the arrow keys and swing it about the reference with ,/. until the shared specimen detail lines up; t cycles the top disk 50% / on / off so you can blink the pair, s cycles the display smoothing, and a accepts the pair and steps on. Only patterns from the first temporal layer are offered: two patterns from different layers were recorded a whole pass apart, so specimen drift sits in the measured offset on top of the scan step, and biasing cf with the very drift this step exists to remove would corrupt every montage built from it. The disks are smoothed for display (normalised convolution, so the masked rim is not dragged inward); none of it reaches the fit, which uses your offsets alone. If a disk looks noisier here than the same disk inside a montage, that is scale, not data: a montage is written at full canvas size and shown ~5× downsampled, so each screen pixel there averages ~25, while this tool magnifies one 192 px disk ~3× so alignment can be judged to a pixel. The default smoothing sits near the effective smoothing a montage view already has. After drift_interactive_pairs of them, Calculate CF fits the values — its panel plots the component of each offset along the fitted direction against separation, not the offset's length, because the fit is a vector least squares: disagreeing per-pair directions shorten the resultant and pull cf below every point while the lengths stay put, so a length plot would draw a line missing its own points. The projection is what cf regresses, so the line passes through by construction, points are coloured by direction error, and a spread over 10° names the worst pair to redo. Apply rebuilds a montage preview at them. Two sliders under the preview also set cf and the rotation directly, with the preview following live — so the montage can be the target itself, sharpening as the values approach the truth. The two routes feed one pair of numbers: Calculate CF pushes its fit onto the sliders and a later slider move overrides it, so a fit can be nudged rather than taken whole. [/] nudge cf and ;/' the rotation (finer with f). Whatever the sliders read at close is what the step uses; closing untouched leaves the stored values standing.

The montage stamps pattern i at rotate(pos_i, theta) * cf, so a specimen point lands on one canvas pixel in every frame only when the shadow content between two patterns is displaced by dp = -cf · R(theta) · dpos. Aligning a pair measures dp directly. Both controls feed that same vector — translating along the separation pins cf, swinging across it pins the rotation — so as complex numbers the set is one linear model z_p = w·z_pos, solved exactly, with cf = |w| and theta = angle(−w). A single aligned pair already determines both (two unknowns, one 2-D offset); more pairs average down aiming error, and the residual only means something from three up. The starting cf need not be close — the offset you drag out is the measurement — but it sets which separations are offered, so that the two disks always still overlap.

Reach for it when the global sweep converges somewhere wrong: that sweep searches cf over a range centred on the stored value, so a stored value far from the truth puts the answer outside the range entirely and re-running cannot help. A minute of hand alignment gives a value to centre the range on. Needs a display; with none, the step says so and falls back to the stored values rather than failing.

Subset: shadow_frames keeps only the first N temporal layers and shadow_area keeps only the given central fraction of the field (the points nearest the centroid of the scan). They compose, and both act before anything is measured — so they shrink the data the rotation/CF search, the MotionCor3 alignment and the corrected positions are all derived from. Use them to get a quick look, or to cut away a bad periphery; not to produce the positions you intend to reconstruct from.

Detector frame: shadow_flip_x, shadow_flip_y and shadow_transpose map the detector axes onto the scan axes. That mapping is a property of the camera, its mounting and the acquisition software rather than of the method, so it has to be set once per setup. They are applied in the fixed order flip X, flip Y, transpose — these do not commute, since transposing and then flipping X is the same as flipping Y and then transposing. The defaults (n, n, y) reproduce exactly the unconditional transpose the step applied before the switches existed, so existing projects are unaffected until one is moved. A wrong mapping does not announce itself: the montage either fails to align at all, or aligns into a mirrored image that looks perfectly plausible until it is compared against something else.

Contrast: shadow_invert (advanced) reverses black and white in every shadow montage the step writes — before, aligned sum, after, and the diagnostics — by negating the bright-field frames they are stamped from. The sign of the defocus decides which way round the contrast falls, so an over-focus and an under-focus dataset come out looking opposite; this flips one onto the other. Nothing measured moves: the sharpness metric is a power spectrum and MotionCor3 aligns by cross-correlation, and neither can see a global sign flip. The same knob is shared with the Shadow Montage engine step, so the two produce montages the same way up.

Output pixel size: shadow_pixel_doit / shadow_pixel_size (global, shared with the Shadow Montage engine step). A shadow montage has no free sampling: one canvas pixel is one detector pixel projected onto the specimen, so the pixel size follows from the refined conversion factor and lands on a different value for every acquisition, defocus and CF refinement (it is printed as Computed pixel size and stamped into the MRC headers). Switching shadow_pixel_doit on resamples the montages the step writes onto the constant shadow_pixel_size, in Å per pixel, so shadow_original, shadow_updated and aligned_sum — and the montages of every other acquisition in the project — come out on one grid and can be compared or stacked directly. Coarser than native downsamples, with the matched Gaussian prefilter applied first so detail past the new Nyquist cannot fold back onto the Thon rings; finer than native only re-grids the montage, since interpolation cannot recover detail the native sampling never recorded (a factor beyond 16× either way is refused as a typo, and the run falls back to native). Nothing measured moves: the rotation/CF search, the MotionCor3 alignment — still run on native-sampling frames at the native -PixSize — and the corrected positions written to positions/shadow are all computed before the resampling. drift_map.png and drift_search_crop.png keep the sampling they were computed at.

Cropping: two independent settings share a similar name. shadow_crop_doit / shadow_crop_nm restrict the rotation and conversion-factor search to a small square at the centre of the scan; the rotation angle and global CF are scan-wide constants, so this yields the same result considerably faster, and the result is applied to the whole dataset. shadow_crop is separate and controls the montage window used for alignment: each montage is reduced to a centred square retaining the middle 1 − 2/shadow_crop of the shorter canvas side.

MotionCor3: mc3_niter (outer refinement passes), mc3_patch, mc3_bft, mc3_Iter, mc3_Tol, mc3_FmRef, mc3_fmdose, mc3_cs, mc3_ampcont, mc3_gpu. The MotionCor3 executable is a step parameter, mc3_exe: the full path to the binary, taken exactly as typed. Its CUDA libraries are resolved by the system loader, so a CUDA in a private prefix must be on LD_LIBRARY_PATH in the environment 4d itself is launched from. Reads mask/frame_id and the calibration values from steps 10/20.

Algorithm

1. Builds a bright-field image per pattern (BF-disk mask, in-disk mean subtracted).
2. Refines the scan-to-image rotation and conversion factor by the chosen method (global sharpness sweep, or local/CTF per-position maps).
3. Splits the patterns into per-frame layers and stamps each layer onto a shared canvas, so that all layers share one set of dimensions. Every montage is reduced to a centred square window and Hann-tapered.
4. Zero-pads the tapered layers to a frame size whose prime factors are small (2, 3, 5, 7) and stacks them into a movie. MotionCor3 constructs its Fourier transforms with cuFFT, which cannot build a plan for a dimension carrying a large prime factor; padding to a compatible size avoids this while preserving the full cropped field of view. Applying the taper before padding leaves no intensity step where the padding begins.
5. Runs MotionCor3 on the movie and removes the padding from the aligned sum; interpolates each frame's shift onto every pattern by acquisition time and applies it to the positions.
6. Repeats for mc3_niter passes until the drift converges.

Results

Images: drift_map.png (drift trajectory + per-pattern applied-drift quiver), and the montages shadow_original, aligned_sum, shadow_updated (as PNG and MRC), plus a defocus-map overlay for local/CTF.

Data written: HDF5 positions/shadow (drift-corrected positions, Å) and, for local/CTF, data/defocus/cf or data/defocus/ctf (per-position defocus map).

Purpose

Lays out the regions of interest (ROIs) for the reconstruction, tiling them as an N × N block over the hole centre, and tags every scan position with its 1-based ROI index so the reconstruction step pulls one ROI at a time. It also estimates the mean applied electron dose, returned as pty_dose.

Parameters

Algorithm

1. Estimates the dose: total electrons (sum of all patterns / CPE) divided by the convex-hull area of the scan positions, scaled by DDC → mean dose in e/Ų.
2. Places exactly N² ROIs over the hole centre — on a square lattice for square ROIs, a hexagonal one for round and hexagon ROIs — auto-sized so the layout tiles the field of view, or at the manual size, then rotated by pty_roi_rot. An odd N puts a ROI on the centre; an even N straddles it.
3. Tags each position with the ROI it falls in (per shape: radius, apothem, or Chebyshev distance).

Results

Image: ptycho_roi.png — the BF map with the hole circle, the numbered ROI outlines, and positions coloured by ROI.

Data written: HDF5 mask/patches/ptycho (per-position ROI index); results/GUI value pty_dose.

Purpose

The main ptychographic reconstruction. It reconstructs one ROI's complex object (amplitude + phase) with py4DSTEM in three recursive "levels," each using progressively fewer scan layers (so less dose/damage), then fuses the levels in Fourier space — the high-dose level supplies robust low frequencies, the low-dose levels supply undamaged high frequencies.

Parameters

ROI & geometry: pty_ROI_id (which ROI), pty_positions_source (units/angs/shadow), pty_defocus_mode (manual/cf/ctf/parallax — averages the chosen defocus map over the ROI) with fallback pty_dF, and the detector-orientation controls pty_detector_orientation, pty_detector_rotation_try, pty_detector_rotation_set.

Per-level lists (comma-separated, one value per level): sharp_lvl (names), sharp_lvl_lrs (scan-layer counts), sharp_lvl_iter, sharp_lvl_batch, sharp_lvl_step (learning rate), sharp_probe_cor (probe-aberration order, level 0 only), positions_max_update and positions_total_distance (per-position drift limits).

Reconstruction & filtering: pty_mode (complex/potential), pty_pure (pure phase), pty_noncoherent_modes (incoherent probe modes; ≥2 → mixed-state), sharp_bcg (high-pass cut-off Å); ripple filter ripple_filter/ripple_width; damage model sharp_dmg_mode (auto dose-weighting vs manual cutoff), sharp_dmg_cut (soft/hard), sharp_lvl_dmg, sharp_lvl_ord; fusion sharp_fusion_mode, sharp_fusion_doseweight; display upsampling pty_upsample_mode/pty_upsample_factor. Reads pixel size, dose, the defocus map, positions, and mask/patches/ptycho from earlier steps.

Algorithm

1. Selects the ROI, resolves the defocus (manual, or ROI-mean of the chosen defocus map), and derives per-level scan-layer counts and per-level dose.
2. Pre-processes: reorients patterns, removes descan (per-pattern centre-of-mass + plane fit), builds the initial probe (mixed-state modes if requested) and object canvas.
3. Level 0 uses all points, fits probe aberrations and refines positions → lvl0_pty.h5.
4. Levels 1 and 2 use subsampled layers, each seeded from a damage-limited fusion of the previous levels → lvl1_pty.h5, lvl2_pty.h5.
5. Between levels the outputs are fused in Fourier space: auto uses a dose-weighted (Grant & Grigorieff critical-dose) Wiener merge; manual uses a hard tanh ring fusion. A ripple band-stop and damage filter are applied per level.

Results

Images: reconstruction.png (per-level and running-fusion amplitude/phase), final_reconstruction.png, publication_figure.png/.pdf (hero phase, amplitude, diffractogram with info-limit ring, radial-PSD resolution estimate), and reconstruction_error.png.

Data written: per-level HDF5 files lvl0_pty.h5, lvl1_pty.h5, lvl2_pty.h5 (each holding the object, cropped object, object FFT, probe, sampling, and error curve). All images are registered to the Results panel.

Purpose

Reconstructs one ROI with Torchslice, a PyTorch/GPU single-slice ptychography engine — the SGD-optimizer counterpart of script 90. It is recursive and multi-level: level 0 uses the full point set with learnable probe aberrations and per-position drift, and later levels use fewer scan layers, each seeded by a dose-weighted Fourier fusion of the prior levels to limit carried-forward damage.

Parameters

ROI & geometry: pty_ROI_id, pty_positions_source (units/angs/shadow), pty_defocus_mode (auto = ROI-mean of the parallax defocus map, or manual) with fallback pty_dF, and pty_dose (from step 80).

Per-level lists: sharp_lvl (labels), sharp_lvl_iter (SGD iterations), sharp_lvl_lrs (scan-layer counts), sharp_lvl_dmg (damage cutoff / fusion band edges Å), sharp_lvl_lrbatch (minibatch size), and per-level learning rates sharp_lvl_lrnet (object), sharp_lvl_lrgrid (positions), sharp_lvl_lrprobe (probe).

Torchslice solver: pty_torchslice_loss (loss function), the optimizers pty_torchslice_netoptim/_gridoptim/_probeoptim (object/positions/probe), and pty_torchslice_cut (hard zeros frequencies past the optimal dose in the dose weighting).

Algorithm

1. Selects the ROI and resolves the defocus; derives per-level scan-layer counts and per-level dose.
2. Builds the microscope/simulation model, an initial probe with the defocus aberration, and a per-pattern descan/tilt model (Zernike fit of the centre-of-mass).
3. For each level, runs a Torchslice SGD reconstruction (chosen loss and optimizers, per-position drift) over that level's point count, then fuses all levels so far by dose-weighting each frequency (Grant & Grigorieff critical dose) to seed the next level.
4. The final fusion combines the three levels' dose-weighted objects with tanh band masks (edges from sharp_lvl_dmg), √dose magnitude weighting, phase preserved.

Results

Images: reconstruction_torchslice.png (a 4×3 grid — levels 0/1/2 and the fused result × amplitude / phase / log power spectrum) and doseweights.png (the per-level dose-weight curves vs spatial frequency).

Data written: per-level MRC files reconstruction_lvl{0,1,2}_{amp,phase}.mrc and the fused reconstruction_fused_amp.mrc / reconstruction_fused_phase.mrc; all images registered to the Results panel.

Purpose

Combines the two channels of the level-fused reconstruction into a single image. Amplitude and phase describe the same object but carry different information about it: the phase resolves fine detail, while the amplitude mostly reports large-scale mass-thickness and is noisy at high frequency. Low-passing the amplitude before the sum keeps its long-range contrast and leaves the fine detail to the phase, instead of adding amplitude noise on top of a sharper channel.

Parameters

Filters: apmerge_amp_lowpass and apmerge_amp_highpass bracket the amplitude into a BAND (10–200 Å by default; the high-pass may be set to 0 to disable it), and apmerge_pha_highpass cuts the phase background at the same 200 Å. All three are Butterworth, the same form as the damage-limiting filter in step 90, so they behave alike. The amplitude high-pass matters because both channels are normalised before being added: the amplitude carries its own slow ramp from uneven illumination and bulk mass-thickness, and left in, that ramp — not the local contrast — sets the scale of everything the amplitude contributes.

Sign & output: apmerge_invert (flip the phase so both channels read dark on a feature and the sum reinforces instead of partly cancelling), apmerge_clip (display percentile pair for the figure only), apmerge_savephase (also write the unfiltered cropped phase). Pixel size comes from the MRC voxel size; pacbed_PIX_itr is used only if that header field is empty, and would be the pre-upsampling value.

Algorithm

1. Finds every amplitude_merged*.mrc / phase_merged*.mrc pair on disk. It carries no ROI setting of its own, so it follows a single-ROI and an all-ROI reconstruction automatically and cannot drift out of step with step 90.
2. Butterworth band-pass on the amplitude and Butterworth high-pass on the phase. The high-passes remove the slow large-scale ramp — residual illumination, sample tilt, the reconstruction’s own low-frequency gauge — which is not specimen contrast and would otherwise set the scale of the sum. The two filters overlap rather than partition: the amplitude supplies everything coarser than its cutoff, the phase everything finer than its own, and both contribute in between.
3. Centre-crops every result to the smallest input, since each ROI’s canvas is sized to its own scan extent and they generally differ by a few pixels. Filtering happens at full size first, so the transforms keep the context outside the crop instead of wrapping against a fresh edge.
4. z-scores both channels and adds them, so their different offsets and contrasts contribute equally rather than whichever has the larger numerical range dominating.

Results

Images: a_p_merge.png — two rows of three panels: the filtered amplitude, the filtered phase and their sum on top, the unfiltered inputs and their straight sum beneath. Every panel is stretched to its own percentiles, so the rows compare what is resolved, not how dark it looks.

Data written: a_p_merge.mrc (uncompressed float32, unclipped, pixel size stamped as the voxel size) and, optionally, phase_cropped.mrc — the phase as reconstructed, unfiltered, on the shared grid and keeping its physical sign. Per ROI these are suffixed _roi1, _roi2, …

Purpose

Picks particles out of the merged ptychographic image, extracts and centres them, and runs a reference-free 2D classification. It stops at the 2D class level — no 3D, no refinement. The point is to answer whether the reconstruction carries interpretable single-particle signal at all, and at what resolution, before any of that effort is spent.

Parameters

Input & particle: spa_source (which image on disk: the amplitude+phase merge, the cropped or merged phase, the merged amplitude, or the potential), spa_diameter (particle diameter in Å — the one setting that must be right; it sets the picking template, the minimum pick separation, the box, the background ring and the alignment mask), spa_dark (particles darker or brighter than background), spa_pickthr (peak threshold in σ of the correlation map), spa_minsep (minimum pick separation in diameters — a true circular distance).

Ice hole: spa_hole keeps only particles over free ice, discarding those on the carbon rim, and spa_hole_margin sets how far inside the measured rim a particle must sit (200 Å by default, about one apoferritin diameter).

Classification: spa_classes, spa_iters (alignment rounds), spa_bin (box binning — also sets the Nyquist limit no FRC can report past), spa_box (extraction box in Å, 0 = auto at 1.6 × diameter).

Screening & validation: spa_notch (periodic-artifact screen, off by default) and spa_control (the phase-randomised control). Pixel size comes from the MRC header, with pacbed_PIX_itr as a fallback.

Algorithm

1. Periodic artifact screen (optional). Finds discrete spikes standing more than 200× above the radial median of their own frequency ring — a median, because a mean would be inflated by the very peaks being looked for. It removes nothing unless the flagged pixels are concentrated in one direction (R > 0.8): a scan error is locked to the raster, while a crystal or a particle packing ring flags at many angles.
2. Picking. Template correlation against a soft disc on a 4× binned copy, with a heavily smoothed percentile mask rejecting contamination first. Picks are separated by 0.85 of a diameter so neighbours are not picked twice.
3. Ice-hole restriction (optional). Particles on the carbon rim are imaged through a much thicker support and reconstruct poorly, so they are dropped. The boundary comes from the calibration, not the image: the ice-hole step (20) measured the hole centre and diameter in Å in the same frame as the scan positions, and each pick is placed in that frame through its ROI's scan extent.
4. Extraction. Boxes are normalised on their background ring, not the whole box, so a box holding a neighbour enters the average on the same footing as an empty one. Every ROI found on disk is picked separately and the particles are then pooled into a single classification, since a single ptychographic field rarely holds enough particles on its own.
5. Centring on a 60–18 Å band rather than the full low frequencies: full-band centring locks onto the particle envelope, which is exactly what a weak particle or a notch filter degrades.
6. Classification. Reference-free multi-reference alignment, seeded by k-means on rotation-invariant polar harmonics so no external reference enters anywhere. Rotation is searched by FFT correlation in polar coordinates with sub-sample peak interpolation; translations are refined against the assigned class reference every fourth round.
7. Validation. Convergence, the phase-randomised control, and per-particle SSNR with a half-set FRC averaged over 12 random splits.

Results

Images: spa_classes.png (class averages sorted by population, with the control classes beneath them), spa_validation.png (SSNR by shell, half-set FRC, and the rotation-convergence trace), and spa_picking_roi<id>.png (the picks drawn on the image, per ROI).

Data written: spa_class_averages.mrc, spa_global_average.mrc, spa_particles.mrc (the extracted, centred, aligned stack) and spa_particles.star — a RELION-style particle list carrying the coordinate, class number, in-plane angle and source ROI for anything downstream.