4d — User Manual

Install with your system package manager (Homebrew on macOS, apt/yum on Linux):

• Build tools: CMake ≥ 3.16, a C++ compiler, gh (GitHub CLI).
• Libraries: Qt 6, FFTW3, HDF5 + tools, LZ4; on Linux, the X11/GL runtime libraries.
• Shell & tools: tcsh, gawk, a text editor (e.g. gedit).
• Python 3.12 in a dedicated virtual environment, with numpy, scipy, scikit-learn, py4DSTEM, torch, h5py, hdf5plugin, blosc2, bitshuffle and the other extensions listed in the installation guide.
• Optional (CUDA): the NVIDIA CUDA toolkit and cupy for GPU-accelerated reconstruction.
• Optional external tools: MotionCor2/3, Zorro, Unblur (drift correction); CTFFIND4, GCTF (CTF); EMAN2, IMOD.

Clone into a dedicated code directory (not ~/4d, which is the default install location) and build:

mkdir ~/Projects && cd ~/Projects
git clone https://github.com/LBEM-CH/4d.git
cd 4d
./build_all

The build script inquires about the install location, checks dependencies, downloads and compiles the Blosc2 compression library, compiles the 4d source tree, installs into your chosen location, and updates your shell environment (e.g. .zshrc). Developers can pass stage arguments, e.g. ./build_all 1 2 1.

Activate your Python virtual environment, then run the 4d binary from the install directory:

source /Users/<you>/venvs/py4d/bin/activate
~/4d/bin/4d

A project is a directory on a fast local disk where all of your data and processing results are stored. When you create a project you choose a project mode, which tailors the tools, parameters, and library columns to a workflow:

Inside a project, project-level configuration lives in a merge/ subfolder, and every dataset lives in its own subfolder. Datasets are organized into image groups (including a special TRASH group). Recently opened projects are remembered and reappear on startup and under File ▸ Recent Projects.

Each imported 4D-STEM acquisition becomes a dataset (historically also called an "image"). All datasets in a project are listed in the Library — a sortable, groupable table where you browse, flag, select, and launch processing. Datasets are stored in 4d's native HDF5 format (see §7), which holds the diffraction patterns, scan positions, correction masks, and derived overview images.

This is the key idea behind 4d. Every processing step is a script (Python .py or C-shell .script/.com) stored in the scripts/ tree. Folders in that tree become the sections and menus you see in the GUI. Each script carries special header comment lines that the application reads:

When you select a script, 4d shows its manual text and its declared parameters (pulling each parameter's label, type, allowed values, and tooltip from the central parameter definitions). You edit the values in the GUI, press Run, and 4d executes the script in your chosen Python environment, streaming its output to a log panel and collecting its result images. The very same scripts are used whether you run them interactively on one dataset (Manual Processing), in a parallel batch (AutoProcess), or automatically during import.

1. Create a project. On launch, the Project initialization wizard opens. Choose Add Project, pick a location on a fast local disk, name it, choose the 4D-STEM mode, and initialize it from master (or copy settings from an existing project).
2. Set your Python environment. Open Preferences ▸ Software and enter the activation command and absolute interpreter path.
3. Import data. Use the Import tab to bring raw HDF5 files into the project, or run the Import scripts on a dataset (Import 4D STEM → Scan Pattern into HDF5 → Get BF and DF image).
4. Browse the Library. Your datasets appear in the Library table. Double-click one to open it for processing.
5. Reconstruct. In the per-dataset tab, work through the SHARP section top to bottom: calibrate, map defocus, correct drift, then run a ptychography reconstruction.
6. Batch or export. Check several datasets in the Library and press Process Selected to run them in parallel via AutoProcess, or use the Export section to hand results to cryoSPARC / RELION.

Launching 4d without arguments opens the Project initialization wizard (pass a project directory on the command line to skip straight to it). The wizard has three pages:

• Open project — a list whose first row is always Add Project; below it, each recently used project shows its mode icon, name, and full path. Use arrow keys and Next.
• Add project — choose the project path (a fast local disk), a name (auto-filled from the folder), and the mode. If the folder is new, a dialog offers Initialize from master or Copy from existing project.
• Loading project — a progress checklist: registering parameters, finding images on disk, initializing parameters, loading the library, placing the widgets.

The title bar shows the version, current project name, and project directory. A horizontal navigation toolbar of large icon buttons switches between full-window views; a header bar above shows the current section's title and an “N Images Selected” count. The navigation buttons, in order:

Menus provide File (Open Project Ctrl+O, Recent Projects, Save Ctrl+S, Quit Ctrl+Q), Options (Preferences), and Help (online help, report an issue, Update, About). Quitting with unsaved changes prompts Save & Quit / Quit Anyway / Cancel.

The Library is the table of every dataset in the project. A toolbar sits on top; below, a splitter divides the dataset table (left) from a preview panel (right).

The table

Rows are grouped by image group, with sortable and drag-reorderable columns. The first two columns are fixed: a Select checkbox and a Flag color. The remaining columns are parameters you choose to display (Preferences ▸ LibraryTable). 4d distinguishes two selection concepts, both shown in the status line:

• Highlighted — rows you clicked/shift/ctrl-selected in the view.
• Selected — the checkbox state; this is the set fed to processing.

Toolbar actions

• Process Selected — queue the checked datasets for batch processing.
• Selection: Show selected only Ctrl+X, Select All Ctrl+A, Deselect All, Invert Ctrl+I, Select highlighted Space.
• Flags: set highlighted rows to none / red / green / blue / gold (N/R/G/B/O).
• Groups & files: Add Group Ctrl+F, Rename Group, Move highlighted Ctrl+M, Trash Ctrl+T, Copy to a second project, Save/Load a selection list, Empty Trash.
• Auto Selection Tool — auto-check datasets whose chosen parameter falls in a numeric range (with optional sign-ignore and flag-color filters).

Preview panel

When a real dataset row is current, the preview panel shows its overview images (bright-field, dark-field, iDPC, thumbnail — configured in Preferences ▸ LibraryPanels) with previous/next navigation and a maximize mode. Double-click a dataset to open it in the Manual Processing tab.

The Manual tab is where you run scripts on individual datasets. It starts empty (“Select entries in the library for processing by double-clicking them”); each opened dataset becomes a closable sub-tab. A sub-tab turns green with a running icon while its script executes and cannot be closed until the script finishes.

Each sub-tab is an execution view — the core “run a script and see its output” surface, also reused by Project utilities, the importer, and custom tabs. Its layout:

• Left tool rail — one button per script module/section (e.g. Prepare, the image sections, project utilities).
• Script list — the scripts in the selected section, with a Python/shell icon (missing scripts show in red) and a Dependent Scripts list you can open in your editor.
• Title bar — the script name, an optional Python environment dropdown (Python scripts only), a Run/Stop play button, Reload & Save results, Save as project default, and Show Help. A progress bar fills the bar while running.
• Setup panel — the script's Manual help text above its declared parameters, with toggles for Run in FAST mode, Show manual, Show advanced parameters, and a parameter search box.
• Output panel — live stdout/stderr with a Verbosity slider (None/Low/Medium/High); double-click to open the external log browser.
• Results / Images panel — result files and image thumbnails; right-click an image to open it in the viewer, open it in the Fourier Analyzer, or reveal it in the file system, plus a file preview / header-information view.

Panel-visibility buttons let you hide or show the Parameters, Output, Results, History, and Status panels.

AutoProcess is the parallel queue processor. Datasets sent with Process Selected from the Library — or produced by the importer — are processed concurrently. While a queue runs, the AutoProcess navigation button pulses yellow→orange and the corresponding Library rows animate through queued and processing states. This runs the same scripts as Manual Processing; only the timing and parallelism differ.

The tab is built from three panels — Concurrency Selection (how much of the machine the batch may use: parallel jobs, live CPU/RAM/disk read-outs, per-GPU use checkboxes and FAST mode), Processing Queue (the datasets waiting, with Add Images, Prioritize, Clear Selected and Clear All), and Current Status (start/stop, the live throughput and remaining-time read-out, the auto-start option, and a per-dataset progress log).

The ? button on this tab's header bar — and the full description of all three panels, the select → submit → queue workflow, the Library row colours, and a worked throughput example — are on the dedicated AutoProcessing page.

The Import tab watches a folder and brings new microscope files into the project as datasets, optionally processing them on arrival. The window is a two-column split of six panels: the left column defines what to import and how (source folder & import mode, import parameters, and the scripts to run on arrival), while the right column drives and monitors the import (status & controls, file-name parsing & delay, and the live files table). The ? button on each panel title — and the one on the IMPORT DATASETS title bar — opens the matching section of the dedicated page.

Available in 4D-STEM mode, the Planning tab helps you design a 4D-STEM experiment before acquisition:

• Ptychoscopy — plan the analysis method (e.g. iterative vs. SSB) and calibration ranges for probe, detector, pixel size, CTF, and camera length.
• ScanPattern — design the probe scan pattern.
• ScanInspect — load a scan-position .star file and visualize the probe scan pattern as dots on a canvas. Controls include the scan-generator coordinate convention, static/animated display, transparency, probe diameter (nm), animation speed/step, zoom (1–100×, with pan and scroll/pinch), and an nm-per-scan-unit scale.

The FT tab embeds the standalone ft Fourier Analyzer (maintained in its own ft repository on github and pulled in when 4d is built). It is an interactive tool for loading an image into a buffer and inspecting its Fourier transform and related analyses, with a full-screen mode. You can also reach it via the “open in Fourier Analyzer” right-click action on result images. Its Load image dialog opens by default in the bundled EXAMPLE_IMAGES directory.

The Project tab manages project-wide settings and maintenance. The header shows the mode, name, path, and image count, with buttons to Rename Project, Change Project Mode, and Repair Project Links, plus image-maintenance actions (Re-index, Reset Parameters, Renumber, Assign Even/Odd). Below are tabs for Search Parameters (view/edit parameters at project or per-dataset scope), Synchronize Image Parameters, and Utility Scripts.

The Preferences dialog has nine pages:

Help ▸ About shows the version, revision, and a short about page. Help ▸ Update checks the release server for a newer version, shows release notes (improvements / bug fixes), and offers an Update Version button; a newer version can also prompt you on startup.