← Back to the Manual Processing tab
Manual Processing — run scripts on one dataset
The Manual Processing tab is where you run individual processing scripts on a single dataset and watch the result. You open a dataset here by double-clicking it in the Library; it appears as a closable tab. You then pick a script category on the left, choose a script, set its parameters, press Run, and inspect the log, computed values and result images.
The numbered markers on the screenshot below point to each region; the legend that follows explains them. The ? button at the top-right of the window (where the dataset count normally sits) opens this page.
1
2
3
4
5
6
7
8
9
10
11
12
13
| # | Region | What it is and does |
|---|---|---|
| 1 | Dataset tab | Each dataset you open (by double-clicking it in the Library) becomes a closable tab here. A tab turns green with a running icon while its script executes and cannot be closed until it finishes. |
| 2 | Script categories | The vertical rail of script modules — Prepare, py4Dstem, TorchSlice, PtychoBones, ePIE, iDPC, Align, WASP, Process, Fibrils, Development, Utility. Pick a category to list its scripts. (See the processing sections for what each contains.) |
| 3 | Script list | The scripts in the selected category. Each row has an icon for the script's language — Python (.py), Bash (a terminal window with a >_ prompt) or C-shell (the plain script icon) — and — when the script declares a help target — a small ? button at the right that opens that script's manual page. Drag the divider at the right edge of the column to make it wider, so that long script titles are shown in full; the width is remembered for every dataset tab. A script edited in an editor while 4d is running is read again the next time it is selected, so its title, manual text, parameters and dependent scripts are shown as they now are in the file. |
| 4 | Script name | The title of the selected script. |
| 5 | Action buttons | From left: the Python-environment selector (for Python scripts), the green Run/Stop button (Run/Stop Script on Local Hardware), the blue Run on Slurm Queue button (see below), Reload & save results, and Save this configuration as project default. A progress bar fills the title while a script runs. |
| 6 | Setup toggles | Use FAST mode (sets fast_mode for the script), Show manual, Advanced parameters, Show Defaults (the default values in (8)), and a Search Parameters box. Preferences > General sets how the last three toggles start (Show manual in its Manual section). |
| 7 | Manual section | The selected script's own help text (from its # MANUAL: lines). Its ? opens the script's manual page. Hide it with the toggle in (6). It opens as tall as its text; drag the divider below it for more or less room. The buttons on its title line, left of the ?, change the size and weight of the text — see The Manual panel. |
| 8 | Parameters | The script's parameters, grouped into sections (e.g. Sample, 4D STEM, Image File). Edit the values here before running; a lock icon marks locked or global (project-wide) parameters. With Show Defaults (6) on, a grey field at the right end of each row, under the heading Default Values, shows the parameter's default value (as defined in Preferences > Parameters), in a lighter grey while the parameter holds that value; the arrow button left of it copies the default into the parameter. After a copy, that arrow turns into a cross (Undo): click it, or press Command-Z or Control-Z, to put the value from before back. Only the last copy can be undone, and only while the parameter still holds the default. The arrow is greyed out while the parameter already holds the default (like the default itself), and the button while the parameter is locked or frozen. |
| 9 | Output | The live standard-output/error log. The Verbosity Level slider chooses how much is shown — Low / Medium / High. A table that a script keeps up to date, such as the job list of the Slurm test, is rewritten in place instead of printed again. Double-click to open the external log browser. |
| 10 | Results | A Parameter → Value table of the values the script wrote back (e.g. detector dimensions, statistics, computed geometry). |
| 11 | Images | The result images the script produced. Right-click for display options (open in the viewer or the Fourier Analyzer, or reveal in the file system); the ★ toggle shows important images only. When a script is selected or has finished, the Results (10) and Images (11) panels shrink to just the values and images listed, but stay at least five lines tall, and the File Preview (12) gets the rest of the column. When the lists are long, the preview keeps at least 40 % of the column (less in a short window, so that each panel still shows five lines) and the two panels scroll. The dividers can still be dragged; a larger window gives its extra height to the preview. |
| 12 | File Preview | A preview of the selected result file, with two lines of header information above it. Images are shown as pictures and animated GIFs play in place; 3D MRC maps and stacks open in an interactive volume viewer — turn them with the mouse, show them as a surface, a projection or a slice, and animate them — and 2D MRC images are shown flat in the same viewer and open, on a double-click, in the full-screen MRC browser with its local and whole-image Fourier transforms and CTF fitting. HDF5 files show their datasets, shapes and compression; text files such as logs (.txt, .log), .results files and scripts show their contents. The File Information button (page-and-eye icon) next to the file name switches to the full header. (details) |
| 13 | Panel buttons | Show or hide the Parameters, Output and Results panels. |
Running a script
- Open a dataset by double-clicking it in the Library — it opens as a tab here (1).
- Choose a category (2) and then a script (3). Its manual text (7) and parameters (8) fill in.
- Pick the Python environment (5) if the script is a Python script, and set the parameters (8) — turn on Advanced parameters (6) if you need the less common ones.
- Press Run (5). The log streams into Output (9); raise the Verbosity Level to see more detail.
- Or run it on the Slurm queue with the blue play button to the right of the green one: the selected scripts then run on a GPU node of the cluster of Preferences ▸ System ▸ Slurm Batch Queue System, one Slurm job each (
srun), with the Python environment of the jobs set there. The job waits in the queue first; its output streams into Output (9) as from a run here, and its results and images arrive as usual. Both play buttons are greyed until the jobs have finished (a greyed button's tooltip ends in (disabled while running)); the job shows on the SlurmQueue tab, where it can be cancelled. This needs a project on a file system the nodes of the cluster mount — the file system of the shared directory of the Slurm settings; for a project elsewhere 4d says so and runs nothing. Python, csh/tcsh and bash scripts can all go to the cluster: a shell script runs there through the shell of its#!line, with${proc_4d}and${app_python}pointing to the copies of the helpers and to the Python that the nodes see. A program it calls must exist on the nodes too. Closing 4d ends the jobs it started this way. - Inspect the results: computed values in Results (10), output images in Images (11) — right-click one to open it in the viewer or Fourier Analyzer — and the header/preview in File Preview (12), where a 3D map can be turned, sliced and animated, and a 2D MRC image double-clicked into the MRC browser to look at its Fourier transform and fit the defocus to its Thon rings.
- Optionally save the parameters as the project default (5) so the next dataset starts from them.
The Manual panel
The Manual section (7) sits above the parameters (8), with a divider between them. Whenever a script is selected, the manual is fitted to its text: it is made as tall as needed to show all of it, as far as the Setup panel allows, and the parameters get the rest. A very long manual can thus leave little room for the parameters; drag the divider up to give them more. The buttons at the right end of the Manual title line, left of the ? button, adjust the text:
| Button | What it does |
|---|---|
| A- / A+ | Smaller or larger manual text, in steps of 1 pt (from 6 to 32 pt). |
| Thin / Normal / Bold | The weight of the manual text; the current one is shown pressed. |
The font family, size and weight are those of Preferences ▸ General ▸ Manual, where the family is chosen. The buttons change the same settings: they are saved for you, apply at once to every open processing window, and the Preferences page always shows the same values. Any change of the font fits the panel to the text again.
Markers, checks and results (Modern style)
With the Modern interface style (Preferences › General › Interface style) the processing window marks what needs attention:
| Mark | Meaning |
|---|---|
| ● blue dot after a parameter name | The value differs from its default. |
| ● orange dot after a parameter name | The value differs from the one the selected step last ran with in this dataset (the values baked into its copy in proc/, with the values its run wrote back). The tooltip of the name gives the earlier value. Run the step again to clear it. |
| ● orange dot on a step or a dataset tab | A parameter of that step, or of any step of that dataset, changed since the step last ran. |
| orange outline and a ⚠ line under the name | The value disagrees with the data: the accelerating voltage does not give the electron wavelength the pattern calibration used, or the probe semi angle is not the one the pattern calibration ran with (its detector cover angle and pixel size follow it). Run Data calibration again, or set the value back. |
| amber hatched field | A parameter flagged please check (ISWRONG) that has not been set yet; editing it clears the flag. |
The Results panel shows the step's files as thumbnails (Images) or the values it wrote back (Values), switched on its title line. There is no File Preview panel in this style, so the Results take the whole column: a click on a thumbnail selects it, a double-click or Space shows it in Quick Look, and a right-click offers its menu (open in the pre-set viewer or the Fourier Analyzer, show in the file system). After a run, the Was column of the values shows the value before the run, struck through, or new for a value the run added (kept until another step is selected). The Output of the scripts sits at the bottom of the right column.
| Feature | How to use it |
|---|---|
| Quick Look | Double-click a thumbnail in Results, or select it and press Space: the file opens large over the window, in the viewer of the File Preview (MRC images with Color, Background, Bar, contrast, brightness and Reset; volumes, animations, text, HDF5). Info shows the File Information. ← → step through the files, Space, Esc, ✕ or a click beside it closes it. |
| Thumbnail size | S, M, L on the Results title line: four, three or two to a row. |
| Run menu (▾ beside Run) | Run this step (Ctrl+R), Run from here (the selected step and every one after it, Ctrl+Shift+R), Run the changed steps (those with an orange dot), and Run this step in all open datasets. The menu names the steps it would run. |
| Progress | While a step runs, a ring on its tab and on its row in the step list shows how far it is; in the stage rows of a step with stages (Run: …) the running stage shows running N % and a progress line, a finished one ✓ done, a failed one failed. |
| Trend | In Values, click ↗ trend beside a value: a chart shows it across the datasets of the group (the open one ringed, an outlier red and named). |
| Focus | The Focus button in the bottom bar hides the group column, the step list and the Results, so the parameters and the Output get the whole width; press it again to bring them back. |
| Thumbnails of MRC files | An MRC result shows its picture in the Results grid (the preview 4d writes next to it, image.mrc.png, made when missing for files up to 100 MB), with a small MRC badge; an HDF5 file shows the .h5.png written next to it, if there is one. |
| Time left | While a step runs, its running stage reads running N % · ≈ 3 min left, the step list shows the time left next to the ring, and the dataset's tab has it in its tooltip. The estimate combines the time the step took when it last finished (kept in LOGS/run_times.ini) with its progress so far. A finished stage shows the time it took, e.g. ✓ done · 3 min 12 s. |
| Error banner | When the last run of the step shown failed, a red strip over its parameters names the error (the Python exception, else the last error line). Show in Output scrolls the Output to it, Copy copies it with the error lines before it, ✕ hides it until the step runs again. A run stopped with the Run button is not reported. |
| Output | A bar over the output: a search field (every hit highlighted, Enter goes to the next one), the number of warnings (⚠) and errors (✕), each a filter that shows only its lines, Follow (keep the newest line in view) and Copy (the lines shown). Under it, while a step runs, a strip names the step, its stage and when it started, and after the run how long it took or that it failed. Six or more lines in a row that differ only in their numbers are folded into one line; click it to show them. Warnings are tinted amber, and names of files that exist are links that open the file. Double-click the title to open the log browser. |
| Page header | The title of the page and its ? are at the right end of the row of dataset tabs, so the parameters get the height of a header row. |
| Last run | Under each step's name, small and grey: when it last ran in this dataset and how long it took (yesterday 16:40 · 3 min 5 s), changed since run or failed with the time, never run, or running. The duration is the one kept in LOGS/run_times.ini. |
| Notification | When a run ends, or a step of it fails, while 4d is in the background, a desktop notification names the step, the dataset and the time it took (a failure also its error), and the window's task bar entry asks for attention. Its button (Show results or Show the error) brings 4d back with the Results or Output panel open. Nothing is shown for a run you halted, nor between the steps of a run. On Linux it needs notify-send (package libnotify-bin); without it, and on Windows and macOS, a tray icon's message is used. |
| What changed | Click the orange dot of a step: a pop-up lists the parameters changed since it last ran, the old value struck through and the new one, each with Revert (not for a locked parameter), and Revert all and Run this step. |
| Not run yet | The Results panel of a step that has not run in this dataset says so, with a Run this step button, instead of an empty grid. A step that ran but wrote no figures says that. |
| Many open datasets | When the tabs no longer fit, a ▾ at the end of the tab strip lists all open datasets, with their orange dots and the progress of a run; the one shown is in bold. |
| Before / After | When a step starts, 4d keeps a copy of the files its last run wrote (up to 64 MB each and 256 MB a step, in the user's cache, not in the dataset). Quick Look of a file the run replaced then offers Before and After (or the B key); the version before has an orange edge. |
| Step icons and the narrow list | Each WASP step has its own icon, the Python logo in its lower right corner. Drag the divider right of the step list to the left: below about 130 pixels the list shows the icons alone (the title in the tooltip, the orange dot and the progress ring beside the icon), and the dependent scripts are hidden until it is wider again. A script names its icon in its header with a line # Icon: name, a picture name.png of 4d's images (apps/resources/images; the WASP icons are drawn by apps/resources/make_wasp_icons.py). |
| Go to (Ctrl+K) | Searches the steps of all groups and every parameter they show, by name or label. ↑ ↓ and Return go to it: the step is selected and a parameter's row is scrolled to, unfolded, lit up for a moment and its field takes the keyboard (the advanced parameters are shown when needed). |
File Preview
The File Preview (12) shows the file selected in the Images or Results panel: pictures as they are, animated GIFs playing in place, text files and scripts as their contents, HDF5 files as their information sheet, and 3D MRC maps and stacks in an interactive volume viewer that turns, slices and animates them. A 2D MRC image is shown flat in that viewer; a double-click on it, or on the file in the Images panel, opens the full-screen MRC browser: the old full-screen viewer with its magnifier, coordinate readout and menu, beside a panel with the display controls, the Fourier transform of a dragged selection or of the whole image, and the fitting of the defocus, astigmatism and its angle to the Thon rings — with spin boxes, sliders, or by dragging the rings. The File Information button in the panel title switches to the file's full header. The panel has a manual page of its own, which the ? button next to that switch opens: File Preview — images, animations and 3D volumes, with the volume viewer's mouse actions, display modes, animation and view buttons, and the MRC browser.