← Back to “Processing sections & scripts”
Utilities & Development
Utilities holds general housekeeping and data-management helpers, and the test of the Slurm batch queue; Development holds experimental focus-pair scripts.
Utilities
- Crop Patterns from HDF5 and save as Numpy Array — crop the central n×n scan positions into an output file.
- Python Environment Test Script — print the active Python environment to verify the environment dropdown.
- Slurm batch queue – test — cut a 10000×10000 test image into 16 tiles, rotate each by 45° in a Slurm job and reassemble the collage; checks the Slurm settings of Preferences › System.
- Merge several HDF5 files into one — merge HDF5 files from the FRAMES directory into one BLOSC2-compressed file.
- Image Inventory — list image files and show them in the Images pane.
- Cleanup — delete non-essential files to save disk space.
- Push Results to STATUS web server — create annotated status images and a log for the 4d status webserver.
10 — Push Results to STATUS web server
Purpose
Publishes the results of the current image to the local status folder — the directory a separate CRON job uploads to the 4d status web server. It writes 400×400 annotated thumbnails of the most informative result images, keeps a short history of the two main thumbnails, and appends one line of key numbers to a per-microscope log that the status page plots as time series. It is normally the last script of an autoprocessing run, either from Utilities or via the equivalent step in the processing queue.
Parameters
| Parameter | Role |
|---|---|
status_folder | Absolute path of the local status folder (Status server settings). none or empty switches the script off — it exits without writing anything. |
status_folder_update | y/n: if n, only the two symlinks are checked/repaired and the script stops before any image or log is written. |
status_microscope | Microscope tag; prefixes every file written, so several microscopes can share one status folder (<tag>-image-1.png, <tag>.data.log, …). |
website_blur_doit, website_blurkernel | Blur the thumbnails that show real sample structure, and the Gaussian kernel diameter (px) used for it. Only the parallax reconstruction, the BF and the DF image are blurred; maps, histogram and FFT never are. |
| values read from earlier steps | parallax_defocus (annotation and log), meanValue, BF_outer, RonchiCenterOffsetX/Y, shadow_cf, shadow_rotation, defocus_RESMAX, flag_local_ctfresolutiontoobad_threshold, image_stddev_relative. |
Algorithm
- Defaults
image_stddev_relativeto0.0if the CTF step never filled it in, and stamps the current time (ms since the epoch) as the status date. - Creates
status_folderif needed, then checks the two symlinks in the image directory —LOCAL_STATUS_thumbnails→<status_folder>/thumbnailsandLOCAL_STATUS_logs→<status_folder>/logs— and creates or re-points them if they are missing, wrong, or not a symlink. - Annotates eight result images into
<tag>-image-1.png…-8.png: resize to 400×400, optionally Gaussian-blur, and draw a yellow caption on a black box in the bottom-left corner. Any missing image is replaced by the placeholderImage-Z.png;-9.png…-12.pngare always placeholders. - Shifts the thumbnail history of rows 1 and 2 one column to the right (
-N-7→-N-8, …,-N-1→-N-2) and copies the new-1.png/-2.pnginto the freed first column, giving the status page an eight-image history strip per row. - Appends one semicolon-separated line to
logs/<tag>.data.log; the Ronchi entry is the length√(RonchiCenterOffsetX² + RonchiCenterOffsetY²)and the defocus is converted from nm to µm. - Registers the twelve thumbnails in the results file so they show up in the Images pane, and appends the current time to
logs/<tag>.last_good.txtwhendefocus_RESMAXis withinflag_local_ctfresolutiontoobad_threshold. - Finally adds read+write permission for user, group and other to everything below the status folder, so any user's processing run can keep the web server fed.
Results
Images (in <status_folder>/thumbnails/): <tag>-image-1 parallax reconstruction (with its defocus in the caption) · -2 average pattern · -3 StdDev of the average pattern · -4 parallax shift map · -5 grey-value histogram · -6 BF image · -7 DF image · -8 FFT of the parallax reconstruction · -9…-12 placeholders. Plus the history copies <tag>-image-1-1…1-8 and <tag>-image-2-1…2-8.
Data written (in <status_folder>/logs/): one line per run in <tag>.data.log with the columns
status_date ; meanValue ; BF_outer ; RonchiCenterOffset ; shadow_cf ; shadow_rotation ; parallax_defocus [µm]
— this column order is what the status page plots, so it must stay in sync with the web front end — and a timestamp per good image in <tag>.last_good.txt.
40 — Slurm batch queue – test
Purpose
A test of the way a processing script hands its work to the Slurm cluster, with the settings of Preferences › System › Slurm Batch Queue System. It needs no data: it makes its own test image and reports what happened at every step, so a wrong host, key, partition or shared directory shows up here before a real processing step depends on it.
Settings
The script has no parameters of its own. Everything comes from the Slurm Batch Queue System section of the preferences: whether the queue is used at all, the login host and its SSH access, the partition and account, the resources of one job (GPUs, CPU cores, memory, time limit), the shared NFS directory as seen here and on the nodes, the environment setup and the Python command on the nodes, and the polling interval.
What it does
- Test image. A 10000 × 10000 image with a gray-value design (a gradient, rings, a checkerboard, a diagonal bar, a disk and thin lines along the tile borders, so that the rotation of every tile is seen at a glance), saved as
slurm_test_input.mrc(MRC mode 1) in the image directory. - Tiles. The image is cut into 16 tiles of 2500 × 2500 pixels, written to a job folder
4d_slurm_jobs/slurm_test_USER_TIMEbelow the shared NFS directory, together with the worker and one batch script per tile. - Submission. The 16 batch scripts are submitted in one call on the login host (over SSH, or on this computer when the login host is empty). Each job runs
dependents/slurm_tile_worker.py, a stand-alone script that needs only numpy (and uses the GPU through cupy when that is installed): it rotates its tile by 45° about the tile centre, bilinearly, into an image of the same size, fills the corners the rotation leaves empty with black, and writes the result as MRC mode 1. Every job is paced to take at least 30 s, so that the progress can be watched. - Watching. Every polling interval the step asks
squeuefor the state of its jobs and reads the small JSON progress file every job rewrites on the shared directory. The log shows the table of the 16 jobs, rewritten in place at every poll, so that it stays one current table however long the jobs run: job id, state (PENDING, RUNNING, COMPLETED, FAILED …), node, device, a progress bar and a note (the pending reason, the run time, the error, or why a job uses the CPU although a GPU was asked for). The progress bar of the step follows the mean progress of the jobs. A job that leaves the queue without a result is reported as failed, with the end of its error log. - Collage. The 16 rotated tiles are read back from the shared directory and assembled into
slurm_test_collage.mrc;slurm_test_overview.pngshows the input image next to the collage. After a successful run the tiles are removed from the job folder; the batch scripts, the logs and the progress files stay.
Without a cluster
With Use the Slurm batch queue off, the 16 jobs run as local subprocesses, several at a time, with the same table and the same results. This checks the tiling, the worker and the collage on any computer.
Stopping and cleaning up
Stopping the step cancels the jobs it submitted (scancel). Their ids are also in slurm_test_jobs.json in the job folder, for a scancel by hand should that fail.
Results
Images: slurm_test_overview.png (input left, collage right), slurm_test_input.mrc, slurm_test_collage.mrc. Log: the job table, and a summary with the nodes and devices used and the run times; a warning lists the jobs that ran on the CPU although a GPU was asked for.