4d — Utilities & Development

Processing scripts · 4d User Manual

← Back to “Processing sections & scripts”

Utilities & Development

Utilities holds general housekeeping and data-management helpers, and the test of the Slurm batch queue; Development holds experimental focus-pair scripts.

Utilities

Each script's ? help button opens its entry below (where one exists).
10 — Push Results to STATUS web server

Purpose

Publishes the results of the current image to the local status folder — the directory a separate CRON job uploads to the 4d status web server. It writes 400×400 annotated thumbnails of the most informative result images, keeps a short history of the two main thumbnails, and appends one line of key numbers to a per-microscope log that the status page plots as time series. It is normally the last script of an autoprocessing run, either from Utilities or via the equivalent step in the processing queue.

This script only makes sense on a computer on which a CRON job is already running that uploads the contents of the status folder to the 4d web server. Without such a job the files are still written, but nobody sees them.

Parameters

ParameterRole
status_folderAbsolute path of the local status folder (Status server settings). none or empty switches the script off — it exits without writing anything.
status_folder_updatey/n: if n, only the two symlinks are checked/repaired and the script stops before any image or log is written.
status_microscopeMicroscope tag; prefixes every file written, so several microscopes can share one status folder (<tag>-image-1.png, <tag>.data.log, …).
website_blur_doit, website_blurkernelBlur the thumbnails that show real sample structure, and the Gaussian kernel diameter (px) used for it. Only the parallax reconstruction, the BF and the DF image are blurred; maps, histogram and FFT never are.
values read from earlier stepsparallax_defocus (annotation and log), meanValue, BF_outer, RonchiCenterOffsetX/Y, shadow_cf, shadow_rotation, defocus_RESMAX, flag_local_ctfresolutiontoobad_threshold, image_stddev_relative.

Algorithm

  1. Defaults image_stddev_relative to 0.0 if the CTF step never filled it in, and stamps the current time (ms since the epoch) as the status date.
  2. Creates status_folder if needed, then checks the two symlinks in the image directory — LOCAL_STATUS_thumbnails → <status_folder>/thumbnails and LOCAL_STATUS_logs → <status_folder>/logs — and creates or re-points them if they are missing, wrong, or not a symlink.
  3. Annotates eight result images into <tag>-image-1.png … -8.png: resize to 400×400, optionally Gaussian-blur, and draw a yellow caption on a black box in the bottom-left corner. Any missing image is replaced by the placeholder Image-Z.png; -9.png … -12.png are always placeholders.
  4. Shifts the thumbnail history of rows 1 and 2 one column to the right (-N-7 → -N-8, …, -N-1 → -N-2) and copies the new -1.png / -2.png into the freed first column, giving the status page an eight-image history strip per row.
  5. Appends one semicolon-separated line to logs/<tag>.data.log; the Ronchi entry is the length √(RonchiCenterOffsetX² + RonchiCenterOffsetY²) and the defocus is converted from nm to µm.
  6. Registers the twelve thumbnails in the results file so they show up in the Images pane, and appends the current time to logs/<tag>.last_good.txt when defocus_RESMAX is within flag_local_ctfresolutiontoobad_threshold.
  7. Finally adds read+write permission for user, group and other to everything below the status folder, so any user's processing run can keep the web server fed.

Results

Images (in <status_folder>/thumbnails/): <tag>-image-1 parallax reconstruction (with its defocus in the caption) · -2 average pattern · -3 StdDev of the average pattern · -4 parallax shift map · -5 grey-value histogram · -6 BF image · -7 DF image · -8 FFT of the parallax reconstruction · -9…-12 placeholders. Plus the history copies <tag>-image-1-1…1-8 and <tag>-image-2-1…2-8.

Data written (in <status_folder>/logs/): one line per run in <tag>.data.log with the columns status_date ; meanValue ; BF_outer ; RonchiCenterOffset ; shadow_cf ; shadow_rotation ; parallax_defocus [µm] — this column order is what the status page plots, so it must stay in sync with the web front end — and a timestamp per good image in <tag>.last_good.txt.

40 — Slurm batch queue – test

Purpose

A test of the way a processing script hands its work to the Slurm cluster, with the settings of Preferences › System › Slurm Batch Queue System. It needs no data: it makes its own test image and reports what happened at every step, so a wrong host, key, partition or shared directory shows up here before a real processing step depends on it.

Settings

The script has no parameters of its own. Everything comes from the Slurm Batch Queue System section of the preferences: whether the queue is used at all, the login host and its SSH access, the partition and account, the resources of one job (GPUs, CPU cores, memory, time limit), the shared NFS directory as seen here and on the nodes, the environment setup and the Python command on the nodes, and the polling interval.

What it does

  1. Test image. A 10000 × 10000 image with a gray-value design (a gradient, rings, a checkerboard, a diagonal bar, a disk and thin lines along the tile borders, so that the rotation of every tile is seen at a glance), saved as slurm_test_input.mrc (MRC mode 1) in the image directory.
  2. Tiles. The image is cut into 16 tiles of 2500 × 2500 pixels, written to a job folder 4d_slurm_jobs/slurm_test_USER_TIME below the shared NFS directory, together with the worker and one batch script per tile.
  3. Submission. The 16 batch scripts are submitted in one call on the login host (over SSH, or on this computer when the login host is empty). Each job runs dependents/slurm_tile_worker.py, a stand-alone script that needs only numpy (and uses the GPU through cupy when that is installed): it rotates its tile by 45° about the tile centre, bilinearly, into an image of the same size, fills the corners the rotation leaves empty with black, and writes the result as MRC mode 1. Every job is paced to take at least 30 s, so that the progress can be watched.
  4. Watching. Every polling interval the step asks squeue for the state of its jobs and reads the small JSON progress file every job rewrites on the shared directory. The log shows the table of the 16 jobs, rewritten in place at every poll, so that it stays one current table however long the jobs run: job id, state (PENDING, RUNNING, COMPLETED, FAILED …), node, device, a progress bar and a note (the pending reason, the run time, the error, or why a job uses the CPU although a GPU was asked for). The progress bar of the step follows the mean progress of the jobs. A job that leaves the queue without a result is reported as failed, with the end of its error log.
  5. Collage. The 16 rotated tiles are read back from the shared directory and assembled into slurm_test_collage.mrc; slurm_test_overview.png shows the input image next to the collage. After a successful run the tiles are removed from the job folder; the batch scripts, the logs and the progress files stay.

Without a cluster

With Use the Slurm batch queue off, the 16 jobs run as local subprocesses, several at a time, with the same table and the same results. This checks the tiling, the worker and the collage on any computer.

Stopping and cleaning up

Stopping the step cancels the jobs it submitted (scancel). Their ids are also in slurm_test_jobs.json in the job folder, for a scancel by hand should that fail.

Results

Images: slurm_test_overview.png (input left, collage right), slurm_test_input.mrc, slurm_test_collage.mrc. Log: the job table, and a summary with the nodes and devices used and the run times; a warning lists the jobs that ran on the CPU although a GPU was asked for.

Paths. The job folder is written here under the local mount of the shared directory, and the batch scripts refer to it by its path on the nodes (Shared NFS directory, as mounted on the cluster nodes). The login host is taken to see it under the same path as the nodes. The progress files are read here through the local mount; NFS attribute caching may delay them by a few seconds.
Development contains experimental focus-pair scripts (import from SerialEM, CTF, drift correction, iciness, process). Most are explicitly marked “Under Development. Do not use.” — treat them as advanced/experimental only.