4d — Utilities & Development

← Back to “Processing sections & scripts”

Utilities & Development

Utilities holds general housekeeping and data-management helpers; Development holds experimental focus-pair scripts.

Utilities

Crop Patterns from HDF5 and save as Numpy Array — crop the central n×n scan positions into an output file.
Python Environment Test Script — print the active Python environment to verify the environment dropdown.
Merge several HDF5 files into one — merge HDF5 files from the FRAMES directory into one BLOSC2-compressed file.
Image Inventory — list image files and show them in the Images pane.
Cleanup — delete non-essential files to save disk space.
Push Results to STATUS web server — create annotated status images and a log for the 4d status webserver.

Each script's ? help button opens its entry below (where one exists).

10 — Push Results to STATUS web server

Purpose

Publishes the results of the current image to the local status folder — the directory a separate CRON job uploads to the 4d status web server. It writes 400×400 annotated thumbnails of the most informative result images, keeps a short history of the two main thumbnails, and appends one line of key numbers to a per-microscope log that the status page plots as time series. It is normally the last script of an autoprocessing run, either from Utilities or via the equivalent step in the processing queue.

This script only makes sense on a computer on which a CRON job is already running that uploads the contents of the status folder to the 4d web server. Without such a job the files are still written, but nobody sees them.

Parameters

Parameter	Role
`status_folder`	Absolute path of the local status folder (Status server settings). `none` or empty switches the script off — it exits without writing anything.
`status_folder_update`	y/n: if `n`, only the two symlinks are checked/repaired and the script stops before any image or log is written.
`status_microscope`	Microscope tag; prefixes every file written, so several microscopes can share one status folder (`<tag>-image-1.png`, `<tag>.data.log`, …).
`website_blur_doit`, `website_blurkernel`	Blur the thumbnails that show real sample structure, and the Gaussian kernel diameter (px) used for it. Only the parallax reconstruction, the BF and the DF image are blurred; maps, histogram and FFT never are.
values read from earlier steps	`parallax_defocus` (annotation and log), `meanValue`, `BF_outer`, `RonchiCenterOffsetX/Y`, `shadow_cf`, `shadow_rotation`, `defocus_RESMAX`, `flag_local_ctfresolutiontoobad_threshold`, `image_stddev_relative`.

Algorithm

Defaults image_stddev_relative to 0.0 if the CTF step never filled it in, and stamps the current time (ms since the epoch) as the status date.
Creates status_folder if needed, then checks the two symlinks in the image directory — LOCAL_STATUS_thumbnails → <status_folder>/thumbnails and LOCAL_STATUS_logs → <status_folder>/logs — and creates or re-points them if they are missing, wrong, or not a symlink.
Annotates eight result images into <tag>-image-1.png … -8.png: resize to 400×400, optionally Gaussian-blur, and draw a yellow caption on a black box in the bottom-left corner. Any missing image is replaced by the placeholder Image-Z.png; -9.png … -12.png are always placeholders.
Shifts the thumbnail history of rows 1 and 2 one column to the right (-N-7 → -N-8, …, -N-1 → -N-2) and copies the new -1.png / -2.png into the freed first column, giving the status page an eight-image history strip per row.
Appends one semicolon-separated line to logs/<tag>.data.log; the Ronchi entry is the length √(RonchiCenterOffsetX² + RonchiCenterOffsetY²) and the defocus is converted from nm to µm.
Registers the twelve thumbnails in the results file so they show up in the Images pane, and appends the current time to logs/<tag>.last_good.txt when defocus_RESMAX is within flag_local_ctfresolutiontoobad_threshold.
Finally adds read+write permission for user, group and other to everything below the status folder, so any user's processing run can keep the web server fed.

Results

Images (in <status_folder>/thumbnails/): <tag>-image-1 parallax reconstruction (with its defocus in the caption) · -2 average pattern · -3 StdDev of the average pattern · -4 parallax shift map · -5 grey-value histogram · -6 BF image · -7 DF image · -8 FFT of the parallax reconstruction · -9…-12 placeholders. Plus the history copies <tag>-image-1-1…1-8 and <tag>-image-2-1…2-8.

Data written (in <status_folder>/logs/): one line per run in <tag>.data.log with the columns status_date ; meanValue ; BF_outer ; RonchiCenterOffset ; shadow_cf ; shadow_rotation ; parallax_defocus [µm] — this column order is what the status page plots, so it must stay in sync with the web front end — and a timestamp per good image in <tag>.last_good.txt.

Development contains experimental focus-pair scripts (import from SerialEM, CTF, drift correction, iciness, process). Most are explicitly marked “Under Development. Do not use.” — treat them as advanced/experimental only.