← Back to the SlurmQueue tab · AutoProcessing page (the LocalQueue tab) · Slurm settings
SlurmQueue — the cluster and your jobs on it
The LocalQueue tab runs scripts on the cores of this computer. Some processing steps can hand their work to a Slurm GPU cluster instead, cut into jobs that run on the cluster's nodes at the same time. The SlurmQueue tab — in the navigation bar right after LocalQueue, with the icon of several computers — is where you see that cluster: how busy its nodes are, which of your jobs wait, run or have finished today, and what became of them. It also holds the Slurm settings, so that the partition or the resources of a job can be changed while looking at the nodes the jobs will run on.
The jobs come from the datasets launched on it from the Library (below), and from the steps, while they run — the test Utilities ▸ 40 Slurm batch queue – test, and WASP ▸ 44 Ptycho – SLURM version, which reconstructs every ROI of its stage B in a job of its own — and the blue Run on Slurm Queue button of the Manual Process tab, which runs any script of an image — Python, csh/tcsh or bash — on a node. The tab itself submits only the short probe jobs of Probe hardware. SlurmQueue watches what was submitted, and lets you cancel it or move it forward among your own jobs. The cluster itself, and the Python environment the jobs need on it, are set up as described in the installation guide; the settings are explained under Preferences ▸ System ▸ Slurm Batch Queue System. While Use the Slurm batch queue is off there, 4d shows nothing of the queue: this tab, the blue button of Manual Process, Launch on Slurm Queue and the Import tab's queue switch are hidden.
The tab is divided in three. A vertical divider splits the top part into the job settings on the left and the nodes on the right; below a horizontal divider lies your queue.
The screenshot shows the tab during a launch of 50 datasets from the Library: 20 of them run their import step on the cluster, one GPU each, every node is full, and the other 30 wait in 4d for room.
1
2
3
4
5
6
7
8
9
10
11
12
| # | Region | What it is |
|---|---|---|
| 1 | SlurmQueue | The tab's button in the navigation bar. It pulses in icy blue–white while you have jobs on the cluster, and it is hidden while the Slurm batch queue is switched off in the preferences. |
| 2 | Settings of a job | Partition, CPU cores, CPU memory, GPUs and time limit of every job 4d submits; edited here, saved to the preferences (below). |
| 3 | Default Values | The default of each setting; the arrow next to a field copies it in. |
| 4 | Cluster Nodes | The nodes of the partition: state, GPU type and memory, GPUs, CPUs and memory in use of the total, load (below); the hover window of a node tells everything Slurm and the probe know about it. Here all five are ALLOCATED: every GPU is taken. |
| 5 | Refresh, Probe hardware | Ask Slurm for the nodes now; run the short probe job that records GPU model, memory and driver of each node. |
| 6 | Busy | The largest of the three uses of a node; red from 85 %. |
| 7 | Queue buttons | Prioritize a pending job, Cancel Selected, Cancel All Mine — running jobs are asked to end, not killed. |
| 8 | Status line | When the queue was last asked, how many of your jobs are pending, running and finished, and during a launch from the Library how many datasets are on the cluster and how many still wait in 4d (here 20 and 30). |
| 9 | Image | The dataset folder a job ran for. Double-click a job to open its dataset in Manual Process, at its script. |
| 10 | Finished jobs | At the top, the oldest first: here the Prepare Folders step of each dataset, completed (white). |
| 11 | Running jobs | Below them, in light yellow: the import step of the 20 datasets on the cluster. The newest jobs are at the bottom, where the table opens. |
| 12 | Log line | The last thing 4d did with the cluster: here the submission of a launched dataset's next script. |
Top left — the settings of a job
The left panel holds the settings that change from one run to the next: where a
job goes and how big it is. They are five of the Slurm Batch Queue System
settings of Preferences ▸ System, the
same values in ~/.4d/4d.cfg, and a change made here counts at once:
the next step that submits jobs uses it. The note at the panel's title says it:
these and all other Slurm settings — the switch that turns the batch queue on, the
login host and SSH access, the account, the shared directory, the Python of the
jobs, the polling interval — are in Preferences ▸ System, which explains every one
with its default. Right of each field is the value 4d was installed with, under
Default Values, with the arrow that copies it into the field, as in the
Manual Process tab. The ? button of the panel opens this page.
| Field | What it does |
|---|---|
| Slurm partition | Where the jobs go. The node table on the right shows the nodes of this partition. |
| CPU cores per job, CPU memory per job, GPUs per job, Time limit per job | What one job asks for. Compare with the node table on the right: more cores or memory than a node's share per GPU leaves GPUs idle, and a job that asks for more than any node has stays pending for ever, with the reason in its Note. A job that runs longer than its time limit is ended by Slurm (TIMEOUT). |
| ? | Opens this page. |
The tab asks for your queue once every polling interval of the preferences (every 5 s by default; never more often than every 2 s), and for the nodes at most every 10 s; both four times less often while another tab is shown, and Refresh asks at once.
Top right — the nodes of the cluster
The right panel lists the nodes of the Slurm partition set on the left — the whole cluster when the partition is empty or names no node — one row each, as Slurm reports them, and refreshes the list at most every 10 s; a changed partition shows at the next refresh. The columns:
| Column | What it shows |
|---|---|
| Node | The node's name without the domain (lbem-gpu2 for lbem-gpu2.epfl.ch). |
| State | Slurm's state of the node as scontrol writes it: IDLE (nothing runs on it), MIXED (part of it is allocated), ALLOCATED (all of it), and with a flag when it is out of service: IDLE+DRAIN or MIXED+DRAIN (draining: no new jobs, out of service once its jobs end), DOWN, DOWN+DRAIN, DOWN+NOT_RESPONDING; the administrator's reason is in the hover window. Such a node is shown in grey. |
| GPU type | The GPU model without the maker's name, e.g. RTX A5000 or A40. It comes from Probe hardware (Slurm does not know it): a grey ? until the node has been probed. |
| GPU RAM | The memory of one GPU, e.g. 24 GB, also from the probe. It bounds how large a job's piece of work may be on that node: the ROIs of a ptychography run that fit on an A40 may not fit on an RTX A5000. |
| GPUs | Used / total, e.g. 1 / 4; empty for a node without GPUs. |
| CPUs | Allocated / total cores. |
| Memory | Allocated / total, in GB. |
| Load | The node's load average, as Slurm last read it: roughly the number of busy cores. A load far below the allocated cores means jobs that wait for the GPU or for the disk. |
| Busy | A bar with the largest of the three uses — GPUs, CPUs, memory — since a node is full as soon as one of them is: green below 50 % use, orange from 50 %, red from 85 %, and grey for a node that is down or drained, whatever it holds. |
Hover a row and a small window opens with everything known about
the node: its full name; the state and, for a node that is down or drained, the
reason, who set it and when; its partitions and features (Ampere, say);
cores and load; memory, total and free; the GPUs, used and total; the Slurm
version, the operating system and the boot time; and the jobs running on it right
now, with their users, so you can see whose work fills a node. Once the hardware
has been probed, the window also shows the GPU model and memory, the driver and
CUDA version, the compute capability, the CPU model, the CUDA toolkits installed
and the root disk.
| Button | What it does |
|---|---|
| Refresh | Asks Slurm for the nodes now, without waiting for the next polling interval. |
| Probe hardware | Submits, after a question, one short job per node that reports the hardware Slurm does not know; see below. |
Where the node information comes from
The table and most of the hover window come from Slurm's own records:
scontrol show node for the nodes and squeue for the jobs
on them. Slurm counts GPUs, cores and memory, but it does not know the GPU model,
the driver or the CPU. Probe hardware fills that gap: after
asking, it submits one short job to each node in turn (srun on that
node, a few seconds of nvidia-smi and lscpu) and records
what the job reports. The probe jobs are ordinary jobs of yours — they appear in
the queue below, and run one at a time. A node that is down or drained is left out. On a node that has no room for even
that at the moment, the probe job waits up to a minute for a slot and then gives
up; probe again later. The results are kept in ~/.4d/slurm_nodes.json on this
computer and merged into the table (GPU type, GPU RAM) and the
hover windows from then on, so one probe suffices
until the cluster's hardware changes.
Bottom — your Slurm queue
Below the horizontal divider is the Slurm queue of this user: every job on the cluster under your account — submitted by a 4d step, by a probe, or by hand in a terminal — and the jobs of yours that finished today. Four buttons sit above the table:
| Button | What it does |
|---|---|
| Prioritize | Moves the selected pending job ahead of your other pending jobs, so that it starts first when the cluster has room. Below, what that can and cannot do. On a dataset waiting in 4d (WAITING), it makes that dataset the next one to go to the cluster. |
| Cancel Selected | Ends the highlighted jobs (below): pending ones are cancelled, running ones are asked to end. When one of them is running, 4d asks first. A finished job is skipped. Pressed again on a job that was asked to end and still runs, it offers to cancel it outright (scancel), with a warning. |
| Cancel All Mine | Ends every job of yours on the cluster the same way, after a question — the probe jobs and the jobs you submitted by hand included. |
| Refresh | Polls now. |
scancel makes Slurm kill the job's processes. A process that is still
writing a large file to the shared storage cannot go before its data has reached the
server; when that takes longer than Slurm waits (60 s on this cluster), Slurm takes
the whole node out of service with the reason Kill task failed, and it stays
drained until an administrator runs
sudo scontrol update NodeName=<node> State=RESUME. So 4d sends a
running job SIGTERM (scancel --signal=TERM) instead: the scripts stop, every
process of the job finishes writing its files, and the job leaves the queue once all of
them have ended — in the table as FAILED with signal 15, after the note
asked to end. Slurm has nothing left to kill. A pending job has no processes
yet and is simply cancelled. The same holds when 4d stops a launched dataset, when it
quits, and for the ROI jobs of the Slurm ptychography.
The columns of the table:
| Column | What it shows |
|---|---|
| Job | Slurm's id; a task of a job array as 8559_3. Empty for a dataset waiting in 4d, which has no job yet. |
| Script | The 4d script the job runs, as the folder of the image scripts and its file (WASP/03_drift_correction.py); for a job that is not a 4d script, its name. The job name is in the row's hover window. |
| Image | The folder of the image the job ran for (0000000313); empty for a job of no image, such as the probe of a node. A job of the Manual Process tab runs in the image folder; the jobs a step submits itself, such as the ROIs of the Slurm ptychography, run in a job folder below 4d_slurm_jobs, whose note 4d_job_origin.json names the image and the step. The hover window shows the whole path and the script. |
| State | PENDING, RUNNING, COMPLETING, COMPLETED, FAILED, TIMEOUT, OUT_OF_MEMORY, CANCELLED, NODE_FAIL … |
| Node | The node it runs or ran on; empty while pending. |
| GPU | The GPU card(s) of that node the job has, numbered as Slurm and nvidia-smi number them, from 0 (0,2,3 for three). Known while the job runs; a job that finished during the session keeps the cards it had. |
| Elapsed | The running time so far, or the total of a finished job, in Slurm's notation (12:34, 1:02:03, 1-02:03:04). |
| Limit | The time limit the job asked for; Slurm kills it when it runs longer (TIMEOUT). |
| CPUs, Memory, GPUs | What the job asked for (16, 60G, 1; the full GPU request, gres/gpu:1, is in the row's hover window). |
| Note | For a pending job, Slurm's reason why it waits: Resources (the cluster is full), Priority (other jobs come first), QOSMaxGRESPerUser and the like (a limit of your account), ReqNodeNotAvail (no node can satisfy the request — compare what the job asks for with the node table). For a finished job, its exit code and when it ended. |
Double-click a job to go to its image: 4d opens the image in the Manual Process tab and shows the script of the job there, selected in its module, with its log and results. A job of an image that is not part of the open project is said so; a job of no image does nothing but say that in the status line.
How the table is ordered and coloured
The rows are in the order in which the jobs leave the queue: the finished jobs at the top, the oldest first, then the running ones, then the pending ones. A new job therefore appears at the bottom and climbs as the jobs above it finish — the table reads like a log, and its last rows are what is still to come. Like a log, it opens at its bottom and stays there as rows come in; scroll up to older jobs and it keeps your place, until you scroll back down to the end.
| Row colour | State | Meaning |
|---|---|---|
| Light blue | Pending | Submitted and waiting for a node; the Note says why. |
| Light yellow | Running | On a node now (or completing). |
| White | Completed | Finished with exit code 0. |
| Light red | Failed | Ended badly: a non-zero exit code, the time limit (TIMEOUT), out of memory (OUT_OF_MEMORY) or a node failure. The log of the step says what the job wrote. |
| Grey | Cancelled | Cancelled — from here, or by a step that was stopped. |
| Pale violet | Waiting | A dataset launched from the Library that waits in 4d for room on the cluster: its job is not submitted yet. The row shows the job it will become, with the settings of now; the Note its place in the line. Cancel Selected drops it from the line, Prioritize puts it at the front. At the very bottom of the table, after the pending jobs; the next 100 of them are listed. |
The finished jobs come from Slurm's accounting (sacct): when the tab
starts polling, it reads every job of yours that ended since the start of the
day, and a job that finishes while 4d runs joins them. They stay listed for the
session, past midnight too; the next start of 4d shows those of that day. The table
lists the newest 100 of them; the status line counts all.
What Prioritize can and cannot do
A Slurm user may lower the priority of their own jobs but not raise it, and the order among the jobs of different users is the scheduler's. Prioritize therefore works only among your own pending jobs: it puts the selected one at the front of them, and leaves everyone else's jobs, and your running ones, where they are.
4d does it in one of two ways. It first tries Slurm's scontrol top,
which moves a job ahead of the user's other pending jobs; a cluster allows that
to users only when its scheduler is configured with enable_user_top. Nor can it help on a cluster that gives every job the same priority, as one without priority weights does (every job shows priority 1): a nice value changes nothing there, the tab says so after trying, and the pending jobs keep the order in which they were submitted.
When the cluster refuses, 4d instead gives your other pending jobs a
higher nice value than the selected one — a positive nice lowers a job's
priority, which users may do — so that the selected job comes first among them
all the same. Either way the change is to the order among your jobs, not to when
the cluster has room for them; and a job the scheduler has already picked may
start before the change counts.
Launching datasets from the Library
Process Selected in the Library opens the
Scripts To Be Executed dialog. Its
Launch on Slurm Queue button (the blue triangle, right of
Launch on Local Queue) runs the chosen scripts on every selected dataset on
the cluster, and brings you to this tab. Each dataset runs its scripts one after the
other, each script as one Slurm job, as the LocalQueue runs them on this computer:
the log of a script goes to the dataset's LOGS folder as it is written,
and its results are saved before the next script starts. Python, csh/tcsh and bash
scripts can all go; the project must lie on the file system the nodes mount, as for
the Manual Process tab. The helpers the scripts call (scripts/proc, the
script's folder, Torchslice) reach the nodes through one copy in the shared directory,
4d_slurm_code, which 4d brings up to date when they have changed.
4d keeps as many datasets on the cluster at once as the partition can run jobs of
the settings at once — for lbem-gpus with 1 GPU, 16 cores and 60 GB
per job that is 20 — and starts the next dataset whenever one has run its last
script. The others wait in 4d, listed at the bottom of the table as WAITING
rows (pale violet), where they can be dropped or moved to the front; the status line
above the table counts both. A dataset launched already is not added again until it
is done.
Cancel Selected on a job of a launched dataset stops that dataset — its
later scripts are not submitted — and Cancel All Mine stops every launched
dataset, the waiting ones included. Closing 4d ends them too: the launched datasets are not kept for the next
start, unlike the queue of the LocalQueue.
The pulsing button, and where the commands run
While you have jobs pending or running on the cluster, the SlurmQueue button in the navigation bar pulses in icy blue–white (the LocalQueue button pulses yellow–orange while its queue runs) — so you see from any tab that the cluster is still working for you. It stops when the last of your jobs has finished.
The tab runs Slurm's commands — squeue, sacct,
scontrol, scancel and, for Probe hardware,
srun — where the steps run theirs: on
this computer when it is a submit host (the login host of the settings empty),
otherwise on the login host over SSH, with the port, user, key and options of the
settings. Nothing is installed on the cluster for it. The window's log line shows
what was last run and what came of it, with Slurm's message when a command
failed.
When the tables stay empty
- Slurm is not switched on. With Use the Slurm batch queue off, the tab polls nothing and the steps work locally. Switch it on in Preferences ▸ System ▸ Slurm Batch Queue System.
- This computer is no submit host and no login host is set. The commands run here and fail (
squeue: command not foundin the log line). Enter the login host, user and key, then check from a terminal thatssh user@host squeue --versionanswers without asking for anything. sacctis not available. On a cluster without accounting storage (noslurmdbd), or where users may not query it, pending and running jobs still show, but no finished ones: the job table empties as your jobs complete, and the exit codes of finished jobs are unknown to the steps too. Ask the cluster's administrator.- Jobs, but no nodes.
scontrol show nodefailed or is restricted on this cluster; the log line says. - A job pends for ever. Read its Note:
ReqNodeNotAvail, or a reason withMaxin it, means that what the job asks for exceeds a node or your limits. Lower GPUs, cores or memory per job in the settings and submit again from the step. - The hover window has no GPU model. The node has not been probed yet: press Probe hardware.