Slurm_tools: shell scripts for cluster monitoring and accounting

A toolbox written for one specific cluster

The README is explicit about the origin. These Slurm tools have been developed for management and monitoring of our cluster, and the description of the repository is simply my tools for the Slurm HPC workload manager. The primary language is Shell.

That framing is the most important fact for an evaluator. This is not a general Slurm administration suite with a test matrix and a support contract. It is the set of commands one site administrator wrote because the built-in Slurm tools did not answer the questions they needed answered, and then made public.

There are fifteen numbered entries in the list, and they cover the territory you would expect. Node and cluster status come first, with pestat printing Slurm nodes status with one line per node including job info, showuserjobs printing the current node status and batch jobs status broken down into userids, and showpartitions printing a partition status overview with one line per partition, and other tools. showuserlimits prints Slurm resource user limits and usage. Then there is the accounting pair, slurmreportmonth for monthly accounting statistics using the sreport command and slurmacct for accounting statistics as an alternative to sreport.

The rest of the list is operational rather than observational: trigger scripts, scripts for managing nodes, for managing jobs, for managing Slurm accounts and users, for managing power_save of nodes, job_submit plugins, tools for InfiniBand and Omni-Path fabrics, and bash command completions for the tools themselves.

The project is archived with a DOI, https://doi.org/10.5281/zenodo.15261429, which is the Zenodo record for the tool set, and the licence is the GNU General Public License v3.0 with a LICENSE.txt at the repository root.

pestat, showuserjobs and showpartitions, one line each

The three status tools share a design idea, which is that a line per thing is the unit that matters. pestat prints Slurm nodes status with one line per node including job info, showpartitions prints a Slurm cluster partition status overview with one line per partition, and showuserjobs prints the current node status and batch jobs status broken down into userids.

That last one is the differentiator. Breaking the cluster down by user id rather than by node answers the question an administrator actually gets asked, which is who is using the machine right now, and it is the question that a per-node listing makes you answer by hand.

The tools are written as shell scripts with names that match the commands they call, and they are the kind of utility that gets aliased into a login shell on day one and then depended on by everyone else on the team. That is also why the bash completion for them exists, and why the Makefile installs it into /etc/profile.d.

What the README does not say is anything about the output format, the sort order, or the exit codes, so the safe assumption is that these are personal scripts whose output format you will want to check before piping them into something else.

Accounting twice, through sreport and through something else

The accounting tools are the part of the collection with a real design decision behind it, because the README describes two tools doing the same job by different routes.

slurmreportmonth generates monthly accounting statistics from Slurm using the sreport command, which is Slurm's own reporting front end over the accounting database. slurmacct generates accounting statistics from Slumb as an alternative to the sreport command, and the Makefile groups it with two companions, slurmacct/jobstats and slurmacct/topreports, alongside slurmreportmonth as the accounting tool set.

So the difference is between delegating the statistics to Slurm's tool and computing them with the project's own code. That matters in practice because the two paths have different failure modes: sreport depends on the accounting database being reachable and its query behaviour, while a self-contained implementation can be pointed at the raw job records and shaped to answer a specific question.

The repository also contains a `database` directory and a `cloud` directory at the top level, which is where the data-side pieces live, and the presence of a `triggers` directory with Slurm trigger scripts suggests the accounting data is also collected automatically rather than only queried on demand.

What the README does not document is which approach to use when, and it does not describe the format of the output either, so if you intend to build reporting on top of these, the tools themselves are the reference.

Job administration scripts, and the names they install

The Makefile is where the full list of commands appears, and it is worth reading because the README lists categories while the Makefile lists every binary.

The job tools are joblist, psjob, showjob and showjobreasons. The job administration tools are jobnice, jobqos, jobtimelimit, notifybadjob, sbadjobs, schedjobs, sratelimit and warn_maxjobs. The naming convention is visible in the list itself: the `s`-prefixed ones and the ones with a `bad` or `notify` in the name are the ones that change state or report something wrong, and the plain ones are the ones that show you what is there.

That distinction matters because the tools are copied into two different directories. The read-only tools, pestat, the user tools, the job tools, the node tools and the partition tools, go to /usr/local/bin, and the administration tools, the accounting tools and the node administration tools, go to /usr/local/sbin. A user gets the reports in their path, and the commands that change jobs, node state or accounting configuration stay behind the administrative path.

The install itself is a copy with no clobber, so an existing binary on the machine is never overwritten silently, which is a small courtesy that also means you have to remove the old file yourself to upgrade.

Node scripts reach the hardware, including IPMI power

The node tools are the part that goes furthest below Slurm. The read-only ones are psnode, shownode, showevents, showpower and showpower_nvidia, and the administration ones are alive, alive_bmc, checktopology, sdrain, spowerdown, spowerup, sreboot, sresume and sversion.

The names make the intent plain. `sdrain` puts a node into drain so that Slurm stops scheduling new work on it, and the power scripts issue the actual power commands rather than asking Slurm to do it, which is what you need when the problem is that Slurm cannot talk to the machine.

The power_save directory carries the complementary set, with power_ipmi, nodesuspend, noderesume and nodefailresume, and the Makefile links those into the node administration group. The `alive` and `alive_bmc` pair suggests a health check that works both through Slurm's own view of the node and through the baseboard management controller, which is the sensible fallback when the node is too sick to report for itself.

There is a separate `slurmibtopology` tool for the InfiniBand topology and a `InfiniBand` directory of tools for managing InfiniBand and Omni-Path network fabrics, so the network side of the cluster has the same treatment as the node side.

Installation is a make target that copies into /usr/local

The Makefile is short and does the install in two steps, with the destinations declared at the top:

BINDIR=/usr/local/bin
SBINDIR=/usr/local/sbin

The user-facing target installs the read-only tools, and the administrative target installs the rest:

install_admintools: ${job_admintools} ${node_admintools} ${accountingtools}
	cp --no-clobber $^ ${SBINDIR}/

The `--no-clobber` flag is the detail to notice. Nothing already on the machine is replaced, which protects an administrator who has customised a script, and it also means that upgrading means removing the old file first.

The third piece is the shell completion, which is installed into a profile directory so that it is sourced by every login shell:

/etc/profile.d/slurm_tools_completion.sh: ${slurm_tools_completion}
	cp --no-clobber $^ $@

and the `all` target chains the completion install with both install targets, so one command does the lot. The `make` targets are not documented in the README beyond the download instructions, which tell you to use the Clone or download button, or to go to the individual tool page, click on the desired file and then click the Raw button to get a link you can use for download. That last route is the one to know if you want a single script rather than the repository.

A deployment HOWTO, and a repository that keeps growing

The README also points at a Slurm HOWTO guide for setting up a Slurm installation, hosted on the DTU Niflheim wiki. The guide is based on RHEL 8 Linux and clones such as Rocky or Alma Linux, and the README says much of the information should be relevant on other Linux distributions as well.

That is a different artefact from the tools, and a more valuable one for many readers, because setting up Slurm correctly is harder than querying it once it is running. A guide written by the people who run the cluster the tools were written for has a different character from vendor documentation, and it is where the reasoning behind a particular configuration is likely to be explained.

The repository root shows the scope. Alongside the tool directories there are `SSH/`, `cloud/`, `database/`, `InfiniBand/`, `plugins/`, `triggers/` and `bash-completion/`, and the `plugins/` directory holds job_submit plugins, which is the Slurm mechanism for intercepting and modifying job submissions. The `triggers/` directory holds trigger scripts, the equivalent mechanism for reacting to state changes in the cluster.

On maintenance, the repository is not archived and the last push was on 2026-10-20, so the collection is still being worked on. There is a single release, v1.0.0 labelled Slurm_tools initial Release from 2025-04-22, and the licence is the GNU General Public License v3.0, which for scripts of this kind means that a site that modifies and redistributes them has to keep publishing the modifications under the same licence, which is the point of the choice.

Editorial conclusion

Slurm_tools fits a site administrator who already runs Slurm and wants readable, scriptable answers to the daily questions about the cluster, and who is comfortable with tools written for a particular site rather than a general distribution. It is the wrong choice if you need a supported product with documentation and support, since the README is a list of the author's tools with a link to their deployment guide. Verify first that the commands match your Slurm version, that the accounting tool you need matches how your cluster stores job records, and that the install targets write only to /usr/local/bin and /usr/local/sbin.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. OleHolmNielsen/Slurm_tools on GitHub
4. README
5. Releases