slurm-docker-cluster runs a whole Slurm scheduler in Docker Compose

A scheduler for development, not for production

The project describes itself as a multi-container Slurm cluster designed for rapid deployment using Docker Compose, and adds that it simplifies the process of setting up a working Slurm environment for development, testing, or lightweight usage.

That scoping sentence is the most important line in the README, and it is easy to skim past. A Slurm installation is normally a multi-machine undertaking with shared storage, a network, an authentication layer and a database, and this project puts all of it inside one Compose stack on one host. What you get is a scheduler you can genuinely submit jobs to, with `sbatch`, `squeue` and `sacct`, and the same node registration and accounting paths a real cluster uses. What you do not get is the resilience, the isolation and the operational maturity of a real allocation.

The implementation is mostly Shell, with the Makefile as the primary interface, and the repository is MIT licensed. The release history is quick: v3.0.2 on 2026-10-19, v3.0.1 on 2026-10-17 and v3.0.0 on 2026-08-09, with the last push on 2026-10-19.

Two version facts frame everything else. Supported Slurm versions are 26.05 and 25.11, and the Makefile names the exact builds it knows about, `25.11.8` and `26.05.5`. Supported architectures are AMD64 and ARM64, auto-detected, which is a genuine convenience given that Apple silicon machines are now a common way to run a Docker workstation.

So the question to answer before anything else is whether you want a scheduler to test your batch jobs against, or a scheduler to run your batch jobs on. This project is clearly built for the first.

Ten containers and six volumes

The container list in the README is short enough to read as an architecture, and each entry has a job.

The database is MariaDB, listed as mysql, and it holds the job and cluster database. The accounting daemon `slurmdbd` sits in front of it, and it records the job history that `sacct` reads. The controller `slurmctld` is the job scheduler. The REST daemon `slurmrestd` provides HTTP and JSON access to the cluster. The compute nodes are `c1` and `c2` for CPU work, dynamically scalable, and `g1` is an optional GPU node with NVIDIA support, also dynamically scalable.

Then four optional services. `ondemand` is the Open OnDemand web portal, `elasticsearch` is for indexing jobs and `kibana` is for visualising that index.

The persistent volumes are the other half of the picture, and there are six. Configuration lives in `etc_slurm`, logs in `var_log_slurm`, job files in `slurm_jobdir`, the database in `var_lib_mysql`, authentication in `etc_munge`, and the Open OnDemand user home in `home_ood`. Separating these means a `make down` keeps your configuration and your accounting history, and only `make clean` removes the volumes.

The Compose file itself shows the dependency ordering done properly. The MariaDB service has a healthcheck that runs `healthcheck.sh --connect --innodb_initialized` with a 30 second start period, and `slurmdbd` declares `depends_on` with `condition: service_healthy`, so the accounting daemon does not start against a database that is not accepting connections yet. The `slurmdbd` service has its own healthcheck based on `pidof slurmdbd`.

That kind of ordering is the difference between a cluster that comes up reliably on the first `make up` and one that requires a retry loop, and it is a good indicator of how the project is put together.

Getting it running with make

The quick start needs Docker and Docker Compose, and offers two routes to the image:

git clone https://github.com/giovtorres/slurm-docker-cluster.git
cd slurm-docker-cluster
cp .env.example .env    # optional: edit to change version, enable GPU, etc.

# Option A: Pull pre-built image from Docker Hub (fastest)
docker pull giovtorres/slurm-docker-cluster:latest
docker tag giovtorres/slurm-docker-cluster:latest slurm-docker-cluster:26.05.5

# Option B: Build from source
make build

# Then, start the cluster
make up

Option A pulls a pre-built image and tags it with the Slurm version the Compose file expects, which is the version string in `.env` and the one used as the image tag in `docker-compose.yml`.

The `.env` file is optional, since the defaults are used automatically when it does not exist, and the example file says so in a comment. But the defaults are development defaults, and the file is where the interesting knobs live. It carries the Docker Compose project name, the Slurm version, the Lmod version at 9.1.2, the Spack version at v1.1.1, the MariaDB credentials, the CPU worker count at 2, and the switches for Elasticsearch, Open OnDemand and GPU. Each of those is a comment away from being enabled, and the Makefile reads them to decide which Compose profiles to activate.

That profile detection is the neatest bit of the Makefile. It greps `.env` for `ELASTICSEARCH_HOST`, `GPU_ENABLE` and `OOD_ENABLE`, and for each one that is set it appends the matching Compose profile to a `PROFILES` variable, which becomes the profile flag passed to `docker compose`. So setting `OOD_ENABLE=true` is enough to bring up the web portal, without editing the Compose file at all.

Running the Makefile with no target prints the list of available commands, which is the discoverability path for everything else.

Compute nodes that register themselves

The scaling model is the part of the design that distinguishes this from a fixed-size Compose file, and it is short to describe.

Compute nodes use Slurm's dynamic registration with `slurmd -Z`, and they self-register with sequential hostnames: c1, c2, c3 and so on for the CPU workers, and g1, g2 and onwards for the GPU workers. Scale up or down at any time without rebuilding.

So the nodes are not listed in the configuration. They announce themselves to the controller when they start, which means adding a worker is a scale operation rather than a configuration edit. The commands are:

# Scale to 5 CPU workers (default is 2)
make scale-cpu-workers N=5

# Or set the default count in .env
CPU_WORKER_COUNT=4
make up

and for the GPU side:

# Scale to 3 GPU workers (requires GPU_ENABLE=true)
make scale-gpu-workers N=3

with `make status` to verify.

There is a limit worth stating. Dynamic registration replaces static node definitions in `slurm.conf`, and the controller tracks the nodes it has heard from. That is fine for a development cluster where nodes come and go with `make scale`, but it also means the controller's view of the cluster is built from announcements rather than from a file you edit and keep under version control. For a cluster you want to declare, this is a different model.

Inside the cluster the usual Slurm commands are what you would expect, and the README shows them from a controller shell:

# Access controller
make shell

# Inside controller:
sinfo                          # View cluster status
sbatch --wrap="hostname"       # Submit job
squeue                         # View queue
sacct                          # View accounting

# Or run example jobs
make run-examples

The `make run-examples` target uses the `run_examples.sh` script and the `examples/jobs/` directory in the repository.

The REST API and the version path that catches people out

The REST API section is where a version detail becomes a practical trap, and the README handles it explicitly.

The API version is auto-detected: v0.0.45 for 26.05.x, and v0.0.44 for 25.11.x. The queries go through the `slurmrestd` container with a JWT token obtained from the controller:

# Get JWT Token
JWT_TOKEN=$(docker exec slurmctld scontrol token 2>&1 | grep "SLURM_JWT=" | cut -d'=' -f2)

# Get nodes
docker exec slurmrestd curl -s -H "X-SLURM-USER-TOKEN: $JWT_TOKEN" \
  http://localhost:6820/slurm/v0.0.45/nodes | jq .nodes

# Get partitions
docker exec slurmrestd curl -s -H "X-SLURM-USER-TOKEN: $JWT_TOKEN" \
  http://localhost:6820/slurm/v0.0.45/partitions | jq .partitions

The URL carries the API version in the path, so a script written against 26.05 will return a 404 against 25.11, and the failure looks like an empty or broken endpoint rather than a version mismatch. If you write anything against this API, pin the path to the Slurm version you pinned in `.env`.

The token flow is also worth noting. The token is minted by `scontrol token` inside the controller and passed as a header named `X-SLURM-USER-TOKEN`, which is the Slurm REST convention, and the token has a lifetime that means your script has to fetch a fresh one rather than caching it forever.

The alternative to the REST API, for anything that does not need to be driven from outside the cluster, is simply to run `sinfo`, `squeue` and `sacct` inside the controller container with `make shell`, which is what most local testing will do.

The optional profiles and what they cost you

Three features are optional and each has its own switch, its own compose profile and its own test target.

Job completion monitoring runs through Elasticsearch and Kibana. Setting `ELASTICSEARCH_HOST=http://elasticsearch:9200` in `.env` enables the monitoring profile, and Kibana is then reachable on port 5601, where the path is Elasticsearch, Index Management, the slurm index, and then Discover. The indexed fields are listed as job ID, user, partition, state, times, nodes and exit code. The test target is `make test-monitoring`.

Open OnDemand gives you a browser interface to the cluster, so you can submit jobs, manage files and watch the queue without a command line. It is enabled with `OOD_ENABLE=true` and served on port 8080, and the README gives the login as `ood@localhost` with the password `password`. The README is explicit about the reason: this setup uses Dex with a static password for development purposes. OOD connects to the cluster as the `ood` user and shares a home directory across the compute nodes for job input and output, and its configuration lives in the `ood/` directory, with changes requiring `make rebuild`.

GPU support uses NVIDIA's official CUDA base images, and the host setup is the part that has to happen outside Docker. The NVIDIA container toolkit repository is added, `nvidia-container-toolkit` is installed, the runtime is configured for Docker and the daemon is restarted. Then in `.env` you set `GPU_ENABLE=true` with `BUILDER_BASE` and `RUNTIME_BASE` pointing at `nvidia/cuda:13.1.1-devel-rockylinux9` and `nvidia/cuda:13.1.1-base-rockylinux9`, and rebuild. The verification is `docker exec g1 nvidia-smi` and the test target is `make test-gpu`.

The README also flags that GPU testing is not included in CI, because the GitHub hosted runners have no GPUs, so `make test-gpu` has to be run manually on a host with an NVIDIA GPU and the toolkit installed. That is an honest gap and it means the GPU path is the least exercised part of the project.

Spack, Lmod and the build stages behind the image

Software installation inside the cluster is handled by Spack with Lmod for the module system, and the README states that Spack is included in the image and integrates with Lmod so installed packages appear immediately as modules, with all nodes sharing the same Spack and module tree.

The workflow is deliberately conventional:

make shell

spack install [email protected]
module avail
module load python/3.14.0
python --version

and modules are available in batch jobs with no extra setup:

sbatch --wrap="module load python/3.14.0 && python3 --version"

There is also a way to add a modulefile that did not come from Spack, by dropping a `.lua` file into the `opt_modulefiles` volume, where it appears on all nodes immediately without a restart:

docker exec slurmctld mkdir -p /opt/modulefiles/myapp
docker cp myapp/1.0.lua slurmctld:/opt/modulefiles/myapp/1.0.lua
module avail

That last point is the design detail that makes this usable. The module tree lives in a volume, so a package installed with Spack once is visible to every node, and a hand written modulefile is picked up without touching the running services.

Behind that, the image is built in three stages, and the Dockerfile explains why. Stage one builds gosu from source with the latest Go, with the comment that pre-built binaries use an old Go version that triggers CVEs, and the build cross-compiles with `CGO_ENABLED=0` and `TARGETOS` and `TARGETARCH` for the multi-arch images. Stage two builds the RPMs on the builder image, enabling the CRB and EPEL repositories for development packages. Stage three installs the RPMs into a clean runtime image.

The base images default to `rockylinux/rockylinux:9`, and the same variables are overridden when `GPU_ENABLE=true`, which is why the GPU instructions change `BUILDER_BASE` and `RUNTIME_BASE`.

The other detail worth naming is where Spack lives. The `.env.example` comments say Spack is cloned at image build time into `/usr/local/spack` and baked into the image, while installed packages and generated Lmod modules are stored in the `spack_root` volume at `/opt/spack`. So the tooling is part of the image and the packages are part of the data.

Editorial conclusion

Adopt slurm-docker-cluster when you need a real Slurm scheduler to develop or test against, and you want it running on one machine with `make up`. Do not adopt it as a production batch system, since the README scopes it to development, testing or lightweight use, the Compose file uses default MariaDB credentials, and the Open OnDemand profile logs in with a static Dex password. Verify first that the Slurm version in `.env` is one the Makefile supports, either 25.11.8 or 26.05.5, that the REST API path you call matches the version the project auto-detects, since 26.05.x uses v0.0.45 and 25.11.x uses v0.0.44, and that you have changed `MYSQL_PASSWORD` before the cluster is reachable from anywhere but your own machine.

Frequently asked questions

Official sources

1. giovtorres/slurm-docker-cluster on GitHub
2. Issues
3. License: MIT
4. README
5. Releases