HyperQueue: running task graphs on a cluster without writing job scripts

The problem it solves

HyperQueue is a tool designed to simplify the execution of large workflows, described as task graphs, on high performance computing clusters. It lets you execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers like Slurm or PBS.

That last clause is the whole problem statement, and it is worth making concrete. The normal way to run a thousand independent simulations on a cluster is to write a job script, put the loop inside it, and ask the scheduler for the resources the loop needs. That works and it is what everybody does. It also has three well known failure modes. The job is one big allocation, so a single task that runs long holds up everything after it. The scheduler sees one job, so the cluster is either idle or oversubscribed, and load balancing has to happen inside your script rather than where it could actually happen. And if a task fails at step nine hundred, you find out about it when the whole job dies, unless you wrote the retry logic yourself.

HyperQueue's answer is to invert the arrangement. You specify what you want to compute, and HyperQueue asks for computational resources and dynamically load balances the tasks across all the allocated nodes and resources. The unit of scheduling is the task, and the scheduler's job becomes to hand out whatever capacity happens to be available, which it does well, and HyperQueue's job is to keep every node busy, which a job script does badly.

And there is a second mode, which the readme states in a single sentence and which is easy to miss: HyperQueue can also work without Slurm or PBS at all, as a general distributed task execution engine. So the same tool that sits on top of a cluster's scheduler can also be the scheduler, for a machine or a set of machines that has none.

The project is written in Rust, is MIT licensed, and is developed by a group of researchers at the Czech National Supercomputing Center, who welcome outside contributions. It has a published paper in a software journal, a poster at a supercomputing conference, and a software archive record, and the readme asks that if you use it in research you consider citing it. That is a good sign about the maturity of the project: it has been through the paper and the citation process, not just shipped.

The resource model, and the four awkward cases

The resource handling is listed first among the features, under the heading of complex resource management, and the sub-points are where the design work is.

The basics are the ones you would expect: load balancing of tasks across all available resources, and automatic submission of scheduler jobs on behalf of the user, so that the tool asks for capacity rather than you asking for it.

Then there are complex and arbitrary task resource requirements, with the number of cores, graphics processors, memory and field programmable gate arrays all named. And under that heading are four cases, and they are the four cases that a job script handles badly.

Non-fungible resources are tasks assigned a specific resource, and the example given is a graphics processor with a particular identifier. This is the request you cannot express in a conventional job script at all, because a job script asks for a number of processors, not for a named device. If two of your tasks must not share a card, or your data is bound to a card, this is the case that matters.

Fractional resources are tasks that require, in the readme's example, half of a GPU. That is a scheduling request the underlying batch schedulers do not express, so a tool that supports it has to keep track of the fractions itself and present them to the scheduler as a whole number of processors.

Resource variants are tasks that can be satisfied either way, and the example is one graphics processor together with four cores, or alternatively sixteen cores. That is a disjunctive requirement, and it is the sort of thing that is easy to say and awkward to schedule, because the two alternatives have different costs and a scheduler will pick the one that is cheapest to give rather than the one you meant.

And related resources are tasks that need, in the example, four cores within the same non-uniform memory access node. That is a locality constraint, and it is the one that matters most for the memory bandwidth of a parallel task, because four cores on one node share memory and four cores on four nodes do not.

So the model has four ways of asking for something that is not just a number of processors, and each of them corresponds to a real situation in which a plain job script is the wrong abstraction. That is what makes the feature list worth reading twice: these are not variations on a theme, they are the cases the design was built around.

The performance claims, and the one that will save you the most pain

The performance section is short, and each of its numbers is a design target rather than a boast.

It scales to hundreds of nodes and workers and to millions of tasks. The overhead per task is below a tenth of a millisecond. And the output of tasks is streamed rather than written to files.

That last one is the item worth understanding, because it solves a problem that every HPC user has. On a distributed filesystem, writing a million small output files is slow, and it is slow in a way that gets worse as the number of files grows, because the filesystem's metadata operations dominate. The usual workaround is to write one file per task and then have a second step to collect them, and that second step is a source of failure, a source of delay, and often the largest single cost in a workflow that computes many small things.

Streaming the standard output and standard error of the tasks removes that step entirely. The data goes where the client is listening rather than onto the shared filesystem, so a workflow of a million small tasks produces no small files at all. The demo in the readme shows this in action with a few hundred commands and a terminal recording, and the terminal is the point: the results arrive in the place you are already looking.

The overhead figure is the other one to sit with. A tenth of a millisecond per task sounds like nothing, and it is nothing for a task that takes an hour, and it is a great deal for a task that takes a millisecond. Which is the point of the tool: HyperQueue is not trying to make the scheduler faster or the compute faster, it is trying to make the overhead of managing a large number of tasks small enough that you can have millions of them. Once the per-task cost is that small, the failure to schedule tasks individually stops being a problem, and the whole design of the tool follows from that number.

The deployment properties follow the same logic. It is provided as a single statically linked binary without any runtime dependencies apart from the standard C library, and no administrative access to a cluster is needed to use it. Both of those matter for a tool whose users are researchers rather than administrators: a single binary means there is nothing to install, and no admin access means a user can run it in their own directory on a login node without anybody's permission.

Three ways to describe a workflow

The interface section is brief, and the first item is the one that determines how long it takes to try the thing.

Task graphs can be defined in three ways: through the command line, through workflow files written in TOML, or through a Python API. Three interfaces for the same model is a deliberate choice, and each one answers a different question about who is driving.

The command line is for the interactive case and for scripts, and the readme's own examples are in that style throughout. A server is started on a login node, a partition, or simply on your own computer. A job is submitted by giving a command. And then resources are asked for, either by starting a worker manually or by configuring automatic submission of workers into the cluster's scheduler.

The scheduler part is two commands, one for each supported scheduler, and both take a time limit and the queue or partition to use.

$ hq alloc add pbs --time-limit 1h -- -q <queue>

$ hq alloc add slurm --time-limit 1h -- -p <partition>

So the allocation is a named thing with a lifetime, and the tool submits and re-submits the underlying scheduler jobs for you while the allocation is alive. That is the mechanism behind the automatic submission mentioned earlier, and it is the piece that turns a tool into something you can leave running for a day.

The results are then read back with two commands, one that waits for the job and one that prints its output.

$ hq job wait last
$ hq job cat last stdout

The word last is the ergonomic touch: you do not have to remember an identifier, so the common case of running one thing and looking at it is two short commands.

The workflow files are for the case where the graph is a thing you want to keep, review and re-run, and the Python API is for the case where the graph is generated by another program, which is the situation in a machine learning pipeline where the list of configurations comes out of a search.

And the third item under a simple user interface is a real-time dashboard for monitoring cluster utilisation. That is a small thing that matters on a cluster, because the question during a long run is always how full the allocation is and whether anything is stuck.

Getting it, and running the first job

Installation is deliberately undramatic, and the readme's tone is the reason.

You download the latest binary distribution, unpack the archive, and use the unpacked binary. There is no build step, no package manager, and no configuration.

$ tar -xvzf hq-<version>-linux-x64.tar.gz

The archive name carries the platform, which is the single-binary deployment showing up in practice, and the readme adds that a nightly build is available for people who want the newest features and are willing to be unlucky.

The first run is three commands and it is worth following in order, because the order carries the meaning.

Start a server, and the readme tells you where: on a login node, on a cluster partition, or simply on your own machine. The server is the thing that owns the queue of tasks, and it is deliberately separable from the workers.

$ hq server start &

Submit a job, and here the example is the smallest possible command.

$ hq submit echo 'Hello world'

Then ask for computing resources, and the readme gives the two options: start a worker yourself, or configure the tool to submit workers into the cluster's scheduler on your behalf.

$ hq worker start &

That separation is the design in miniature. The server holds the state and the task queue. The workers hold the capacity. And the connection between them is the only coupling, which is what lets the server run on a login node while the workers live inside scheduler jobs, and also lets the whole thing run on one laptop for a test.

The three-part structure also explains why the tool can work without a scheduler at all: in that case you start workers yourself, and the server talks to them directly, and you have a distributed task engine with no batch system involved.

Once the job is done you wait for it and print its output, and the example of a command whose only job is to print a greeting is the right choice for a quick start, because the point is to prove the whole path works before you point it at anything expensive.

The project around the code

The readme's closing sections describe a research project that has been through its formal stages, and the details there are useful to a reader deciding how seriously to take the software.

There is a journal article in a software journal with a digital object identifier, whose title is about efficient and ergonomic task graphs on high performance computing clusters, and a poster from a supercomputing conference whose title is about overcoming the limitations of high performance computing job managers. Those two titles together state the argument of the project: the problem is not that the schedulers are slow, it is that their unit of work is wrong for this kind of workload.

There is a software archive record, which matters for a research tool because it means a specific version can be cited and retrieved years later, and the readme gives a BibTex entry with the full author list, the journal, the volume, the article number, the year, the ISSN and the digital object identifier. That is a project that expects to be cited in a thesis.

The support routes are split three ways, and the split is worth noting: the issue tracker for bugs, a discussion forum for general questions and feature requests, and a Zulip chat server for talking to the developers. The separation of a bug report from a feature request from a conversation is a small piece of project hygiene that saves a maintainer a great deal of triage.

The funding acknowledgement names two sources: a European project on advanced computing, and a national infrastructure programme in the Czech Republic. And the team is a group of researchers at a national supercomputing centre who say they welcome outside contributions, which is an invitation that is worth taking up given the size of the codebase.

The repository layout shows a Rust workspace with several crates, including the main one, a second one with a short name, and one that appears to be a Python binding, alongside a benchmarks directory, a documentation directory with its own site configuration, a test directory with a Python test configuration, a linting configuration, and a spell checking configuration. The last of those is the smallest file in the list and probably the one that saves the most annoyance, because a project with an acronym in every crate name would otherwise spend a surprising amount of time failing a spell check.

Editorial conclusion

HyperQueue fits a researcher with a workflow of many thousands of similar tasks who is currently writing a job script per task or fighting a scheduler with a single multi-step job. It does not fit a workload that is a handful of long jobs, because a scheduler is the right tool for those. Verify first whether your cluster's scheduler is one of the two that HyperQueue can submit to on your behalf, whether the resources your tasks need are expressible in the model's terms, especially the fractional and variant cases, that you can run a server on a login node or a partition, since the server is what owns the queue, and that the streaming of output suits your analysis rather than fighting it, because the alternative of many small files on a shared filesystem is exactly the problem the design is avoiding.

Frequently asked questions

Official sources

1. It4innovations/hyperqueue on GitHub
2. License: MIT
3. Project website
4. README
5. Releases