AiiDA: a provenance graph wrapped around your HPC jobs

What problem AiiDA is built around

AiiDA is a workflow manager for computational science with a strong focus on provenance, performance and extensibility. The audience is implied by the rest of the project: simulation codes run on high performance computing clusters by people who have to publish results and defend them later.

The problem is not job submission. Any number of tools submit a script to SLURM. The problem is the trail. Six months after a calculation, somebody asks why this number is what it is, which inputs produced it, whether it was run with the same parameters as the run before it, and what the code was actually doing when it ran. Answering that from a shell history and a folder of output files is a bad afternoon.

AiiDA answers it by construction. Every calculation is tracked with its inputs, its outputs and its metadata, and those records form a provenance graph that can be queried and exported. The README claims fast queries on graphs containing millions of nodes, and separately claims the event based workflow engine supports tens of thousands of processes per hour with full checkpointing.

Those two numbers are the reason this is not a general purpose pipeline tool. It is infrastructure for a specific scientific setting: many small and medium calculations, each one cheap to run but expensive to keep track of, executed repeatedly with variations.

The second half of the pitch is portability. The README says you can move your calculations to a different computer by changing one line of code, and lists SLURM, PBS Pro, torque, SGE and LSF as schedulers that work out of the box.

The provenance graph as the central data structure

The graph is not a reporting layer bolted on afterwards. It is the store that the workflow engine itself is built from, and that is why the project can make claims about query speed.

Each node corresponds to a process and carries the inputs, outputs and metadata of that calculation. When one calculation consumes the output of another, the relationship is recorded in the same structure, so the graph ends up holding the dependency tree of a whole study. From that graph you can walk backwards from a published number to the code version, the input parameters and the machine that produced it.

Two consequences follow from making the graph the primary artefact rather than a side effect. First, workflows become self-documenting: the README describes writing complex, auto-documenting workflows in python. Second, exporting is a first class operation, since subsets of the provenance graph can be shared with peers or published on the Materials Cloud for anyone to inspect.

That export capability is what makes AiiDA interesting for more than one lab. A graph subset is a portable record: a reviewer can check the provenance of a result without installing the code that produced it, and a student can pick up a colleague's graph and see how a study was assembled.

The cost of this design is that AiiDA assumes a database. The dependency list includes psycopg with binary extras and SQLAlchemy, and the storage layer includes an object store component, so a deployment needs a PostgreSQL instance and a place for large data objects. That is a real operational commitment before you have written a single workflow.

An event based engine with checkpointing

The execution model is event based rather than step based. Instead of a script that runs top to bottom, a workflow is a set of processes with dependencies, and the engine reacts to events such as a process completing, a process failing, or a remote job changing state.

This is the design decision that makes the rest work. When a process finishes, the engine wakes the dependents, writes the outputs into the store, and updates the graph in the same transaction. Checkpointing then falls out naturally, because the state of the whole workflow is in the database rather than in the memory of a running process. A workflow survives a crash of the daemon, a restart of the cluster frontend, or a laptop that went to sleep.

For scientific work, the practical effect is that a campaign of thousands of calculations can be interrupted and resumed without recomputing anything already done, and that the failure of one calculation does not take the rest of the campaign down with it.

The workflow itself is written in Python and linked to arbitrary executables on local and remote computers. AiiDA does not interpret the science. It runs your program, watches for it to finish, and records what it saw. That boundary is what allows arbitrary simulation codes to be used, and it is also why the plugin interface exists to handle the messy parts of parsing inputs and outputs for each code.

Schedulers out of the box, plugins for everything else

The HPC interface is the part that determines whether AiiDA works in your group. The README states that you move calculations to a different computer by changing one line of code, and names SLURM, PBS Pro, torque, SGE and LSF as compatible schedulers that work out of the box. That list covers the majority of university and national laboratory clusters, and the SSH transport appears in the project topics alongside the scheduler names, which points to remote execution over SSH rather than a bespoke agent.

The plugin interface is the extension point. AiiDA plugins add support for new simulation codes, including input generation and parsing, new data types, schedulers, transport modes and more. The registry is a separate site from the core repository, so the core does not need to change when a new code is supported.

That division of labour is a deliberate design choice and it has a cost for a new user. Out of the box, what you get is the engine, the graph, the scheduler adapters and the storage. Getting a specific simulation code to run means finding or writing a plugin that knows how to build its input files and read its outputs, and for codes that have no plugin yet, that work falls on you. For a group standardising on a handful of codes, that is a reasonable trade. For a group with a long tail of one-off scripts, it is the main obstacle.

The project also describes itself as open source in the sense of sharing the graph rather than only the code, which is the Materials Cloud export path described above. For a research group, publishing provenance alongside a paper is often the goal that matters.

Installing from a release, never from main

The README installation section is short, and it contains the most important operational instruction in the whole document. It points to the documentation at aiida-core.readthedocs.io for installation steps, and then warns in bold not to use AiiDA directly from the GitHub main branch, because it changes rapidly and may put your AiiDA storage in an unrecoverable state.

That warning is the practical answer to how you install. The repository publishes to PyPI and to conda-forge, as the release badges in the README show, and the recent release history shows tagged versions v2.9.2 on 2026-09-03 and v2.9.3 on 2026-10-02, so a stable coordinate always exists to pin against. The repository itself is not archived and the last push was on 2026-10-22, so the main branch is genuinely moving.

The packaging metadata shows what you are installing. The build backend is flit_core, and the project metadata declares support for Python 3.10 through 3.14 on Linux and MacOS X, and marks the development status as Production/Stable:

[build-system]
build-backend = 'flit_core.buildapi'
requires = ["flit_core >=4.0.2,<5"]

[project]
authors = [{name = 'The AiiDA team', email = '[email protected]'}]

The dependency set is large and telling. It includes SQLAlchemy and psycopg for the database, alembic for schema migrations, circus as a process controller, asyncssh for remote access, cloudpickle and dill for serialisation, pydantic for validation, and graphviz for rendering. AiiDA is a server-side system with a database and a daemon, not a library you import and forget.

The same concern applies to the database. AiiDA keeps its state in PostgreSQL and migrates it with alembic, which means upgrading AiiDA is a schema migration of a live database, not a pip upgrade. Plan for a dump and a tested restore path before you move between versions.

Where AiiDA is the wrong tool

The obvious case is scale mismatch. If a group runs a handful of calculations a week and keeps its parameters in a spreadsheet, AiiDA adds a database, a daemon, a plugin dependency and a migration path to solve a problem that a Makefile and a directory of output files already solve adequately. The overhead of learning the workflow model is not repaid.

The second case is interactive work. AiiDA is built around submitted and tracked processes. If your work is exploratory and the analysis changes shape while you look at it, the model of a workflow with fixed dependencies fights the way you actually think.

The third is code availability. If a plugin does not exist for your simulation code, and writing one is more work than the tracking is worth, the project does not help. The plugin interface is a strength when a community maintains plugins for common codes and a burden when it does not.

There is a fourth, quieter risk in the design itself. Because the database is the source of truth and the workflow engine mutates it as jobs run, an incorrectly configured or partially applied upgrade can leave the store in a state the README warns about. Teams that cannot afford to rebuild provenance data should treat the main branch warning as a hard rule rather than advice.

The alternative to AiiDA is not another framework so much as a different practice: hand written shell scripts driving a scheduler, with parameters and provenance recorded by convention in text files and a run log. That approach is cheaper to start and far harder to scale, because the bookkeeping is manual and breaks as soon as several people run variations of the same study in parallel. AiiDA replaces the convention with a schema.

Licence, governance and who maintains it

AiiDA is released under the MIT open source license, with the licence text in LICENSE.txt and a separate open_source_licenses.txt listing other open source components included in the distribution. That second file matters in practice: MIT covers the AiiDA code itself, while the bundled dependencies carry their own terms, which is why the project tracks them separately.

The project is a NumFOCUS affiliated project, supported by the MARVEL National Centre of Competence in Research and the MaX European Centre of Excellence among others. NumFOCUS affiliation brings governance and a neutral home for the project, which is the main answer to the question of who maintains a workflow engine that a national lab depends on.

The activity signal is consistent. Releases v2.8.3 and v2.9.2 were published on 2026-09-03 and v2.9.3 on 2026-10-02, the repository is not archived, and the last push was on 2026-10-22. The CI setup runs a code build workflow and a coverage report, and the tree also contains a.devcontainer directory, a pre-commit configuration and an AGENTS.md file, which tells you the project expects a conventional developer environment rather than a bespoke setup.

For users who cite the work, the README asks them to cite the 2020 Scientific Data paper on AiiDA 1.0, and the earlier 2016 paper if they reference the ADES concepts. A workflow manager that appears in a paper's methods section is infrastructure in the strongest sense, which is both the argument for adopting it and the reason to plan upgrades carefully.

Editorial conclusion

Adopt AiiDA when your group runs thousands of simulations across a batch cluster and needs to answer later where a number came from, because the provenance graph and the event based engine are the point of the project and the scheduler adapters ship with it. Do not adopt it for a handful of scripts, for a pipeline whose steps are simple and stateless, or if you need a graphical workflow editor at install time, since the README describes the interface as written in Python. Before installing, read the warning about the main branch, take a tagged release from PyPI or conda-forge instead of the repository, and confirm the version of your database and plugins matches the release you choose, because AiiDA stores its state in a PostgreSQL database that must be migrated when the version changes.

Frequently asked questions

Official sources

1. aiidateam/aiida-core on GitHub
2. Issues
3. Project website
4. README
5. Releases