redun: a workflow engine that treats workflows as lazy Python expressions

The contrarian bet: write the workflow as code, not as a graph

redun describes itself as yet another redundant workflow engine, and the name is deliberately self-deprecating, a reference both to the number of existing workflow engines and to the redo build system that inspired it. The project is from insitro, is written in Python, and is licensed Apache-2.0.

The design argument is explicit. The README takes the position that writing dataflows directly is unnecessarily restrictive, and that doing so loses abstractions that most modern high-level languages rely on: control flow, composability, recursion and higher order functions. Instead of a separate graph description, a workflow is a lazy expression, and a scheduler evaluates it.

That evaluation is what produces the graph. Lazy expressions emit dynamic directed acyclic graphs when they are evaluated, so the shape of the workflow is a consequence of running the Python code rather than something you declare up front. The code you write stays in Python, and the README's example notes that beyond the `@task` decorator the code follows typical Python conventions and is organised like a sequential program.

The audience follows from that choice. A bioinformatics or data science team that already writes its processing in Python, and that is tired of hand-maintaining a separate pipeline definition file next to the code, is the clear user. A team that wants a YAML DAG, a UI, or a job server with a queue of its own is not.

The listed use cases match that reading: bioinformatics, cheminformatics through an AWS Glue example, web and API data extraction, and general data science, each with a directory under `examples/`.

How the scheduler decides what to rerun: hashing, code changes and caches

The interesting engineering is in what redun considers a change. Three mechanisms carry most of the weight.

First, data changes are detected by file hashing, and this covers in-memory values as well as external data sources such as files and object stores. A `File` value carries a path and a hash, and the example output shows those hashes travelling with every argument, so a task is a function of content rather than of a filename.

Second, code changes are detected by hashing individual Python functions and comparing them against historical call graph recordings. A function whose body changed hashes differently, so the recorded history of what ran before is enough to decide that the previous result no longer applies. This is a different rule from the usual one, and it is what lets redun claim incremental computation that is reactive to code changes as well as data changes.

Third, past intermediate results are cached centrally and reused across workflows. The cache is not scoped to a single run.

The fourth feature follows from the third. Past call graphs serve as a data lineage record and can be queried for debugging and auditing. The repository layout makes the persistence layer visible: there is a `db/` directory, a `docker-compose.yml` that runs a `redun_postgres` image, and the dependency list includes SQLAlchemy and Alembic, which is the migration tool. The Makefile has `upgrade` and `revision` targets that call Alembic against `redun/backends/db/alembic.ini`, so the recording backend is a real, versioned database schema rather than a local cache file.

The trade-off is straightforward. Central caching and cross-workflow lineage require a shared recording backend, which means a Postgres instance in production and a migration step when the schema changes.

Writing the first workflow: a compiler in about twenty lines of Python

The README's small taste compiles a C program, and the structure is the same as any other redun workflow. You import `task` and `File`, declare a namespace, and decorate each unit of work.

@task()
def compile(c_file: File) -> File:
    """
    Compile one C file into an object file.
    """
    os.system(f"gcc -c {c_file.path}")
    return File(c_file.path.replace(".c", ".o"))

The decorator is what turns an ordinary function into a workflow node. The body of the function is unremarkable Python, including the shell call, and the return value is another `File`, which is how the dependency edge to the next task is expressed.

Tasks compose by calling each other, and the top level task gathers results for several programs.

@task()
def make_prog(prog_path: str, c_files: List[File]) -> File:
    """
    Compile one program from its source C files.
    """
    o_files = [
        compile(c_file)
        for c_file in c_files
    ]
    prog_file = link(prog_path, o_files)
    return prog_file

Note what the laziness buys. The list comprehension calls `compile` for each source file, but the calls are not executed at that moment; the scheduler resolves them, which is where the DAG is emitted and where caching decides whether a call actually needs to run. A sequential reading of the code is still correct, so the code stays reviewable in a way that a separately declared graph would not.

The example lives in `examples/02_compile/`, and the repository carries a large set of siblings, including `examples/07_control_flow/`, `examples/eager_eval/`, `examples/context/`, `examples/custom_executor_config/`, `examples/federated_task/` and `examples/etl/`, which cover the parts of the framework the README does not explain.

Running it: one command, a job log and a status table

A workflow is executed with the `redun run` command, taking the script and the entry point.

redun run make.py make

The output is a log of jobs, and every line carries a job identifier, the fully qualified task name with its arguments, and the executor the job ran on.

[redun] | JOB STATUS 2021/06/18 10:34:29
[redun] | TASK                             PENDING RUNNING  FAILED  CACHED    DONE   TOTAL
[redun] |
[redun] | ALL                                    0       0       0       0       8       8
[redun] | redun.examples.compile.compile         0       0       0       0       3       3
[redun] | redun.examples.compile.link            0       0       0       0       2       2

The columns are the state of each task: pending, running, failed, cached, done and total. The cached column is the one worth watching, since a run where most tasks are cached is redun telling you that the inputs and the code are unchanged.

Two other lines in the sample output are worth reading. One shows the configuration directory, ending in a `.redun` folder next to the workflow, and another shows a database upgrade being applied at startup, with the schema version moving from a negative placeholder to 2.0. The recording backend is therefore created and migrated on first use rather than requiring a separate provisioning step, at least for the default local configuration.

The version shown in that sample output, 0.4.15, is from an older run. The package metadata in the repository now lists version 0.46.0, and the two most recent GitHub releases listed for the project are 0.27.0 and 0.26.0, both dated 2025-05-23. The mismatch between the sample output, the listed releases and the metadata version is worth keeping in mind when you read any blog post or issue about redun, since the engine has moved on considerably since that example was captured.

Where the work runs: threads, processes, AWS Batch and Spark

Tasks are not tied to one machine. The README states that workflow tasks can be executed on a variety of compute backends, naming threads, processes, AWS Batch jobs and Spark jobs among them. In the sample run, jobs are dispatched to an executor called `default`, so the backend is a configuration choice rather than a property of the code.

The example directory shows how far this goes. There are `examples/05_aws_batch/`, `examples/aws_batch_array_jobs/`, `examples/aws_batch_multinode/`, `examples/aws_glue/`, `examples/azure_storage/`, `examples/conda/` and `examples/docker/`, which together cover batch arrays, multi node jobs, Glue, non-AWS object storage, Conda based environments and containerised tasks.

The dependency list explains the AWS orientation. `aiobotocore` with the `boto3` and `awscli` extras is pinned to a minimum version, with a comment explaining that the extra dependencies help the solver find a working combination because aiobotocore requires pinned versions of boto3 and awscli. `botocore` is pinned with an explicit exclusion of version 1.28.0, and `s3fs` and `aiohttp` are there for object store and network access. `alembic` and `sqlalchemy` cover the recording backend, and `rich` and `textual` cover terminal output.

The Postgres backend is optional and installed as an extra.

pip install redun[postgres]

That variant assumes `pg_config` from the `postgresql-devel` package, which on Ubuntu means `libpq-dev`, and a C compiler, because the database driver is built locally. The repository's own compose file starts a Postgres service for development.

  db:
    image: redun_postgres
    build:
      context: db/
      dockerfile: Dockerfile
    environment:
      - POSTGRES_PASSWORD=postgres
    ports:
      - "5432:5432"

The service is built from the `db/` directory in the repository and listens on port 5432. For a local development setup that is the whole story, and it is also the point at which the project stops being a pure library and starts needing a service you can forget to back up.

Developing on redun with uv, tox and pre-commit

The project uses uv for dependency management and tox for the test matrix, and the Makefile is the entry point for both.

.PHONY: setup
setup:
	uv sync --all-extras
	uv run pre-commit install
	uv tool install tox --with tox-uv

Setup syncs every optional dependency group, installs the pre-commit hooks into the local checkout, and installs tox as a uv tool with the tox-uv plugin. The Makefile also carries a `teardown` target that removes `.venv/` and `.tox/`, which is the manual way to get back to a clean state when the environment has drifted.

Tests run through tox and then invoke a separate Postgres test script, so the database tests are part of the default run rather than an optional extra.

.PHONY: test
test:
	tox
	make test-postgres

Release engineering is scripted too. The `release` target refuses to run unless the current branch is `main`, reads the version out of `pyproject.toml`, refuses to continue if a tag with that version already exists, and then creates and pushes an annotated tag. The `publish` target runs lint, test and build in that order before calling a publish script, which is a useful detail for anyone mirroring releases: the version in `pyproject.toml` is the single source of truth.

The package metadata shows the supported surface.

[project]
name = "redun"
version = "0.46.0"
description = "Yet another redundant workflow engine."
readme = "README.md"
requires-python = ">=3.10"
license = {text = "Apache-2.0"}

Python 3.10 is the floor, and the classifiers list support for 3.10, 3.11 and 3.12, with the development status marked as Beta. The licence is Apache-2.0, which is permissive and carries the usual obligations around notices and modification, and the metadata names one author and one maintainer, both at insitro.

The alternative approach, and where redun is the wrong tool

The real alternative to redun is not another engine so much as the approach redun argues against: writing the dataflow directly, as an explicit graph, or writing a sequence of shell commands and calling it a day. The difference is where the dependency information comes from. A hand-written graph makes the author responsible for declaring every edge, and that is exactly the bookkeeping that goes stale. Redun derives the graph from evaluating the code, so the edge and the code that creates it are the same artifact.

The other alternative is the make-like incremental build, which is where the inspiration comes from, and redun is a strict superset of that idea in one respect: it hashes code as well as data, so a change to a function invalidates the work that function produced.

There are real cases where redun is the wrong choice, and the README helps because it is honest about scope.

A team that wants a visual DAG editor, a job server with its own queue, or a product with a support contract is not looking at this project. The README does not document a multi-user server mode, a REST API, or an authentication and authorisation model, and its own examples are run from a command line in a local checkout.

The dependency footprint is also a factor. A plain `pip install redun` pulls in aiobotocore with boto3 and awscli extras, SQLAlchemy, Alembic, s3fs, requests, rich and textual, plus a set of version pins and exclusions that the maintainers have clearly fought with. Installing it is easy; keeping that dependency set conflict free inside a larger environment is the part that takes attention.

And there is a project-health dimension that the metadata does not soften. One author and one maintainer are listed, the package is marked Beta, and the last push to the repository was on 2026-07-17. The two GitHub releases listed are 0.27.0 and 0.26.0, both from 2025-05-23, while the metadata version is 0.46.0, so release tagging and version numbering have not stayed in step. None of that makes the framework unusable, and the design documentation, tutorial and the long list of examples give a real sense of how it works, but a team betting on it should plan for the possibility of maintaining a fork.

Editorial conclusion

redun fits a Python-first data team that wants one file to describe both the computation and its dependencies, and that needs caching and lineage across runs rather than inside one session. It does not fit a team that wants a declarative DAG file, an air-gapped environment without a Postgres recording backend, or a supported product with more than one maintainer, since the package metadata lists a single author and maintainer. Verify first that your execution backends and object stores are covered by the executors you need, and that the recording backend you run in Postgres is reachable, because caching and lineage both depend on it.

Frequently asked questions

Official sources

1. insitro/redun on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases