# Toil runs CWL, WDL and Python workflows through one engine

> Toil is a pipeline management system written entirely in Python that accepts workflows in Common Workflow Language, in Workflow Description Language, or as its own Python API, and can run the same job locally, on an HPC cluster or in the cloud. The breadth is the point, and the cost is a set of optional extras you have to choose deliberately.

**DataBiosphere/toil** — A WDL, CWL and Python API supporting easy-to-use workflow engine. It is scalable, efficient and cross-platform (Linux/macOS).

- Repository: https://github.com/DataBiosphere/toil
- Website: http://toil.ucsc-cgl.org/.
- Stars: 936 · Forks: 246
- Language: Python
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/databiosphere-toil

## Three ways to describe a workflow, one engine to run it

Toil is described as a scalable, efficient, cross-platform pipeline management system written entirely in Python, and it is built around the principles of functional programming. The distinctive part is not the execution engine but the number of ways you can hand work to it. It supports running workflows written in either Common Workflow Language 1.0 to 1.2 or Workflow Description Language 1.0 to 1.1, and it also has its own rich Python API for writing workflows against.

That third option is often ignored in comparisons, and it changes what Toil is. A CWL or WDL workflow is a document, portable across engines by design. A Toil Python API workflow is a Python program that imports Toil and builds the job graph in code, which gives you the whole language for control flow and data handling but ties the workflow to the engine that runs it.

The execution target is equally plural. The documentation states that Toil supports running workflows locally on your system, for example a laptop, on an HPC cluster, or in the cloud. The same workflow description therefore survives a change of execution environment, which is the practical argument for a system that accepts both standard languages and one proprietary-ish API.

The scope is worth stating plainly, because it is not a general data tool. It is a pipeline runner for genomics and data processing work, packaged with an Apache-2.0 licence and the copyright held by the Regents of the University of California.

## The extras split tells you exactly which backend you are buying

The repository root is where Toil's real architecture shows through. Instead of one dependency list, there is a main requirements.txt plus one file per optional capability: requirements-aws.txt, requirements-cwl.txt, requirements-dev.txt, requirements-encryption.txt, requirements-google.txt, requirements-htcondor.txt, requirements-kubernetes.txt, requirements-mesos.txt, requirements-server.txt and requirements-wdl.txt.

The setup script picks the right file at install time by name.

```python
def get_requirements(extra=None):
    """
    Load the requirements for the given extra.

    Uses the appropriate requirements-extra.txt, or the main requirements.txt
    if no extra is specified.
    """
    filename = f"requirements-{extra}.txt" if extra else "requirements.txt"
```

So a CWL installation and a WDL installation pull different dependencies, and the Kubernetes and Mesos backends pull their own. The base requirements.txt is small and unsurprising for a workflow runner: dill for serialising objects, requests and urllib3 for HTTP, docker for container work, pyyaml and ruamel.yaml for configuration, configargparse for command line parsing, psutil for process information, coloredlogs and prompt_toolkit for terminal output.

One detail in setup.py is worth reading twice, because it explains an oddity you will otherwise hit on a Mac. The comment says htcondor is not supported by apple, and notes that it is tricky to conditionally support in the all extra due to how wheels work, so it is not included in all. Condor support therefore has to be asked for by name rather than arriving with the blanket extras set.

## make develop extras=[all] and why a virtualenv matters

Toil does not ship a single install command in the README. The build instructions live in the Makefile, and they start with a constraint: all build targets require a virtualenv to be active. The supported targets are prepare, develop, docs, dist, clean, test, docker and push_docker.

The develop target creates an editable install of Toil and its runtime requirements in the current virtualenv, which means changes to the source code immediately affect the virtualenv. The extras variable is how you add backend support, and the Makefile gives the full-coverage form.

```bash
	make develop extras=[all]
```

The Makefile also warns that some tests require extras to be installed, and points to setup.py for the list of supported extras. That warning explains a common first-run failure: a partial install can pass a simple workflow and then fail inside the test suite on a missing backend dependency.

The build itself uses setuptools with setuptools_scm, so the version is derived from the source tree rather than typed into a file.

```toml
[build-system]
requires = ["setuptools>=64", "setuptools_scm>=8"]
build-backend = "setuptools.build_meta"
```

Two other targets matter for anything beyond local experimentation. The dist target creates a source distribution and a wheel, and it is used for some unit tests and for installing the currently checked out version into the appliance image. The docker target exists as a separate target, and the repository carries a docker/ directory plus a setup_gitlab_docker.py script, so a container image is a first-class distribution path rather than an afterthought.

## Version 9.5.0, a Python 2 story, and a licence that lets you keep changes private

The release history is short and recent. Version 9.5.0 was released on 2026-06-02, 9.4.1 on 2026-05-07 and 9.4.0 on 2026-05-05. The last push to the repository was on 2026-10-21, so the project is still receiving work rather than sitting on a historical tag.

The README keeps a short notes section about how the project has moved, and it is useful for judging migration cost. Toil moved from the BD2KGenomics organisation to DataBiosphere on July 5th, 2018, and Toil dropped Python 2.7 support on February 13, 2020, with the last working py2.7 version being 3.24.0. That is a hard boundary: if you inherited a pipeline that still depends on the py2.7 era, the version ceiling is 3.24.0, not the current release.

The licence is Apache-2.0, and the repository carries both a LICENSE file and the standard Apache header in each source file, with copyright the Regents of the University of California. Apache-2.0 is permissive and includes an explicit patent grant, which is why it is a common choice for academic and commercial use alike. The repository also includes an attic/ directory and a dashboard/ directory, alongside docs/, contrib/ and examples/, which tells you the project keeps older code around rather than deleting it.

The remaining question a reader should settle is not legal but operational. The documentation and community channels are spread across the project website, Read the Docs, a low volume announce mailing list, a Google Groups community forum, a blog and a BioStars channel, so plan where you will track release notes rather than assuming one feed is enough.

## CWL or WDL or the Python API: choosing between the three fronts

The three interfaces are real alternatives with different consequences, and the choice is a fork in the road for a new project.

A Common Workflow Language workflow is a document that other engines can also read. Given that Toil carries a CWL 1.2 conformance badge in the README, and that the description lists CWL among the topics, CWL is the interface with the strongest outside validation. If your pipeline might need to run somewhere other than Toil later, CWL is the safer bet, and the cost is a JSON or YAML file with a schema to learn.

A Workflow Description Language workflow is the alternative for teams who prefer that syntax, supported across versions 1.0 to 1.1, with a dedicated requirements-wdl.txt for its dependencies. Choosing WDL means accepting a narrower portability story than CWL and gaining a workflow description that many bioinformatics groups already write.

The Python API is the different bet. A workflow written against Toil's own API can express things a declarative document cannot, and it inherits everything Python gives you. The price is portability: that workflow is coupled to Toil, so the CWL and WDL arguments in reverse. Where a pipeline needs custom control flow, that coupling is often the right trade; where a pipeline needs to move between execution sites, it usually is not.

The same split shows up in execution backends. Local execution on a laptop, an HPC cluster and the cloud are not interchangeable at the operational level, and the extras lists show which of them carry their own dependencies, from htcondor through Kubernetes and Mesos to the cloud providers.

## Linux and macOS only, and no Windows story

The one hard constraint that will end an evaluation early is the platform. The project describes itself as cross-platform, but the parenthetical is explicit: Linux and macOS. There is no Windows in that list, and the repository shows how the macOS limitation is handled in practice, with htcondor deliberately excluded because apple does not support it.

The repository layout reinforces that this is a Linux-first system built for cluster environments. It carries a gridengine-related topic, mentions Slurm among its topics, and keeps integration scripts named setup_gitlab_ssh.py and setup_gitlab_docker.py. The extras names line up with the deployment targets a cluster or cloud user actually has: aws, google, kubernetes, mesos, htcondor, server, encryption, cwl, wdl and dev.

The other limitation worth naming is a documentation split. The README itself is short and points outward, describing the project and linking to the website for a description of Toil and its features, with full documentation for the latest stable release on Read the Docs. If you are evaluating from the README alone, you are reading the entry point rather than the reference, and the practical details of job store configuration, restart semantics and backend flags live in the hosted documentation.

One more constraint applies to anyone building from source rather than installing a release: the Makefile is explicit that build targets need an active virtualenv, and the dist target is what produces the artifact you would install into an appliance image.

## Conclusion

Toil fits teams that already have workflows in more than one description language, or that want to move a working CWL or WDL pipeline between a laptop, a Slurm or Mesos cluster and a Kubernetes deployment without rewriting it. It is the wrong tool for a single small pipeline in one language, where the extras and the job store overhead are pure cost, and it will not run on Windows at all. Before adopting, check which extras your backend needs against the requirements files in the repository root, install with make develop extras=[all] so the test suite has what it needs, and confirm that version 9.5.0 from the 2026-06-02 release still speaks the CWL and WDL versions your existing workflows use.

## FAQ

### How do I install Toil from source?

The build instructions live in the Makefile rather than the README. All build targets require an active virtualenv, and the develop target creates an editable install, with extras added through the extras variable as in make develop extras=[all].

### Which workflow languages can Toil run?

Toil supports Common Workflow Language 1.0 to 1.2 and Workflow Description Language 1.0 to 1.1, and it also offers its own Python API for writing workflows against.

### Can Toil run on Windows?

No. The project describes itself as cross-platform but limits that to Linux and macOS, and its extras list reflects the same orientation, with cluster and cloud backends such as htcondor, Kubernetes and Mesos.

### What licence is Toil released under?

Apache-2.0, with copyright held by the Regents of the University of California, as stated in the repository LICENSE file and the headers on the source files.

### Which Python versions does Toil support?

Toil dropped Python 2.7 support on February 13, 2020, and the last working py2.7 version is 3.24.0. Current releases require a modern Python 3 environment with setuptools 64 or newer and setuptools_scm 8 or newer for the build.

## Sources

- [DataBiosphere/toil on GitHub](https://github.com/DataBiosphere/toil)
- [License: Apache-2.0](https://github.com/DataBiosphere/toil/blob/master/LICENSE)
- [Project website](http://toil.ucsc-cgl.org/.)
- [README](https://github.com/DataBiosphere/toil/blob/master/README.md)
- [Releases](https://github.com/DataBiosphere/toil/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/databiosphere-toil