Parsl turns decorated Python functions into a workflow across cores and nodes

Decorators turn calls into futures, and futures chain

The core idea fits in a dozen lines. You decorate a function with python_app and calling it no longer runs the function, it returns a Future. Inside a parsl.load() context the decorated calls run concurrently, and a decorated function can take the futures of other decorated functions as its arguments, which is what makes a chain of dependent steps into a workflow rather than a pile of threads.

    import parsl
    from parsl import python_app


    # Make functions parallel by decorating them
    @python_app
    def f(x):
        return x + 1

The documentation is careful about where the real value lies. Parsl can be used like Python's parallel executors, but across multiple cores and nodes, and the real power of Parsl is in expressing multi-step workflows of functions, since it lets you chain functions together and will launch each function as inputs and computing resources are available. That last clause is the important design decision. The scheduler is free to start a step before the previous one has finished if the data dependencies allow it, so the shape of your data flow, not the order you wrote the lines in, determines what runs when. The example in the README shows two calls to f issued together and their results then passed into g, and the assertion on the result value confirms the semantics are the same as a serial function call once you wait on the future.

Configuration decides where the work runs

Parsl ships no executors of its own in the base install. Instead you tell it how to use your computing resource, and the documentation points new users at a configuration quickstart for that purpose, and only afterwards to the parallel computing patterns guide for deciding how to use parallelism in an application. That ordering is a fair summary of the learning curve. The library itself is installed with a single pip command.

pip3 install parsl

The optional dependency groups in setup.py are the clearest evidence of how wide the supported resource space is. There are groups for aws with boto3, kubernetes with kubernetes, azure with azure and msrestazure, google_cloud with google-auth and google-api-python-client, and a set of research platforms: globus_compute, globus_transfer, radical-pilot pinned to 1.90, workqueue, flux, proxystore, and gssapi for python-gssapi. There is also a monitoring group pulling in sqlalchemy between 2 and 2.1, and a visualization group with pydot, networkx, Flask, flask_sqlalchemy, pandas and plotly. So Parsl is not a local parallel library with optional remoting, it is a scheduler abstraction whose transport is a configuration choice, and the difference between a laptop run and a cluster run is a file.

The runtime rests on ZeroMQ, typeguard and dill

The core requirements file is short and tells you what the machinery is made of. pyzmq at 17.1.2 or newer provides ZeroMQ, which is the message passing layer the executors and the data transfer channel are built on. typeguard is pinned to at least 2.10, excluding the short-lived major version 3 and capping below 5, and the comment in the file explains why: major version 3 was rapidly replaced by major version 4, and versions 2 and 4 are both seen in the wild in complex deployments. typing-extensions at 4.6 or newer, dill for serialising Python objects that the standard pickle cannot handle, tblib for pickling tracebacks, requests, sortedcontainers, psutil at 5.5.1 or newer, setproctitle and filelock. The ZeroMQ dependency is the one to notice, because it means Parsl's executors are communicating processes rather than processes sharing a machine's memory, and that is what lets the same code run against a local thread pool and a remote cluster without the application changing.

Usage tracking is opt-in, and that is stated up front

The README carries a note that Parsl uses an opt-in model to collect usage statistics for reporting and improvement purposes, and it links to a usage tracking guide for understanding what is collected and how to enable it. Two things are worth noting about how that is framed. The collection is not on by default, so an installation does not report anything until someone turns it on, and the documentation points to a specific guide rather than a summary, which suggests the statistics list is detailed enough to need a page of its own. For a library that runs inside a workflow engine on institutional clusters, that distinction between opt-in and opt-out is the one that matters to administrators, and it is the first thing to check when a site's policy review asks what a dependency sends home. The project is also a NumFOCUS affiliated project and carries CZI Essential Open Source Software for Science badges, plus several NSF award badges, which is a reasonable signal about the governance around it.

The developer setup is a Makefile with an environment you activate

Contributing to Parsl starts with a clone and a make-driven environment, and the README walks through it as a numbered list. You clone the repository, move to the parsl directory, then run make to list the available targets, make virtualenv to create a virtual environment, source .venv/bin/activate to activate it, make deps to install the Python dependencies from test-requirements.txt, make test to run the tests, and make clean to remove the virtualenv and all test and build artefacts. The Makefile itself shows the same structure with targets for lint through parsl/tests/lint-inits.sh, isort and flake8 checks over parsl/, mypy with a MYPYPATH pointing at the mypy-stubs directory, and separate test targets such as local_thread_test and gce_test that run pytest against specific executor configurations. The presence of a local_thread_test target next to a gce_test target is a good sign about how the project is tested, since the same workflow code has to pass against several different executors. The repository also carries a CITATION.cff and a codemeta.json, so the project is prepared to be cited as a software artefact.

Where a simpler tool is the better answer

Parsl is a heavy answer to a problem, and it is worth knowing when the problem does not need it. If your work fits in one process, Python's own concurrent.futures and multiprocessing do the same job with less configuration and no ZeroMQ layer, and if your work is a single embarrassingly parallel loop over a local dataset, the extra machinery buys nothing. The library earns its weight when you have a dependency graph of steps that need different resources, a mix of CPU and GPU or HPC and cloud work, or a workflow that has to be reproduced later on different infrastructure. The cost is concrete. Every execution environment becomes a configuration surface with its own version pins and its own failure modes, and a workflow that crosses a network has to handle the transfer of data between stages, which is a real cost in time and in storage even though the API hides it. Parsl also asks you to define the resource layout before you write the workflow, so a team that has not settled how it wants to allocate compute will spend its first week on configuration rather than on the science. The library is a good fit when the workflow shape is the hard part, and an unhelpful one when it is just the inner loop that needs speeding up.

Editorial conclusion

Parsl suits research and HPC groups who want to express a multi-step workflow in ordinary Python and run it on a cluster, a cloud provider or a laptop, and it suits teams who need one API that covers both local multiprocessing and remote execution. It does not suit applications whose parallelism fits comfortably inside one process, and it does not suit teams unwilling to learn the configuration layer that tells Parsl where work can run. Before adopting it, install parsl with pip3, run the decorator example inside a parsl.load() block, and read the configuration quickstart, because the library's behaviour off a laptop depends almost entirely on that file.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Parsl/parsl on GitHub
4. Project website
5. README