# PaSh compiles POSIX shell pipelines instead of wrapping them

> PaSh is a data-parallel shell processing system that parses shell scripts, finds dataflow regions and compiles them to run in parallel, with a just-in-time runtime as a second path. It is POSIX first, bash needs a beta flag, and the runtime is compiled C.

**binpash/pash** — PaSh: Light-touch Data-Parallel Shell Processing

- Repository: https://github.com/binpash/pash
- Stars: 604 · Forks: 54
- Language: Shell
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/binpash-pash

## PaSh rewrites the pipeline rather than scheduling it around

PaSh is a system for parallelizing POSIX shell scripts, hosted by the Linux Foundation, and it describes itself as light-touch data-parallel shell processing. The distinction that matters is how it works. Most attempts at shell parallelism wrap commands in a job scheduler and hope the stages line up. PaSh goes further: it reads the script, works out where the data flows, and compiles those regions so the stages can run at the same time.

That makes it a source-to-source compiler rather than a runner. The two academic write-ups the project asks you to cite say so directly. The first is the EuroSys 2021 paper PaSh: Light-Touch Data-Parallel Shell Processing, and the second is the OSDI 2022 paper on practically correct just-in-time shell script parallelization. The second paper describes a different execution strategy for the same front end, which is the subject of the runtime section below.

The intended users are the same people who have a working shell pipeline that has become too slow: system administrators and developers who inherited a script nobody wants to rewrite in C or Python because the semantics of the pipeline are the point.

## Four directories, four stages of the pipeline

The repository layout describes the architecture more clearly than any diagram would. The main Python package lives under src/pash/, and inside it four components split the work.

The preprocessor parses shell scripts, expands variables, and identifies dataflow regions for compilation. The compiler translates those shell dataflow regions into intermediate representations and applies the parallelization transformations. The jit_runtime is a just-in-time runtime that executes compiled regions and manages shell state. The runtime directory holds the runtime components, and the README names them as eager, split, and associated combiners.

That ordering matters for anyone debugging a failure. A problem in variable expansion is a preprocessor problem, a wrong dependency between stages is a compiler problem, and a shell state issue at execution time belongs to the jit_runtime. The evaluation directory holds the shell pipelines and example scripts used for evaluation, and the scripts directory holds the continuous integration, deployment and testing automation.

The Python side is only half the system. The runtime layer is C, and the package data declaration in the project metadata says so explicitly: pash.runtime ships C sources, headers, shell scripts and a Makefile, while pash.jit_runtime ships shell scripts.

## Installing with pip and running a parallel job

The recommended installation is pip, followed by a one-liner check that the command is wired up.

```sh
pip install pash
pash -c "echo hi"
```

The command is invoked as pash, and the entry point in the project metadata is pash.cli:main. To actually parallelize a script, the flag that matters is -w, which sets the parallelization degree. The example in the project runs a script from the evaluation directory at degree 2.

```sh
pash -w 2 ./evaluation/intro/hello-world.sh
```

For local development the project documents an editable install and a separate test entry point.

```sh
pip install -e .
pash --help
pash -c "echo hello | cat"
./scripts/run_tests.sh
```

More detail on manual installation and on other platforms sits in the installation instructions under docs/install. The pash --help output is the place to look for the full set of commands and flags.

## Bash syntax needs the beta flag, and the dependency list explains why

PaSh targets POSIX shell, and the difference shows up when a script uses bash extensions. The project's guidance is that if the script contains bash specific syntax, you add the beta --bash flag to enable support.

The dependencies listed in the project metadata explain what that support is built on. Alongside graphviz, libdash, shasta and sh-expand, there is libbash, pinned to a 0.1 series, and a separate pash-annotations package pinned to a 0.2.4 range.

```toml
dependencies = [
    "graphviz",
    "libdash",
    "pash-annotations~=0.2.4",
    "shasta~=0.5",
    "libbash~=0.1.14",
    "sh-expand~=0.2.0",
]
```

Two things follow from that list. The annotation library is where the knowledge about specific commands lives, and the project documents a separate workflow for running against a local checkout of it, so you can extend or debug the annotations without reinstalling. The libbash dependency is the reason the bash support is labelled beta rather than being a default, and it is worth noting that a version of libbash in the 0.1 series is an early dependency to build on.

The interpreter the runtime leans on is dash, through the libdash package, which is the POSIX shell rather than bash.

## Installing from source compiles C binaries, so a toolchain is required

The pip install is not a pure Python operation. The setup script exists to compile the C runtime binaries during installation, and it checks for the tools it needs before it starts. The required set is gcc, make and git, and the script raises an error naming each missing one instead of failing later in the build.

The build compiles a fixed list of runtime targets in the runtime source directory.

```python
    targets = ["eager", "split", "r-merge", "r-wrap", "r-split", "r-unwrap", "set-diff", "dgsh-tee"]
```

The names describe the shape of the runtime. eager and split are the runtime components named in the repository structure, and the r-prefixed targets are the combiners that glue compiled regions back together. dgsh-tee is called out separately in the script because it needs git to clone during the build, which is why git is on the required tool list at all.

The practical consequence is that PaSh cannot be installed in an environment without a C toolchain, even though the package itself is managed by pip and setuptools. The build backend is setuptools with the src directory as the package root, and the optional dev extra adds pytest, black, mypy, build and twine.

The project metadata also pins the interpreter. The requires-python field asks for 3.12 or newer, and the classifiers list both Python 3 and Python 3.12, so an older 3.11 environment is not supported.

## The alternative is rewriting the pipeline, and the difference is who owns the semantics

The realistic alternative to PaSh is not another shell parallelizer, it is replacing the script. If a pipeline is the bottleneck, the conventional fix is to rewrite the hot part in C, in Python, or in a streaming framework, which is a change in ownership as much as a change in language.

With a rewrite, the shell script stops being the source of truth. You end up maintaining two artefacts, the script that still runs in places the rewrite does not reach, and the rewrite itself, and the parity between them is yours to maintain. PaSh takes the opposite route. The shell script stays the artefact you edit, the preprocessor and compiler derive the parallel version from it, and there is no second copy to keep in step.

There is a middle path inside the project as well. The compiler path transforms regions ahead of execution, while the jit_runtime path executes compiled regions and manages shell state at run time, which is the subject of the OSDI paper on practically correct just-in-time parallelization. Which one you get is a property of the toolchain rather than something you configure per script.

The cost of the compile-in-place approach is that the transformation is only as correct as the annotation library. Unknown commands, unusual shell constructs and the difference between what a shell does and what the annotations say it does are where surprises live, which is why the annotation library has its own local development workflow.

## A beta project with a long enough history to be worth trying

On status, the honest summary is that PaSh is a research system that is usable in production by people who accept a beta label and read the annotations. The project metadata classifies it as Development Status :: 4 - Beta, targets POSIX systems including Linux and MacOS X, and lists both developers and system administrators as the intended audience.

Activity continues. The repository is not archived and the last push was on 2026-09-28. The release history shows v0.16 on 2026-01-22, v0.15 on 2026-01-18, and before that v0.14 on 2024-11-19, a gap that shows the project does not ship on a fixed schedule. The current version in the project metadata is 0.16.6.

The licence is MIT, with the licence file referenced from the project metadata, so a fork or an internal deployment carries no obligation beyond attribution. The repository also carries a contribution guide under docs/contributing and a code of conduct, and development happens in the open over Discord.

One rough edge in the documentation is worth flagging: the chat link in the readme is written with a malformed scheme, so the invite link next to it is the one that works.

## Conclusion

PaSh fits infrastructure teams with a slow POSIX shell pipeline they cannot rewrite in another language, and who can accept a beta classification, a Python 3.12 floor and a C build step on the machine that runs it. It is the wrong tool for bash-heavy scripts that will not work with the --bash flag, for environments where you cannot install build tools, or for anyone who needs guaranteed identical output today rather than a compiled approximation of it. Verify first that your script uses POSIX syntax, that the pash -w degree matches the resources you want to spend, and that the runtime binaries compile on your target before you point a production job at the tool.

## FAQ

### How do I install PaSh?

Run pip install pash, then check the installation with pash -c "echo hi". The command entry point is pash.cli:main, and the installation instructions under docs/install cover manual installs and other platforms.

### Does PaSh support bash scripts?

Only through the beta --bash flag, which the project says is needed when a script contains bash specific syntax. The support depends on the libbash dependency, pinned to the 0.1 series.

### How do I run a shell script in parallel with PaSh?

Use the -w flag with the parallelization degree, for example pash -w 2 ./evaluation/intro/hello-world.sh. Run pash --help to see the rest of the available commands.

### What does PaSh need installed on the machine to work?

Python 3.12 or newer, plus gcc, make and git, because the setup script compiles the C runtime binaries during install and checks for those tools before building. The runtime targets include eager, split, several r-prefixed combiners and dgsh-tee.

## Sources

- [binpash/pash on GitHub](https://github.com/binpash/pash)
- [Issues](https://github.com/binpash/pash/issues)
- [License: MIT](https://github.com/binpash/pash/blob/main/LICENSE)
- [README](https://github.com/binpash/pash/blob/main/README.md)
- [Releases](https://github.com/binpash/pash/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/binpash-pash