# Bacalhau: scheduling compute where the data already lives

> Bacalhau is a Go framework for running jobs on the machines that hold the data instead of copying the data to the cluster. One binary acts as client, orchestrator and compute node, jobs reach Docker or WebAssembly engines, inputs and outputs travel through S3, HTTP, IPFS or local storage, and the last push on 2026-10-21 sits behind a v1.9.5 tag from nine days earlier.

**bacalhau-project/bacalhau** — Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

- Repository: https://github.com/bacalhau-project/bacalhau
- Website: https://docs.bacalhau.org
- Stars: 876 · Forks: 105
- Language: Go
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/bacalhau-project-bacalhau

## The idea in one line: compute over data, not data over compute

Bacalhau is described as an open-source distributed compute orchestration framework designed to bring compute to the data. The README is explicit about the alternative it rejects: instead of moving large datasets around networks, Bacalhau executes jobs close to where the data's location is, which reduces latency and resource overhead.

That framing is not a marketing position so much as an economic one. The README's list of reasons gives ingress and egress costs as the second item, after fast job processing, and the petabyte-scale claim follows from the same logic: if the data does not travel, the size of the data stops being a constraint on the job.

The security argument is the same argument seen from a different angle. The README lists data scrubbing and security happening before migration with a granular, code-based permission model, and separately lists data sovereignty, meaning processing sensitive data inside security boundaries without it leaving the premises. Cross-organizational computation is the third variation on the theme, allowing specific vetted computations on protected datasets without exposing the raw data.

The repository description puts it as a community-driven framework for fast, cost-effective distributed compute over data, and the topics attached to the repository name the domains it is aimed at: bioinformatics pipelines, gene sequencing, AI data collection, data engineering, video processing and IoT.

## One binary that is client, orchestrator and compute node

The first key feature in the README is single binary simplicity: Bacalhau is a single self-contained binary that functions as a client, an orchestrator and a compute node, which the README says makes it easy to set up and scale.

That is the architectural decision that shapes everything else. There is no separate control plane binary to deploy, no agent to install on worker hosts and no configuration file describing which nodes exist. A node is a node is a node, and the role it plays is decided by how you invoke it.

The third feature names the model explicitly: a dedicated orchestrator coordinates job scheduling while compute nodes run tasks. So the roles exist, they are just not different programs. The orchestrator knows about the fleet and decides where a job should run; the compute node is the thing that talks to the execution engine.

The repository layout backs up the Go and CLI story. There is main.go at the root, a cmd/ directory, a clients/ directory, and then a pkg/ directory that holds the substantial parts of the implementation. A go.work file sits alongside go.mod and go.sum, which indicates a Go workspace, and the module path is github.com/bacalhau-project/bacalhau. The go.mod declares Go 1.26.9.

There is also a webui/ directory and a python/ directory at the top level, alongside a pyproject.toml that configures Poetry for the Python tooling. The web interface is a separate part of the tree from the Go binary, and the Python side appears to be developer tooling rather than part of the runtime.

## Execution engines and storage integrations

The second key feature is modular architecture, with support for multiple execution engines, Docker and WebAssembly, and for storage providers, reached through clean interfaces.

Two engines is a narrow but meaningful choice. Docker is the general case: anything with a container image runs. WebAssembly is the interesting one, because a WebAssembly module has no system dependencies and no filesystem assumptions beyond what the runtime provides, which makes it a natural fit for running untrusted or portable computation close to someone else's data. The dependency list confirms both are real: docker/docker v28.5.2 is a direct requirement, and the Go module list includes the WebAssembly tooling along with the rest of the stack.

Storage is the other half, and the README names S3, HTTP/HTTPS, IPFS and local storage systems. The IPFS support is not incidental. The go.mod file carries the IPFS stack directly: kubo v0.43.1, boxo, go-cid, go-ipld-format, go-unixfsnode, the CAR format and the DAG codec packages. That is a dependency footprint for content-addressed data rather than a thin wrapper around an HTTP client.

The consequence is that inputs and outputs are described as addresses rather than as paths. A job is given where its input lives and where its result should go, and the storage integration resolves them. The README lists publisher support as its own feature, with outputs going to local volumes, S3 or other storage backends, which is the output half of the same model.

## Job types and the two ways to submit work

Bacalhau supports several job types, listed as batch, ops, daemon and service jobs for different workflow requirements.

Batch is the familiar case: run this container over this input, write the result. Daemon and service cover long-running work, a process that stays up rather than exits, which is what you want for a stream processor or an API. Ops is the one the README does not define further, and it is worth noting that the repository has a top-level ops/ directory, so the term is structural rather than merely a label.

Submission comes in two styles. Declarative submission defines jobs in YAML, and imperative submission passes arguments via the command line. The documentation site has separate pages for installation, submitting your first job and publishing results, which suggests the YAML form is the fuller one.

The YAML route matters for more than convenience. Declarative job definitions are files, which means they can be reviewed, diffed, stored in version control and applied by a pipeline. A framework whose whole argument is reproducibility across an organisation needs a way to describe a job that two people in two organisations would run identically, and a YAML document is that way.

The documentation index in the README lists the three pages you need to start, and the project also maintains a Quick Start guide for running a first job.

## Installing the CLI and checking the version

The installation path in the README is a shell pipeline followed by a version check:

```bash
# Install Bacalhau CLI (Linux/macOS)
curl -sL https://get.bacalhau.org/install.sh | bash

# Verify installation
bacalhau version
```

Two things are worth noticing. The installer is fetched over HTTPS and piped straight into a shell, which is the shortest possible path to a working binary and also the one that requires you to trust the host and the moment. The comment marks it as the Linux and macOS path, and the README links a fuller Installation Tutorial for the cases where a script is not what you want.

The bacalhau version command is the first thing to run, and because the binary is the client as well as the orchestrator and the node, it is also the quickest way to see which build you have on all three roles. That matters more than it sounds in a fleet, where the orchestrator and the compute nodes have to agree.

Building from source is a Go build, and the Makefile at the root shows how the project does it. The Makefile exports GOOS and GOARCH from the environment, sets CGO_ENABLED to 0 so the result is a static binary, and handles platform details such as mapping armv6 and armv7 to arm with the right GOARM value, and appending .exe on Windows. It also derives the version, commit, repository URL and branch from git at build time and stamps them into the binary through the version package, with the build date set from the clock.

For local development the project expects Poetry, and the pre-commit hook runs through it. The pyproject.toml pins Python at ^3.10.5 and lists the tooling dependencies, including ruff and pre-commit, and the repository carries separate linting configuration for Go, Python, shell, YAML and Markdown.

## Where Bacalhau stops, and where the claims thin out

Two limits are visible from the README and the dependency list.

The first is the engine set. Two execution engines means a job is either a container or a WebAssembly module. There is no mention of running a plain binary, a Kubernetes job spec, or a Slurm job on an existing HPC allocation, and given that the topics include gene sequencing, that omission is going to matter to somebody. The same applies to storage: S3, HTTP, IPFS and local are a good spread, but a job reading from a mounted NAS, a Parquet lake on HDFS or a database over a wire is not covered by anything described here.

The second is that the README quantifies almost nothing. It claims fast job processing, low cost, petabyte-scale processing, reduced latency and resource overhead, and it names use cases across log processing, distributed data warehousing, fleet management, distributed machine learning and edge computing. There are no benchmark numbers, no cost figures, no throughput figures and no comparison table. The repository does contain a benchmark/ directory at the top level, so there is at least some measurement infrastructure. It just is not in the README.

A third, quieter point concerns the trust model. The pitch includes running vetted computations on other organisations' protected data, and the README mentions a granular, code-based permission model, but the details of how a remote node is authorised to read a specific input are not described here. The go.mod does show JWT libraries and key handling in the dependency list, and the Makefile refers to a private and public key file, so there is a signing story, but a reader cannot verify the authorization model from the README alone.

## Bacalhau against a shared cluster and against a workflow engine

The comparison worth making is with the two things you would otherwise build.

The first is the conventional cluster with a shared filesystem. You copy the data into object storage once, submit a job to a scheduler, and read the result back. That works well when the data is already centrally held and the jobs are large. Bacalhau is for the opposite situation, where the data is distributed across sites and copying it is the dominant cost. If your data is already in one place, Bacalhau adds a scheduling layer and a set of storage abstractions without removing any work, because the copy it is designed to avoid was never the problem.

The second is a workflow engine such as an Airflow or Prefect style orchestrator. The difference is the unit of work. A workflow engine sequences steps that each run on infrastructure you control, and it is good at dependency graphs, retries and schedules. Bacalhau's unit is a job that can run anywhere, including on a node operated by a third party, and the reason it can is that the job carries its execution environment with it, as a container or a WebAssembly module, and reaches its data through an address rather than a mount. That is a fundamentally different answer to the problem of running code near data.

The practical consequence is that the two are closer to complementary than to competing. A workflow engine can be the thing that decides what should happen, and Bacalhau can be the thing that runs a particular step in a place where the data already is. The cost of that combination is two systems to operate and a mapping between their models, which is the tax every layered design pays.

## Conclusion

Adopt Bacalhau when your workload is large enough that moving the bytes costs more than running the job, when the data already sits inside organisations you do not own, and when a job that reads a petabyte should never cross the network. Do not adopt it for a single cluster with a shared filesystem, because that is the case it is designed to avoid. Verify first that your data source is reachable through one of the supported storage integrations, that the execution engine you need is Docker or WebAssembly rather than something else, and that the orchestration layer you are joining has the nodes you need.

## FAQ

### What is Bacalhau and what problem is it meant to solve?

Bacalhau is an open-source distributed compute orchestration framework that runs jobs close to where the data lives rather than moving large datasets over a network. The README lists reduced latency, lower ingress and egress cost, data sovereignty and cross-organizational computation among the reasons for that design.

### How do I install the Bacalhau CLI and check it works?

On Linux and macOS the README gives a single command, curl -sL https://get.bacalhau.org/install.sh | bash, followed by bacalhau version to verify the installation. A fuller Installation Tutorial is linked for other cases.

### Which execution engines and storage backends does Bacalhau support?

The README names Docker and WebAssembly as the execution engines, reached through clean interfaces, and S3, HTTP/HTTPS, IPFS and local storage for data. Job types are listed as batch, ops, daemon and service, submitted either as YAML or through command line arguments.

## Sources

- [bacalhau-project/bacalhau on GitHub](https://github.com/bacalhau-project/bacalhau)
- [License: Apache-2.0](https://github.com/bacalhau-project/bacalhau/blob/main/LICENSE)
- [Project website](https://docs.bacalhau.org)
- [README](https://github.com/bacalhau-project/bacalhau/blob/main/README.md)
- [Releases](https://github.com/bacalhau-project/bacalhau/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/bacalhau-project-bacalhau