# Gearpump maps streaming work onto the Pekko actor hierarchy

> Gearpump is a Scala streaming engine that models jobs as actors inside Apache Pekko and ships as a single assembled archive. It suits teams that want one runtime for message passing, but the docs stop well before deployment.

**gearpump/gearpump** — Lightweight real-time big data streaming engine over Apache Pekko

- Repository: https://github.com/gearpump/gearpump
- Website: https://gearpump.github.io/gearpump/
- Stars: 755 · Forks: 150
- Language: Scala
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/gearpump-gearpump

## Gearpump models streaming inside the Pekko actor hierarchy

Gearpump describes itself as a lightweight real-time big data streaming engine built on top of Apache Pekko, and its central design decision is stated in a single line: streaming is modelled within the Pekko actor hierarchy. That choice matters for how you write and reason about a job. The unit of concurrency is an actor, the same abstraction used by the actor system already running in your Scala services, rather than a record pushed between stages by a separate scheduler.

The repository layout follows that idea. Alongside the top level core directory there are streaming, services, gearpump-hadoop and unmanagedlibs modules, plus a dev-tools directory and a separate examples tree holding beam, distributedshell, pagerank and streaming samples. The top level also keeps a conf directory for configuration and a bin directory for the scripts that ship with the assembled archive.

For engineers coming from a batch or micro-batch engine, the practical consequence is that state, failure handling and message routing are expressed as actor behavior. Whether that reads as simpler or as a layer of ceremony depends entirely on how familiar the Pekko programming model already is to the team picking it up.

## Two gears and a benchmark figure of 18 million messages per second

The project name is a deliberate reference to a gear pump, which the README describes as a super simple pump made of only two gears that is very effective at streaming water. The metaphor sets expectations for the codebase: a small number of moving parts rather than a broad catalog of connectors and abstractions.

Performance claims are stated as initial benchmarks, and the wording deserves care. The README reports near 18 million messages per second at 100 bytes per message with an 8ms latency on a 4-node cluster, and points readers to a performance report page hosted on the documentation site for the steps needed to reproduce the test. No hardware specification, no JVM tuning notes and no test method are given in the repository itself, so the figure is best treated as a directional claim rather than a number to plan capacity against.

Two screenshots are embedded in the README, one of a dashboard animation and one of a static dashboard image. Both point at an operational console for inspecting running jobs, which suggests that the project expects a human to watch a deployed cluster rather than only a headless pipeline.

## Building the pack archive with sbt assembly

There is no package manager install path given for Gearpump. The build instructions start from a clone of the repository, and the build itself is driven by sbt rather than by Maven, even though the README links to a Maven dependencies page for consumers who want to depend on the published artifacts.

Clone the source and move into the directory:

```bash
  git clone https://github.com/gearpump/gearpump.git
  cd gearpump
```

Then produce the distributable archive:

```bash
  ## Please use JDK 17 or 21; the build currently uses Scala 2.13.
  ## The target package path is under output/target/.
  sbt clean +assembly 'gearpump-pack / packArchiveZip'
```

After that step a package file named gearpump-${version}.zip appears under the output/target folder. Scala API documentation is generated separately:

```bash
   ## Will generate the scala doc under target/scala_2.xx/unidoc/
   sbt unidoc
```

That is the whole setup path the project offers. There is no quickstart command, no container image and no sample submission command in the repository, so the first real use means following the external guide for writing a first application and then the guide for submitting it.

## JDK 17, Scala 2.13 and the proxy steps the build expects

The build carries three explicit environment constraints. The comment inside the assembly command names JDK 17 or 21 and says the build currently uses Scala 2.13, which narrows the toolchain far more than the version numbers of the releases would suggest. A second comment states that the target package path sits under output/target, so a packaging change moves artifacts rather than the command that produces them.

The third constraint is network access. The build note warns that a network connection is required, and that anyone behind an enterprise proxy must set the proxy in the environment before running the build commands. Two shell specific blocks are given.

For windows:

```bash
set HTTP_PROXY=http://host:port
set HTTPS_PROXY= http://host:port
```

For Linux:

```bash
export HTTP_PROXY=http://host:port
export HTTPS_PROXY=http://host:port
```

Note the space after the equals sign in the Windows HTTPS_PROXY line. It is reproduced here exactly as written, and it is worth fixing in your own environment before assuming a proxy problem, because a stray space can silently produce a malformed URL for the second protocol.

## The documentation stops before cluster deployment and job submission

Everything the project says about actually running a job lives outside the repository. The README's useful resources list links to a Basic Concepts page, a guide for developing a first application, a guide for submitting a first application, a Maven dependencies page and the documentation site, and the performance report is hosted on that same site rather than in the tree.

What the repository does contain is the building block structure: a conf directory, a bin directory, the ReleaseProcess.md file, a KEYS file and a pubring.asc file alongside a sealed key, a checkstyle-config.xml and a scalastyle-config.xml, a codecov.yml and a .coveragerc. The presence of the signing artifacts indicates a release process that verifies and signs artifacts, and the style and coverage configuration files show that the same code path is expected to pass static checks and coverage collection in CI.

What is not documented in the tree is the operational surface a new adopter needs first: how a cluster is formed, how a submitted application is scheduled across nodes, what happens to a job when a node dies, and how state is recovered. The Basic Concepts page is the place to look for those answers, and the fact that this core content lives off repository is the single biggest constraint on evaluating the project.

## Apache-2.0 covers Gearpump, and LICENSE.bin covers the libraries

Gearpump itself is licensed under the Apache License 2.0. The README separates the project license from the licenses of the libraries it uses, pointing readers to a LICENSE.bin file in the repository for the dependency set. Two pieces of borrowed code are acknowledged explicitly: the netty transport work is based on Apache Storm, and the cgroup code work is based on JStorm, with thanks extended to the contributors of both.

Those acknowledgements have a practical consequence. Netty based transport and cgroup handling are exactly the layers where the underlying project's license terms travel with the code, so an organization that already has a policy about Storm derived or JStorm derived components has to trace that policy into the packaged archive, not just read the top level LICENSE file. The Apache 2.0 grant on the project as a whole does not by itself answer questions about every binary shipped inside the archive.

To see the actual dependency list the project offers a task:

```
  sbt dumpLicenseReportAggregate
```

Running that against your own build produces the report of all dependencies, which is the correct starting point for a license review rather than an assumption that everything bundled is covered by the same terms.

## A seven year gap separates 0.9.0 from 0.10.0

The release history is lumpy. Version 0.8.1 shipped on 2016-09-01, version 0.9.0 shipped on 2019-02-12, and version 0.10.0 shipped on 2026-08-12. Between the two middle releases there is a gap of close to seven years, so the project has a habit of long dormancy punctuated by a burst of work rather than a steady release train.

The repository itself is not archived, and the last push was on 2026-10-14, which means work continued in the weeks around the 0.10.0 release. Whether that pace holds is a question only time and the commit history can answer, but the pattern in the version list is the more useful signal for planning: assume the code you pin today may not see a follow up release for a long stretch, and expect breaking changes to arrive all at once when one does come.

The version numbering has stayed below 1.0 throughout its life, which is a reasonable signal that interfaces are still expected to move. For a team considering adoption, that argues for treating the actor APIs as something to wrap in an internal layer of your own rather than to expose directly to application code.

## How the actor model differs from a batch first streaming engine

The alternative approach worth weighing is not a single competitor but a different shape of solution: an engine whose unit of work is a record passing through a sequence of processing stages, with resources and offsets managed by the framework rather than by actors that the application code writes.

The difference shows up in what you own. In Gearpump you write actors and place them in a hierarchy, so backpressure, message routing and failure handling are expressed in code that you can read and test as ordinary Scala units. In a stage based engine, those concerns are typically handled by the framework, which reduces the amount of application code but removes your ability to shape the behavior directly.

There is a middle option that deserves a straight comparison here: using the actor system you already run, without a streaming layer at all. For a modest stream, a set of Pekko actors with a broker in front of them may cover the requirement with far less to operate and no dependency on a project whose releases arrive in multi year gaps. Gearpump earns its place when the stream needs the specific things its packaging provides, such as the assembled archive layout, the dashboard console and the bundled Hadoop module, rather than for the actor abstraction by itself.

## Conclusion

Gearpump fits teams that already run Scala services on Pekko and want their streaming code in the same actor model, and it is the wrong choice for anyone who needs a documented production deployment path, since the README hands off every operational detail to an external documentation site and last shipped a release on 2026-08-12. Before committing, check the current Apache Pekko version your branch depends on, read the Basic Concepts page for the deployment topology, and run sbt dumpLicenseReportAggregate to see what Apache-2.0 does and does not cover for the bundled libraries.

## FAQ

### What execution model does Gearpump use?

The README states that streaming is modelled within the Pekko actor hierarchy, so jobs are built from actors inside an Apache Pekko actor system rather than from stages that an external scheduler moves records between.

### How do I build Gearpump from source?

Clone the repository and run sbt clean +assembly 'gearpump-pack / packArchiveZip' with JDK 17 or 21, since the build currently uses Scala 2.13. The resulting gearpump-${version}.zip package file is written to the output/target folder.

### What throughput does Gearpump claim?

The README reports initial benchmarks of near 18 million messages per second at 100 bytes per message with an 8ms latency on a 4-node cluster, and links to a performance report page with the steps to reproduce the test.

### Which license applies to Gearpump and to the libraries it bundles?

Gearpump itself is licensed under the Apache License 2.0, while the licenses of the libraries it uses are listed in the LICENSE.bin file. The netty transport code is based on Apache Storm and the cgroup code is based on JStorm, and sbt dumpLicenseReportAggregate generates a report of all dependencies.

### How recently was Gearpump released?

Version 0.10.0 was released on 2026-08-12, and the last push to the repository was on 2026-10-14. The previous release, 0.9.0, dates from 2019-02-12, so the gaps between releases have historically been long.

## Sources

- [gearpump/gearpump on GitHub](https://github.com/gearpump/gearpump)
- [License: Apache-2.0](https://github.com/gearpump/gearpump/blob/master/LICENSE)
- [Project website](https://gearpump.github.io/gearpump/)
- [README](https://github.com/gearpump/gearpump/blob/master/README.md)
- [Releases](https://github.com/gearpump/gearpump/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/gearpump-gearpump