Wallaroo wally: Pony APIs for distributed stream processing

Why Wallaroo wally exists for stateful stream processing

Wallaroo wally addresses the coordination work required to process continuous data across several machines. Its documented goals include resilient distributed computing, in-memory application state, low-latency processing, portability, and scaling while an application is running. The intended audience is therefore not limited to developers learning stream-processing concepts. Wally is aimed at teams that want the framework to coordinate distribution while application code defines transformations, keys, state, ingress, and egress.

The repository presents Wally as a mono-repository containing all Wally source. Its top-level layout is broad. Directories cover connectors, orchestration, monitoring, testing, deployment assets, documentation, and examples. That structure is consistent with a platform that includes more than a small data-processing library. It also means an adoption review must consider more than the runtime API, because deployment, monitoring, and connector behavior may affect operational fit.

Wally separates application logic from some distributed plumbing. A developer still defines how records enter the topology, how they are transformed, which key owns a state entry, how that state changes, and where results go. The framework connects those computations into a named application through `Wallaroo.build_application`. This can reduce the work involved in building a custom service around TCP, partitioning, state coordination, and live scale-out. The README does not, however, document delivery guarantees, retry behavior, or the exact process for recovering state after a worker failure. Those semantics should not be inferred from broad statements about resilience.

How Wally connects a TCP source to keyed state

The README's word-count application shows Wally as a typed pipeline. A `TCPSourceConfig[String]` creates a string source, `.to[String](Split)` splits each input into multiple strings, and `.key_by(ExtractWord)` assigns each emitted word as the key for state. The pipeline then applies `AddCount`, which implements `StateComputation[String, RunningTotal, WordTotal]`, and sends the result to a TCP sink.

The source has an explicit wire protocol. `StringFrameHandler.header_length()` returns `4`, while `payload_length` reads the payload length with `Bytes.to_u32`. `decode` converts the payload bytes into a Pony `String`. An application using a different protocol must provide framing behavior that matches that protocol. This is not an incidental detail because the source parser and payload decoder sit directly on the network boundary.

State handling is equally visible. `WordTotal` stores a `U64` count. `AddCount.initial_state()` creates a count of zero, and `apply` increments the value before returning a `RunningTotal` containing the word and updated count. `ExtractWord` returns the input word itself as a `Key`, so the framework can route updates for the same word to the corresponding state entry.

The final stage uses `TCPSinkConfig[RunningTotal]`, whose encoder formats each result as a word, comma, count, and newline. This gives the example a complete path from framed TCP input through parsing and stateful processing to framed TCP output. What remains undocumented is equally important: the README does not define at-least-once or exactly-once processing, source replay, state persistence, or transaction boundaries. Those details must be checked before relying on Wally for use cases where duplicate or lost updates would be costly.

Installing wally with the versioned Wallaroo Up route

The README directs users to the Wallaroo Up installation page. The repository's `Dockerfile` supplies a more concrete installation recipe based on release `0.6.1`. The file pins that version with `WALLAROO_VERSION`, downloads `wallaroo-up.sh`, makes it executable, selects the Docker source, and requests all installation components.

ENV WALLAROO_VERSION 0.6.1
ENV PATH /wallaroo-bin:$PATH
ENTRYPOINT ["env-setup"]

The installer commands in the container file are:

curl https://raw.githubusercontent.com/WallarooLabs/wallaroo/${WALLAROO_VERSION}/misc/wallaroo-up.sh -o wallaroo-up.sh -J -L
chmod +x wallaroo-up.sh
export WALLAROO_UP_SOURCE=docker
export CUSTOM_WALLAROO_BUILD_ARGS="target_cpu=x86-64"
echo y | ./wallaroo-up.sh -t all

The image starts from `ubuntu:xenial-20171006`, which is an Ubuntu 16.04-era base. It installs Python 2 and Python 3 development packages, `virtualenv`, and `virtualenvwrapper`, then runs the installer. At the end of the build, the image links a `wallaroo-tutorial/wallaroo-0.6.1` directory at `/wallaroo-src`, copies `docker/env-setup` into `/wallaroo-bin`, and declares `/src/wallaroo` as a volume.

That is a reproducible starting point for the repository's declared container environment, but it is not evidence of compatibility with a current workstation or cloud image. The README does not provide a complete host CLI invocation for running the word-count application. It also does not document the source and sink options represented by `TCPSourceConfigCLIParser` and `TCPSinkConfigCLIParser`. Those details should be read from the linked installation documentation and the files under `examples/pony/word_count/` before building a local launch command.

A first wally use through the Pony word-count topology

The README identifies `examples/pony/word_count/` and `examples/pony/market_spread/` as useful starting points. The word-count example offers the clearest introduction because its stages have narrow responsibilities. `Split` implements `StatelessComputation[String, String]`, lowercases each line, removes leading and trailing punctuation, splits on punctuation, and returns an array of words. Keeping this operation stateless prevents parsing concerns from being mixed into the counter state.

`ExtractWord` then uses each resulting string as its `Key`. `AddCount` is the stateful component. It receives a word and a `WordTotal`, increments `state.count`, and returns `RunningTotal(word, state.count)`. The sink encoder writes the word and count as a text line. For a continuous stream, this is a running count after each processed word, not a final summary produced after end-of-stream.

The example also exposes a business-rule limitation. The shown splitting logic does not filter empty strings. Depending on the input stream, punctuation-only or malformed lines may therefore produce keys that a conventional word counter would exclude. Developers using the example as a starting point need to decide whether normalization, empty-token removal, Unicode handling, and counter overflow belong in their application. The framework supplies the state model, but it does not define those semantics.

Because the README excerpt omits the complete command-line parser setup and launch command, it would be inaccurate to invent a run command or port. Start with the repository's complete example directory, confirm the arguments expected by its source and sink parsers, and only then replace the demonstration logic with application-specific computations.

Wally's language and operational boundaries

Wally's clearest boundary is its primary Pony API. The repository also contains `examples/python/`, but the README does not explain whether Python has equivalent state, connector, topology, scaling, or deployment APIs. A team with an established Python or Go service should not assume that the existence of an example directory establishes feature parity. It would first need a documented binding or a separately maintained integration process.

The visible release history is another boundary. Releases `0.6.1`, `0.6.0`, and `0.5.4` were published on 2018-12-31, 2018-11-30, and 2018-10-31. The repository is not archived, and its last push was on 2026-09-03. Those facts show current repository access and recent code activity, but they do not establish release compatibility, deployment support, or a maintained package ecosystem.

A conventional service is a credible alternative when the workload needs one process, local state is small, and distribution is not worth the operational cost. Such a service can use a normal TCP server and an in-process database or data structure without implementing a distributed topology. Wally becomes more relevant when keyed in-memory state, multi-worker coordination, and live scale-out are central to the application.

It is also the wrong choice if exact delivery, replay, persistence, or failure semantics cannot be established from the documentation. The README calls the framework resilient, but it does not specify the recovery contract. Operators should not approve a production topology until they know what happens to source offsets, in-memory state, and TCP writes when workers or connections fail.

Apache 2.0 licensing and upgrade work for wally

Wallaroo wally uses the Apache License 2.0. The repository identifies that license, and the README links to the Apache 2.0 license text. Teams incorporating Wally should include that license in their dependency review. The contribution process has an additional condition: the README says pull requests are not accepted unless the contributor accepts a Contributor License Agreement linked as `cla`.

Upgrade work cannot be assessed from version numbers alone. The Dockerfile is tied to `WALLAROO_VERSION 0.6.1`, while installation depends on a version-specific `wallaroo-up.sh` script. The container also installs packages associated with its Ubuntu Xenial base. Moving to another release or host environment therefore requires checking the installer, runtime activation, source and sink behavior, examples, orchestration files, and container assumptions together.

The monorepo layout can widen that work. Connectors, orchestration, monitoring, testing, demos, documentation, and language examples live under the same repository. A change in deployment tooling may affect components that an application team initially considers unrelated. The repository contains `CHANGELOG.md`, `ROADMAP.md`, and `VERSION`, but their contents are not quoted in the README, so this article cannot claim a documented compatibility policy.

The first concrete upgrade check is alignment among the chosen Wally release, `wallaroo-up.sh`, the generated runtime, and the application API. For release `0.6.1`, the repository `Dockerfile` provides the documented starting environment. Any departure from that environment should be treated as a compatibility test rather than assumed support.

Editorial conclusion

Wallaroo wally fits teams that accept Pony and want a framework to coordinate keyed stream-processing state across multiple workers. It is a poor fit for a service built around Python or Go unless a maintained language binding is verified. Before adoption, confirm that `wallaroo-up.sh`, the generated runtime, the container dependencies, and the intended release still work together. The last push was on 2026-09-03, but the documented releases and installation material date from 2018, so deployment compatibility deserves direct verification.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. WallarooLabs/wally on GitHub