# Streamz builds pipelines where every node is a stream you can fork, join and rate-limit

> Streamz is a Python library for building pipelines that manage continuous streams of data, offering simple use in simple cases and branching, joining, flow control, feedback and back pressure in complex ones, with optional Pandas and cuDF dataframe support. It fits a data scientist who wants live pipelines over DataFrames, and it does not fit a batch pipeline or a project that has standardised on another stream library.

**python-streamz/streamz** — Real-time stream processing for python

- Repository: https://github.com/python-streamz/streamz
- Website: https://streamz.readthedocs.io/en/latest/
- Stars: 1,306 · Forks: 151
- Language: Python
- License: BSD-3-Clause
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/python-streamz-streamz

## One abstraction: a stream of data you can attach operations to

Streamz helps you build pipelines to manage continuous streams of data. The README states the design ambition plainly: it is simple to use in simple cases, but also supports complex pipelines that involve branching, joining, flow control, feedback, back pressure, and so on.

The abstraction underneath is small enough to state in a sentence. A stream is a sequence of elements that something else can subscribe to, and every operation you attach returns another stream. That is the whole idea, and it is why the same object works as a source, a transformation and a sink.

The consequence is that a pipeline is a graph rather than a chain. A node with two subscribers fans out, and a node with several inputs is a join. Flow control and back pressure come from the same mechanism, because a slow consumer applies back pressure upstream rather than being buffered without limit. Feedback loops are possible too, which is the feature that lets you build a stream whose output feeds its own input, at the cost of being able to hang your program.

This is a different shape from the batch model most Python data work uses. In a batch pipeline you write a function that takes a collection and returns a collection, and the framework decides when to run it. In a stream pipeline you write the same function and it runs per element, and the framework is responsible for the scheduling, the buffering and the back pressure. The library is maintained by Matthew Rocklin, who is also associated with Dask, and the same problem-solving shows up in the design.

It is BSD-3 licensed, at version 0.6.6, and the last push was on 2026-09-22.

## The dataframe integration is the reason most people arrive

Streamz can optionally work with both Pandas and cuDF dataframes, to provide sensible streaming operations on continuous tabular data. That is the practical entry point for most users, because Pandas is already the data structure in their code and the question they are trying to answer is how to keep a rolling computation over it live.

The cuDF support is the more interesting half, because cuDF is the GPU dataframe library from the RAPIDS stack, and the README carries a badge for the RAPIDS custreamz gpuCI. The cudf repository includes a custreamz directory with a kafka.py, and the badge in the README links straight to it. So the GPU path is tested in CI by the RAPIDS project rather than only by Streamz itself.

That has an architectural consequence worth understanding. cuDF dataframes are in GPU memory, and moving data between the CPU and the GPU for every operation is expensive enough to matter. A stream of cuDF-backed elements therefore cannot be treated as a stream of small Python objects, and the operational characteristics of the two backends differ more than the shared API suggests.

The repository has a streamz.dataframe package, and the packages list in setup.py is streamz plus streamz.dataframe, with the test packages derived from those. The examples directory includes examples/dataframes.ipynb for the Pandas path.

The three runtime dependencies are tornado, toolz and zict. toolz provides the functional helpers the operations are built from, and zict provides the buffer abstraction that lets a stream hold recent elements with a memory-based or disk-based spillover, which is how a stream with a window keeps bounded state.

## The examples show the three concurrency models the library runs under

The examples directory is the best available map of what Streamz is for, and three files stand out because they run the same problem under different concurrency models.

examples/fib_asyncio.py, examples/fib_thread.py and examples/fib_tornado.py all compute something Fibonacci-shaped, and the difference is the execution model. One runs the stream on asyncio, one on threads, and one on Tornado. There is also examples/fibonacci.ipynb for the notebook version.

That trio is a deliberate statement about the library's scope. A stream pipeline has to run somewhere, and the same pipeline code can be driven by an event loop, by a thread pool, or inside an async web framework. A library that only worked on asyncio would be unusable inside a synchronous Django view or a blocking data pipeline, and a library that only worked on threads could not be driven from an async coroutine.

The other examples show the range of use cases. examples/network_wordcount.py counts words over a network stream, examples/scrape.py scrapes, examples/plot_graph.py plots the graph of a pipeline, and examples/river_kmeans.py and river_kmeans.ipynb do online clustering over a stream, which is the clearest statement of what streaming buys you over batch. A batch implementation of k-means over a growing collection has to refit; a streaming one updates.

examples/map_failure_modes.py is the one worth reading before writing production code, because it enumerates what happens when a map operation raises. A stream where one element kills the pipeline is a common design, and this example is the documentation of that behaviour.

The examples also include examples/iterators-and-streams.ipynb, which is where the difference between a plain Python iterator and a Streamz stream is worked out, and examples/mystream.png, a rendered picture of a pipeline.

## Back pressure and flow control are the parts that need thought

The feature list of the README names branching, joining, flow control, feedback, back pressure and so on, and those are the parts of a stream library that are not automatic.

Back pressure is the mechanism that keeps a slow consumer from being drowned. When a downstream node cannot keep up, the signal to pause propagates upstream, and the upstream source stops producing. That is a different failure mode from unbounded buffering, where a slow consumer causes memory to grow until the process dies, and the choice between them is a design decision rather than a library setting.

Flow control is the knob on top of that. A rate limit on a source means the pipeline deliberately produces slower than the data arrives, which is how you keep a downstream database from being the bottleneck that decides the throughput.

Feedback is the feature that lets a stream's output feed its own input, and it is the one to be careful with. A feedback loop with any gain can run away, and because the stream is lazy, the runaway shows up as a growing queue rather than an exception. A pipeline that loops needs a termination condition, and the library will not supply one.

Windows are the other thing that needs thought, because a rolling computation over a stream has to keep state, and the zict dependency is what allows that state to spill to disk when it exceeds memory. A stream that keeps unbounded state is a memory leak with a nicer syntax, and the size of the window has to be chosen deliberately.

## The build environment is a Kafka and conda container, which tells you about testing

The Dockerfile is more informative than it first appears, because it is the test environment rather than a production image.

It starts from python:3.14-slim and installs wget and vim. It then downloads and unpacks Apache Kafka, with the Scala and Kafka versions as build arguments, 2.13 and 4.2.0, and exposes port 9092. Kafka is in the image because the Kafka source needs a broker to test against, and the kafka.py in the custreamz directory is a real part of the ecosystem.

Then it installs Miniconda from the Anaconda repository, accepts the conda terms of service for the main and r channels, updates conda, creates an environment named by the STREAMZ_ENV argument, and installs optional dependencies from conda-forge including jupyterlab, numpy and pandas. The streamz source is copied in and built from source.

The choice of Python 3.14 in the image and the presence of a py314 environment file show the library is being kept current with new interpreter releases, and the conda route is a consequence of the RAPIDS dependency, since the cuDF stack is distributed through conda-forge rather than pip.

The Python floor is 3.10, declared as python_requires in setup.py, and the setup.py reads the install requirements directly out of requirements.txt, splitting the file on newlines. The package list is streamz and streamz.dataframe, with the matching test packages, and the maintainer is Matthew Rocklin.

The licence file is LICENSE.txt with a BSD-3 Clause notice in the README, and the project also has a Dockerfile, a docker/ directory, a ci/ directory, a conftest.py at the root, a .codecov.yml and a .coveragerc, and documentation built from the docs/ directory and published to readthedocs.

## Where Streamz is the wrong tool

The first wrong-tool case is batch processing. Streamz is built for data that arrives continuously. If your pipeline runs over a finite historical dataset, a batch framework is the right shape, and using a stream library for it means reimplementing chunking, ordering and parallelism guarantees that a batch engine already gives you.

The second is an ecosystem mismatch. There are several stream libraries in Python, and a team that has already standardised on one, or on the reactive extensions of a web framework, should not adopt a second abstraction for the sake of one feature. The cost is not the dependency, it is that engineers now reason about two models of data flow in the same codebase.

The third is the data volume shape. A stream library processes element by element, which is efficient for high-rate low-latency data and inefficient for a few large batches. Operations that need to see the whole dataset, a global sort, a full-window aggregate over a large collection, are awkward in a stream, and the streaming implementations of those operations are often the slowest part of the pipeline.

The fourth is failure behaviour, and the example file map_failure_modes.py is the evidence. An exception raised inside a map operation propagates through the pipeline, and whether the stream continues, dies, or retries is a decision you make. A library that turns a single bad record into a stopped pipeline is fine for a monitoring stream and wrong for one where dropping a record is unacceptable, and you have to know which you are building before you rely on it.

## Conclusion

Streamz fits a data scientist or researcher who needs a live pipeline that reacts to arriving data rather than a batch job, particularly one already working with Pandas or cuDF, and it does not fit a team that processes historical data in batches or that has standardised on another streaming library. Before adopting it, check the examples directory for a pipeline shape close to yours, because the library's fit varies a lot between the simple case and the complex one, and note that it is BSD-3 licensed at version 0.6.6 with the last push on 2026-09-22.

## FAQ

### What is Streamz used for?

Streamz helps you build pipelines to manage continuous streams of data. It is simple to use in simple cases, and supports complex pipelines involving branching, joining, flow control, feedback and back pressure. It can also work with Pandas and cuDF dataframes for streaming operations on continuous tabular data.

### How do I install Streamz?

The package is published on PyPI as streamz, at version 0.6.6, and requires Python 3.10 or newer. Its runtime dependencies are tornado, toolz and zict. The repository also contains a Dockerfile that sets up a full test environment with Apache Kafka and a conda environment.

### Does Streamz support GPU dataframes?

Yes. Streamz can work with cuDF, the GPU dataframe library from the RAPIDS stack, as well as Pandas. The README carries a RAPIDS custreamz gpuCI badge linking to the cudf repository's custreamz directory, which includes a kafka.py, so the GPU path is covered in CI by the RAPIDS project.

### Can Streamz pipelines run under different concurrency models?

Yes. The examples directory contains fib_asyncio.py, fib_thread.py and fib_tornado.py, which run the same kind of computation under asyncio, threads and Tornado respectively, plus a fibonacci.ipynb for notebook use.

## Sources

- [Issues](https://github.com/python-streamz/streamz/issues)
- [License: BSD-3-Clause](https://github.com/python-streamz/streamz/blob/master/LICENSE)
- [Project website](https://streamz.readthedocs.io/en/latest/)
- [python-streamz/streamz on GitHub](https://github.com/python-streamz/streamz)
- [README](https://github.com/python-streamz/streamz/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/python-streamz-streamz