Pathway runs your Python ETL on a Rust engine, and the free tier only promises at least once

Python code, Rust engine, and a wheel assembled by maturin

You write a pipeline in Python and a Rust engine runs it. The engine is not a wrapper around your code: the Cargo manifest names the library pathway_engine, builds it as a cdylib and a lib, and pulls Differential Dataflow from a path inside the repository at external/differential-dataflow rather than from crates.io. That is where the incremental computation lives, and it is why the same pipeline definition can serve a batch job, a stream, and a stream replay.

The Python distribution is not pure Python. pyproject.toml declares the build backend as maturin and requires maturin >= 1.4:

[build-system]
requires = ["maturin >= 1.4"]
build-backend = "maturin"

So `pip install -U pathway` puts a compiled artifact into your environment, and if no wheel matches your Python, your platform or your architecture you are compiling the engine yourself. That build is pinned to a recent toolchain:

rust-version = "1.99"
differential-dataflow = { path = "./external/differential-dataflow" }

The crate is also marked publish = false, so you are never going to resolve the engine as a normal Rust dependency; it is consumed as a Python package or built from this tree.

The free build gives at least once, the enterprise build gives exactly once

This is the line that decides whether pathway fits your pipeline, and it is stated in one sentence in the feature list. The free version gives at least once consistency. The enterprise version gives exactly once.

The mechanism behind that sentence is that pathway handles time for you: late and out-of-order points are managed by updating the results whenever new data arrives, so a corrected event does not need a separate batch job to patch yesterday's output. What the free tier does not do is suppress the duplicate. On a pipeline that bills customers, increments a counter, or writes into a table with a unique key, at least once means your consumer has to be idempotent, and that work moves from the framework into your code.

Two other parts of the feature list matter here: the Airbyte connector, which is the route to more than 300 data sources, and the LLM xpack, which is documented on its own page and holds the wrappers, parsers, embedders, splitters, the in-memory real-time vector index and the LlamaIndex and LangChain integrations.

The pipeline stays in memory while only the computation state is persisted

Two statements in the README pull in opposite directions, and the combination sets your sizing. The deployment note says the whole pipeline is kept in memory and is easy to run under Docker and Kubernetes. The persistence feature says it saves the state of the computation so you can restart a pipeline after an update or a crash.

Read together, they mean the rows live in the memory of the machine running the pipeline, and persistence lets a restart resume from saved computation state. That is a good trade when a crash should resume instead of replay from the beginning, and a bad one when the working set is larger than the node.

The dependency list shows where the engine reaches outside itself. The Rust side embeds duckdb with the bundled and json features, deltalake with datafusion and s3, apache-avro, aws-sdk-dynamodb, aws-sdk-kinesis and async-nats. In other words, the same pipeline that updates incrementally can also read and write files, tables and object storage, but the incremental part is the part that holds your data in memory.

pyarrow is capped below 26 and deltalake below 2, so the resolver decides your environment

The base dependencies of the Python package are not small, and several of them carry an upper bound rather than a floor:

"pyarrow >= 10.0.0, < 26.0.0; python_version < '3.15'",
"deltalake >= 1.6.0, < 2.0.0",
"google-cloud-bigquery >= 3.29.0, < 4.0.0",
"beartype >= 0.14.0, < 0.23.0; python_version < '3.15'",

The same list also brings fastapi, uvicorn, sqlalchemy, sqlmodel, panel, jupyter_bokeh, scikit-learn, shapely, h3, geopy, python-sat, networkx, pydantic, the OpenTelemetry API, SDK and gRPC exporter, google-cloud-pubsub, google-api-python-client and diskcache. Optional extras add more: the pyfilesystem extra carries setuptools < 82.0.0.

What that means in practice is that adding pathway to an existing analytics environment is a resolver negotiation, not a clean install. If your team already pins pyarrow, deltalake or the BigQuery client to a version outside those ranges, pip will either refuse or move your pins, and the extra dependency set is installed for everyone whether or not you use Google Cloud, plotting or the FastAPI surface. The cap on pyarrow is the one most likely to collide, because pyarrow is exactly the package a data platform pins hard.

pip install -U pathway, then a template in a notebook or a container

The install is one line, and pathway asks for Python 3.10 or above, which pyproject.toml confirms with requires-python = ">=3.10":

$ pip install -U pathway

For a first run, the project does not ask you to scaffold a project. It sends you to a template gallery where the examples come in notebook and docker formats and launch from the browser. The event-processing ones cover a Kafka ETL job, log monitoring with alerting, and linear regression on a Kafka stream. The LLM ones cover unstructured data turned into SQL on the fly, a private RAG stack on Ollama with Mistral AI, adaptive RAG, and multimodal RAG on gpt-4o.

That choice has a consequence for how you evaluate the framework. A hosted notebook is the fastest way to see a connector and a transformation work, and the slowest way to find out what the dependency set does to your own environment, because the notebook has its own. The repository itself carries the code: examples/ holds notebooks/, templates/ and projects/, next to its own LICENSE and CODE_OF_CONDUCT.md and a slide deck named Pathway-PyData_Global_2022.pdf. Reading from there costs you an afternoon and tells you more about the current API than a notebook does.

Any Python function is allowed in the middle of a Rust transformation

Pathway ships stateful transformations, joins, windowing and sorting, many of them implemented directly in Rust, and then it says you can use any Python function instead, your own or one from any library. That openness is the reason it slots into an existing ML stack. It is also the place where the performance story gets complicated.

The engine is described as multithreaded, multiprocessing and distributed, and a Python function in the middle of that pipeline is the step that runs in the interpreter. A UDF that calls scikit-learn or a pandas expression puts your workload back onto the object the framework was built to get you out of. Treat the multithreading claim as a property of the Rust transformations, and count your Python steps separately before you promise throughput.

The same split shows up in connectors. Kafka, GDrive, PostgreSQL and SharePoint have native connectors, and the Airbyte connector opens up more than 300 sources. If neither has what you need, the README points you at the Pathway Python connector. So a source can be native, Airbyte-backed or yours.

The engine is BUSL-1.1 while the repository page reads NOASSERTION

The licensing picture needs two files and a caveat. The Cargo manifest for the engine declares license = "BUSL-1.1", the Python project classifies itself as License :: Other/Proprietary License, and the repository carries LICENSE.txt, which the README links from its License section. GitHub's own license field for the repository reads NOASSERTION, so the repository page will not tell you which terms apply to the part you want to ship. Read LICENSE.txt and decide for yourself; this is not legal advice.

Versions tell a similar story. The Cargo manifest is at 0.33.0, the newest published release is v0.32.1 from 2026-08-01, and before that come v0.31.1 from 2026-06-12 and v0.31.0 from 2026-05-25. The project has not reached 1.0, the two tiers of consistency are tied to the same commercial boundary, and the default branch is main with a last push on 2026-10-30. Pin an exact version for anything you depend on, and expect the engine and the Python package to move at their own pace between those tags.

Editorial conclusion

Take it when your pipeline has to keep updating itself while you keep writing it in Python, and accept duplicate delivery or budget for the enterprise tier. Do not take it for a batch job that a single query would finish, or if you need permissive licensing without reading LICENSE.txt first. Check three things before you commit: that a wheel exists for your Python and platform, that pyarrow below 26 and deltalake below 2 resolve against what you already run, and that BUSL-1.1 in the Cargo manifest matches your legal reading of the licence.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes