Estuary Flow: capture, materialize, derive on one streaming substrate

Three verbs, and the file-plus-stream trick underneath

Estuary describes itself as the Right-Time Data Platform that integrates all of the systems you use to produce, process and consume data, unifying today's batch and streaming paradigms so that your current and future systems are synchronized around the same datasets, updating in milliseconds. The primary language of the repository is Rust.

The three operations are the whole model. Capture pulls data from your systems, services and SaaS into collections, which are millisecond-latency datasets stored as regular files of JSON data in your own cloud storage bucket. Materialize turns a collection into a view within another system, such as a database, key/value store, Webhook API, or pub/sub service. And Derive creates new collections by transforming from other collections, using the full gamut of stateful stream workflow, joins and aggregations, in real time.

What makes those three fit together is explained in the How does it work section. Estuary builds on a real-time streaming broker created by the same founding team called Gazette, and because of that the collections are both a batch dataset, stored as a structured data lake of general-purpose files in cloud storage, and a stream, able to commit new documents and forward them to readers within milliseconds. New use cases read directly from cloud storage for high-scale backfills of history and switch to low-latency streaming on reaching the present.

That dual nature is the design idea worth understanding. A backfill reading history and an incremental update reading the present are not two code paths, they are the same collection read two ways, and the topics attached to the repository spell out the vocabulary: change-data-capture, elt, etl, data-pipeline, data-integration.

The performance techniques, and the mixed language stack

The README lists the architectural techniques it uses to get throughput without adding latency, and each one is a specific engineering decision rather than a slogan.

Optimistic pipelining uses the natural back-pressure of the systems to which data is committed. Reduce annotations group collection documents by key wherever possible, in memory, before writing them out. Derivation states, the registers, are co-located with derivation compute, living in an embedded RocksDB that is replicated for durability and machine re-assignment, updating in memory and writing out only at transaction boundaries. Work done in external Remote Procedure Calls and even process-internal operations is vectorized.

And then there is the language choice, which is the most unusual thing in the repository. The approach is described as marrying the development velocity of Go with the raw performance of Rust, using a zero-copy CGO service channel. And the repository layout confirms both are real: there is a Cargo.toml workspace with a crates/ directory for the Rust side, and a go.mod module github.com/estuary/flow for the Go side, alongside a go/ directory.

The Go module dependency list shows where the Go code sits. It pulls in the Google Cloud storage client, grpc and grpc-gateway, the Prometheus client, an etcd client, a RocksDB binding, go-sqlite3, logrus, and go.gazette.dev/core, which is the Gazette broker library itself.

The Rust workspace is a 2024 edition workspace requiring rust-version 1.95, with the licence field set to BSL, and its dependency list is long: axum for the HTTP layer, async-stripe for billing, async-compression, apache-avro for the Avro format, and the usual async and serde stack. The Cargo manifest even carries comments explaining why particular features are enabled and why others are stubbed out, which is the kind of detail a maintained dependency list accumulates.

The UI, the CLI, and where the configuration lives

Estuary gives you two interfaces over the same platform, and the README is clear that both are needed.

The low-code web application for essential workflows is at dashboard.estuary.dev, and the flowctl CLI gives fine-grain control over pipelines. Together they comprise the unified platform, and the point made is that you can switch between them as you build and refine pipelines and collaborate with a wider breadth of data stakeholders.

The configuration is a YAML file. The examples directory shows it plainly, with a flow.yaml at the top level of the examples, a flow_generated/ directory beside it, and a separate derive-patterns/ directory. There is a flow.schema.json at the repository root, which is the schema for those files.

The examples are worth looking at as a catalogue of what the platform can express. There is an acmebank/ example, citi-bike/, stock-stats/, temp-sensors/, and a reduction-types/ directory, and the README notes that many examples cover derivations, reductions and other flow.yaml examples. The reduction-types directory is the interesting one, because reductions are the mechanism behind the reduce annotations that group documents by key, and having a directory of them suggests the shapes matter.

The docs are not in this repository, which the README states plainly. The connector reference pages are in the estuary/connectors repository with pull requests welcome, and the other pages are in the estuary/docs repo, which is internal to the Estuary team, so a documentation problem is reported as an issue in this repository instead.

The tutorial list points at practical flows: a basic dataflow, PostgreSQL CDC streaming to Snowflake, and real-time CDC with MongoDB.

Connectors, and the Airbyte compatibility path

Captures and materializations use connectors, plug-able components that integrate Flow with external data systems, and the in-house connectors focus on high-scale technology systems and change data capture, which the README glosses as databases, pub-sub and filestores. They are published as GitHub packages under the orgs/estuary/packages repository, and there is a separate estuary/connectors repository for the rest.

The compatibility path is the interesting part. Estuary can run Airbyte community connectors using airbyte-to-flow, and that is what allows a greater variety of SaaS systems to be supported than the in-house set would cover on its own.

So there are two tiers of source, and the distinction has consequences for what you can do. A high-scale system with a change data capture interface, a database or a pub/sub bus, is served by a first-party connector written for the platform. A SaaS application is more likely to be served by an Airbyte connector adapted to the platform, which gives you coverage but at the cost of a translation layer.

The full list of currently supported connectors lives on the website rather than in the repository, and if what you need is not there the README points at a request form in the connectors repository, with a query parameter that pre-fills the capture from or materialize to side and the favourite system name. That template tells you how the project thinks about connectors, as a direction of travel rather than a generic component.

The topics carry the change data capture tag, and the tutorials include a PostgreSQL CDC to Snowflake example, which is the canonical shape of a problem the platform is built for.

The BSL, and the deployment choices

The licensing deserves a paragraph of its own because it is unusual for a project of this shape and it is the first thing to read.

The Cargo workspace manifest sets the package licence to BSL, and the repository root carries both a LICENSE-BSL and a LICENSE-APACHE, with the README saying to see the BSL license for information on using Estuary outside the managed offering. The licence reported for the repository is one GitHub cannot classify, so the files are the authority and the BSL is the one that governs the platform.

The deployment choices are the other half. You can publish your data flows in Estuary's shared SaaS environment, or you can use a private deployment or a BYOC deployment for enterprise-ready security, and the documentation links both. A free account is available through the sign-up link, and the dashboard is at dashboard.estuary.dev.

The distinction between a shared SaaS and a BYOC deployment is the practical one, because it decides whether your data flows through someone else's environment. A private deployment keeps the runtime in your own network, and BYOC means you bring your own cloud, which is the model a regulated data team is likely to need.

The project also has the usual support structure: a Slack community, an email address, a contact page, and a documentation comparison reference for people evaluating it against other data platforms.

Editorial conclusion

Adopt Estuary Flow when your batch and streaming pipelines are two systems doing one job, because the same collection is both a file in your bucket and a live stream, so a backfill and an incremental update are the same object. Do not adopt it expecting a permissively licensed library, because the workspace manifest declares the BSL licence and the repository carries both LICENSE-BSL and LICENSE-APACHE, and the README points you at the BSL for using Estuary outside the managed offering. Before you build, decide between the shared SaaS environment, a private deployment and a BYOC deployment, because that choice determines whether you run the software yourself or buy a managed service, and check the connector list for the systems you actually need.

Frequently asked questions

Official sources

1. estuary/flow on GitHub
2. Issues
3. Project website
4. README
5. Releases