supabase/etl: a Postgres replication engine that drops the broker

Two phases, one process, and a store that remembers

Every published table travels the same route. A publication feeds an initial sync that copies the rows already in the table, then catches up whatever changed while that copy was still running. Once the table is ready, ongoing replication captures inserts, updates, deletes and truncates and delivers them to the destination as ordered events.

Copy and change data capture sit underneath as two paths rather than two customer-visible stages. Initial sync uses the copy path followed by CDC catch-up; ongoing replication uses the CDC path on its own.

A store persists checkpoints, schemas, destination metadata and table state, which is what lets the engine restart without beginning again from zero. Delivery is at-least-once, so after a crash a destination can see the same change twice. Nothing else has to run: no Kafka, no Flink, no Debezium, no coordination service kept alive beside it.

Publications and logical WAL decide what moves

Filtering happens in the database, not in the destination. Postgres publications select tables, columns, rows and operation types, and the engine reads those publications directly, so a pipeline that only wants two columns or only deletes does not need its own filter logic downstream.

Two gates sit in front of any run. The source server must set `wal_level = logical`, and the replication user needs the `REPLICATION` role. Neither is optional, and neither can be worked around from the ETL side.

Supported versions run from PostgreSQL 14 through 18, but that floor rises twice for specific paths. PostgreSQL 15 or newer is recommended for column and row publication filters, and PostgreSQL 16 or newer is required the moment the replication connection points at a physical read replica. A team moving replication onto a standby has to check the standby's major version before writing any pipeline config at all.

Adding the engine to a Cargo build from Git

The crate is not on crates.io yet, so the dependency points at the Git repository. Two entries are enough to start:

[dependencies]
etl = { git = "https://github.com/supabase/etl" }
tokio = { version = "1", features = ["full"] }

Read the version fields before you build anything. The workspace declares `rust-version = "1.95.0"` and `edition = "2024"`, and a `rust-toolchain.toml` at the root pins the compiler used for development. An older toolchain will fail on the workspace before it reaches your code.

[workspace.package]
license = "Apache-2.0"
edition = "2024"
rust-version = "1.95.0"

One upgrade hazard is visible in the dependency line itself: nothing pins a commit, so a fresh resolve takes whatever the default branch holds at that moment. A lockfile check-in is the cheapest guard here.

Tests need environment variables, and the repository ships a template for them. Copy it, then load it into the shell before running cargo test:

cp .env.example .env
source .env

Local ClickHouse through the bundled xtask helper

ClickHouse is named the fastest destination to start locally, and two xtask invocations set it up:

cargo x init
cargo x setup replicator

The first starts ClickHouse at `http://localhost:8123` with `user etl / etl`, and the second points the replicator at it by default. The template environment file spells the same values out, alongside the two Postgres servers the replication tests use:

export TESTS_DATABASE_PORT=5430
export TESTS_DATABASE_REPLICA_PORT=6430
export TESTS_CLICKHOUSE_URL=http://localhost:8123
export TESTS_CLICKHOUSE_USER=etl

Note those Postgres ports. Tests bind the primary to 5430 and the physical replica to 6430 rather than to the port a stock Postgres install claims first, which matters before you point a local server at the harness.

Spelling of the helper is inconsistent across files. The README uses `cargo x init`, while the benchmark instructions in the environment template use `cargo xtask benchmark --destination clickhouse`, both for the workspace member `crates/xtask`. Copy the spelling from the file in front of you. A wrong alias name fails before any container starts.

Benchmarks run against the same container, which the template starts separately:

docker compose -f scripts/docker-compose.yaml up -d clickhouse

BigQuery work is a separate path with separate credentials. Those tests are skipped by default and run with:

cargo test -p etl-destinations --features bigquery -- --ignored

The service account behind them needs the BigQuery Data Editor and BigQuery Job User roles.

One destination is Stable while the project warns about churn

Five destinations ship in the repository, and the maturity column is not uniform. `bigquery` is marked Stable. `clickhouse`, `ducklake` and `snowflake` are marked Private alpha. `iceberg` is marked Deprecated. Private alpha is a statement about managed-service access only, and it does not restrict local or self-hosted use of the open-source destinations.

That table sits next to a note saying the project is under active development and that APIs and setup steps may change before anything is declared stable. A framework that has not reached a stable version already ships one connector labelled Stable and another labelled Deprecated, so the per-destination label is the only status worth reading. Treat engine maturity and connector maturity as two separate questions.

For a cloud target, BigQuery is named the most mature option in the destination set. For a first run on a laptop, ClickHouse is named the fastest to start, which is why its xtask path exists. DuckLake and Snowflake carry the same Private alpha label as ClickHouse without getting the same first-run tooling, and their individual limits live in the destinations reference page rather than in the repository files.

The benchmark used a custom destination, not BigQuery

The headline comparison comes from an independent CDC benchmark run by Hardbyte, with Debezium Server 3.1.3.Final on the other side: 26 ms against 848 ms of tail latency, and 13 MB against 1,629 MB of measured runtime memory, at 200 source changes per second across four fast HTTP consumers. Those are the figures behind the claims of about 33 times lower latency and about 125 times smaller memory.

The configuration is the part worth reading. The engine build tested was `d206d66e` from July 2026, running a custom `HttpDestination` with `MemoryStore`. Neither is one of the five shipped destinations, and a memory store discards on disk exactly the durable state that the checkpoint design depends on. The repository itself notes that versions, polling, batching, destination, store, workload and deployment all affect the outcome, which means the ratio describes that pairing and cannot be read as a statement about ETL pointed at BigQuery or Snowflake.

The alternative worth naming is Debezium, and the difference is about what has to run alongside it. ETL keeps the pipeline in one process behind typed Rust traits for destinations and stores. Debezium appears in these pages as a server distribution measured under a configuration the repository does not document, next to Kafka and Flink as services ETL claims not to require. A team already running Kafka Connect is not the audience for this engine.

Ten workspace members and a pinned toolchain

The workspace lists ten members, running from the framework crate through `etl-config`, `etl-destinations`, `etl-postgres`, `etl-replicator` and `etl-telemetry`, plus benchmarks, examples, `etl-maintenance` and `xtask`. Destination targets are cargo features of `etl-destinations` rather than separate packages, since the BigQuery tests switch one on with `--features bigquery` instead of naming a different crate.

Correctness lints are set to `deny` instead of warn, and the list shows what the project refuses to compile: `await_holding_lock`, `large_futures`, `clone_on_ref_ptr` and `unused_async` are all rejected outright. In an engine that carries table state across await points during catch-up, those are the four that would matter. Dependency policy has its own file in `deny.toml`, the graph is committed as `dependencies.dot`, and development environments are described through `flake.nix`.

Two maintenance facts matter for adoption. The workspace is Apache-2.0, so a team embedding the crates inside a closed-source service carries the notice and licence terms with what it ships. The last push landed on 2026-09-28, and there is no GitHub release history to check a version against, even though `cliff.toml` sits at the root of the tree waiting to turn conventional commits into versioned notes.

Where the replicator is the wrong choice

Several situations rule it out. A physical read replica path needs PostgreSQL 16 or newer while the engine supports 14 through 18, so a mixed-version estate cannot point every pipeline at it. An Iceberg pipeline has no forward path, since that destination is marked Deprecated. Recovery is at-least-once, which means the destination has to absorb a repeated change, and the README does not document rollback. Store implementations are pluggable, so persistence behaviour varies with the store you choose.

If the goal is a managed pipeline rather than a process you operate yourself, the same README redirects you elsewhere: availability, pricing and operations for the hosted product live in the Supabase Pipelines documentation, not in this repository. That is the real alternative, and the difference is who carries the upgrade work rather than which engine is faster.

Before committing, verify the source server version against the specific gate you will hit rather than the general 14 to 18 range, confirm the replication role exists on the user you plan to use, and pin the git dependency to a commit you have actually read.

Editorial conclusion

Adopt supabase/etl if your team already writes Rust and controls the source publication, because sync, CDC, state and delivery stay inside one process behind typed destination and store traits. Leave it alone if you need a JVM toolchain, if your target is the Iceberg connector marked Deprecated, or if nobody can set wal_level = logical. First check that your server falls between PostgreSQL 14 and 18, then pin the git dependency to a commit you have read, since nothing in that snippet carries a rev.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. supabase/etl on GitHub