SQLFlow: streaming pipelines where the transform is a DuckDB query over each batch

Three parts and one YAML file

SQLFlow is described as a stream processing engine that lets you define pipelines with just SQL. It consumes a stream, runs DuckDB SQL over each batch, and writes the result out, and the README characterises it as a lightweight, single-binary Flink.

The architecture has three parts. The source is Kafka, a WebSocket or webhooks, modelled as a stream of messages. The handler is DuckDB SQL executed over a batch of that stream, and it can filter, aggregate, enrich or drop data. The sink is where the SQL result goes, which is Kafka, ClickHouse, Iceberg, the console, or through the sqlcommand sink anywhere DuckDB can write.

A config file names all three, and it can also carry optional commands that run before the pipeline starts, such as attaching databases and creating tables, plus optional tables the engine manages across the pipeline's lifetime.

That shape is the whole appeal. A pipeline is a YAML file and a SQL query, with the batch semantics supplied by DuckDB rather than by a framework's own windowing API. The README's own summary of the operating model is one Go binary, one Docker image, one YAML file per pipeline, and no cluster.

The CLI has four commands. run runs a pipeline against a live source. serve serves a config's named SQL datasets over HTTP. validate checks a pipeline offline and reports every fault at once. dev invoke runs a pipeline's handler against a static file, which is the one that makes local development possible without a broker.

A five minute path that avoids a broker entirely

The quick start is unusually well thought out for a stream processor, because the second step does not need Kafka at all.

First you build the binary:

make sqlflow

Then you validate a pipeline against test data, without a broker. The dev invoke command runs the config's handler over a JSONL fixture and prints the result:

./bin/sqlflow dev invoke dev/config/examples/basic.agg.mem.yml dev/fixtures/simple.json

{"city":"New York","city_count":28672}
{"city":"Baltimore","city_count":28672}

That step is the important one for anyone evaluating the project. You can exercise your SQL, your config and your sink logic against a file, with no Kafka, no Docker and no cluster, and see the rows your query produces. The same command is the one the Docker image runs in its documentation example.

After that the quick start starts Kafka locally with docker-compose, publishes ten thousand test messages with a Python publisher driven by uv, watches the output with a Kafka console consumer, and finally runs the pipeline with a maximum message count:

./bin/sqlflow run -c dev/config/examples/basic.agg.mem.yml --max-msgs=10000

The consumer then prints one row per city, with the counts, which is the same aggregation the fixture produced in step two but computed over the live stream.

The libduckdb dependency decides your deployment

SQLFlow reaches DuckDB through the Arrow ADBC driver manager, which dlopens libduckdb at runtime. The README is explicit that the binary is not standalone and that wherever you run it, that shared library has to be present.

The environment variable SQLFLOW_DUCKDB_LIB points at the library. Without it, SQLFlow looks in /opt/homebrew/lib/libduckdb.dylib on macOS and /usr/local/lib/libduckdb.so on Linux. The pinned DuckDB version lives in one place, the DUCKDB_VERSION file.

There are three ways to satisfy that. The published Docker image bakes in a matching libduckdb.so and sets SQLFLOW_DUCKDB_LIB, so nothing else is needed, and the images are multi-arch for linux/amd64 and linux/arm64:

docker run --rm \
  -v $(pwd)/dev:/tmp/conf \
  turbolytics/sql-flow:latest \
  dev invoke /tmp/conf/config/examples/basic.agg.mem.yml /tmp/conf/fixtures/simple.json

Prebuilt release binaries are published for linux and macOS on amd64 and arm64, and the install sequence downloads the library, points the variable at it and checks the binary:

chmod +x sqlflow_<version>_<os>_<arch>
./scripts/install-libduckdb.sh /usr/local/lib
export SQLFLOW_DUCKDB_LIB=/usr/local/lib/libduckdb.so
./sqlflow_<version>_<os>_<arch> version

One warning matters here: scripts/install-libduckdb.sh always fetches the linux libduckdb.so, for whichever architecture it detects, and it is for linux hosts and containers only. On macOS, brew install duckdb puts the library at the default path and no environment variable is needed.

Building from source needs Go 1.26 or newer, a C toolchain because cgo is mandatory, and libduckdb.

What the sinks cover and what the use cases look like

The sink list is where the design shows its ambition. Alongside Kafka, ClickHouse, Iceberg and the console, the sqlcommand sink can write anywhere DuckDB can COPY to, which the README lists as PostgreSQL, S3, parquet, MotherDuck and DuckLake.

The use cases in the documentation are given as concrete configurations in the examples directory. Streaming data transformations clean data and types and publish the new data. Stream enrichment adds data to an input stream. Data aggregation reduces volume by aggregating input batches. Tumbling window aggregation buckets data into arbitrary time windows such as hour or 10 minutes.

Then there are the cases that make it more than a Kafka to ClickHouse pipe. Running SQL against the Bluesky Firehose executes SQL against any websocket source, with an example config for the Bluesky firehose. Streaming to an Iceberg catalog, streaming to ClickHouse, enriching streams with Postgres data by querying postgres during stream processing, and sinking Kafka to Postgres all have their own example configs.

The Postgres example is worth a second look, because it is the pattern that separates SQLFlow from a simple filter. If your handler can query Postgres, you can join a stream against a relational dimension table inside the batch, which is the kind of thing that otherwise forces you to denormalise everything into the event payload or maintain a separate lookup topic.

The Go dependency list explains the sink coverage. There is the ClickHouse Go driver, the Iceberg Go library, the pgx PostgreSQL driver, the Arrow ADBC driver manager, the franz-go Kafka client with its kmsg package, the Paho Eclipse MQTT client, the coder websocket library, gonja for templating, Prometheus and OpenTelemetry clients for metrics, cobra for the CLI, and YAML for configuration.

How the project is built and tested

The build is a multi-stage Dockerfile with a comment-heavy explanation of each decision, which is unusual and useful.

The builder runs on the build platform and cross-compiles for the target, so a multi-arch build never runs a compiler under emulation. The Go side is handled by Go itself and the cgo side is Arrow's C data interface and the ADBC driver manager, plain C and C++ that Debian's cross toolchain compiles. libduckdb is never linked, only dlopened, so the target's copy is downloaded as a file, and the runtime stage only copies, so nothing runs under emulation at all.

Two environment settings in the builder deserve attention. GOTOOLCHAIN is set to local, and the comment explains why in detail: under the automatic setting, a go.mod directive newer than the image silently fetched its own toolchain, so the shipped binary was built by a compiler the pin did not name, which is how go1.26.0 reached a release image pinned to golang:1.25. A directive the image cannot satisfy now fails the build, and the fix is to move GO_IMAGE. And GOFLAGS is set to -buildvcs=false because the source is copied in without its .git, and stamping VCS info would fail.

The Makefile pins the DuckDB version in one place, the file of the same name, read from there by the container build, the benchmark script and CI. The version is derived from git describe for the image tag, and the build info is stamped into the binary through linker flags.

The Python side is described as test and tooling support for the Go engine, not a second implementation. The pyproject.toml is explicit that the shipped artifact is the Go binary and that Python remains for the release suite, the coverage matrix generator and the Kafka fixtures those two share.

That coverage matrix is worth a look. The README links a coverage and invariant matrix document that says what is tested and, separately, what is proven, regenerated from the suites on every push, and the Makefile has a target that runs a generator module from the scripts directory to produce it.

Where SQLFlow is the wrong choice

The first limitation is the batch model. SQLFlow runs SQL over each batch of the stream, and the batching is not the same thing as a windowing engine with state. A tumbling window aggregation is available, and it buckets data into time windows, but the general question of maintaining state across an unbounded stream with checkpointing and exactly-once semantics is not what DuckDB's per-batch execution provides. If your pipeline needs a long window, or state that survives a restart, you are in a different class of tool.

The second is the DuckDB version coupling. Because libduckdb is dlopened at runtime and the version has to match, a deployment has two things to keep in step: the SQLFlow binary and the DuckDB library beside it. That is a smaller problem than a cluster and a much smaller one than a streaming platform, but it is a problem, and the fix of a single pinned version file only works if nothing else in your environment wants a different DuckDB.

The third is the operational envelope. There is no cluster, which is the point, but there is also no checkpointed recovery, no backpressure story beyond what the source provides, and no multi-node execution. The README gives a throughput figure of about 900k messages per second on a laptop in roughly a quarter GiB of memory, which is a useful scale anchor, but it is a single-process number.

The comparison with Flink or Kafka Streams is therefore the honest one. SQLFlow wins when your transform is expressible as SQL over a batch and you want no cluster to run. It loses when your pipeline needs durable state, exactly-once semantics, or a window that spans more than a batch. For a team that has outgrown a simple consumer transform but does not want to operate a stream processor, it is a genuine middle option, and the cost is that you are running a database engine as your stream runtime rather than a stream engine with SQL bolted on.

Editorial conclusion

SQLFlow fits a team that wants streaming aggregation and enrichment with SQL semantics and no cluster to operate, and that is comfortable with the DuckDB library being a runtime dependency rather than something bundled. It does not fit a workload that needs stateful windowing across restarts, or a team that wants a managed stream processor with a support contract. Before adopting it, read the libduckdb requirement carefully, because the binary dlopens a shared library at runtime and the version must match, then pin the DuckDB version through the DUCKDB_VERSION file, and check the coverage and invariant matrix to see which of the documented behaviour is actually tested.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. turbolytics/sql-flow on GitHub