ArkFlow: A Rust Stream Engine That Runs SQL and Python UDFs Over Kafka, MQTT and SQL Sources

The gap ArkFlow fills between a message broker and a stream processor

A broker moves bytes. It does not filter, reshape, join, or decode them. The usual response is to put a stream processor in front of the broker, and the usual stream processors want a cluster, a JVM, and a state backend you have to operate. ArkFlow takes the other position: one Rust binary, one YAML file, and a list of streams. Each stream declares an input, a pipeline, an output, and an error output. That is the whole mental model, and it is visible in the quick start config, which wires a synthetic generator into a JSON-to-Arrow conversion and a SQL filter before writing to stdout. The target reader is an engineer who already runs Kafka or NATS and wants a transform stage that starts with `./target/release/arkflow --config config.yaml` rather than a deployment topology. The README also frames the product around AI inference on streaming data, listing anomaly detection and complex event processing alongside the ordinary plumbing. That framing is broader than the quick start demonstrates; the quick start is a filter, not a model call.

How a stream is assembled: input, pipeline, output, error_output, buffer

The architecture is a per-stream directed graph expressed in YAML. A stream object has five slots. `input` produces records. `pipeline` holds a `thread_num` value and an ordered list of processors. `output` consumes the result. `error_output` accepts any output component and receives messages that failed processing, which is the mechanism that keeps a bad record from stalling the stream. `buffer` handles backpressure and temporary storage. The pipeline is where the interesting work happens. Processors run in the order written, and the README's example chains `json_to_arrow` before `sql`, which tells you the SQL processor operates on Arrow tables rather than raw JSON. The query string is plain SQL over a table named `flow`, as in `SELECT * FROM flow WHERE value >= 10`. That ordering constraint matters: if you put `sql` before a conversion step, the engine has nothing tabular to query. Parallelism is configured per stream through `thread_num`, set to 4 in the example. The README does not state whether processor order is preserved across threads within a single stream, so if your pipeline depends on strict sequential semantics, that is something to confirm in the docs rather than assume.

Delivery guarantees and the per-stream write-ahead log

The feature list states that ArkFlow is at-least-once by default through per-stream WAL durability, with optional exactly-once for transactional sinks. This is the most consequential design decision in the project, and it deserves more scrutiny than the one-line description gives it. At-least-once means a record can be delivered more than once after a restart or a failure, so any downstream sink that is not idempotent needs a deduplication key. The README does not name which sinks qualify as transactional, and it does not describe the WAL's retention, flush interval, or recovery procedure. Those are exactly the parameters an operator needs before putting this in front of a payments or inventory system. The buffer components are the other half of the durability story. The README lists a memory buffer for high-throughput scenarios and window aggregation, and a session window that groups messages by activity gaps and closes after a configurable period of inactivity. The list is visibly truncated in the README at "Sliding Wi", so the full set of window components has to be read from the docs site rather than the repository front page. Treat the buffer catalogue as incomplete until you check.

Getting it running: clone, build, and a config that actually exercises the pipeline

The installation path is source-only in the supplied material. There is no published crate or container image mentioned. The commands are `git clone https://github.com/arkflow-rs/arkflow.git`, then `cd arkflow`, then `cargo build --release`, with `cargo test` available for the test suite. The binary lands at `./target/release/arkflow` and takes `--config config.yaml`. A minimal config that exercises the real path looks like the quick start: a `logging` block with `level: info`, then a `streams` list containing one stream. Inside that stream, the input uses `type: "generate"` with a `context` string holding a JSON literal, an `interval` of `1s`, and a `batch_size` of 10. The pipeline sets `thread_num: 4` and chains two processors, `json_to_arrow` and `sql` with a `query`. The output and error_output are both `type: "stdout"`. Swapping in a real source means replacing the input block; the Kafka example in the README uses `brokers`, `topics`, `consumer_group`, `client_id`, and `start_from_latest`. The Kafka output example nests a `topic` object with `type: value` and an inner `value` object, which is a template-style indirection the README does not explain in prose. Copy that structure literally rather than simplifying it.

The connector surface, and where it is thinner than the feature list suggests

Inputs cover Kafka, MQTT, HTTP, files (CSV, JSON, Parquet, Avro, Arrow, with cloud storage support), a synthetic generator, SQL databases (MySQL, PostgreSQL, SQLite), NATS with JetStream, Pulsar, Redis streams, lists and pub/sub, WebSocket, Modbus, an in-memory source for testing, and a multiple-input mode that merges several streams into one pipeline. Outputs cover Kafka, MQTT, HTTP, InfluxDB 2.x, NATS, Pulsar, Redis, SQL databases with batch inserts and UPSERT, stdout, and a drop sink. That is a wide list for a project at version 0.5.0. The processing side is where the README is least specific. It names JSON processing, SQL, Protobuf, batch processing, VRL, and Python user-defined functions over a batch. For VRL it links out to Vector's documentation rather than describing the integration, which is a reasonable choice but means the semantics are inherited, not owned. For the Python processor, the README gives no example of a UDF signature, no statement about which Python version is embedded, and no note on whether it is compiled in by default or gated behind a Cargo feature. Given that Python embedding in Rust normally pulls in a runtime dependency and complicates cross-compilation, assume it is optional and verify before you plan around it.

Where ArkFlow is the wrong tool

The clearest failure mode is a pipeline that needs exactly-once into a sink the README does not classify as transactional. The documentation says exactly-once is optional and available for transactional sinks, without listing them, so the safe assumption for any unlisted sink is at-least-once plus your own idempotency handling. A second limitation is operational. The README advertises an optional Hub and web console to observe, configure, and operate multiple ArkFlow compute nodes as a fleet, but nothing in the supplied material describes enrollment, authentication, or how configuration is pushed to nodes. If fleet management is the reason you are looking at ArkFlow, you are buying an unverified capability. A third is the version cadence. The releases listed are v0.3.1 in May 2025, v0.4.0-rc1 in June 2025, and v0.5.0 in October 2025. A release candidate appearing between two stable versions suggests the project is still settling its interfaces, and YAML schema churn between minor versions is a real cost when your configs live in a repository. Finally, if you need a connector that is not on the list, you are writing it yourself in Rust against the component traits, which is a different kind of commitment than adding a plugin to an established processor.

The honest alternative: Vector, and the actual difference in approach

Vector is the obvious comparison, and the README makes it for us by linking to Vector's VRL documentation for the VRL processor. The difference is in the centre of gravity. Vector is built around observability pipelines: collecting logs and metrics, transforming them with VRL, and shipping them to sinks. Its transform language is VRL, a purpose-built expression language. ArkFlow puts a SQL engine over Arrow tables at the centre, with VRL as one processor among several. If your transform is naturally a SELECT with a WHERE clause over structured records, ArkFlow's model is a shorter path than writing VRL remap rules. If your transform is log parsing and enrichment, VRL is the more direct tool and Vector has the larger operational history behind it. The second axis is the runtime. Both are Rust, so neither asks for a JVM, but Vector's distribution model is a packaged binary with a long release history, while ArkFlow as described here is built from source with `cargo build --release`. The third axis is AI. ArkFlow's stated purpose includes loading and executing machine learning models on streaming data. If that is your requirement, Vector is not in the same category, and you should read ArkFlow's model-loading documentation directly because the README only asserts the capability.

Licence, maintenance cost, and what to check before you commit

ArkFlow is Apache-2.0, which permits commercial use and modification and includes a patent grant. That is the permissive end of the spectrum and it does not impose copyleft obligations on your own code. This is a description of the licence identifier, not legal advice; if you are redistributing a modified binary or embedding ArkFlow in a product, have counsel read the LICENSE file. The maintenance cost that matters here is not the licence, it is the build. Source-only installation means your CI needs a Rust toolchain and the build time that comes with it. If you enable the Python processor, expect that dependency to affect build and packaging. Upgrades are the other recurring cost: a project that shipped a release candidate between stable releases will change YAML schema, and your config files are the thing that breaks. Pin the version you build, keep configs in version control, and diff them against the docs before each bump. The first thing to verify is the buffer and window component list, since the README truncates it mid-word. The second is whether the Python UDF processor needs a Cargo feature flag. The third is which sinks the project considers transactional for exactly-once. All three are answerable from the docs site at arkflow-rs.com.

Editorial conclusion

Adopt ArkFlow if you are comfortable compiling Rust binaries, your sources are Kafka, NATS, MQTT, Redis, Pulsar or SQL databases, and you want SQL or Python transforms declared in YAML rather than a JVM cluster. Do not adopt it if you need a broad, long-stable connector catalogue or a hosted control plane, or if Python UDFs are load-bearing for you. Before committing, verify three things in the repository: whether the Python UDF processor is behind a Cargo feature flag, the exact buffer and window component list in the docs, and whether your sink is one of the transactional ones that the README says can use exactly-once. The README also promises a Hub and web console, but nothing in the supplied material shows how a fleet is enrolled or upgraded, so treat that piece as unverified until you read the docs.

Official sources

1. arkflow-rs/arkflow on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases