Apache Iggy: A Rust Message Streaming Platform Built from the Ground Up

What Apache Iggy Is and Why It Was Built from Scratch

Most message streaming systems at scale either extend an existing message queue or add a streaming layer on top of a database engine. Apache Iggy takes a different approach: the README explicitly states it is 'not yet another extension running on top of existing infrastructure, such as Kafka or SQL database.' It is built from the ground up using low-level I/O with a thread-per-core shared-nothing architecture, io_uring for Linux kernel-level async I/O, and the compio crate for maximum efficiency. The README explains that io_uring and compiled Rust with no garbage collector together produce predictable resource usage and low latency. The name is an abbreviation for the Italian Greyhound, described in the README as a small but extremely fast dog. Apache Iggy is a Linux Foundation project under the Apache Software Foundation umbrella, as indicated by the ASF_LICENSE.txt and NOTICE files in the repository.

Transport Protocols, Multi-Tenancy, and Consumer Groups

The README lists four supported transport protocols: QUIC, WebSocket, TCP with a custom binary specification, and HTTP with a REST API. This is a meaningful architectural choice: QUIC provides encrypted, multiplexed connections without the head-of-line blocking that affects TCP, while the HTTP REST API allows integration with any language or tool that can make HTTP calls without a dedicated SDK. The README describes TLS support across all four transports. Iggy organises data into streams, which group topics, providing multi-tenant isolation: different applications or teams can use the same Iggy server while keeping their data in separate streams. Topics divide further into partitions, enabling horizontal scaling through consumer groups. Consumer groups enforce message ordering and distribute messages across connected clients. Polling supports reading by offset, by timestamp, by first or last N messages, or by next N messages for a specific consumer, with optional auto-commit of offsets.

Running Iggy with Docker

The repository includes a docker-compose.yml that builds and runs the server:

docker-compose up -d

The compose file maps four ports: 3000 for HTTP, 8080 for QUIC, 8090 for TCP, and 8092 for WebSocket. It sets the advertised address via `IGGY_NODE_ADVERTISED_ADDRESS=localhost` and uses a named volume for persistent storage. The Dockerfile performs a two-stage build: the first stage builds the server binary and the web UI in a Rust builder image, and the second stage copies the binaries into a slim Debian runtime image. The environment variables `IGGY_HTTP_ADDRESS`, `IGGY_QUIC_ADDRESS`, `IGGY_TCP_ADDRESS`, and `IGGY_WEBSOCKET_ADDRESS` control which interfaces each transport binds to.

The CLI can be installed separately via Cargo:

cargo install iggy-cli

This installs the `iggy` binary for managing streams, topics, partitions, and consumers from the command line without requiring the full server build.

SDKs, Connectors, and the Kafka Gateway

The README lists published client SDKs in Rust (on crates.io), Python (on PyPI), Java (on Maven Central), .NET (on NuGet), and JavaScript (on npm). This coverage means most teams can consume and produce messages without implementing the binary protocol directly. The connector system uses custom Rust plugins and supports a wide range of sinks: ClickHouse, Delta Lake, Elasticsearch, InfluxDB, MongoDB, PostgreSQL, Redshift, S3, and others visible in the Cargo.toml workspace member list. Connectors run with OpenTelemetry logs and traces in the connectors runtime. The Kafka gateway allows Kafka producers and consumers to connect to Iggy without code changes, which enables incremental migration from a Kafka-based system. The README also mentions an MCP server for providing context to language models, located in the core/ai/mcp workspace member. The server supports optional data encryption using AES-256-GCM on both the server side and the client side. Optional archiving to disk or S3-compatible cloud storage is listed as a feature in the README. User authentication and authorization with granular permissions and Personal Access Tokens is included in the server, giving teams a way to isolate access across streams without running separate broker instances.

Key Differences from Apache Kafka

The most direct comparison for Apache Iggy is Apache Kafka. Kafka is a mature, widely deployed platform with a large ecosystem of connectors, managed cloud offerings from all major providers, and years of production use at scale. Iggy is a single Rust binary with no external dependency on ZooKeeper or a controller quorum process. Kafka's storage model uses a partitioned commit log and relies on the broker coordinating replicas via a metadata service; Iggy's thread-per-core shared-nothing design means each CPU core handles its own workload without locking. Kafka does not natively support QUIC or WebSocket as first-class transports; Iggy does. The trade-off is ecosystem maturity: Kafka has far more third-party connectors, managed offerings, and documented operational patterns than Iggy has today. Teams migrating from Kafka can use the Iggy Kafka gateway as a compatibility layer, but it is a shim, not a drop-in replacement for Kafka's full API surface.

Limitations and Features Not Yet Available

The README is direct about two notable gaps. Server-side message compression is not yet implemented: the README states that topic compression values are reserved for future disk and network compression support, and that users who need compression today should apply it manually using message headers. OpenTelemetry export from the main server is also listed as unavailable, pending a runtime integration, though it is available for the connectors runtime. These are concrete constraints to verify before depending on either feature in a production deployment. The README also notes that processing guarantees depend on the application's ordering of message processing and offset commits when using optional poll auto-commit, which means exactly-once semantics require care at the application level rather than being a built-in guarantee.

Licence, Clustering, and Maintenance

Apache Iggy is licensed under Apache-2.0. The repository last pushed on 2026-09-27, with server-0.9.0 released on 2026-09-18. The README includes a section on clustering, which the Apache Iggy documentation site at iggy.apache.org covers in detail. The repository lists benchmarking tools (core/bench and its dashboard) as workspace members, allowing performance testing of specific configurations before deployment. The Docker Compose file includes `SYS_NICE` capability and `seccomp:unconfined` security options, which the README implies are needed for optimal io_uring performance on Linux. Teams running in environments that restrict these capabilities should verify compatibility before deploying. The repository structure is a large Cargo workspace with members spanning the core server, CLI, binary protocol, configuration, connector SDK, multiple sink implementations, and the benchmarking dashboard. The gateways/ directory at the root contains the Kafka gateway, and the core/ai/mcp directory contains the Model Context Protocol server. The repository also includes a Helm chart under helm/ for Kubernetes deployments. Example code is available in csharp/, go/, java/, node/, php/, python/, and rust/ under the examples/ directory, covering producer and consumer patterns in each supported language. The project uses the Apache mailing list for governance discussions, with the list address documented at iggy.apache.org/community/mailing-lists/.

Editorial conclusion

Apache Iggy suits teams building new streaming infrastructure who want a self-contained binary with no external broker dependency, fine-grained control over transport protocols, and the option to write consumers and producers in Rust, Python, Java, .NET, JavaScript, or Go. Teams whose existing pipelines depend on Kafka producer and consumer APIs can use the Kafka gateway to migrate incrementally, but the gateway is a compatibility shim, not a full Kafka replacement. Before adopting, note that server-side message compression is listed as not yet supported in the README, and OpenTelemetry export from the main server is pending a runtime integration.

Frequently asked questions

Official sources

1. apache/iggy on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases