AutoMQ: Diskless Kafka on S3, and What That Means for Your Broker Bill

The problem AutoMQ targets: broker disks that hold data you would rather not own

In a conventional Kafka deployment, each broker owns a set of local disks, and a partition lives on the brokers that host its replicas. Scaling storage means adding brokers or attaching volumes, and cross-availability-zone replication between replicas shows up on the network bill. The README frames the project around exactly that: "Diskless Kafka on S3, Offering 10x Cost Savings and Scaling in Seconds." The cost claim is the vendor's, and the README links to its own benchmark and cloud bill comparison pages, so verify the multiplier against your own workload.

The audience is narrower than "anyone running Kafka." AutoMQ is for teams whose stream data is already a derived copy of something durable elsewhere, and who want broker count to track CPU and network rather than retained bytes. Log pipelines, change-data-capture fan-out and event buses fit. A system where the broker disk is the only copy of the data, with no upstream replay available, is a different conversation, and the README does not claim to solve it.

How the diskless design splits the brokers from the bytes

The repository layout tells most of the story. There is a top-level s3stream/ directory, and a storage/ directory alongside the familiar core/, metadata/, raft/, connect/ and streams/ modules inherited from the Kafka codebase. The naming points at the architecture: stream data is written to object storage through an S3 stream layer, while the Kafka protocol surface stays where clients expect it.

That split is what makes the scaling claim possible. A broker that does not own durable bytes can be replaced without copying partitions, because a new broker reads from the same object storage the old one wrote to. It also explains the project's framing around multi-AZ availability and the absence of cross-AZ traffic cost: replicas that would otherwise replicate bytes between zones are instead reading a shared object store. The trade-off is that every fetch path now depends on object storage latency and on the credential and endpoint configuration being correct. Single-digit millisecond latency is the claim in the repository description, so a reader should measure it against their own access pattern rather than assume it.

Metadata is a separate concern from stream data. The presence of metadata/ and raft/ at the top level suggests coordination state is still managed by the cluster rather than pushed to S3, which is consistent with how a Kafka-compatible system has to behave: consumer group state and partition leadership cannot live in a blob store with the same semantics.

Getting AutoMQ running when the README has no quickstart

The README has a local Quick Start meant only for evaluation, not production: it needs Docker 20.x or later, Docker Compose v2, at least 4 GB RAM allocated to Docker and ports 9092 and 9000 free, and it runs docker/docker-compose.yaml, a single node acting as controller and broker alongside MinIO for S3 storage. It also points to the documentation site and to an AutoMQ Playground account for a hosted trial, and the repository ships build and packaging paths. The Gradle wrapper is present at the root, so building from source is also possible. The repository also includes a container/ directory, a docker/ directory and a chart/ directory, which is where a container image and a Helm chart would come from; the Quick Start uses the automqinc/automq image, and the chart/ directory is the place to look for Helm details.

The repository pins a Java version in .java-version, and the bin/ directory at the root holds scripts. Configuration lives in config/, so read the files there before editing them.

Once a broker is up, the client side is unchanged, which is the point of the project. The examples/ directory at the root holds a README.md, a bin/ directory and a src/ directory, and that is where the repository puts runnable sample code. The Quick Start in the root README runs the Kafka producer performance test from inside the automq_net Docker network with bootstrap.servers=server1:9092; read examples/README.md before writing your own producer.

If a topic can be created and records accepted, the broker is speaking the Kafka protocol. What that does not prove is that records landed in object storage rather than a local volume; the Quick Start pairs the node with MinIO for S3 storage, so verify it by inspecting the bucket you configured.

Where the diskless model becomes the wrong choice

The first limitation is operational, not architectural. A diskless broker is only as available as its object storage and its credentials. If the S3 endpoint is unreachable or the IAM role expires, the failure is not a slow disk, it is a stalled fetch path across the cluster. The README's multi-AZ availability claim addresses zone failure; it does not address a regional object storage outage..

The second is the release cadence. The listed releases are 1.7.5-rc3 from 2026-11-02, 1.7.5-rc2 from 2026-10-28, and a nightly build from 2024-08-23. That is a young line with a release candidate in flight, and the repository shows no long-term support policy in the material here. A team that needs a version it can sit on for two years should read the release/ and docs/ directories before assuming one exists.

The third is scope. AutoMQ is not a drop-in answer for every Kafka workload. If your consumers rely on features the project has not carried over, or if your deployment is a single node on a laptop, the object storage dependency adds configuration and a network hop for no benefit. The safest way to know is to test your own client and connector set.

AutoMQ vs WarpStream and vs stock Kafka: three different storage bets

The comparison that matters is about where bytes live and who pays for moving them. Stock Apache Kafka keeps partition data on broker disks and replicates between brokers, often across zones. AutoMQ keeps the Kafka protocol and API but moves stream data to object storage through its s3stream layer, leaving brokers without durable local state. WarpStream, takes the same object storage premise but is written in Go rather than Java, which changes the operational footprint and the amount of the Kafka codebase you inherit.

That last difference is the practical one. AutoMQ's repository is a Java project with Kafka's module structure intact (core/, connect/, streams/, raft/, clients/), so existing JVM-side tooling and the Kafka client ecosystem are close at hand. A Go implementation makes a different trade: a smaller runtime to deploy, but a separate codebase from the one your Kafka-adjacent tools were written against. Neither choice is free, and the README does not benchmark AutoMQ against WarpStream, so the repository offers no data to settle it.

Licence, maintenance and the cost of upgrading

AutoMQ is Apache-2.0, and the repository carries LICENSE, NOTICE and NOTICE-binary files at the root. The NOTICE-binary file is the one to read if you redistribute the built artifacts, since it records third-party components bundled into the binaries. Apache-2.0 permits commercial use and modification; it also means the project carries no warranty, which is standard and worth remembering when the storage layer is your object store.

Maintenance is active by the only measure available here: the last push was on 2026-11-02, and the repository is not archived. That says the code is being touched, not that any given release is stable. Upgrading means tracking the 1.7.x line and reading the release notes in release/ for each step.. If you build from source, note that the pinned Java version in .java-version is part of your upgrade surface, not just your build environment.

There is an operational cost that does not appear in a licence file. Diskless brokers shift spend from provisioned EBS volumes to object storage requests and egress. The README advertises no cross-AZ traffic cost, but a high-throughput small-message workload can move the bill in the other direction. Model that before treating the cost claim as a budget line.

Editorial conclusion

Adopt AutoMQ if you already run Kafka clients and want broker storage off the critical path, you are comfortable operating on AWS object storage, and you can accept a Java project whose README Quick Start is explicitly for evaluation and not suitable for production. Do not adopt it if your workload is a single-node development cluster, if you need a long-term support release rather than the 1.7.x line, or if you cannot run the S3, IAM and network setup the diskless design assumes. Verify four things before committing: that your Kafka client versions and custom serializers work against an AutoMQ broker, that your object storage settings match your bucket and credentials, that the container image or Helm chart you intend to use is the one you tested, and that a broker restart under load leaves your consumer offsets where you expect them. The last one is the whole point of the design, so test it first.

Frequently asked questions

Official sources

1. AutoMQ/automq on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases