Gazette: one append log that is both a live stream and a folder of S3 files

A journal is simultaneously a stream and a data lake folder

Gazette's central abstraction is the journal, which the README describes as a streaming append log represented using regular files in a BLOB store such as S3. That single representation choice is the whole design. Because a journal is written as an append log and periodically stored as immutable, organized files in cloud storage, the same bytes can be consumed as a low-latency stream by a running service and read later by a SQL engine or a batch job as ordinary files in a data lake.

For teams that already keep data in object storage, the appeal is that the stream and the historical archive stop being two separate pipelines with two separate copies of the truth. The README states that Gazette has served production use cases for nearly five years, with deployments scaled to millions of streamed records per second. That figure comes from the project itself, so treat it as a claim about their deployments rather than an independent measurement.

The intended users are platform engineers, application teams and analysts who want to work from a common catalog of data in whichever way suits them, rather than a single prescribed processing paradigm.

The broker service and the consumers framework in the Go package layout

Two pieces sit on top of the journal. The first is a journal broker service, and the second is a consumers framework for building streaming applications in Go.

The top-level repository layout gives a workable map of that structure. Directories named broker/, consumer/, allocator/, keyspace/, labels/, message/, server/ and task/ appear alongside testing helpers brokertest/, consumertest/ and etcdtest/, with cmd/ holding the command line entry points and examples/ holding runnable programs. The Makefile generates protobuf sources with gogo/protobuf and grpc-gateway, and go.mod lists both google.golang.org/grpc and grpc-ecosystem/grpc-gateway, so the broker speaks gRPC and also exposes a gateway rather than a bespoke wire protocol.

The practical consequence is that a Go service can subscribe to a journal, receive records, and process them inside the same process that owns the consumer logic, instead of running a separate stream client library.

Getting the broker running from the Dockerfile in the repository

The README gives no install command. Its Where to Start section points to the official documentation at gazette.dev and the API Godocs at godoc.org/go.gazette.dev/core, so a first run starts from those pages rather than from a copy-and-paste line. The repository itself is a Go module named go.gazette.dev/core, and the module file pins the language version:

module go.gazette.dev/core
go 1.26.0
toolchain go1.26.9

Building from source means producing two binaries, the broker and the command line client. The Dockerfile in the repository root shows exactly which artifacts the image expects:

FROM ubuntu:24.04 AS broker
ARG TARGETARCH

and then it drops privileges for the running process:

RUN useradd gazette --create-home --shell /usr/sbin/nologin
USER gazette
WORKDIR /home/gazette

The broker stage is deliberately thin, installing only ca-certificates and curl on top of ubuntu:24.04, so object storage and etcd access rely on client libraries compiled into the binary. Protobuf generation is a separate step, and the Makefile builds its own plugins first:

${WORKDIR}/protoc-gen-gogo:
	go mod download github.com/golang/protobuf
	go build -o $@ github.com/gogo/protobuf/protoc-gen-gogo

That target assumes protoc is already present on the machine, because the rule invokes it directly.

The examples that ship in the tree: bike-share, word-count and stream-sum

The repository ships runnable programs under examples/, and they are the fastest way to judge whether the journal model fits your data. The directory listing names examples/bike-share/, examples/word-count/ and examples/stream-sum/, plus a standalone script examples/batch_download.py and a shell helper examples/generate-zonemap-gke.sh.

The Dockerfile gives the examples their own stage and names the shipped binaries explicitly:

COPY \
     ${TARGETARCH}/bike-share \
     ${TARGETARCH}/chunker \
     ${TARGETARCH}/counter \
     ${TARGETARCH}/gazctl \
     ${TARGETARCH}/integration.test \
     ${TARGETARCH}/summer \
     ${TARGETARCH}/wordcountctl \
     /usr/local/bin/

Two details are worth noticing. The wordcountctl, counter and summer binaries correspond to the word-count and stream-sum example programs, so the counters are part of the shipped surface rather than test fixtures. The examples stage also installs librocksdb8.9 and libsqlite3-0, which the broker stage does not, suggesting the examples exercise local storage engines beyond what the broker itself needs. A zone map generator script running under GKE suggests that journal files carry per-range metadata, which is what lets a SQL engine skip file regions it does not need instead of scanning everything.

What go.mod reveals about the operating dependencies

The module file is the clearest available description of what the broker actually depends on, and the list is long. Storage clients for Google Cloud Storage v1.57.0, Azure Blob at github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.2 and the AWS SDK at github.com/aws/aws-sdk-go v1.55.8 all appear, which matches the README's BLOB store framing and lets one deployment point at whichever provider you already run.

Coordination goes through etcd. Both go.etcd.io/etcd/api/v3 and go.etcd.io/etcd/client/v3 sit at v3.6.5, and the repository carries an etcdtest/ package, so etcd belongs to the runtime environment rather than being an optional extra. Local state and encoding come from github.com/jgraettinger/gorocksdb, github.com/jgraettinger/urkel and github.com/jgraettinger/cockroach-encoding, and sqlite3 shows up as github.com/mattn/go-sqlite3. Metrics come from github.com/prometheus/client_golang v1.23.2, and github.com/soheilhy/cmux is present for multiplexing. A kustomize/ directory sits at the top level, which by convention holds the Kubernetes manifests for that layout.

Every one of these is a compile-time dependency, so the upgrade surface of a Gazette deployment is effectively the surface of this module list.

Where the journal model costs more than it saves

Two constraints matter when sizing a deployment. First, the journal is stored in object storage, so the write path touches a remote BLOB store instead of a local append file, and any reader that skips the live stream pays object storage latency and egress costs.

Second, the consumers framework targets Go. A polyglot organization ends up running Go services to feed non-Go consumers over the journal, rather than sharing one client library across languages.

Documentation gaps are worth pricing in. The README does not document journal compaction thresholds, retention windows or a rollback procedure, so operational defaults have to be confirmed against gazette.dev rather than assumed. Record format evolution is not described there either, and while a CHANGELOG.rst file exists at the repository root, release notes are the only place a format change is likely to surface.

Tooling adds friction as well. Building protobuf sources needs protoc installed locally, and the Go floor is unusually high, since go.mod declares go 1.26.0 with toolchain go1.26.9. An older corporate Go toolchain cannot build the project as it stands, which is a concrete migration cost for teams still on a pinned, slower-moving release.

A recently pushed repository with slower moving release tags

The repository is not archived, and the last push was on 2026-10-21, which is recent enough to say the project is receiving changes. Release tags move on a slower clock: the recorded releases are v0.103.0 on 2026-03-23, v0.102.0 on 2026-02-17 and v0.101.0 on 2025-10-08.

That gap between a recent push and a March release tag matters for operators, because the code on the default branch can sit well ahead of the last tagged build. Anyone who wants a pinned, reproducible deployment should build from a tag rather than from master.

The licence is MIT, with a LICENSE file at the repository root. That is permissive for commercial use and modification, and the obligation that survives adoption is attribution and preservation of the licence notice. As always, the interaction between that notice and the rest of your stack is a question for your own counsel rather than for this article.

Editorial conclusion

Teams that already run object storage and want one catalog to serve streaming and analytical reads, and that can commit to Go for their consumer services, should evaluate Gazette against their own latency and retention needs. Teams with polyglot consumer fleets, an older Go toolchain, or a requirement for sub-millisecond ingestion should look elsewhere. Before adopting, check three things: whether the journal layout and compaction defaults in the documentation match your retention rules, whether etcd and your chosen BLOB provider are acceptable operating dependencies, and how much of the unreleased work since the v0.103.0 tag of 2026-03-23 you would be inheriting, given the last push was on 2026-10-21.

Frequently asked questions

Official sources

1. gazette/core on GitHub
2. License: MIT
3. Project website
4. README
5. Releases