lovoo/goka: a Kafka stream processing library in Go that keeps its state in Kafka

What Goka solves for a Go service that already runs Kafka

A Go service consuming Kafka normally has to answer three questions on its own: which partitions do I own, where does my per-key state live, and what happens to that state when this process dies. Goka answers all three by extending the consumer group concept. It binds a state table to the group and persists that table in Kafka, so the partitioning of the input topics and the partitioning of the state stay aligned.

The README describes the target as reducing the complexity of building "highly scalable and highly available microservices." That framing is accurate about where the work goes. You write a callback that receives a key and a deserialized value, read the current state with ctx.Value(), write a new one with ctx.SetValue(), and Goka handles assignment, rebalancing, recovery and the local cache. The audience is a team that already operates Kafka and writes Go, and that would otherwise be building the same partition-assignment and state-recovery plumbing for the third time.

Processors, group tables and views: the data flow

The core object is a processor group. You define it with goka.DefineGroup, declaring its inputs, its outputs and its persistence codec. One or more instances of the processor form the group, and Goka distributes the partitions of the input topics across those instances. A message arriving on a partition is handled by exactly the instance that owns it.

State lives in a group table, described in the README as "a partitioned key-value table stored in Kafka that belongs to a single processor group." Because both the input partitions and the table partitions are assigned together, a callback for key K always reads and writes the table partition that holds K. When an instance fails, the remaining members take over its table partitions and recover them from Kafka.

Two other objects complete the picture. Emitters push key-value messages into Kafka without being part of a group, which is how an external service, say a database handler, feeds state changes into the pipeline. Views are read-only local caches of an entire group table, useful for serving external requests, for example over gRPC.

Local storage keeps a copy of the table partitions on disk to speed recovery and cut memory use. The default is LevelDB, with an in-memory map and a Redis-backed implementation also available in the repository.

Installing Goka and running a counter processor

The README gives a single install command. It fetches the module and its dependencies into your Go module cache.

go get -u github.com/lovoo/goka

Goka does not talk to Kafka directly. It relies on Sarama, so the first thing to configure is the Kafka version, otherwise you inherit whatever default the library ships with. The README shows the global replacement pattern.

cfg := goka.DefaultConfig()
cfg.Version = sarama.V2_4_0_0
goka.ReplaceGlobalConfig(cfg)

If different components need different settings, you pass a builder to the constructor instead. The README demonstrates this with goka.WithConsumerGroupBuilder and goka.ConsumerGroupBuilderWithConfig.

The README's own example is a counter. An emitter writes the value "some-value" under the key "some-key" into the example-stream topic every second, and a processor increments a per-key counter held in the example-group-table topic. The processor side is short: inside the callback, ctx.Value() returns the stored counter for the message's key, ctx.SetValue() writes the incremented value back, and the group is declared as an Input on the stream plus a Persist with an Int64 codec.

The examples directory ships numbered examples from 1-simplest through 10-visit, along with a docker-compose.yml for Zookeeper and Kafka and a Makefile. The README says to run make start from the examples folder to bring those local containers up. After the emitter and processor are both running, the processor logs a line per message showing the key, the counter and the message.

At-least-once delivery and the state you cannot put in a table

The README states plainly that messages are delivered with at-least-once semantics. That is the limitation to internalize before adopting Goka, because it shapes what a callback is allowed to do. A callback can run more than once for the same message, so any side effect it triggers, a payment call, an email, an insert into an external system, needs its own idempotency key. The group table update itself is safe to repeat, since incrementing a counter twice under the same key is not, and that is exactly the kind of bug at-least-once semantics produce.

There is a second boundary. Goka's state model is a key-value table, one value per key, optionally with a TTL. It is not a query engine. If you need joins across two streams, windowed aggregations, or a declarative topology with a scheduler, Goka gives you callbacks and a table and expects you to build the rest. The copartition strategy file in the repository suggests the library does reason about co-partitioned topics, but the README does not document a join operator.

Finally, Goka assumes Kafka is present and healthy. There is no embedded broker and no fallback. If your workload is a batch job over files, or a stream that arrives over HTTP, Goka is the wrong shape entirely.

Kafka Streams as the alternative, and the real difference

The obvious comparison is Kafka Streams. Both put processor state in Kafka topics and both partition that state alongside the input. The difference is the programming model and the runtime. Kafka Streams is a Java library that runs inside your JVM process and gives you a DSL with operators like map, filter, join and window, plus a topology you can inspect before it starts. Goka is a Go library that gives you a callback per message and a table, and leaves the composition to your own code.

That has a practical consequence for a Go shop. Adopting Kafka Streams means either running a JVM service alongside your Go services or rewriting the service in Java. Adopting Goka means adding one entry to go.mod. The repository's go.mod pins github.com/IBM/sarama v1.46.3 and github.com/syndtr/goleveldb v1.0.0, so the dependency surface is small and inspectable.

The trade is expressiveness. Kafka Streams will compute a windowed join for you; with Goka you would emit to an intermediate topic and maintain the second table yourself. If your processing is genuinely a topology of joins and windows, the Java library earns its runtime cost. If it is per-key state plus a decision, Goka is the shorter path.

Maintenance, releases and the BSD-3-Clause licence

The repository is not archived, and the last push was on 2026-09-24. Releases are infrequent rather than constant: v1.1.17 on 2026-08-06, and before that v1.1.16 and v1.1.15, both on 2026-02-06. The version numbers stay in the 1.1.x line, which tells you the maintainers are comfortable with the API surface as it stands and are shipping fixes rather than redesigns. The repository also carries a MIGRATION.md file, which is where you would look before moving between major versions.

Upgrade cost is mostly the Sarama version. Because Goka delegates all Kafka communication to Sarama, a Kafka feature you need may be blocked until Goka bumps that dependency, or until you pass a custom builder with your own config. The README points at the Sarama Config documentation for the full set of knobs, and at a wiki page on configuring log compaction for table topics. That wiki page matters: if the table topics are not compacted, the group table grows without bound.

Licensing is BSD-3-Clause, a permissive licence that permits use in closed-source products provided the copyright notice and disclaimer are retained. The repository also depends on LevelDB under a BSD-style licence and on Redis client code under gopkg.in/redis.v5. This is a description of the licence text, not legal advice; your own counsel should confirm obligations for your distribution model.

Editorial conclusion

Adopt Goka if you are already running Kafka, your services are written in Go, and you want partitioned state to live in the same log system as your messages rather than in a separate database. Do not adopt it if you need exactly-once processing, if your team is not on Go, or if you want a SQL-like DSL instead of callback functions. Before committing, verify three things: that the table topics backing your group are created with log compaction, that your Kafka version is set explicitly on the Sarama config, and that at-least-once redelivery is acceptable for every side effect your callbacks perform.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. lovoo/goka on GitHub
4. README
5. Releases