Brooklin moves data between systems that do not speak to each other

A general purpose streaming layer

Brooklin is described as a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale, and the design goal is multitenancy, so that one cluster can simultaneously power hundreds of data pipelines across different systems. The word general purpose is doing real work in that sentence, because a large part of the appeal is that a pipeline is not tied to a particular pair of systems. A stream can start in a relational database and end in a message bus, or in a key value store, and the same operational machinery, the same REST management, the same diagnostics, applies to all of them. The project describes itself as extensible, which means the set of supported systems is a matter of adding a connector rather than a matter of writing another pipeline.

Sources and destinations can be mixed

The first distinguishing feature is extensibility for any source and destination, and the elaboration matters. Brooklin offers a flexible API that can be extended to support a wide variety of source and destination systems, and it is not confined to a single type of source or destination system. The consequence stated in the documentation is that source and destination systems can be freely mixed and matched, and they do not have to be the same. That is the design that makes the Kafka mirroring use case a special case of a general capability rather than the product itself. It also means that the connector library, not the core, is where the engineering effort of using Brooklin for a new system goes, which is a trade a team should make consciously.

Scale comes with isolation

The second distinguishing feature is scalability, and it is described in three parts. Brooklin supports creating an arbitrary number of data streams that are processed concurrently and independently, such that errors in one stream are isolated from the rest. It supports partitioned data streams throughout its core implementation and APIs. And it can be deployed to a cluster of machines, which is described as scale out, to support as many data streams as desired. The isolation claim is the one that makes multitenancy safe. In a system where a hundred pipelines share a cluster, the difference between one poisoned stream and an outage is whether a failure in one stream can take the others down with it, and partitioning plus independent processing is how that is achieved. The Kafka use case states the same idea at a finer grain, since one bad partition will not affect an entire topic and mirroring continues for all the other healthy partitions.

Operating it over REST

The third distinguishing feature is that the system is meant to be operated, not just run. Brooklin exposes a REST endpoint for managing data streams, offering a rich set of operations on them in addition to CRUD, and the examples given are pause and resume. It also exposes a diagnostics REST endpoint that enables on-demand querying of the status of a data stream. That is a management surface rather than a deployment-and-forget one, and it is what a platform team needs when a hundred pipelines are running and someone has to find out why one of them stopped. The pause and resume operations are also what make a rolling operation possible, since a stream can be held while the cluster around it is reconfigured.

Kafka mirroring is the flagship

The use case section leads with mirroring Kafka clusters, and it is where the battle testing claim sits, with Brooklin providing capabilities for reading and writing massive amounts of data to and from Kafka with high reliability at scale. Four capabilities are listed. Multitenancy, where a single Brooklin cluster can be used to mirror data across several Kafka clusters. Fault isolation across topic partitions. Whitelisting topics using regular expressions, so the topics to mirror are selected by pattern against their names rather than by a hand maintained list. And pausing and resuming individual partitions through the Datastream Management Service, which the documentation describes as finer control over replication pipelines, with a link to the architecture page for the REST endpoints. The tutorial in the project wiki covers mirroring Kafka clusters in more detail, and the project also has step-by-step tutorials for running Brooklin locally in a few example scenarios.

Change data capture, and the bridge to stream processors

The second use case is change data capture. Brooklin supports propagating change data capture events from data stores, such as relational databases and key value stores, and it supports bootstrapping data from a datastore, which is streaming a snapshot of the existing data before any change events. The bootstrap is what separates a usable CDC implementation from an unusable one, because a consumer that starts receiving change events without the prior state has no way to make sense of them. MySQL support is stated to be currently under development, which is a fair signal about how complete the connector set is. The third use case is as a bridge to stream processing systems, where Brooklin is the underlying streaming infrastructure feeding systems such as Apache Samza, Apache Storm, Apache Spark and Apache Flink, which positions it as the transport layer under a processing engine rather than the engine itself.

The module layout and the release gap

The repository is a Gradle multi module Java project, and the module names describe the system. The datastream-server/ module holds the server, with datastream-server-api/ and datastream-server-restli/ providing the API and the REST layer, datastream-client/ the client, datastream-directory/ the directory of streams, and datastream-common/ and datastream-utils/ the shared pieces. The connectors are separate modules, with datastream-kafka-connector/, datastream-kafka-factory-impl/ and datastream-file-connector/ for the systems that ship today, and there are modules for testing, tools and a kafka facade. The build is build.gradle with a settings.gradle, the Gradle wrapper, a Vagrantfile for a local environment, checkstyle/ and findbugs/ directories for static analysis, a scripts/ directory, a config/ directory, and a HEADER file for the licence header, with a NOTICE file and a .pullapprove.yml for code review approval. The development story is the part worth flagging. The listed releases are 5.4.3 from 2023-10-31, 5.4.0 from 2023-08-15 and 5.1.0 from 2023-01-20, while the last push to the repository was on 2026-08-19, so the code is being worked on but no new tagged release has been published in a long time. The licence is BSD 2-Clause, copyright LinkedIn Corporation.

Editorial conclusion

Brooklin fits a platform team that has to move data between a database and a message bus, or between two Kafka clusters, and wants one system that handles hundreds of pipelines with per-stream isolation instead of a bespoke bridge per pair of systems. It does not fit a team that needs an active development line, because the newest release in the project is 5.4.3 from 2023-10-31 even though the last push was on 2026-08-19, and the MySQL change data capture support is described as under development. Before you adopt it, read the deployment tutorials in the project wiki, decide whether the connector set covers the systems you need, because the value of the system is in the connectors rather than in the core, and check whether the release you would install matches the Kafka client version your producers use.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-2-Clause
3. linkedin/brooklin on GitHub
4. README
5. Releases