# Bifrost: MySQL binlog replication to Redis, ClickHouse, Kafka and nine more targets

> Bifrost acts as a MySQL replica to read binlog events, then fans those events out to up to twelve different target stores in parallel. DDL sync works for nine of those targets but not for Redis, MongoDB or Memcache.

**brokercap/Bifrost** — Bifrost is a production-oriented heterogeneous middleware that syncs MySQL, MariaDB and Kafka data to Redis, MongoDB, ClickHouse, StarRocks, Doris, Kafka and other services.

- Repository: https://github.com/brokercap/Bifrost
- Website: https://www.xbifrost.com
- Stars: 2,092 · Forks: 341
- Language: Go
- License: Apache-2.0
- Published: 2026-10-09 · Updated: 2026-10-09 · Language: en
- Canonical page: https://hysenlabs.com/projects/brokercap-bifrost

## Bifrost registers as a MySQL replica and reads binlog events directly

Bifrost connects to MySQL using the replication protocol, registers itself as a replica, and receives the binlog stream in sequence. It does not poll the database, so it places no additional load on the query path. Every INSERT, UPDATE, DELETE and schema change logged by the primary arrives at Bifrost as a binlog event rather than through a scan or trigger.

The project targets production environments, with full sync and incremental sync both available. Full sync copies the current table state first. Incremental sync via binlog follows all subsequent changes from the recorded position. After a shutdown or crash, Bifrost recovers both its configuration and that binlog position automatically, resuming from where it stopped.

MariaDB and Percona both work as sources alongside MySQL, using the same binlog-plus-select mechanism. MySQL versions 5.1.73, 5.5.42, 5.6.36, 5.7.18 and 8.0.21 are listed as tested in the README. Building from source requires Go 1.21.8 at minimum, as declared in go.mod. The binlog position is the critical piece: starting from the wrong offset either replays events already applied or skips changes that never arrived at the targets.

## Kafka as a source: four message formats and incremental-only delivery

Kafka is a source type in Bifrost, not only a target. It arrives at Bifrost in four distinct message formats, each handled by a different ingestion path. A plain string message is stored as a queue_data field without transformation. The canal format is converted automatically to Bifrost's internal representation. Bifrost's own format is used when Bifrost itself produced those Kafka messages, which enables a Bifrost-to-Kafka-to-Bifrost relay path. The debezium format covers cases where a debezium instance writes to Kafka and Bifrost consumes that stream.

MongoDB is a fifth source, reading from the oplog rather than a binlog. Full sync is not available for MongoDB or for any of the four Kafka source paths; all five of those combinations are incremental only. MySQL, MariaDB and Percona are the three sources that support both full and incremental modes.

A sixth type, Mock, exists specifically to support plugin development. It generates MySQL binlog events without any real database connection, so a new output plugin can be built and tested without standing up a source instance. This reduces the friction during plugin development, since the full source setup is not required to iterate on the output side.

## Twelve output targets, and three of them receive no DDL changes

The target list has twelve entries: Redis, MongoDB, MySQL, ClickHouse, Memcache, RabbitMQ, Kafka, ActiveMQ, a custom HTTP service, an Hprose RPC service, StarRocks and Doris. StarRocks and Doris are both included, connecting to Bifrost via the MySQL wire protocol rather than a dedicated connector.

DDL synchronization is not uniform across those twelve targets. MySQL, ClickHouse, RabbitMQ, Kafka, ActiveMQ, the HTTP plugin, the Hprose RPC plugin, StarRocks and Doris all receive DDL changes. Redis, MongoDB and Memcache receive row changes but no schema events. A CREATE TABLE or ALTER TABLE on the MySQL primary propagates to nine targets and silently skips three. For pipelines where those three are the primary stores and schema changes are frequent, a separate process is needed to keep them in sync after each DDL event.

Kafka output supports three message formats on the output side: the canal format, the debezium format and a plain Kafka message string. That makes it possible to serve both older consumers expecting canal and newer consumers expecting debezium from a single MySQL source, without an extra translation layer between them.

## Binary, Docker and source-build paths to the port 21036 web interface

Binary installation starts with downloading a release archive, extracting it and marking the binaries executable:

```bash
wget https://github.com/brokercap/Bifrost/releases/download/v2.3.12-beta/bifrost_v2.3.12-beta_Linux-amd64-bin.tar.gz
tar -zxvf bifrost_v2.3.12-beta_Linux-amd64-bin.tar.gz
cd bifrost_v2.3.12-beta_Linux-amd64-bin/bin && chmod a+x ./Bifrost*
```

Docker works with the jc3wish/bifrost image and a volume mount for persistent configuration and data:

```shell
docker pull jc3wish/bifrost
mkdir -p /data/BifrostData
docker run -d -p 127.0.0.1:21036:21036 -v /etc/localtime:/etc/localtime -v /data/BifrostData:/linux/data jc3wish/bifrost
```

This loopback mapping limits exposure under Docker’s default networking. Docker releases older than 28.0.0 can expose localhost-published ports to other hosts on the same network segment; verify the Docker version and firewall settings. Change the default credentials before exposing the interface.

Building from source uses the repository and its Makefile:

```sh
git clone https://github.com/brokercap/Bifrost.git
cd Bifrost
make install prefix=./target
```

Start and stop the server with:

```shell
./Bifrost-server start
./Bifrost-server stop
```

The web interface is at https://127.0.0.1:21036. Default credentials are username Bifrost and password Bifrost123, set in etc/Bifrost.ini. A live demo runs at https://www.xbifrost.com/demo/goto.html using those same defaults. Change the credentials before exposing any instance to a network outside a local test environment.

## Bifrost.ini controls user roles, queue sizes and a file-queue overflow option

Bifrost.ini is divided into three sections. The [user] section maps each username to its password. The [groups] section assigns roles: administrator permits add and delete operations on the configuration, while monitor is read-only. Runtime behavior goes in [Bifrostd], which holds the queue, network and retry settings.

Queue sizes affect throughput under load. toserver_queue_size controls the buffer sending data to downstream plugins and defaults to 5000. channel_queue_size sets the per-channel buffer and defaults to 1000. When the in-memory queue fills repeatedly, file_queue_usable can be set to true to spill excess events to disk. The measurement window for that trigger is file_queue_usable_count_time_diff, given in milliseconds, and file_queue_usable_count sets how many times the queue must fill within that window before disk spill activates.

HTTPS requires tls=true together with tls_key_file and tls_crt_file, which accept relative or absolute paths. Certificate generation steps are at wiki.xbifrost.com/install/https_certificate/. The listening address and port are set by listen=0.0.0.0:21036 and can be changed there.

Two retry settings govern error handling for plugins: plugin_commit_timeout, in seconds, controls how long to wait before committing when no data is flowing, and plugin_sync_retry_time, in seconds, sets the pause between retry attempts when a sync fails.

## One binlog parsing thread distributes events to all plugins in parallel

Bifrost runs a single binlog parsing thread, which keeps events in the exact sequence MySQL logged them. Once parsed, each event is handed to multiple output plugins that run in parallel, so a slow or unavailable Redis instance does not block the ClickHouse plugin from processing the same row change. Each plugin has its own queue, so one slow target does not hold up others.

Position tracking is per-session, stored alongside the configuration on disk. On restart, Bifrost reloads configuration and returns to the last saved binlog position automatically. Events in flight at crash time may be re-delivered, so target stores that cannot accept duplicate writes need a deduplication layer on their end. No built-in exactly-once delivery guarantee is documented, so the responsibility for deduplication stays with the target store or the application layer above it.

Monitoring alerting does not require a separate tool: Email and WeChat Enterprise alerts are both supported out of the box. An HTTP API also gives third-party monitoring platforms access to status information, protected by HTTP Basic authentication. The full endpoint list is at wiki.xbifrost.com/api/http_api/.

## Dynamic plugins require Linux, and every recent release is labeled beta

The dynamic_plugin setting in Bifrost.ini controls whether plugins can be loaded at runtime as shared objects. This setting is valid only on Linux, where Go can load shared objects at runtime. On macOS, Windows or any other platform, the setting is ignored and the value is always false, regardless of what the configuration file says. Plugin updates on those platforms require a full rebuild rather than a hot-load.

All three recent releases carry a beta tag: v2.4.5-beta (released 2025-10-19), v2.4.3-beta (released 2025-08-03) and v2.4.1-beta (released 2025-04-26), with no stable-tagged release in the repository within that period. Deployment policies that require a stable release tag before promoting to production need to account for this before scheduling adoption.

The DDL gap in Redis, MongoDB and Memcache adds a second operational concern. Schema changes on the MySQL side reach nine of the twelve target types and bypass three of them without any logged notification from Bifrost. Each ALTER TABLE or CREATE TABLE that runs on the primary requires a manual check of those three targets to determine whether their schema is still consistent with the source.

## Apache-2.0 licence, a Gitee mirror and a plugin development API

The licence is Apache-2.0, with the text in the repository's LICENSE file. A separate file named 免责声明.txt sits at the repository root alongside it.

Plugin development has a dedicated guide at wiki.xbifrost.com/plugin/plugin_dev/. The repository layout reflects this: a plugin/ directory sits at the top level alongside the input/, server/ and sdk/ directories. The Mock source type in the source list exists precisely to support plugin authors, removing the dependency on a real MySQL deployment during development.

Source code is mirrored on Gitee for teams where GitHub access is unreliable. A QQ group (号 70331601) is listed as the primary community channel; no GitHub issue tracker is mentioned as a reporting route. The last push was on 2026-06-20, and the repository is not archived.

Key dependencies from go.mod include github.com/ClickHouse/clickhouse-go v1.4.3, github.com/Shopify/sarama v1.29.0 for Kafka, go.mongodb.org/mongo-driver v1.17.2 for MongoDB, and github.com/go-redis/redis/v8 v8.7.1 for Redis. Each of those pins a specific version, so updating any of them requires testing against the corresponding target type.

## Conclusion

Bifrost suits teams that need to fan MySQL or MariaDB changes to multiple stores without separate CDC pipelines. The binlog-slave approach adds no query load, and parallel plugin execution means one slow target does not stall the others. Skip it if Redis, MongoDB or Memcache are your primary targets and you have frequent schema changes: DDL events do not reach those three. All releases from v2.4.1 through v2.4.5 carry a beta label, so confirm that is acceptable before a production deployment. Verify your MySQL version against the tested range (5.1.73 through 8.0.21) and configure TLS in Bifrost.ini before exposing port 21036 to a network.

## FAQ

### What source databases does Bifrost support?

MySQL, MariaDB and Percona are supported via binlog and select, both full and incremental. Kafka works as a source in four formats (string, canal, bifrost and debezium) and MongoDB works via oplog, both incremental only.

### Does Bifrost synchronize DDL changes to all output targets?

No. DDL sync is supported for MySQL, ClickHouse, RabbitMQ, Kafka, ActiveMQ, the HTTP plugin, the Hprose RPC plugin, StarRocks and Doris. Redis, MongoDB and Memcache do not receive DDL changes.

### How can Bifrost be installed?

Three paths are documented: download a binary release archive and extract it, use the jc3wish/bifrost Docker image with a data volume, or clone the repository and run make install. The web interface starts at https://127.0.0.1:21036 after the server is running.

### How does Bifrost recover after a crash or restart?

On restart, Bifrost reloads its configuration from disk and returns to the last saved binlog position. Events that were in flight at the time of the crash may be re-delivered, so target stores should be able to handle duplicate writes.

### Does Bifrost run on platforms other than Linux?

The binary and Docker paths work on the platforms the release provides, but the dynamic_plugin feature is Linux-only. On other platforms that setting is always false regardless of the configuration file, so runtime plugin loading is not available.

## Sources

- [brokercap/Bifrost on GitHub](https://github.com/brokercap/Bifrost)
- [License: Apache-2.0](https://github.com/brokercap/Bifrost/blob/master/LICENSE)
- [Project website](https://www.xbifrost.com)
- [README](https://github.com/brokercap/Bifrost/blob/master/README.md)
- [Releases](https://github.com/brokercap/Bifrost/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/brokercap-bifrost