MongoShake reads the oplog once and can send it to MongoDB, Kafka, TCP or a file

The oplog is the only input MongoShake asks for

MongoShake does not scrape collections. It attaches to the oplog of a source MongoDB, reads entries after a configured timestamp, and hands them onward. That single decision is what makes it fast to start and awkward to use for anything but change replication: a logical dump of existing documents is not what the tool does.

The source may be a single mongod, a replica set, or a sharded cluster. The target may be a mongod or a mongos, and several mongos instances are allowed on the target so that data is hashed across them for availability. Two operational notes come straight from the README. If the source is a replica set, the project suggests reading from a secondary or hidden member to take pressure off the primary. If the source is sharded, every shard has to connect to MongoShake separately, which means N processes for N shards.

Internally the code divides along the module boundaries you can see in the tree: collector, oplog, journal, quorum, tunnel, receiver and executor. Filtering happens after fetching, driven by whitelist and blacklist entries on database and collection namespaces.

shad_key decides whether replay is per document or per collection

Parallel replication is controlled by a setting called shad_key, and this is the single most consequential knob for correctness. `__id__` makes the concurrency granularity the document, so independent writes to different documents proceed in parallel. `__collection__` serializes at the collection or table level, preserving order within a collection. `__auto__` inspects the collections: if any collection carries a unique index it switches to `__collection__`, otherwise it uses `__id__`.

That auto mode is convenient but opaque in advance. A migration started with `__auto__` on a schema you have not fully inventoried may order writes differently from the same migration run a week later. If a target has unique constraints that the source does not, or the reverse, the collection level setting can hide an ordering bug while the document level setting exposes it as a duplicate key error on the target.

Delivery semantics are at least once. The README is explicit that all oplogs are transferred at least once and that this is acceptable because oplog DML is idempotent. Acks are tracked with sequence and acknowledgment numbers, described as similar to the sequence and acknowledgment numbers in TCP, so the receiver confirms batches rather than individual writes.

Tunnels turn the replicator into a log shipper

The `direct` tunnel writes into the target MongoDB directly and covers the ordinary backup and disaster recovery case. Beyond that, `rpc` and `tcp` connect synchronously to a receiver over RPC or raw TCP, `file` writes the output to a file, `kafka` sends asynchronously, and `mock` throws all data away and exists purely for testing a pipeline.

Receivers for rpc, tcp, file, mock and kafka are shipped with the project, but the contract is explicit that a user writing their own receiver has to implement the interface for whichever tunnel they pick. The tunnel type is pluggable in both directions, so a new transport is a code contribution, not a plugin drop-in.

Gzip, zlib and deflate compressors work on batched oplogs before sending, which matters most for the kafka and file tunnels where the batch is what crosses the wire. Because oplogs are batched together in the handling pipeline and both worker concurrency and executor concurrency are adjustable, throughput tuning is a knob-turning exercise rather than a code change, and the performance document linked from the README is where the numbers live.

Building collector from the Makefile or the Dockerfile

The repository builds with the standard Go toolchain. The Makefile's default target runs build.sh, and a platform specific target exists for Linux:

make linux

That target depends on `clean`, which removes bin, pprof and output files, log directories, diagnostic and data directories, and stray pid files. Running it removes your previous binary first, so keep copies elsewhere if you pin a build.

The container route does the same work in two stages. The build stage installs bash, make, git, zip, tzdata, ca-certificates, gcc and musl-dev, then invokes make linux. The runtime stage keeps only tzdata, ca-certificates and musl, copies the products into /app, and starts the collector:

FROM alpine:3.12
WORKDIR /app
COPY --from=golang /app/bin .
EXPOSE 9100
ENTRYPOINT ["/app/collector"]

Port 9100 is exposed for metrics. Note that the entrypoint is the collector binary, so the container runs one replication job per container and a sharded source needs one container per shard. The module path is github.com/alibaba/MongoShake/v2, with go 1.21 declared and toolchain go1.24.5 pinned, and the default branch is develop rather than main.

Checkpoints and the master_quorum election

High availability is built around two pieces. MongoShake periodically persists its context into a register center, which by default is the source database, and that context is the checkpoint marking the last successfully replayed oplog position. Separately, a hypervisor mechanism restarts the process when it dies, configured through `master_quorum` in the configuration.

There is a hard coupling to note: `master_quorum` only works when `checkpoint.storage=database`. Switch checkpoint storage to a file and the election silently has nothing to coordinate through. When `master_quorum=true`, you must also set `master_quorum.election_id` to a valid MongoDB ObjectID and keep it unique per sync job or HA group, so that two independent jobs do not elect against each other.

Monitoring is available three ways. A RESTful API exists for both monitoring and debugging. A Prometheus exporter written by a third party covers replication metrics. Since v2.8.8 MongoShake ships its own prometheus metrics, documented under docs/monitor in the repository.

DDL syncing works only from a replica set, and gid is missing

Since version 1.5, MongoShake can replicate DDL using a global barrier. When a DDL oplog is fetched, it injects a barrier that holds every later oplog in the queue until the DDL has landed on the target and the checkpoint has advanced. That is the correct mechanism, and the constraint is the problem: DDL syncing is supported only when the source is a ReplicaSet. The target may be a replica set or sharding, but a sharded source is explicitly deferred.

The other gap is global id, also called gid. In the Aliyun internal version, gid marks the database id and lets MongoShake fetch only oplogs equal to the source database id, which is what stops two databases that are backups of each other from looping. The README states this is unsupported in the current open source version, limited by modifications to the MongoDB kernel. Without gid, every oplog is fetched. Active-active without gid is therefore possible but requires the procedure described in the FAQ document rather than a configuration flag.

Versions, branches and what an upgrade costs you

The versioning rule is unusual and worth internalising before you pin anything. Numbers follow a.b.c, where a is the major version, c is the bugfix version, and b is the minor version where an even number means stable. So 1.2.x, 1.4.x and 2.0.x are stable while 1.5.x and 2.1.x are not. Release tags follow a pattern of release-v2.8.8-20260428, with the date embedded, and the releases spaced widely: v2.8.6 on 2025-08-25, v2.8.7 on 2025-12-18, and v2.8.8 on 2026-04-28. The last push to the repository was on 2026-10-10, so commits land on the develop branch well ahead of the newest tag.

That gap is the real upgrade cost. Code on develop can be months newer than the latest release tag, and the even-minor stable rule means an odd-minor line is not somewhere a production deployment belongs. The licence is GPL-3.0, which matters if you intend to embed the binary in a product rather than run it as a service alongside your database.

Editorial conclusion

MongoShake suits an operator who needs a full cluster migration, a filtered copy into a second deployment, or an oplog feed into Kafka for downstream consumers, and who is comfortable tuning worker and executor concurrency. It does not suit teams that need exactly-once delivery, unsupported DDL from a sharded source, or an out-of-the-box active-active topology, because the open source build has no global id and relies on at-least-once replay. Verify first that your source is a replica set if you need DDL, and read the FAQ page referenced in the README for the loop-avoidance procedure that replaces gid.

Frequently asked questions

Official sources

1. alibaba/MongoShake on GitHub
2. Issues
3. License: GPL-3.0
4. README
5. Releases