ape-dts: one Rust binary for snapshot migration, CDC, and data repair across engines

ape-dts bundles migration, change streaming, and repair in one process

ApeDTS is a standalone Rust binary for moving data between databases, message queues, and analytical engines. The README calls it a production-ready binary of roughly 17 MB compressed and describes it as a lightweight, standalone data migration and CDC engine handling snapshot migration, continuous synchronization, data validation, correction, and in-flight transformation. The audience is infrastructure and database engineers who need a controlled move between engines, often for disaster recovery or a migration off one system onto another, and who would rather not assemble a pipeline of connectors, a broker, and a consumer application to do it. The design stance is stateless and cloud-native: the process runs on its own, with no external runtime components and no intermediate storage. That is the single most important architectural fact in the project, because it decides where the operational burden sits.

The task matrix shows exactly which engine pairs support which operations

The most useful page in the README is the task type table, because it is honest about coverage. Snapshot and CDC are marked as supported across the whole listed set of paths: mysql to mysql, pg to pg, mongo to mongo, redis to redis, mysql to kafka, pg to kafka, mysql to starrocks, mysql to clickhouse, mysql to tidb, pg to starrocks, pg to clickhouse, mysql to doris, and pg to doris. The data check, revise and review row is narrower. It is marked only for mysql to mysql, pg to pg, mongo to mongo, and mysql to tidb, with the Kafka, Redis, StarRocks, ClickHouse and Doris columns empty. Structure migration is marked for the homogeneous pairs, for mysql to starrocks, mysql to clickhouse, mysql to tidb, and for the pg to starrocks, pg to clickhouse and pg to doris routes, but not for the Kafka or Redis columns. So the headline claim of mature multi-engine coverage has a specific shape: broad for moving data, narrow for validating and correcting it.

The Rust workspace splits connectors, parallelism, pipeline, and tasks

Based on the repository layout, the implementation is a Cargo workspace with nine members: `dt-common`, `dt-precheck`, `dt-connector`, `dt-parallelizer`, `dt-pipeline`, `dt-task`, `dt-main`, `dt-cli`, and `dt-tests`. The naming makes the data flow legible. `dt-connector` holds engine specific connectivity, `dt-parallelizer` applies the source and task specific parallelism the README describes, `dt-pipeline` moves records between them, `dt-task` models the task types, and `dt-main` is the module the build produces. `dt-precheck` is where the environmental checks before a migration would live. The dependency list reinforces the multi-engine claim: a forked `sqlx` with the mysql and postgres drivers, `mysql-binlog-connector-rust` pinned to tag v0.3.4 with openssl-tls for reading the MySQL binary log, a forked `rust-postgres` set for the Postgres side, and `tiberius` with `bb8` for SQL Server. The Rust workspace also lists `dt-task` and `dt-tests` separately, which is what lets the test suite exercise tasks end to end.

dtscli is the local control surface for tasks

The CLI is called `dtscli`, and the README describes it as a lightweight local tool for creating and managing ApeDTS tasks. Its documented capabilities are to generate task configs, start `dt-main`, list tasks, stream logs, and stop, restart or delete local task records. In practice that is the whole workflow surface: you describe a task in a config file, the CLI generates or holds that config, and it launches the engine process that does the work. Installation instructions for the CLI are not in the top level README, which points to `dt-cli/README.md` for both installation and detailed usage, so the download and install step has to come from there. The repository does ship a sample task config path in the Makefile, along with the module it builds:

CONFIG_PATH ?= ./images/example/mysql_snapshot_sample.yaml
MODULE_NAME ?= dt-main

The `CONFIG_PATH` default points at a MySQL to MySQL snapshot sample, which is a sensible first target to study if you are evaluating the tool on a database pair you already understand.

The container build pins a Rust toolchain and mirrors for dependencies

Building the project yourself means a Rust toolchain, and the Dockerfile pins one:

ARG RUST_VERSION=1.85
FROM --platform=${BUILDPLATFORM} rust:${RUST_VERSION}${RUST_IMG_ALT} as builder

The build stage installs `cmake` and `libclang-dev` before compiling, and the file also carries a `LIBC` argument that can be switched to musl for Alpine based distributions. The Dockerfile defaults to mirrors hosted at `mirrors.ustc.edu.cn`, for apt packages and for the Rust static distribution, while the Makefile defaults its apt mirror to `mirrors.aliyun.com`, and the Makefile sets `RUSTUP_DIST_SERVER` to that mirror as well. That is a practical detail for teams outside mainland China, because the defaults will slow a first build elsewhere, and the override points are named in the files rather than left implicit. The Makefile also exports `DOCKER_CLI_EXPERIMENTAL=enabled` and builds with buildx across `linux/arm64,linux/amd64`, with the image named `apecloud/ape-dts` and tagged `latest` by default.

The published MySQL to MySQL benchmark measures a narrow workload

The README includes one benchmark section, described as `ape-dts: 2.0.26.1` on Alibaba Cloud in the same VPC, with MySQL 8.4.11 on an 8c16g node. The workload is a full load of 8 tables by 4 million rows, then CDC with 8 tables, 1 million baseline rows and 4 million UPDATEs. For the MySQL to MySQL snapshot case, the table reports 109067 rows per second on 1c2g, 131783 rows per second on 2c4g, and 133283 rows per second on 4c8g. The same rows show source MySQL CPU between 7.28 and 9.18 percent while target MySQL CPU climbs from 112.59 percent to 172.36 percent, which tells you the write side is where the pressure lands. Two limits are worth naming. The CDC table in that section has headers and no rows, so the change stream numbers are not published here, and the whole exercise covers one engine pair and one schema shape.

Lua transforms and column filtering run in flight, not in a separate stage

Beyond moving rows, the tool filters and routes at the database, table, and column level and applies lightweight Lua transformations while data is in flight. The documentation tree lists a dedicated page for Lua based modification under both the tutorial section and the data processing section, which suggests the transform runs inside the transfer path rather than as a separate downstream job. The same in-flight model appears in the structure side, where there is a page for checking structure with Liquibase, an external tool, alongside the built-in structure check page. Other documented capabilities include a heartbeat written back to the source database, generating SQL statements from the change stream, two-way data sync, resuming at a breakpoint, running multiple tables in parallel, and a custom consumer path that goes from mysql or pg into kafka and on to a consumer. Monitoring is documented too, with pages for monitor information and for position information.

AGPL-3.0 and a fast release cadence are the two costs to weigh

The project is licensed under AGPL-3.0, which is the practical constraint for anyone considering it inside a hosted service, and it deserves a conversation with your own counsel rather than a general reading of this article. On the engineering side, the cadence is fast and the versioning is dense: the release list shows v2.0.25.1 on 2026-07-14, v2.0.26 on 2026-08-03, and v2.0.26.1 on 2026-09-29, and the last push was on 2026-10-15. A patch release in the 2.0.26 series a few days after the base release suggests fixes are landing quickly. That is good for a project absorbing new engine behaviour and awkward for a team that wants a frozen version, so pin an exact tag and read the changelog before upgrading. The repository also carries a `CHANGELOG.md`, `rust-toolchain.toml`, a `Cross.toml` for cross compilation, and a dedicated `dt-tests` member whose own README describes how the tests are run.

Editorial conclusion

ape-dts fits teams moving a database into an analytical engine or running a continuous change stream between two engines the project has already validated, and it fits teams that want a single stateless process instead of a Kafka deployment for the pipeline itself. It does not fit heterogeneous pairs outside the listed matrix, and it does not give you check, revise or review on the Kafka, Redis or analytical sink paths. Before adopting, confirm your exact source and target pair appears in the task table, read the checkpoint and resume documentation for that path, and settle the AGPL-3.0 question with your own counsel, since the licence is the one thing no amount of engineering changes.

Frequently asked questions

Official sources

1. apecloud/ape-dts on GitHub
2. Issues
3. License: AGPL-3.0
4. README
5. Releases