Ghostferry copies selected MySQL rows between servers without a long lock

A copy tool between two MySQL instances, not a schema migration

Ghostferry is a library that lets you selectively copy data from one MySQL instance to another with a minimal amount of downtime. The repository description calls it the swiss army knife of live data migrations.

The lineage is stated up front. It is inspired by GitHub's gh-ost, but with one decisive difference: instead of copying data from and to the same database, Ghostferry copies from one database to another, and it can copy only part of the data. That is the whole reason the project exists as a separate tool rather than a configuration of gh-ost.

The primary language is Go, and the module is github.com/Shopify/ghostferry. The go.mod file pins go 1.27.2 and depends on go-mysql-org/go-mysql for the binlog side, go-sql-driver/mysql for plain SQL, Masterminds/squirrel for query building, gorilla/mux for the control server HTTP routes, and both zerolog and logrus for logging, with DataDog/datadog-go for metrics. Several logging packages coexist, which is a hint that this codebase absorbs code from more than one era.

There is an example application, ghostferry-copydb, under the copydb directory, which demonstrates the library by copying an entire database from one machine to another.

The component names describe the copy algorithm

The root directory is a Go project with most of its files at the top level rather than tucked into packages, and the file names map directly onto the phases of a migration.

There is a binlog_streamer.go and a binlog_writer.go, which read the source's binary log and write the resulting changes, plus dml_events.go for the data change events themselves. The reading side of a copy is data_iterator.go, with data_iterator_sorter.go alongside it for ordering, and batch_writer.go for the write batches. filter.go is the piece that decides which rows are in scope, which is what makes a partial copy possible. throttler.go limits the rate of the copy so the source is not swamped.

Verification is a first class concern and it has its own files. There is an inline_verifier.go, an iterative_verifier.go, a target_verifier.go and a compression_verifier.go, with verifier.go as the shared interface. state_tracker.go keeps the position in the migration, and cursor.go and pagination_key.go handle resumable paging. There are also table_schema_cache.go, wait_until_replica_is_caught_up_to_master.go and sharding/ for shard aware copying.

The copy itself is orchestrated by ferry.go, with error_handler.go, http_callback.go, metrics.go and progress.go covering the operational surface. A simplified model of the high level copy algorithm is written in TLA+ under the tlaplus directory, and the README is careful about what that model proves: it is a small finite model with explicitly stated simplifying assumptions, and checking it is not a proof of correctness of the current Go implementation.

A control server with a web UI for watching and steering the copy

control_server.go sits at the centre of the operational story, and the copy is not a fire-and-forget process. It exposes an HTTP interface, with gorilla/mux handling the routes, and the copydb binary ships a web UI whose templates are loaded from a webui directory.

The README is specific about a detail that bites people. The web UI templates are loaded from webui below ControlServerConfig.WebBasedir, which defaults to the current directory. Debian packages built with make copydb-deb instead compile in the base directory /usr/share/ghostferry and install webui beneath it, and the base directory is the parent of the webui directory rather than webui itself. If you move the binary away from where you built it, the UI templates will not be found.

There is also a callback mechanism, http_callback.go, which lets something outside the process be notified, and metrics.go for instrumentation through the Datadog client.

The operational picture is that a migration is watched rather than waited on. The README documents an example configuration, examples/copydb/conf.json, that binds the UI to 127.0.0.1:8000 and adds two Custom Script buttons, which is how the tool lets an operator run their own commands at chosen moments during the copy.

Running copydb against two local MySQL servers

The development setup is fully scripted, which makes it easy to see what a real run looks like. The prerequisites are Go 1.27.2, Git, Make and a MySQL client to build and run ghostferry-copydb, plus Docker or Podman for the local MySQL servers. The Go and Ruby versions are pinned in .tool-versions, which both mise and asdf read.

With a version manager, the setup is two commands.

mise install    # or: asdf install

Without one, any Go 1.21 or newer works, because go.mod requires Go 1.27.2 and the go command downloads that toolchain itself. The Ruby side is needed for the tests and the documentation site, and requires a C compiler toolchain plus the MySQL client development libraries for the mysql2 gem.

Two disposable MySQL 8.0 servers come up from the compose file.

docker compose -f docker-compose_8.0.yml up -d mysql-1 mysql-2
# or: podman-compose -f docker-compose_8.0.yml up -d mysql-1 mysql-2

They listen on 29291 for the source and 29292 for the target, with a passwordless root account, and the README is explicit that these are throwaway test servers and not a template for production credentials. The default docker-compose.yml instead uses percona:5.7 images with row based binlog format and GTID enabled, and a separate file exists for MySQL 8.4.

Once both accept connections, you build the binary and run it against the example configuration.

ghostferry-copydb -verbose examples/copydb/conf.json

Verifiers, and the SkipTargetVerification flag that turns one off

The verifier files are the part of the design that distinguishes this from a simple row copier. A copy can appear to succeed and still leave the target with different data than the source, and the verifiers exist to detect that.

The example configuration in examples/copydb/conf.json uses the Inline verifier, and the README points out that it sets SkipTargetVerification to true, which disables target-write monitoring, and notes that the tutorial intentionally keeps the protected default. In other words, the example shipped for convenience takes a shortcut the tutorial does not want you to copy into production thinking.

That single flag is the most important thing in this section. With verification on, the copy is checked against the target as it runs, and a mismatch is a failure you can see. With it off, the copy is faster and quieter, and you are trusting the binlog stream. If you are moving data that has to be correct, keep the verification on and treat SkipTargetVerification as something you switch on deliberately for a throwaway run.

The README does not document the failure modes of each verifier in the pages shown, so read inline_verifier.go, iterative_verifier.go and target_verifier.go before you choose one for a production migration.

TLA+, Ruby integration tests and a Jekyll documentation pipeline

Ghostferry treats the algorithm as something worth formalising. The tlaplus directory holds a simplified model of the high level copy algorithm together with a TLC model configuration in tlaplus/ghostferry.toolbox, and the comment at the top of tlaplus/ghostferry.tla states the simplifying assumptions explicitly. The README insists that model checking it is not a proof of correctness of the Go code, which is the right caveat to keep in mind.

The test suite is not only Go. There is a Rakefile and a Gemfile, and the README notes that test/test_helper.rb loads pry-byebug from the development group unless CI is set, which is why bundle install should not exclude the test, development or docs groups. The Ruby integration tests take a DEBUG=1 option to see detailed output from a live Ghostferry run rather than only on failure. The Makefile has separate test-go and test-ruby targets, and the default goal is test. Running the suite against the 8.0 servers requires exporting MYSQL_VERSION=8.0.

The documentation is built with Jekyll and the Just the Docs theme, with settings, page titles and navigation order in docs/_config.yml. The build writes to build/docs and htmlproofer fails on broken internal links or anchors, and the live preview is served at http://127.0.0.1:4000/ghostferry/main/. The changelog page is generated at build time from the root CHANGELOG.md, which is the only file to edit for release notes, and because the Jekyll watcher only watches docs/, you have to restart the preview after changing it.

Prerelease tags, a 1.4.0 version string and an IRC channel from another era

The release conventions are unusual and worth understanding before you pin anything. The recent tags are named release-4a99fc3, release-332115b and release-3308a69, which carry a commit hash rather than a version number, and the README states that most published packages are prereleases.

The version that ends up in the binary comes from the Makefile, where VERSION is 1.4.0 and the string is assembled from that value, a UTC timestamp and the short commit SHA, with a +dirty suffix when the tree is not clean. The same variable is injected at link time through the VersionString symbol. Debian packages are built by make copydb-deb and named after the target architecture, and they are published on the project's GitHub Releases page.

The README also points at an IRC channel on irc.freenode.net, and Freenode is long gone as a chat network, which tells you how much of the surrounding documentation has aged. The publishing date on the repository is current, with the last push on 2026-10-19 and release tags on the same day and on 2026-10-14, so the code is being worked on even though the docs have stale links.

Licensing is MIT with a LICENSE file at the root, which matters if you embed the library in your own migration tooling.

Where Ghostferry loses to a plain dump and restore

The alternative approach is the obvious one: dump the source with the MySQL client, load it into the target, and cut over. The difference in approach is that a dump and restore moves everything and takes the source out of the equation, while Ghostferry moves selected rows while the source keeps taking writes and streams those writes through the binlog so the target converges.

That difference is what makes the tool expensive to use. The source has to be running with binlog enabled in row format, as the compose files configure, with server ids, GTID settings and log-slave-updates turned on. The copy competes with production traffic for IO and CPU, which is why a throttler exists and why the README points at waiting for a replica to catch up to the master as a distinct step. The process also needs a control server, a web UI and enough operational attention to know whether the copy is healthy.

For a full database move with a maintenance window, a dump and restore is simpler and safer. For a partial copy, where a handful of tables or a filtered set of rows must land in another instance without pausing writes, Ghostferry is doing something a dump cannot.

The other boundary is scope. It copies MySQL to MySQL. There is no support here for copying from another database engine, and no schema migration story, since gh-ost is the tool for changing a table inside one database.

Editorial conclusion

Ghostferry is the right tool for a team that needs to move a subset of rows, a single table or a filtered slice of a database from one MySQL instance to another while the source keeps serving traffic, since its filter, throttler and verifier machinery is built for exactly that and nothing else. It is the wrong tool for a schema change inside one database, which is the case gh-ost covers, or for a full logical backup. Before adopting it, read the technical overview for the component model, test it against your own table sizes, and read the source in this repository rather than the published API docs, which the README says track only tagged versions.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. Shopify/ghostferry on GitHub