pgcopydb for Parallel PostgreSQL Migrations

Why pgcopydb Exists for PostgreSQL Server-to-Server Copies

pgcopydb targets a specific gap between PostgreSQL command-line tools. A dump in directory format creates local files, so a simple pipeline combining parallel pg_dump and pg_restore is not available. The README describes the desired arrangement as parallel dumping connected directly to parallel restoring, but explains that directory-format pg_dump output must be written and read through local storage. pgcopydb therefore handles schema work with pg_dump and pg_restore while implementing its own multiprocess streaming for table data.

The intended users are PostgreSQL administrators and engineers responsible for migrations between two running servers. The project can copy a selected database, copy an entire instance with `clone --all-databases`, or continue applying changes after the initial copy. That makes it more specific than a file-oriented backup utility and more capable than a basic logical dump-and-restore sequence when throughput and migration stages matter.

The project also separates several operational steps. The command list includes `clone`, `fork`, `follow`, `snapshot`, `compare`, `dump`, `restore`, `copy`, and `stream`. Those names reflect a migration workflow rather than a single-purpose backup command. Schema comparison, data comparison, index creation, sequence handling, and logical change replay are all exposed as parts of that workflow.

How pgcopydb Transfers Tables, Indexes, and WAL Changes

The central design decision is to retain familiar PostgreSQL utilities for schema handling while taking control of the data path. According to the README, pgcopydb uses pg_dump and pg_restore for schema parts and implements its own multiprocess streaming for data. It also bypasses pg_restore index building and drives index creation internally so indexes can be built concurrently.

That division has a practical consequence. Table copying and index copying are exposed as separate options in the example command, allowing the operator to choose different concurrency levels for each phase. The command list further shows dedicated operations for tables, table parts, blobs, sequences, indexes, constraints, roles, and extensions. This gives an operator control over migration order instead of treating every object as one indivisible dump.

Change Data Capture uses PostgreSQL Logical Decoding. The default `pgoutput` plugin is built into PostgreSQL core from version 10 and requires no extension installation, while `test_decoding` and `wal2json` are also supported. WAL changes are stored in structured SQLite databases named `output.db` and `replay.db` in the pgcopydb work directory. The README states that this storage supports reliable processing across process restarts, file rotation, and the `stream prune` command for reclaiming disk space.

The `follow` command implements a logical replication client. `stream setup`, `stream receive`, `stream apply`, and related subcommands divide change handling into preparation and execution. The `clone --follow` combination represents an online migration, while `clone` alone represents the offline approach.

Building the C Tool and Running a First Clone

The repository exposes a Makefile rather than a documented package installation command in the README. The Makefile can build the binary and install it, while the Dockerfile builds a multi-architecture Debian-based image and installs PostgreSQL development dependencies. The source build defaults to PostgreSQL 16 through `PGVERSION ?= 16`.

From a source checkout, the available build and install entry points are:

make
make install

The first command delegates to `src/bin/` to produce `pgcopydb`, and the second delegates the installation step to the same source subtree. The Makefile does not specify a universal system package name in the displayed build rules, so operators should use the repository build or the documented packaging route rather than inventing an installation command.

A first offline copy can follow the README example. Set both connection strings as environment variables, then select table and index concurrency:

export PGCOPYDB_SOURCE_PGURI="postgres://[email protected]/dbname"
export PGCOPYDB_TARGET_PGURI="postgres://[email protected]/dbname"
pgcopydb clone --table-jobs 8 --index-jobs 2

The README says that this copies the database from the source to the target. The connection URIs identify the source and target databases, while `--table-jobs` and `--index-jobs` control separate parallel phases. The README does not show the complete expected output, so operators should verify process status and target contents rather than assume that a successful process return proves application-level correctness.

Offline Copies Versus Online Migration with pgcopydb

The clearest project-specific boundary is between `clone` and `clone --follow`. The README says `clone` implements the offline migration approach. It is the appropriate starting point when the source can be quiesced or otherwise kept consistent for the copy. The table and index controls let an operator tune the copy phase, and the separate schema, data, sequence, index, and constraint commands support more deliberate workflows.

Online migration changes the problem. `clone --follow` adds logical decoding and change replay, which means the source continues to generate WAL changes that must be retained, processed, and applied to the target. The README explicitly warns that online migrations involve much more complexity than offline migrations and suggests implementing an offline migration first. This is a practical warning, not a claim that online mode is unsafe. It means that replication lag, change ordering, target state, restarts, and storage for SQLite CDC files need deliberate verification.

The `stream prune` command is particularly relevant for long-running online operations because it removes already-applied CDC files to reclaim disk space. Operators should establish where the work directory lives and how much disk it consumes before assuming that CDC storage is transient. The README does not specify capacity planning formulas or target recovery behavior, so those points must be checked in the project documentation or tested before production use.

Operational Boundaries of the Migration Command Set

pgcopydb offers useful migration primitives, but the README does not turn it into a complete data integrity checker. The `compare` command family includes `schema` and `data`, which provides a way to inspect differences between source and target. The README does not state whether comparison is exhaustive, how long it takes on large databases, or whether it handles every PostgreSQL object type, so those limits should not be inferred.

The command structure also shows that some operations are separate and potentially order-sensitive. Roles and extensions have dedicated copy commands. Sequences have their own copy operation, and indexes are intentionally handled outside pg_restore's normal index-building behavior. Constraints and schema objects can therefore require attention to dependencies and migration order. The `list` command can expose schemas, tables, table parts, sequences, indexes, dependencies, extensions, and collations, which provides inspection points before or during a copy.

For a tool comparison, the real alternative is a conventional `pg_dump` and `pg_restore` workflow. That approach uses PostgreSQL's directory format and local intermediate files, with `pg_dump --jobs=N` and `pg_restore --jobs=N` options as described in the README. It is a more direct fit when intermediate files, standard restore tooling, or the familiar command pipeline matter more than pgcopydb's streaming design. pgcopydb's difference is its server-to-server multiprocess data path, separate index concurrency, and CDC workflow.

Maintenance, License, and Upgrade Considerations

The repository is not archived, and its last push was on 2026-10-13. The listed release history includes v0.18 on 2026-06-27, v0.16 on 2024-05-22, and v0.17 on 2024-08-07. These dates establish release and repository activity, but the README does not describe a support policy, compatibility matrix, migration path, or upgrade procedure. The project is a C codebase with source and Docker build paths, so upgrades should account for PostgreSQL client and server compatibility, required privileges, and changes to logical decoding behavior.

The repository identifies the license as Other, with a custom `LICENSE` file that GitHub cannot classify. The Makefile also carries a PostgreSQL License notice. This is not enough to determine how the custom license affects redistribution or linking obligations. Operators should read the `LICENSE` file directly and review the notices in the relevant source files before distributing a binary or incorporating it into a product. The README does not provide legal interpretation, so no broader license claim is warranted.

The available build variables include `PGCOPYDB`, `TOP`, and `PGVERSION`, with PostgreSQL 16 as the default. The Makefile provides `clean`, `install`, `docker`-based `build`, `force-build`, and `clean-images` targets. These targets show that container builds and version-specific rebuilds are part of the repository workflow, but the README does not document the operational effort required to upgrade an existing installation.

Editorial conclusion

Choose pgcopydb when moving one database or a complete PostgreSQL instance with separate table and index concurrency is the priority. Do not treat its online mode as a simple replacement for an offline migration, since the README explicitly warns that logical replication adds substantial complexity. Verify the supported PostgreSQL versions, required privileges, logical decoding setup, target compatibility, and the behavior of clone and follow before using it for production data.

Frequently asked questions

Official sources

1. dimitri/pgcopydb on GitHub
2. Issues
3. README
4. Releases