# facebook/wdt: a C++ library for moving data over many TCP paths

> Warp speed Data Transfer is an embeddable C++ library and command line tool for sending files or directory trees between two machines over parallel TCP connections. It targets fast storage and fast links, and its build and release cadence are worth checking before adoption.

**facebook/wdt** — Warp speed Data Transfer (WDT)  is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths.

- Repository: https://github.com/facebook/wdt
- Website: https://www.facebook.com/WdtOpenSource
- Stars: 2,958 · Forks: 400
- Language: C++
- License: NOASSERTION
- Published: 2026-09-24 · Updated: 2026-09-24 · Language: en
- Canonical page: https://hysenlabs.com/projects/facebook-wdt

## The transfer problem WDT was written for

Single-stream TCP transfer tools leave bandwidth on the table on long, high-latency links, because one connection cannot fill the pipe while it waits for acknowledgements. WDT's answer is to open several TCP paths at once and treat the aggregate as the transfer. The README states the goal plainly: the lowest possible total transfer time, limited by hardware (disc or network bandwidth) rather than latency, with low CPU and memory use. The stated production case is a RocksDB snapshot moved between hosts at a throttled 600 Mbytes/sec across long distance, high latency links such as Sweden to Oregon, which the README describes as 3x the speed of the previous HTTP-based solution. WDT is aimed at engineers who own both ends of the wire and can run a process on each side. It is not a drop-in replacement for scp in a locked-down environment, and it is not a sync tool: the command line tool sends a directory tree, and the receiver writes it to a destination directory. The README also notes the project has been optimized for servers with fast IOs, in particular flash or in-memory read and write, and that disk throughput will not be as good.

## How WDT splits a transfer across threads and ports

The architecture is visible in the top-level file layout. SenderThread and ReceiverThread both inherit from WdtThread, which holds the settings they share. Sender and Receiver inherit from WdtBase, which holds what both sides need. On the sending side, a SourceQueue produces ByteSource objects, and DirectorySourceQueue is the concrete implementation that walks a directory and returns files sorted by decreasing size, so the largest file starts moving first. The README notes you can begin pulling from the queue before discovery finishes. FileByteSource wraps a file identified as a relative path from a root directory, and that relative path is what gets sent to the receiver. Each sender thread binds to a certain port, and the receiver side accepts connections on those ports. When a connection breaks, ThreadTransferHistory lets a thread ask the receiver how far into the history the data got, so the transfer can resume rather than restart. Protocol.cpp and Protocol.h define the wire format, WdtTransferRequest carries the connection details between the two sides, and WdtResourceController is an optional factory that caps how many senders or receivers exist at once. The design choices are deliberate: no exceptions, blocking thread IOs, and a preference for moderately structured C over heavy C++ features, all to keep control flow predictable and minimize system calls.

## Building facebook/wdt with CMake and running a first transfer

The README points to build/BUILD.md for build instructions and lists CMake as the build driver. It also lists the dependencies: gflags only for the command line tool, gtest only for tests, glog for logging, parts of Folly (conv, threadlocal and checksum support), and the crypto part of openssl-1.x for encryption. The README states you can build and embed wdt as a library with as little as a C++11 compiler and glog, and that glog could be macro-ed away or replaced with printing to stderr. The exact CMake invocation is not reproduced in the README, so read build/BUILD.md before running anything. Once the binary exists, the receiver starts first and names the destination directory:

```bash
wdt -directory /data/users/ldemailly/transfer1
```

The sender then names the source directory and the destination host:

```bash
wdt -directory /usr/bin -destination devbig074.prn2
```

The README shows the sender printing a progress bar and a summary line with transfer status, file count, data Mbytes, header overhead and throughput. In the README's own example, 1887 files and 1582.08 Mbytes moved at 588.816 Mbytes/sec, and the README cautions that this small-file case is slower than a large transfer because TCP still has ramp-up time. Note the terminology section: WDT prints Mbytes meaning 1024*1024 bytes, so the numbers are binary megabytes rather than decimal.

## Where WDT is the wrong tool

The README is candid about the storage side: WDT has been optimized for servers with fast IOs, and disk throughput will not be as good, with disk optimization listed as future work. If your source or destination is a spinning disk, the parallel TCP paths will not save you, because the bottleneck is the platter. The second limitation is packaging and maintenance. The most recent release listed in the repository metadata is v1.27.1612021, dated 2016-12-05, and the two before it are from 2016 as well. The last push to the default branch was on 2026-09-15, so commits continue, but the release tags do not reflect that activity. Anyone who pins versions or expects semantic versioning gets a stale tag. The third limitation is operational: WDT opens multiple TCP ports, so a firewall that permits a single port will break it, and the receiver must be started before the sender. The README does not document rollback, partial-failure cleanup, or what happens to a destination directory when a transfer is interrupted mid-file, which matters if you point it at a live data directory.

## WDT compared with rsync and scp

The honest comparison is with rsync, because both move a directory tree and both can resume. rsync's mechanism is a delta algorithm over a single connection: it compares checksums between source and destination and transfers only the changed blocks, which makes repeated syncs of a mostly-identical tree cheap. WDT does not do delta compression in the open source version; it sends whole files, and its resume logic works at the level of ThreadTransferHistory, tracking how far a thread got before a connection broke. That is a different trade-off: WDT wins on raw throughput for large, fresh datasets over high-latency links, and rsync wins on incremental updates and on working through a single port. scp is the simpler baseline, one stream, no resume, and the README's own numbers put WDT well above an HTTP-based approach for its production case. There is also wcp.sh in the repository, described as a script to use wdt like scp for single big files, which does the splitting for you pending splitting support inside wdt proper, and the README says to install it as "wcp".

## Licence, maintenance and upgrade cost

The repository metadata reports the license as NOASSERTION, which means the automated classifier could not match the LICENSE file to a known identifier. Read LICENSE yourself before shipping WDT inside a product; the README does not discuss licensing at all, and nothing here is legal advice. On maintenance, the picture is mixed and worth stating precisely: the repository is not archived, and the last push was on 2026-09-15, but the newest release listed is v1.27.1612021 from 2016-12-05. That gap means the tagged artifacts are old even though the tree is touched. For upgrade cost, the practical issue is that WDT is a C++ library you compile into your service, so a source update means rebuilding against the same glog, Folly and openssl-1.x dependencies listed in the README. The README says dependencies are kept minimal to maximize portability and keep the binary small, and that this also reduces compile time. If you embed the library, you also inherit the WdtOptions object, which the caller mutates to change behavior, while the standalone tool changes behavior through gflags in wdtCmdLine.cpp. Any option rename between versions lands on your code, not on a config file.

## Conclusion

Adopt facebook/wdt if you control both endpoints, can build C++ from source with CMake, and your bottleneck is a fast NIC rather than a spinning disk; the README states the project is optimized for flash or in-memory I/O. Do not adopt it if you need a packaged binary, a documented upgrade path, or a rolling release cadence, since the newest release listed is v1.27.1612021 from 2016-12-05. Before committing, verify that your toolchain satisfies the CMake and glog requirements in build/BUILD.md, and confirm whether the LICENSE file's terms fit your distribution plan, because the repository metadata reports the license as NOASSERTION.

## FAQ

### How do I build facebook/wdt?

The README directs you to build/BUILD.md and lists CMake as the build driver, with gflags for the command line, gtest for tests, glog for logging, parts of Folly, and the crypto part of openssl-1.x for encryption. The README states the library can be built with as little as a C++11 compiler and glog.

### Does facebook/wdt support resuming an interrupted transfer?

Yes, at the thread level. ThreadTransferHistory is described as letting a thread talk to the receiver after a connection breaks to find out how far into the history the data was sent. The README does not document rollback or cleanup of partially written files.

### What does Mbytes mean in facebook/wdt output?

WDT uses Mbytes everywhere in its output to mean 1024*1024 bytes, that is 1048576 bytes. The README acknowledges this is technically the mebibyte (MiB) standard but keeps Mbytes for readability.

## Sources

- [facebook/wdt on GitHub](https://github.com/facebook/wdt)
- [Issues](https://github.com/facebook/wdt/issues)
- [Project website](https://www.facebook.com/WdtOpenSource)
- [README](https://github.com/facebook/wdt/blob/main/README.md)
- [Releases](https://github.com/facebook/wdt/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/facebook-wdt