Gloo under the hood: collectives, rendezvous and a maintenance-only note

Where Gloo sits under the PyTorch distributed API

Gloo exists to give machine learning code collective primitives that work across several machines. The algorithms named in the README are a barrier, broadcast and allreduce, and every collective is exposed through a transport layer that hides whether data travels over plain IP or something faster. Read that sentence closely, because it defines the whole design: IP always works, and InfiniBand or RoCE gets used when the machines have it. Nobody at the call site picks a fabric.

The intended user is therefore not someone shopping for a messaging library. The project homepage points at the PyTorch distributed documentation, which means most people meet Gloo from inside a PyTorch training job rather than by linking to it directly. That placement has a consequence worth stating plainly. Gloo exposes no user-facing configuration surface of its own here; everything a user sets, such as the rank and the machine count, is passed down from the framework above it.

The primary language is C++, and the top-level layout shows the shape of the project: a `gloo/` source directory, a `cmake/` directory, a `docs/` directory for detailed documentation, and a `tools/` directory. There is no packaging manifest at the top level, so there is nothing to install from this repository.

GPU buffers skip the host copy, host buffers do not

The mechanism that matters most for training throughput is the doubled implementation. Where applicable, each algorithm has one implementation that works with system memory buffers and one that works with NVIDIA GPU memory buffers. For the GPU variant there is no need to copy memory between host and device, and the algorithm implementations take care of it.

That is a real saving, and it is also a maintenance cost that is easy to skim past. Two code paths per collective means two places for a bug, two places to test, and a third dependency set for anything that touches the second path. The optional dependency list names CUDA and NCCL for CUDA aware algorithms, tests, and benchmark, which puts the weight of that path on the NVIDIA stack rather than on Gloo itself.

The transport story extends the same idea one hop further. If the InfiniBand transport is used, GPUDirect can be used to accelerate cross machine GPU-to-GPU memory transfers. Gloo does not require that path: the README is explicit that IP is usable at all times, so a cluster without RDMA hardware degrades in transport rather than failing outright. Whether GPUDirect is actually available in a given build is not spelled out in the build instructions, so verify that against your own configuration before assuming it.

Building the tests: Google Test 1.8, Ubuntu 17.10, two binaries

The build path is CMake, and the tests have one real prerequisite beyond the compiler. Building them requires Google Test version 1.8 or higher. On Ubuntu that version ships with 17.10 and up, so a current system already satisfies it. An older system needs Google Test installed separately, and then the `GTEST_ROOT` CMake variable has to point at the custom install.

Where the packaged version is acceptable, conda covers both libraries in one command:

conda install -c anaconda gmock gtest

With that in place, the test build is the standard CMake sequence, with `BUILD_TEST=1` switched on:

mkdir -p build
cd build
cmake ../ -DBUILD_TEST=1 -DGTEST_ROOT=/some/path (if using custom install)
make
ls -l gloo/test/gloo_test*

The final ls is the check that matters: the binaries land under `gloo/test/`, and the glob is how you confirm the build produced something. Note the wording of the CMake line. The parenthesised note sits inside the command itself, which tells you the command is meant to be edited before use rather than pasted verbatim when you rely on a conda or system install.

For the CUDA algorithms, add `USE_CUDA=ON` as well. Those tests are built at `gloo/test/gloo_test_cuda`, a separate binary from the host-memory one, which confirms the split between the two implementations all the way down to the test layout.

Rendezvous through Redis, and why Redis Cluster is refused

The benchmark tool is the part of Gloo you can exercise without a PyTorch training loop, and it also shows the rendezvous constraint most clearly. Its dependency list is short: on Ubuntu, one package.

sudo apt-get install -y libhiredis-dev

The benchmark binary is built with its own flag and checked the same way the tests are:

mkdir build
cd build
cmake ../ -DBUILD_BENCHMARK=1
make
ls -l gloo/benchmark/benchmark

The benchmark tool depends on Redis via Hiredis for rendezvous, and it depends on both CUDA and NCCL when benchmarking the CUDA algorithms. Four steps precede a run: copy the tool to every participating machine, start a Redis server on any host including a client machine, pick a unique identifier for the run with the `uuid` tool or some number, then launch on each machine.

The hard limit is worth isolating: Redis Cluster is not supported. One reachable Redis endpoint is enough, and a clustered deployment is not an option.

./benchmark \
  --size <number of machines> \
  --rank <index of this machine, starting at 0> \
  --redis-host <Redis host> \
  --redis-port <Redis port> \
  --prefix <unique identifier for this run> \
  --transport tcp \
  --elements <number of elements; -1 for a sweep> \
  --iteration-time 1s \
  allreduce_ring_chunked

`--transport tcp` is the interesting flag here. Setting it explicitly is how you pin the run to IP even on fabric that could carry InfiniBand, which makes a result reproducible but also means the output below says nothing about the RDMA path. Setting `--elements` to -1 sweeps sizes instead of fixing one. The sample table was captured on 4 machines with a 40GbE network and reports min, p50, p99, max and sample counts per size: at 1000 elements the p50 reads 267 microseconds, and at 5000000 elements min is 10030 microseconds across 92 samples. Those numbers describe that one setup and cannot be carried to another network.

Vendoring the CMake project root beats installing a package

Gloo's own recommendation for consuming it is unusual and deliberate. Since it is a library, the README says it is most convenient to vendor it in your own project and include the project root in your own CMake configuration. That is a copy-the-tree workflow, not an install-then-link workflow, and it changes who owns updates.

Vendoring gives you a pinned source tree with a visible commit. It also gives you the merge work: every upstream change arrives as a conflict to resolve against your own build configuration, on a project whose README already asks maintainers to discuss new use cases. For a library small enough to read, that is a reasonable trade. For a fork with a fast-moving framework above it, it is a recurring cost with no version tags to guide it.

The top-level entries bear this out. There is a `CMakeLists.txt`, a `cmake/` directory with the build logic, `.clang-format` for formatting and `.tsan-suppressions` for thread sanitizer runs, plus `CLAUDE.md`, `CODE_OF_CONDUCT.md`, `CONTRIBUTING.md` and `SECURITY.md`. The `docs/` directory holds the detailed documentation the README points to. Nothing at the top level packages or publishes the library, so vendoring is not merely the easiest route, it is the only route the project describes.

Feature complete means new use cases need a conversation

The first substantive thing the README states is a scope limit, and it sits above the feature description in a highlighted note: Gloo is considered feature complete and in maintenance-only mode, and for new use cases or non-bugfix changes you should reach out to the maintainers to discuss.

Read together with the rest of the layout, that note explains a lot. There is no numbered version you can pin, track or diff against, and no grouped record of what changed when. Maintenance activity continues without producing something versioned for a consumer to consume. The last push was on 2026-09-17.

Practically, this means three things for an adopter. Bug fixes land without ceremony, so staying on the default branch is the only way to get them. New collective algorithms, new transports and new buffer types will not arrive because you need them. And any gap you hit in a PyTorch configuration above Gloo has to be patched in your vendored copy, because there is no feature request path that ends in a merged contribution.

Linux, libstdc++ and a README that names the old owner

The constraints are narrow and stated without hedging. Gloo is built to run on Linux and has no hard dependencies other than libstdc++. The README then concedes the obvious catch in the same breath: in practice it will generally only be useful when combined with one of the optional dependencies. Bare, Gloo gives you a collective library and nothing to coordinate the machines with, because rendezvous needs either Redis through Hiredis or MPI.

So the wrong-tool cases are concrete. Windows is out. A cluster without a Redis server or an MPI runtime has no documented rendezvous path. A workload that needs many algorithms beyond barrier, broadcast and allreduce will not find them listed. And anyone who wants a managed transport, a monitored library or a supported upgrade path is asking for something this project states it does not do.

One detail rewards a closer look, because the repository metadata and the README disagree about the terms: the repository is recorded with a custom licence that GitHub cannot classify, while the README states that Gloo is BSD-licensed. Those are not the same claim, and the difference matters more for a vendored C++ tree than for a remote dependency, since the tree lands inside your own source repository. Settle it against the file at the root of that tree before you commit to it.

A second detail sits in the links rather than the prose. The README's documentation link, the PyTorch distributed link and the CI badge all still point at `facebookincubator/gloo`, while the project now lives under the pytorch organisation. The build instructions themselves are unaffected, but the badge resolves to a workflow path in the old location, so treat green CI status there with care.

Editorial conclusion

Gloo is the right pick when you are already inside PyTorch distributed and want a small, vendorable collective layer you can read end to end; the maintenance-only note means features will not arrive on request, so fork it only if you can carry patches yourself. Do not reach for it on Windows, where it is unsupported, or as a standalone general-purpose transport, since rendezvous still needs Redis or MPI. Before vendoring the project root, settle the licensing question against the file at the root of the tree, and confirm your cluster can supply the Google Test 1.8 build path the test target requires.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. pytorch/gloo on GitHub
4. README