OpenDiLoCo: the training code for low-communication distributed pretraining

A paper's code, with a notice that it is finished

The repository contains the training code and the experiment results for the paper OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training, which is linked to its arXiv entry, and the project is written in Python with an Apache 2.0 licence.

The first thing the README says is the most important fact about it. There is an important notice that OpenDiLoCo is no longer maintained, and for a production-ready distributed training solution it points at prime, the successor project, which the notice says offers improved fault tolerance, bandwidth utilization and scalability.

That makes the repository's role clear. It is a released artefact of a research paper, kept for reference and for reproduction, rather than a system under active development. The last push was on 2026-07-17, the repository is not archived, and there are no GitHub releases, so there is no version history to track and the code sits in the state the paper used.

The audience is therefore narrow and specific: someone who wants to understand or reproduce a low-communication training setup, or a reader who wants to see how the approach is wired up in practice. The main content is the open_diloco directory with the training script, the run_training.sh helper and the experiment configurations the README describes as the ones used for the results in the paper.

The paper link is on the project homepage, and the repository also carries a Dockerfile, a scripts directory, tests, a pre-commit configuration, a requirements file and a development requirements file.

The mechanism: many local steps, then one averaging round

The design of DiLoCo, and therefore of this code, is visible in the flags of the experiment commands rather than in prose.

Each worker runs a number of local steps before it talks to anyone, set with the --hv.local-steps argument, and the README shows two configurations for the same 150m model, one with 500 local steps and one with 50. The workers then exchange weights rather than gradients, which is what makes the scheme low-communication: the network traffic is a weight update every few hundred local steps instead of every step.

The averaging itself is done by the hivemind library, which performs distributed weight averaging. The README shows the alternative as well, a plain DDP baseline run with torchrun and train_fsdp.py directly, using the same model and the same total batch size of 512, the same 88,000 total steps, the same bf16-mixed precision and the same 4e-4 learning rate. The baseline exists so the low-communication variant has something to be compared against on the same terms.

The parallel implementation is FSDP, and the sharding strategy is a flag. The 150m runs use NO_SHARD, and the 1b baseline uses the hybrid sharded zero 2 strategy, with a total batch size of 8192 instead of 512 and a per device batch size of 16. The model paths point at PrimeIntellect's llama-150m-fresh and llama-1b-fresh checkpoints on the hub.

The README is explicit that the torchrun arguments and the per device train batch size can be changed to match your machine without affecting the final results unless stated otherwise, which is a useful property: the experiment is a recipe you can resize to your hardware.

Hivemind needs a DHT and one reachable initial peer

The distributed averaging needs a peer to peer network underneath it, and the README explains that some of the experiments use the hivemind library to perform the weight averaging, which requires a Distributed Hash Table.

The DHT has to be initialised, and an initial peer is required to do that. That initial peer must be reachable from all the machines taking part in the distributed training, and on the chosen machine you run the DHT instance with a fixed identity file and the address it should listen on.

hivemind-dht     --identity_path fixed_private_key.pem     --host_maddrs /ip4/0.0.0.0/tcp/30001

The output lists the visible multiaddresses, and the README explains that the multiaddress strings printed after the visible multiaddresses line are the ones you use to initialise your training processes. The PEER environment variable is then set to one of those strings, and every worker gets the same value.

Two practical points follow. The initial peer is a single point of failure at startup, so the machine running it has to be up and reachable before the other nodes start, and a private key file has to exist on it, since the identity path is given explicitly rather than generated. And because the peer address is baked into the launch command, moving the initial peer means re-exporting PEER on every worker.

Two environment paths, and a container that assumes GPUs

There are three documented ways to get a working environment, and they differ in how much they assume about your machine.

The first is to clone the repository with its submodules, which matters because the experiment scripts depend on content that is not in the main tree.

git clone https://github.com/PrimeIntellect-ai/OpenDiLoCo.git --recursive

Then the environment is created, either with conda on Python 3.11 or with a plain virtual environment.

conda create -n OpenDiLoCo python=3.11 -y && conda activate OpenDiLoCo

python -m venv .venv
source .venv/bin/activate

The dependencies are installed with pip, and one of them comes from the PyTorch nightly index.

pip install .
pip install --pre torchdata --index-url https://download.pytorch.org/whl/nightly/cpu

Flash Attention 2 is optional and needs a CUDA compiler on the system.

# (Optional) flash-attn
pip install flash-attn>=2.5.8

The alternative is a pre-built image, and the command is worth reading for what it assumes.

docker pull primeintellect/open_diloco:main
docker run -d --name open-diloco --ipc=host --network=host --gpus=all primeintellect/open_diloco:main
docker exec -it open-diloco bash

The container is run with all GPUs passed through, host networking and host IPC, which is the configuration a distributed training job needs for shared memory and for the peer to peer networking to work between containers. The image itself is based on the PyTorch 2.3.1 CUDA 12.1 cuDNN 8 development image, so it assumes an NVIDIA host.

Resuming and stopping are the two things you will need

Two operational details in the README are worth pulling out, because both are the kind of thing that only matters when a run has been going for hours.

The first is stopping. The current implementation of hivemind does not handle the Ctrl+C keyboard interrupt well, and the README says you can stop the runs using pkill.

pkill -f torchrun

That is a blunt instrument, since it matches every torchrun process on the machine rather than your own run, but on a dedicated node it is the documented way out of a run that will not respond to an interrupt.

The second is resuming. The training script takes a --resume-from-checkpoint argument pointing at a checkpoint directory, and the README shows a run that resumes from a saved model step.

torchrun --nproc_per_node=8 \
    train_fsdp.py \
    ...
    --resume-from-checkpoint checkpoints_1b/2024-06-20/hivemind_1b/bm5zjkzr/model_step_6000

The checkpoint path in that example encodes the date, the run and the model step, and the runs also write checkpoints on an interval through the --ckpt.interval flag with a destination directory set by --ckpt.path. Because the checkpoint naming is derived from the run, the layout is predictable, and you can point a new run at an old directory rather than at a random file.

The other helper worth knowing is run_training.sh, which the README explains calls train_fsdp.py eight times with the right arguments to simulate eight workers locally. That is how the low-communication configurations are demonstrated on a single machine, and it is the fastest way to see the method work before you attempt a real multi-machine run.

What the pins and the licence say about using this code

The requirements file is where the age of the project shows. The dependencies are transformers at roughly 4.40, datasets at 2.19.1 or newer, wandb at 0.16.4 or newer, cyclopts at 2.6.1 or newer, fsspec with GCS support, torch pinned exactly to 2.3.1, and then two dependencies taken directly from git: hivemind at a specific commit of the learning-at-home repository and pydantic_config at a specific commit of another fork.

The torch pin at 2.3.1 is the one to look at first, because it fixes the CUDA and hardware behaviour of the whole stack, and the Docker image matches it. Pinning hivemind and pydantic_config to commits rather than releases means the behaviour of the experiment is reproducible, at the cost of those two packages having no upgrade path. The pyproject is otherwise minimal, with a setuptools build, the package list set to open_diloco, the dependencies read dynamically from requirements.txt, and a ruff line length of 120.

The licence is Apache 2.0, which permits use, modification and redistribution including in commercial products provided the notice travels with the code.

The realistic assessment is that this is code to read and to reproduce, not code to operate. The successor project named in the notice, prime, is where the production path is, and the notice says it offers improved fault tolerance, bandwidth utilization and scalability, which are exactly the three things a research code release cannot promise. The difference in approach between the two is that OpenDiLoCo is a paper implementation with fixed experiment configurations, while the successor is positioned as a system, and the difference in what you inherit is the difference between a fixed torch and a maintained one.

Editorial conclusion

OpenDiLoCo fits a researcher who wants to reproduce or study a low-communication training setup, and it does not fit a team that needs a maintained distributed training system, since the README states it is no longer maintained and points to prime instead. Before running anything, read the requirement pins, because torch and the hivemind and pydantic_config dependencies are fixed to specific versions and git commits, and those pins are the first thing that will break.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. PrimeIntellect-ai/OpenDiloco on GitHub
4. Project website
5. README