torchft: per-step fault tolerance for PyTorch training without stopping the world

The goal is to keep training through an error rather than restart it

torchft implements techniques for doing a per-step fault tolerance so you can keep training if errors occur without interrupting the entire training job. The tag line is Easy Per Step Fault Tolerance for PyTorch.

That framing is the whole point, and it is worth being precise about the difference from what exists in stock PyTorch. Distributed data parallel training is synchronising: every step, every rank has to participate, and if one rank dies the collective hangs and the job ends. Recovery means going back to the last checkpoint, which on a large model can mean losing hours of work and re-doing a lot of data.

torchft's answer is to detect the failure at step granularity and replace the failed worker, so the surviving workers keep going.

The project describes itself as providing the primitives required to implement fault tolerance in any application or train script, as well as the primitives needed to implement custom fault tolerance strategies. That is a deliberate framing. It is not only a training framework; it is a set of parts you can assemble into your own strategy.

Out of the box it ships Fault Tolerant DDP, Fault Tolerant HSDP with fault tolerance across the replicated dimension and any mix of FSDP, TP and others across the other dimensions, LocalSGD and DiLoCo. The last two are marked experimental, which matters when you are deciding what to build on.

The implementation is based on the large scale training techniques presented at PyTorch Conference 2024, and the repository links a poster for them. So the design is not theoretical: it comes out of running large training jobs.

The licence is recorded as a custom licence that GitHub cannot classify, with a LICENSE file at the root, and the homepage is meta-pytorch.org/torchft. The last push was on 2026-10-15 and the only release tag is v0.2.0 from 2026-08-27.

Three reusable components, and the lighthouse is the one that matters

The Overview section names three key reusable components, and understanding them tells you what the library actually is.

First, coordination primitives that can determine which workers are healthy via heartbeating on a per-step basis. Second, fault tolerant ProcessGroup implementations that report errors sanely and can be reinitialized gracefully. Third, checkpoint transports that can be used to do live recovery from a healthy peer when doing scale up operations.

The first is the heart of the design. A standard PyTorch process group has no notion of a member being dead; a rank that stops participating makes the collective hang until a timeout kills the job. A per-step heartbeat changes that, because membership becomes a question with an answer at every step rather than a failure you discover through a timeout.

The lighthouse is the coordinator that does this. torchft implements a lighthouse server that coordinates across the different replica groups, and then a per replica group manager and fault tolerance library that can be used in a standard PyTorch training loop.

The payoff is stated directly: this allows for membership changes at the training step granularity which can greatly improve efficiency by avoiding stopping the world training on errors.

Stopping the world is the expensive thing. In a synchronous job, one dead rank means every other rank waits. With the lighthouse, the group can detect the failure, agree a new membership, and continue, so the remaining workers are not idle for the length of a restart.

That also explains the checkpoint transports. If a worker is replaced, the new one needs the weights. Rather than having the replacement read from a shared filesystem checkpoint written before the failure, a live recovery can pull the state from a healthy peer over the transport. That is what makes recovery fast rather than merely possible.

The Rust dependency list backs the picture. tokio with the full feature set for the async runtime, tonic and prost for the gRPC service that the lighthouse exposes, axum as an HTTP server, pyo3 with the extension-module feature for the Python bindings, and askama for templating. So the lighthouse is a gRPC service in Rust, and the Python side talks to it through the extension module.

Running the lighthouse, and pointing a training loop at it

The lighthouse is started as a separate process, and the README gives the invocation.

RUST_BACKTRACE=1 torchft_lighthouse --min_replicas 1 --quorum_tick_ms 100 --join_timeout_ms 10000

The three flags are the parameters that decide how quickly the group reacts. min_replicas is how many replicas the group needs to consider itself healthy, quorum_tick_ms is the heartbeat tick, and join_timeout_ms is how long a worker has to join before it is given up on. The RUST_BACKTRACE=1 is there because the lighthouse is a Rust binary and a panic in it would otherwise be opaque.

Then the training loop points at it through an environment variable.

TORCHFT_LIGHTHOUSE=http://localhost:29510 torchrun --master_port 29501 --nnodes 1 --nproc_per_node 1 train_ddp.py

TORCHFT_LIGHTHOUSE gives the address of the lighthouse, and the rest is an ordinary torchrun invocation. Notice that the example uses a single node and a single process, which is a deliberate choice: it means you can see the mechanism working on a laptop before you put it across a cluster.

The Python side of the integration is what train_ddp.py shows, and it is short.

from torchft import Manager, DistributedDataParallel, Optimizer, ProcessGroupGloo

manager = Manager(
    pg=ProcessGroupGloo(),
    load_state_dict=...,
    state_dict=...,
)

m = nn.Linear(2, 3)
m = DistributedDataParallel(manager, m)
optimizer = Optimizer(manager, optim.AdamW(m.parameters()))

for i in range(1000):
    batch = torch.rand(2, 2, device=device)

    optimizer.zero_grad()

    out = m(batch)
    loss = out.sum()

    loss.backward()

    optimizer.step()

The integration surface is the Manager, and it takes a process group, a load_state_dict and a state_dict. Everything else is torchft's DistributedDataParallel and Optimizer wrappers around your normal module and your normal AdamW. The training loop body is unchanged from a plain loop, which is the design goal: you swap in a wrapper and a coordinator, and the step logic stays yours.

The Manager's load_state_dict and state_dict arguments are also where you plug in live recovery, since they are how a replacement worker obtains the weights it missed.

The README describes this as a minimal DDP train loop that highlights all of the key components in torchft, and the HSDP variant of the same idea is available out of the box in torchtitan, which provides an out of the box fault tolerant HSDP training loop on top of torchft that can be used to train models such as Llama 3 70B.

Installing means Rust, protoc and a PyTorch nightly, in that order

The build is not a pure Python install, and the prerequisites section is specific about it.

You need Rust with the necessary dependencies, protobuf-compiler and the corresponding development package for Protobuf, and PyTorch 2.7 RC+ or Nightly. The README also warns that the Rust versions available in many conda environments may be outdated, and recommends downloading the current one directly.

curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh

The protobuf packages are per distribution.

sudo apt install protobuf-compiler libprotobuf-dev

sudo dnf install protobuf-compiler protobuf-devel

The published wheels are the easy path.

pip install torchft

And there is a nightly package on PyPI for the versions that track main.

pip install torchft-nightly

With the development dependencies on top of that.

pip install torchft-nightly[dev]

Installing from source goes through maturin, and the pyproject.toml names it as the build backend with the pyo3 extension-module feature and the module named torchft._torchft.

pip install .

The README adds a specific troubleshooting note for the source path: if the installation command fails to invoke cargo update due to an inability to fetch the manifest, it may be caused by the proxy, proxySSLCert and proxySSLKey settings in your .gitconfig file affecting the cargo command, and the fix is to temporarily remove those fields from .gitconfig before running the installation.

That is a specific and useful piece of advice, because a corporate git configuration with a proxy is common and the failure mode, a cargo manifest that cannot be fetched, points at the network rather than at the git config.

For editable installs with the Rust extensions and development dependencies, the command is pip install -e '.[dev]'.

The Dockerfile shows the shape of the build environment, and it is short: it starts from the maturin image, then downloads protoc 25.1 for linux-x86_64 and unzips it into /usr. So the protobuf compiler is pinned to a specific version in the container path.

Where torchft is the wrong tool

The first wrong-tool case is a training job on one node or a small cluster where restarting is cheap. The whole value of torchft is that it avoids a restart, and it achieves that by adding a coordinator service, replacing process group implementations, and routing weight recovery through transports. On a job that takes minutes to restart from a checkpoint, that machinery costs more than it saves, and it adds a service that can itself fail.

The second case is a team that cannot operate a coordinator. The lighthouse is a separate process that must be reachable, must have a quorum configuration, and must be up before the workers start. In some environments, notably inside a managed training platform where you control only the container, that is not something you can deploy.

The third case is asynchronous or pipeline-parallel training, where the failure model is different. torchft is designed to allow for fault tolerance when using training with replicated weights such as in DDP or HSDP, and the fault tolerant HSDP implementation covers fault tolerance across the replicated dimension with any mix of FSDP, TP and others across the other dimensions. The design doc and the diagrams in the repository are for the replicated case, and there is nothing in the visible documentation about pipeline parallel.

The fourth case is a team that wants a fully finished recipe rather than primitives. The README says torchft provides the primitives required to implement fault tolerance in any application or train script, and the examples point at an integration test for LocalSGD and DiLoCo as the reference. You are expected to write the loop, or adopt torchtitan's. The two out-of-the-box algorithms that are ready, LocalSGD and DiLoCo, are labelled experimental.

On maintenance, the project has one release tag, v0.2.0 from 2026-08-27, against a last push of 2026-10-15. The Cargo manifest shows the same version 0.2.0, so there is a tagged artefact and current work. The library is young and its API is likely to move, which is why the nightly package exists alongside the stable one.

There is also a dependency floor worth checking. The pyproject requires torch>=2.7, and the README asks for PyTorch 2.7 RC+ or Nightly, which puts a recent nightly or release candidate into your environment. That is a real constraint for a production training stack, where you generally want a stable, pinned PyTorch.

The telemetry dependencies are worth knowing about too. The runtime dependencies include the OpenTelemetry exporter for OTLP over HTTP plus the SDK and API packages, both at 1.39.0 or newer. The lighthouse and the library emit metrics through that stack, which is useful, and it also means the package pulls an HTTP exporter into your environment whether or not you intend to export anything.

On licence, the repository records a custom licence that GitHub cannot classify, with the LICENSE file as the authority, and it is a Meta PyTorch project rather than the pytorch/pytorch repository. Anything beyond reading the code is a question for whoever reads that file.

Editorial conclusion

torchft suits a team training a large model across many machines where a single node failure costs more than the training run itself, since the lighthouse lets the group replace a failed replica at step granularity rather than restarting the job. It does not suit a single-node or small-cluster setup where a restart is cheap, and it does not suit a team that cannot add a coordinator service. Before adopting, read the design doc linked from the README, because the membership and quorum behaviour is where the design decisions are, and check that your training loop can accept a state dict and a load state dict through the Manager.

Frequently asked questions

Official sources

1. Issues
2. meta-pytorch/torchft on GitHub
3. Project website
4. README
5. Releases