# YaFSDP: Yandex's sharded data parallel framework for transformer training

> YaFSDP is a sharded data parallelism framework from Yandex, built for transformer like architectures and aimed at cutting communication and memory operation overhead. The README backs it with a benchmark table against FSDP on 64 to 256 GPUs, and the package itself is small, a torch dependency, a strict mypy configuration and a justfile for checks.

**yandex/YaFSDP** — YaFSDP: Yet another Fully Sharded Data Parallel

- Repository: https://github.com/yandex/YaFSDP
- Stars: 986 · Forks: 49
- Language: Python
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/yandex-yafsdp

## A sharding framework aimed at transformer architectures

YaFSDP expands to Yet another Fully Sharded Data Parallel. The README describes it as a Sharded Data Parallelism framework designed to work well with transformer like neural network architectures, and states that it is developed and maintained by Yandex.

The name is honest about the category. This is not a model, a trainer or a serving stack. It sits at the layer where a large model is split across devices and the training step has to coordinate gradients and parameters across all of them. The target is clear from the framing: pretraining and fine tuning of large language models, where the parameter count makes sharding mandatory rather than optional.

The design goal is stated as reducing communications and memory operations overhead, and that phrasing is the key to understanding what the project claims. In a sharded setup, the theoretical work is small compared to the cost of moving tensors between devices, so an implementation that changes when and how data moves can move the wall clock time of an iteration more than any amount of tuning inside the model would.

For the internals, the README points to two blog posts, one on Medium and one on Habr. That is the honest answer to how the mechanism works: the repository itself is a thin package around the interesting design work, and the writing is where the design is explained.

## The benchmark table and what it actually measures

The README compares YaFSDP with FSDP across pre-training setups ranging from 7B to 70B parameters, 64 to 256 devices, and 2048 to 8192 tokens per sequence, and it publishes the raw numbers rather than a summary.

Several rows are worth reading closely. For Llama 2 34B on 256 GPUs at 2048 tokens, the table gives a 21.99% speedup with an iteration time of 3.51 seconds against 4.50 seconds for FSDP. For Llama 2 70B on 256 GPUs at 2048 tokens with 10 checkpointed layers, the speedup is 21.48%, with 6.97 seconds against 8.87 seconds. The largest figure in the table is 26.60% for Llama 3 70B on 256 GPUs at 2048 tokens with 20 checkpointed layers, where iteration time goes from 9.76 seconds to 7.17.

That last number is worth flagging against the headline. The README says YaFSDP is up to 20% faster, while the table contains a 26.60% entry, so the summary and the data are not quite aligned. Read the table rather than the sentence.

The conditions are equally important. Each run uses a per device batch size of 1, speedup means the relative iteration time decrease between YaFSDP and FSDP runs, num-ckpt-layers is the number of transformer layers to which activation checkpointing was applied, and performance was measured on a cluster of hosts with A100 80 GB GPUs.

## The gains shrink with longer sequences and with checkpointing

The benchmark table has an internal pattern that is more informative than the headline number, and it is the part you should use when deciding whether the result applies to your setup.

Take Llama 2 7B on 64 GPUs. The speedup is 9.92% at 2048 tokens, 3.43% at 4096 tokens and 2.68% at 8192 tokens. The same model on 128 GPUs gives 9.57%, then 2.42%, then 2.32%. The gap narrows as the sequence gets longer, and it narrows in exactly the regime where a single training step already takes more than two seconds, which is the Llama 2 7B row at 8192 tokens where iteration time is 2.23 seconds against 2.29.

The same pattern shows up at the top end. Llama 2 70B on 256 GPUs at 2048 tokens gives 21.48%, while at 4096 tokens with 50 checkpointed layers it gives 7.17%, with 11.07 seconds against 11.93 seconds. Llama 2 34B on 256 GPUs at 2048 tokens gives 21.99%, and at 4096 tokens with 5 checkpointed layers it gives 8.35%.

The reading is that YaFSDP's advantage is largest when the per-iteration work is small relative to the coordination cost, and shrinks when the model is doing heavy computation per token or when activation checkpointing changes the memory traffic profile. If your workload sits in the second half of that table, the benefit will be smaller than the 20% headline suggests, and if your model is smaller than 7B or your cluster is under 64 devices, nothing in the table tells you what to expect.

## A small package: torch, Python 3.12 and strict typing

The distribution is deliberately thin. The project metadata names the package ya-fsdp at version 0.1.0, requires Python 3.12 or newer, and lists torch as its only runtime dependency. Development extras add codespell, mypy, ruff and torchfix.

The build uses uv_build, with the uv_build requirement pinned to a narrow range, and the module root is set to the package directory. The repository root holds the ya_fsdp/ package directory, docs/, examples/, docker/, patches/, assets/, a pyproject.toml, a justfile, a CITATION.cff and a .devcontainer.json.

The quality tooling is configured strictly. Ruff is the linter, with an extensive rule selection covering bugbear, comprehensions, pydocstyle, refurb, logging, naming, performance, pylint, raise, ruff specific rules, type checking, tidy imports, pyupgrade and warnings, with docstring and some pylint rules ignored. Mypy runs in strict mode over the ya_fsdp directory, with the two large generated or excluded modules ya_fsdp/ya_fsdp.py and ya_fsdp/meta_param.py excluded from both ruff and mypy, which tells you where the bulk of the code sits and that it is not hand written. A separate mypy override ignores missing imports for the yccl.* modules.

The justfile is the developer entry point, with recipes for format-check, format-fix, lint-check, lint-fix, type-check and spell-check, a combined check target and a fix target, and test running pytest.

## The examples need a patched Hugging Face stack in Docker

The README points to two example documents in the examples folder, clm.md for causal pre-training and sft.md for supervised fine-tuning, and it flags a requirement that catches people out.

Both examples require a Docker image, which is built using the docker/build.sh script. The image is based on the NVIDIA PyTorch image, with the release notes linked for that base, and with some patched Hugging Face libraries. The patches for those libraries are in the patches folder.

That constraint is the real adoption cost of YaFSDP. The framework itself is a package you import, but the examples that demonstrate it are built around a container with modified library dependencies, which means the path from the README to a running job involves building an image and accepting patched versions of the training stack. That is a meaningful difference from a package you can add to an existing environment.

The examples folder also holds fsdp_config.yaml, which sits alongside the two markdown guides and points at the configuration shape the examples expect. There is no equivalent YaFSDP configuration file shown in the repository listing, so the two guides and the Docker image are the specification for how the framework is meant to be driven.

## FSDP is the baseline, and the comparison is about overhead

YaFSDP is measured against FSDP, and the README places two diagrams side by side, ya_fsdp.png and fsdp.png, to show the two approaches.

The difference in approach, as the README describes it, is about overhead rather than about what is sharded. Both approaches fully shard, and both are aimed at the same problem of holding a model that does not fit on one device. What YaFSDP changes is the cost of the machinery around the sharding: fewer and cheaper communication operations, and fewer memory operations, in a framework whose design targets transformer architectures specifically rather than being general purpose.

That framing has consequences for how you should evaluate it. A general purpose sharding layer has to make compromises to stay general. A framework written for one architecture family can assume things about the model, for example how layers are structured and how gradients flow, and can specialise the communication pattern around those assumptions. The claim is that this specialisation is where the iteration time reduction comes from.

It also means the comparison in the table is a comparison of two frameworks rather than of two configurations of one framework, and the numbers were taken on A100 80 GB hardware with a per device batch size of 1. A different interconnect, a different model family or a different batch size is a different experiment, and the README does not claim otherwise.

## Version 0.1.0, no releases, and a citation request

The version number is the honest headline about maturity. The package is at 0.1.0, the repository publishes no GitHub releases, and the last push was on 2026-10-19 on the main branch. If you depend on it, you pin a commit or a package version, because there is no release stream to track.

The project is developed and maintained by Yandex, and the README asks that anyone using the codebase cite it, with a BibTeX entry for the paper style reference:

```bibtex
@misc{YaFSDP2024,
  author =       {Mikhail Khrushchev and Anton Frolov and Ruslan Vasilev},
  title =        {YaFSDP: Yet another Fully Sharded Data Parallel},
  howpublished = {\url{https://github.com/yandex/YaFSDP}},
  year =         {2024}
}
```

The licence is Apache-2.0, which is permissive and includes an explicit patent grant, so embedding it in an internal training stack carries no copyleft obligation and no licensing fee.

Support is handled through GitHub issues, and the internals are documented in blog posts rather than in the repository. Given the version number and the release situation, the realistic read is that this is a framework from a company that uses it internally, published so that others can try it, with the examples and the blogs doing the work that a fuller documentation site would do in a more established project.

## Conclusion

Adopt YaFSDP if you pre-train or fine-tune transformer models on many GPUs and can run the patched Hugging Face stack in the Docker image the README describes, and if the benchmark table's configuration matches yours closely enough to trust. Do not expect a drop-in replacement for FSDP without that image, since the examples require patched libraries. Before starting, check that you are on Python 3.12 or newer, note that the package version is 0.1.0 with no GitHub releases and the last push was on 2026-10-19, and read the Apache-2.0 licence terms.

## FAQ

### What is YaFSDP and who maintains it?

YaFSDP is a sharded data parallelism framework designed for transformer like neural network architectures, and the README states it is developed and maintained by Yandex.

### How much faster is YaFSDP than FSDP according to the README?

The README says up to 20% faster for pre-training LLMs, and the benchmark table lists speedups from 2.32% to 26.60% across Llama 2 and Llama 3 setups on 64 to 256 A100 80 GB GPUs, with per-device batch size 1.

### What do the YaFSDP examples require to run?

Both examples, clm.md for causal pre-training and sft.md for supervised fine-tuning, require a Docker image built with the docker/build.sh script, based on the NVIDIA PyTorch image with patched Hugging Face libraries taken from the patches folder.

### What are the YaFSDP package requirements and dependencies?

The package is named ya-fsdp at version 0.1.0, requires Python 3.12 or newer, and depends on torch at runtime, with codespell, mypy, ruff and torchfix as development extras.

### What licence does YaFSDP use and is there a released version?

The project is licensed under Apache-2.0. The repository publishes no GitHub releases, so the package version 0.1.0 and the commit you choose are the only version markers, and the last push was on 2026-10-19.

## Sources

- [Issues](https://github.com/yandex/YaFSDP/issues)
- [License: Apache-2.0](https://github.com/yandex/YaFSDP/blob/main/LICENSE)
- [README](https://github.com/yandex/YaFSDP/blob/main/README.md)
- [yandex/YaFSDP on GitHub](https://github.com/yandex/YaFSDP)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/yandex-yafsdp