# facebookresearch/optimizers: Shampoo and GPA-AdamW as research code

> The facebookresearch optimizers repository collects PyTorch optimization algorithms built for external collaboration, currently Distributed Shampoo and GPA-AdamW, and is packaged as torch-shampoo. It suits a research team studying preconditioning or low communication training who wants to read and modify the algorithm rather than call a black box.

**facebookresearch/optimizers** — For optimization algorithm research and development.

- Repository: https://github.com/facebookresearch/optimizers
- Stars: 583 · Forks: 67
- Language: Python
- License: NOASSERTION
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/facebookresearch-optimizers

## A repository for developing optimizers, not a catalogue of them

The facebookresearch optimizers repository is described as a GitHub repository of PyTorch optimization algorithms, designed for external collaboration and development. The phrase that matters is the last one. This is not a package that collects the optimizers you might reach for, it is a place where optimizer research happens.

Two algorithms are listed as included. Distributed Shampoo, which is a preconditioning method that keeps per-parameter matrix statistics, and GPA-AdamW, described as Generalized Primal Averaging. Both are research-grade choices with a specific purpose, and the repository exists to make them available and modifiable rather than to compete with the optimizers that ship inside PyTorch itself.

The audience follows. A researcher who wants to test a variation of Shampoo, or a team experimenting with low communication training across many machines, wants the implementation in a form they can read, patch and cite. Someone looking for a drop-in AdamW is in the wrong place.

The operational facts are modest and worth knowing before you start. The project is owned by Meta Platforms, the copyright line in the README says so, and the package name in the manifest is `torch-shampoo` even though the repository holds two separate optimizers. The Python floor is 3.12, the torch requirement is 2.8.0 or newer, and the classifiers mark the project as Development Status 3, Alpha.

The last push was on 2026-09-14, and there is exactly one release tag, v1.0.0, dated 2024-02-12. So the code is being worked on while the version history has not moved, which is a normal shape for a research repository but something to plan around.

## Distributed Shampoo and what grafting actually does

The usage example for Distributed Shampoo is short, and the interesting part is the last argument.

```python
import torch
from distributed_shampoo import AdamPreconditionerConfig, DistributedShampoo

model = ...  # Instantiate model

optim = DistributedShampoo(
    model.parameters(),
    lr=1e-3,
    betas=(0.9, 0.999),
    epsilon=1e-8,
    grafting_config=AdamPreconditionerConfig(
        beta2=0.999,
        epsilon=1e-8,
    ),
)
```

The constructor takes the model parameters, the usual Adam style learning rate, betas and epsilon, and then a `grafting_config`.

Grafting is the mechanism that makes a preconditioning method usable in the early phase of training. A full Shampoo update with matrix preconditioners is a heavier computation than plain Adam, and early in training the statistics a preconditioner depends on are not yet reliable, so the preconditioned direction can be worse than the adaptive one. Grafting lets the implementation borrow the direction from a simpler method, here the Adam preconditioner configured by `AdamPreconditionerConfig`, while the Shampoo machinery handles the rest.

The configuration object takes its own `beta2` and `epsilon`, which means the grafting method's own hyperparameters are set separately from the optimizer's. That separation is the point of the design, and it is the kind of detail that only shows up in a real implementation, not in a paper's abstract.

The README defers for the rest, pointing to the additional documentation in the Distributed Shampoo README and specifically to its How to Use section. The distribution aspect, which is the harder half of the algorithm, is therefore documented in a subdirectory rather than in the root document, and the root document does not describe the distributed setup at all.

## GPA-AdamW: two interpolation coefficients and a mode switch

GPA-AdamW is the second optimizer, and its usage example is even shorter.

```python
import torch
from gpa.gpa_adamw import GPAAdamW

model = ...  # Instantiate model

optim = GPAAdamW(
    model.parameters(),
    lr=1e-3,
    train_interp_coeff=0.7,
    eval_interp_coeff=0.9967,  # GPA mode (DiLoCo-equivalent of 32 local steps)
)
```

The two coefficients are the interesting part. `train_interp_coeff` and `eval_interp_coeff` set the interpolation used during training and during evaluation respectively, and the comment on the second one identifies the regime: at 0.9967 this is GPA mode, described as the DiLoCo equivalent of 32 local steps. The comment is a single line, but it tells you the whole point of the hyperparameter, which is that a higher interpolation coefficient approximates more local computation before the update is applied.

The README then states a requirement that will bite anyone who forgets it. GPA-AdamW requires switching between train and eval modes through `optimizer.train()` and `optimizer.eval()`, so that gradients are computed on the y-sequence and evaluation runs on the x-sequence. This is not a cosmetic API detail. If the mode is not switched, the interpolation is applied to the wrong sequence, and the failure is a silently degraded run rather than an exception.

For the full story the README points to the GPA-AdamW README for details on Schedule-Free versus GPA mode, hyperparameter tuning and distributed training. So the two algorithms are documented separately, each in its own directory with its own README, and the root document is the entry point rather than the manual.

## Installation is a clone and a pip install, with an extra for examples

The installation instructions use a git clone rather than a PyPI install, which is consistent with a repository meant to be developed in.

```bash
git clone git@github.com:facebookresearch/optimizers.git
cd optimizers
pip install .
```

There is a variant for people who also want to run the examples, and the README names where those examples live, in the Distributed Shampoo and GPA-AdamW directories. To include their dependencies, the last line of the install changes to the extras form.

The extras themselves are visible in the manifest, and they are more than a convenience. The examples group pulls in torchvision, tensorboard and hydra-core, which is a strong hint about how the examples are configured, since hydra is a configuration framework and tensorboard is a logging one. The dev group adds the examples group plus pre-commit, ruff, usort and mypy, so the project uses ruff for linting, usort for import sorting and mypy for types, with pre-commit to run them.

The manifest also shows the build and the packaging decisions. The build backend is setuptools with a version floor of 61.0, and the project metadata is minimal.

```toml
[project]
name = "torch-shampoo"
version = "1.0.0"
dependencies = [
    "torch>=2.8.0",
]
requires-python = ">=3.12"
```

A single runtime dependency on torch, and a hard Python floor of 3.12, is the whole requirement list. The setuptools configuration then shows something that explains the package layout: `py-modules` lists four top level modules, `commons`, `matrix_functions`, `matrix_functions_types` and `optimizer_modules`, and the package discovery includes `distributed_shampoo*` while excluding examples and tests. The GPA implementation is therefore not packaged the same way as Shampoo, which matches the fact that the two are documented as separate pieces.

## Continuous integration, licence details and the wrong-tool cases

The badges at the top of the README describe the continuous integration, and there are five separate workflows rather than one. There is a tests workflow, a gpu-tests workflow, a pre-commit workflow, a type-check workflow and an examples workflow. The split matters in a research repository, because the expensive checks and the cheap checks fail at different times, and a contributor with no GPU still gets the fast feedback.

The licence needs a note. The README says the project is released under the BSD license and links to the LICENSE file, while the licence field for the repository on GitHub is reported as a custom licence that GitHub cannot classify. The MIT style classifiers are not used, and the manifest points at the LICENSE file rather than embedding a licence expression, so the text in that file is the authority. This is not legal advice, and the discrepancy is probably a classifier or detection issue rather than a substantive one, but a company that scans licences will not read it that way and should look at the file.

There are also governance files, with a CONTRIBUTING.md that the README points to as the place to start if you want to help, a CODE_OF_CONDUCT.md and a CHANGELOG.md, plus a `makefile` at the top level, a `distributed_shampoo/` directory and a `gpa/` directory.

The cases where this repository is the wrong tool are clear. A production training stack that needs a stable optimizer with a support contract should use what ships in PyTorch, because the single v1.0.0 tag from February 2024 and the Alpha status classifier are the honest signals of maturity here. A team on Python 3.11 cannot install it at all, given the 3.12 floor, and a team pinned to an older torch cannot satisfy the 2.8.0 requirement.

And the honest research caveat applies. Both optimizers exist because the standard adaptive methods are suboptimal in specific regimes, which means they are also the methods where the defaults are least likely to be right for your problem. The documentation for each one exists precisely because the hyperparameter choices need explaining, and the README is candid that tuning guidance lives in the subdirectory READMEs rather than in the summary.

## Conclusion

facebookresearch/optimizers is a good fit for a research group that intends to study or adapt preconditioning and interpolation schemes, and that is comfortable working inside a research repository with a single listed release and a BSD license file. It is the wrong choice for a production training stack that needs a stable, supported optimizer with a release cadence, and for a team on Python 3.11 or an older torch. Verify first which of the two optimizers matches your problem, since Distributed Shampoo and GPA-AdamW solve different ones, read the per-algorithm READMEs that the root document defers to, and check the LICENSE file yourself, because the licence field GitHub reports does not match the licence named in the README.

## FAQ

### What optimizers does the facebookresearch/optimizers repository provide?

Two: Distributed Shampoo and GPA-AdamW, which is described as Generalized Primal Averaging. The repository is described as a place for PyTorch optimization algorithms designed for external collaboration and development.

### How do I install the facebookresearch optimizers?

Clone the repository with `git clone git@github.com:facebookresearch/optimizers.git`, change into the optimizers directory and run `pip install .`. To include the examples for Distributed Shampoo and GPA-AdamW, use `pip install ".[examples]"` instead of the last line.

### What does the grafting_config option do in Distributed Shampoo?

It selects the method used for grafting, and the example passes an AdamPreconditionerConfig with its own beta2 and epsilon, so the grafting method's hyperparameters are set separately from the optimizer's own betas and epsilon.

### What is the difference between train mode and eval mode in GPA-AdamW?

The README says GPA-AdamW requires switching between them with optimizer.train() and optimizer.eval() so that gradients are computed on the y-sequence and evaluation runs on the x-sequence. The two interpolation coefficients, train_interp_coeff and eval_interp_coeff, set the behaviour in each mode.

### What are the Python and PyTorch requirements for the facebookresearch optimizers?

The manifest requires Python 3.12 or newer and torch 2.8.0 or newer, and torch is the only runtime dependency. The project is packaged as torch-shampoo and is classified as Development Status 3, Alpha.

### Under which licence is the facebookresearch optimizers released?

The README states the project is released under the BSD license and links to the LICENSE file in the repository. That file holds the authoritative terms, since the licence reported for the repository is described as a custom one that GitHub cannot classify.

## Sources

- [facebookresearch/optimizers on GitHub](https://github.com/facebookresearch/optimizers)
- [Issues](https://github.com/facebookresearch/optimizers/issues)
- [README](https://github.com/facebookresearch/optimizers/blob/main/README.md)
- [Releases](https://github.com/facebookresearch/optimizers/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/facebookresearch-optimizers