torchforge: PyTorch-native reinforcement learning infrastructure

The problem is the infrastructure, not the algorithm

torchforge exists to separate infra concerns from model concerns so that reinforcement learning experimentation is easier. The README frames it as a PyTorch-native agentic RL library that lets you focus on algorithms rather than infrastructure, and it delivers that by providing clear RL abstractions plus one scalable implementation of them. The distinction the project draws is between two kinds of need. When you want fine-grained control over placement, over fault handling and redirecting training loads during a run, or over communication patterns, the primitives are there. When you do not want that control, you can concentrate on the RL algorithm itself. The three stated goals follow from that: usability for rapid research, isolating the RL loop from infrastructure; hackability, since every part of the loop can be modified without interacting with infrastructure; and scalability, the ability to shift between async and synchronous training and across thousands of GPUs. The hosted documentation lives at meta-pytorch.org/torchforge, and a Discord badge points to the OpenEnv community channel.

A pinned dependency set that defines the shape of the project

The pyproject.toml shows the project name is forge, described as a PyTorch native platform for post-training generative AI models, and it requires Python of at least 3.10 and under 3.13. The dependency list is where the architectural choices are visible. It pins `torch==2.9.0`, `torchdata>=0.8.0`, `torchtitan==0.2.0` and `torchmonarch==0.2.0`, requires `vllm>=0.13.0,<0.14.0`, and pulls in `torchstore`. The comment next to torchstore in the file refers to switching to a nightly build, which tells you the dependency set is expected to move. Around the core sit the usual research stack: `datasets>=2.21.0` and `tokenizers` for Hugging Face integrations, `omegaconf` for configuration, `wandb` for experiment tracking and `hf_transfer`. The optional dependency groups mirror that structure, with a dev group carrying pytest, pytest-cov, pytest-timeout, tensorboard, expecttest, anyio, pytest-asyncio and multiprocess, and a docs group with sphinx and the extensions it needs. The repository layout shows the same split: `src/` for the library, `apps/` for runnable examples, `benchmarks/`, `experimental/`, `tests/`, `scripts/`, `docs/` and `assets/`.

Installing with conda, and the ROCm path

The README states that torchforge requires PyTorch 2.9.0 with Monarch, vLLM and torchtitan, and the documented install path is conda. You create an environment on Python 3.12, activate it and run the bundled install script:

conda create -n forge python=3.12
conda activate forge
./scripts/install.sh

ROCm users get a dedicated script with the same first two steps:

conda create -n forge python=3.12
conda activate forge
./scripts/install_rocm.sh

The ROCm notes are the useful part. `PYTORCH_ROCM_ARCH` and `ROCM_VERSION` are auto-detected where possible and can be set manually, with gfx942 given as an example architecture. For ROCm 7.x, PyTorch stable 2.9.0 is not available, so the script defaults to nightly wheels, and you can override with `PYTORCH_CHANNEL=stable|nightly`. ROCm builds install Monarch with `USE_TENSOR_ENGINE=0`, which means RDMA and distributed tensor features are disabled for now, and the script accepts optional flags `--use-sudo` for system packages and `--skip-amdsmi`. The script installs system dependencies as well and uses DNF, which the README notes could be extended to other Linux distributions.

Pixi, and the uv gap the project admits to

There is a second supported path through Pixi, which the README describes as combining the benefits of uv with access to conda forge for system dependencies. The repository ships a pixi.toml manifest with build tasks, where install is the combined install all task. Installing Pixi itself is a curl one-liner:

curl -fsSL https://pixi.sh/install.sh | bash

And the install of torchforge itself is a single command:

pixi run install

The honest note in the README is that the team is actively working on enabling pure uv installation, that conda is the recommended approach at the moment, and that uv support is not fully working but is tracked in a GitHub issue. The install script also has a system packages variant, since by default the packages installation uses conda and you can pass `--use-sudo` to install system packages on the target machine instead, for example `./scripts/install.sh --use-sudo`.

Your first run is a GRPO job on two GPUs

After installation the README gives a concrete check. Running the GRPO application with a YAML config should print output confirming that GRPO training is running, and the config is for a Qwen3 model at 1.7B:

python -m apps.grpo.main --config apps/grpo/qwen3_1_7b.yaml

The caveat is a hardware minimum, because you need at least two GPU devices. Supervised fine tuning has its own entry point in the same apps layout, this one for a Llama3 8B model:

python -m apps.sft.main --config apps/sft/llama3_8b.yaml

Both are module invocations with a config file rather than a long argument list, which is consistent with the omegaconf dependency in the project metadata. The `apps/` directory at the top level of the repository is where these live, and the same directory carries the config YAMLs, so the intended workflow is that you copy or edit a config and run the module against it.

Why the abstractions, and what the repo layout implies

The value of the abstraction layer is best read against what a research team otherwise writes itself. A typical RL training loop needs a policy under training, a rollout engine to generate experience, a way to place those rollouts across accelerators, an environment or simulator, and a training step that consumes the results. When the environment is remote or the rollout engine sits on a different cluster, the loop also needs fault handling, the ability to redirect training loads mid-run, and explicit communication patterns. The README says those primitives exist and that all parts of the RL loop can be modified without interacting with infrastructure. The `experimental/` directory is the visible consequence of that hackability claim, and the `benchmarks/` directory is the visible consequence of the scalability claim. The architecture here is a set of seams rather than a single pipeline, which is the right shape for research code and a poor shape for a product someone else has to maintain.

The development pause is the first fact to weigh

The README opens with a notice that development in Forge has paused, and that LLM training at PyTorch is being consolidated in torchtitan. That single line changes how you should read everything else. The pinned dependency set already references torchtitan, the repository publishes no GitHub releases, and the version is carried in a version.txt file at the top level rather than in release tags. For an existing user this means dependency versions will not move as they otherwise would, and the roadmap for new features is elsewhere. For someone starting a new project, the sensible read is that the functionality is moving to torchtitan rather than that it is being extended here. The library is BSD 3 licensed, though the README points out that other content linked from the repository, such as third-party data and models, carries its own licence and terms of service, so the code licence is not a blanket clearance for everything the project references.

Editorial conclusion

torchforge suits researchers who want to write a reinforcement learning algorithm and have placement, rollout and communication handled for them, and who value being able to replace any part of the loop. It is the wrong starting point for new work, because the README states that development in Forge has paused and that LLM training at PyTorch is being consolidated in torchtitan. Before you adopt it, check the pinned dependency set in pyproject.toml, in particular torch==2.9.0, torchtitan==0.2.0, torchmonarch==0.2.0 and vllm>=0.13.0,<0.14.0, and confirm the two GPU minimum for the GRPO example applies to your hardware. The last push was on 2026-09-09 and the repository publishes no GitHub releases, so versioning lives in version.txt.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. meta-pytorch/torchforge on GitHub
4. README