rl4co, a reinforcement learning benchmark for combinatorial optimization

A benchmark, which is a different promise from a library

RL4CO is described as an extensive Reinforcement Learning benchmark for Combinatorial Optimization, and the stated goal is to provide a unified framework for RL-based CO algorithms and to facilitate reproducible research in the field, decoupling the science from the engineering.

That last phrase is the whole design brief. In practice, RL papers for routing and scheduling problems tend to differ in their environment implementations, their training loops, their evaluation protocol and their baselines, which makes cross-paper comparison unreliable. A benchmark fixes the surrounding scaffolding so that the variable is the policy.

The library is built on four foundations, each chosen for a specific reason. TorchRL is the official PyTorch framework for RL algorithms and vectorized environments on GPUs, which matters for a benchmark where throughput decides how many configurations you can run. TensorDict handles heterogeneous data such as states, actions and rewards. PyTorch Lightning wraps the training loop. Hydra configures the experiments.

The topics attached to the repository name the problem families it covers: the vehicle routing problem and the travelling salesman problem, scheduling, and electronic design automation. The environment and policy examples live under examples/ in the repository, with separate directories for datasets, modeling and advanced usage, and a README that indexes them.

The licence is MIT, the current version is 0.7.0, and Python 3.10 or newer is required. The package metadata classifies it as Development Status 4 - Beta, with an intended audience of developers and of science and research.

Constructive policies and improvement policies

The policy implementations are organised by what the policy does to a solution rather than by architecture, and the division has two branches.

Constructive policies learn to construct a solution from scratch. Within that, the autoregressive variant constructs solutions one step at a time through a decoder, which is the family that includes the attention model and the POMO policies. The non-autoregressive variant learns to predict a heuristic, such as a heatmap, and then constructs the solution from that prediction, which is cheaper at inference time and is the natural fit when you want a fast first solution rather than a sampled one.

Improvement policies learn to improve a pre-existing solution, which is a different problem from building one. There the reward signal applies to a modified solution rather than a constructed one, and the useful baselines are the local search operators you would otherwise write by hand.

The names in the minimal example show the interface. AttentionModelPolicy is constructed with the environment name and a number of encoder layers, and POMO is the RL model that wraps the environment and the policy. The point of that split is that the model handles the training loop and the rollout machinery while the policy handles the architecture, so a new architecture is a new policy class rather than a new training script.

The benchmark also modularises reusable components, and the example the README highlights is environment embeddings, which can be swapped to solve new problems. The repository has a notebook dedicated to that, examples/3-creating-new-env-model.ipynb, which is the fastest way to see how much work a new problem actually is.

Hydra carries every experiment setting

The configuration story is Hydra, and it is the mechanism that makes the benchmark comparable across policies and problems.

A default run is one command:

python run.py

which trains the default configuration, an attention model on the TSP environment. Everything else is an override on the command line, and the overrides follow the group.key=value form. You can pick an experiment configuration from configs/experiment/ and change individual settings:

python run.py experiment=routing/am env=tsp env.num_loc=50 model.optimizer_kwargs.lr=2e-4

The README points out that the environment can be swapped on the command line, for instance with env=cvrp, or by editing the corresponding experiment file such as configs/experiment/routing/am.yaml. There is a notebook, examples/advanced/1-hydra-config.ipynb, for the fuller Hydra workflow.

Two command lines deserve attention because they show how the framework handles awkward cases. Disabling the logger is done with logger=none and a tilde-prefixed key to switch off a callback that would otherwise need a logger:

python run.py experiment=routing/am logger=none '~callbacks.learning_rate_monitor'

And a hyperparameter sweep uses the multirun flag, where the -m short form turns a single value into a comma separated list:

python run.py -m experiment=routing/am model.optimizer.lr=1e-3,1e-4,1e-5

That last line is the one that makes this a benchmark rather than a script. A learning rate sweep over three values, launched from the terminal, with the same code path as a single run, is the smallest unit of the kind of experiment the project exists to support.

Installing it three ways, depending on how you intend to use it

The release on PyPI installs with pip:

pip install rl4co

For the bleeding edge of the main branch, which the README recommends when a bug has been fixed since the last release but no new release has been rolled out yet, the install is direct from git:

pip install -U git+https://github.com/ai4co/rl4co.git

For local development the recommendation is uv, the package manager the README calls blazing-fast:

git clone https://github.com/ai4co/rl4co && cd rl4co
uv sync --all-extras
source .venv/bin/activate

That creates a virtual environment in .venv/ and installs all dependencies, which matters for a project with optional extras around environments and datasets.

For people who would rather not install anything, there is a quickstart notebook that opens in Colab, and a fuller training notebook beside it, plus a plain Python script version of the training flow in examples/2b-train-simple.py and a meta-training example in examples/2d-meta_train.py. The examples directory is therefore the practical documentation.

The minimal example in the README shows how little code is needed once the pieces are in place. A generator makes the instances, the environment wraps it:

from rl4co.envs.routing import TSPEnv, TSPGenerator
from rl4co.models import AttentionModelPolicy, POMO
from rl4co.utils import RL4COTrainer

generator = TSPGenerator(num_loc=50, loc_distribution="uniform")
env = TSPEnv(generator)

The rest of the snippet constructs the policy with the environment name and the number of encoder layers, wraps them in the model, and hands the whole thing to the trainer, which is the Lightning wrapper the library is built on.

What a benchmark settles, and what it does not

The value of rl4co is easiest to see by imagining what it replaces. A research group comparing an attention model against an improvement policy on the CVRP would otherwise write two training scripts, two environment implementations and two evaluation loops, and any difference in the results could come from the harness rather than the policy. With the benchmark, the harness is fixed and the comparison is about the policy.

What it does not do is make the RL approach competitive with the classical solvers. The benchmark exists to make the learning research comparable, not to argue that a learned policy should replace an exact method on an instance set where an exact method is available. The classical baselines still have to be in the comparison table, and the README does not claim otherwise.

There is also a scope boundary worth naming. The library covers environments, policies, the training loop and configuration. It is not a solver library, it does not produce a certificate of optimality, and it does not manage the operational concerns of a routing system in production, such as dispatching, replanning on failure or the reporting an operations team needs.

The alternative approach, then, is to build on TorchRL and PyTorch Lightning directly and write the environment and the evaluation protocol yourself. That is entirely reasonable for a single problem with a single policy, and rl4co is the better choice when the number of problems, policies or configurations is large enough that keeping the harness identical matters more than owning every line of it.

Editorial conclusion

rl4co suits a research group that wants to compare RL policies for combinatorial optimization under one training loop, one configuration system and one set of environments, and who want to plug in a new problem without writing a training harness. It does not suit a production routing system that needs a solver guarantee, and it does not replace the classical optimisation baselines you should still be comparing against. Before you build on it, check the environment embeddings for the problem you care about, read the advanced Hydra configuration notebook, and pin the version, since the package is classified as beta and was at 0.7.0 when released in September 2026.

Frequently asked questions

Official sources

1. ai4co/rl4co on GitHub
2. License: MIT
3. Project website
4. README
5. Releases