OfflineRL-Kit: A Pure PyTorch Offline RL Library for Algorithm Researchers

The gap OfflineRL-Kit fills between a paper and a runnable script

Offline reinforcement learning papers usually ship as a single repository per algorithm. Each one brings its own replay buffer, its own network definitions, its own logging and its own evaluation loop. Reproducing a comparison across CQL, IQL and TD3+BC therefore means reconciling three codebases that disagree about how a batch is sampled and how a return is reported. OfflineRL-Kit's stated purpose is to remove that reconciliation work: it collects five model-free algorithms (CQL, TD3+BC, IQL, EDAC, MCQ) and four model-based ones (MOPO, COMBO, RAMBO, MOBILE) under one code structure. The README describes the target user directly as researchers, and lists parallel tuning and a log system among the features it considers researcher-friendly. The benchmark table in the README covers nine D4RL locomotion tasks, from halfcheetah-medium-v2 through walker2d-medium-expert-v2, with four seeds per cell. That table is the library's own reported result set, and the README marks it as ongoing, which is an honest signal that the numbers are still being filled in rather than frozen.

How a training run is assembled from components

The architecture visible in the README is composition rather than inheritance. A run starts by creating a Gym environment and converting it to an offline dataset with qlearning_dataset(env). That dictionary goes into a ReplayBuffer constructed with buffer_size, obs_shape, obs_dtype, action_dim, action_dtype and device, and then loaded with buffer.load_dataset(dataset). Networks come next: MLP backbones for the actor and the two critics, a TanhDiagGaussian distribution with latent_dim taken from the actor backbone's output_dim, and ActorProb and Critic wrappers. Optimizers are plain torch.optim.Adam instances passed into the policy. The policy object, here CQLPolicy, is where the algorithm lives: it takes the actor, both critics and their optimizers plus algorithm hyperparameters such as cql_weight, temperature, max_q_backup, deterministic_backup, with_lagrange, lagrange_threshold, cql_alpha_lr and num_repeart_actions. The trainer, MFPolicyTrainer, receives the policy, an eval_env, the buffer, the logger, and the loop parameters epoch, step_per_epoch, batch_size and eval_episodes. Swapping CQL for IQL or TD3+BC means swapping the policy class and the hyperparameters it accepts; the buffer, trainer and logger stay in place. That is the mechanism behind the README's claim that a new algorithm can be built in a few lines from existing components.

Installation pulls in MuJoCo and D4RL before OfflineRL-Kit itself

The install sequence has three stages, and the first two are outside the project. You install the MuJoCo engine from mujoco.org and then mujoco-py, whose version, the README notes, depends on the engine version you installed. Next comes D4RL from the Farama Foundation: git clone https://github.com/Farama-Foundation/d4rl.git, cd d4rl, pip install -e . Finally OfflineRL-Kit: git clone https://github.com/yihaosun1124/OfflineRL-Kit.git, cd OfflineRL-Kit, python setup.py install. Two things stand out. The setup.py install path is the older distutils-style entry point rather than a pyproject-based build. And the dependency chain means a fresh environment can fail at the MuJoCo step long before any OfflineRL-Kit code is exercised. The library is MIT licensed, so the code itself imposes few obligations, but D4RL and MuJoCo carry their own licences and their own installation conditions, and nothing in the README addresses that split. If you are evaluating this for a team rather than a solo project, the licence review has to cover the full chain, not just the MIT badge in the README. This is not legal advice; check the upstream licences yourself.

Logging and Ray tuning as the experiment management layer

Logging is configured through make_log_dirs(args.task, args.algo_name, args.seed, vars(args)) followed by a Logger built from that directory and an output_config dictionary. The README's example sets three sinks: consoleout_backup to stdout, policy_training_progress to csv, and tb to tensorboard. The logger also exposes log_hyperparameters(vars(args)), so the full argument namespace is written alongside the run. The directory naming embeds task, algorithm name and seed, which means runs are separated on disk by construction rather than by convention. For tuning, the README points at Ray and shows a tune.run call over run_exp with a config dictionary. The example grids real_ratio over [0.05, 0.5] and seed over range(2), names the experiment tune_mopo, and requests 0.5 GPU per trial. The real_ratio parameter in that example is specific to the model-based side, which fits MOPO: it is the fraction of real data mixed into model rollouts. The half-GPU resource request is the detail worth noticing, because it implies two trials can share one device, which is only safe if the per-trial memory footprint stays low.

Where the model-based algorithms behave differently from the model-free ones

The README's benchmark table is the most informative artefact in the repository, and it is worth reading for variance rather than for means. On hopper-medium-v2, MOPO reports 62.8 with a spread of 38.1 and RAMBO reports 82.1 with a spread of 38.0, while MOBILE reports 103.6 at 1.0. On walker2d-medium-expert-v2, RAMBO drops to 78.4 with a spread of 45.4. Those are the library's own four-seed numbers, and they show that the model-based entries are far more seed-sensitive than the model-free ones on the same tasks. CQL on hopper-medium-v2 sits at 59.1 with a spread of 4.1; EDAC on the same task is 101.8 at 0.2. A researcher comparing algorithms from this table should treat the model-based rows as noisier evidence, not as a clean ranking. The table is also labelled ongoing, so some cells may be revised. The practical consequence is that a single-seed model-based run in this library tells you very little; the README's own numbers make that case better than any external benchmark could.

The constraints you inherit: D4RL, MuJoCo and the absence of releases

The data path is the sharpest limitation. The quick-start example builds its dataset from gym.make(args.task) and qlearning_dataset(env), which is the D4RL route. Offline RL in practice often means logged data from a simulator, a recommender system or a robot, in formats that have nothing to do with D4RL. Nothing in the supplied material shows an adapter for arbitrary offline datasets, so using this library outside the D4RL benchmark suite means writing your own loader and confirming that the ReplayBuffer's obs_shape, obs_dtype and action_dim conventions match your arrays. The second constraint is version coupling. mujoco-py's version depends on the MuJoCo engine version, and D4RL in turn depends on mujoco-py, so a mismatch surfaces as an import failure rather than as a clear error message. The third is release hygiene: no releases were retrieved for this repository, so there is no tagged version to pin, no changelog and no compatibility statement. You are tracking main. The repository also links a newer sibling project, VLARLKit, for VLA reinforcement learning, which suggests attention is being split, though the README gives no statement about OfflineRL-Kit's own maintenance plans. For a library whose value is partly that it is current with the algorithm literature, that absence of releases is the thing to weigh.

How it differs from D4RL and from CORL-style reference implementations

The nearest comparison is D4RL itself. D4RL is a benchmark and dataset package: it defines the tasks, the offline datasets and the evaluation protocol, and it expects you to bring your own algorithm. OfflineRL-Kit consumes D4RL rather than competing with it, and its contribution is the algorithm side: the policy classes, the shared buffer, the trainer loop and the logger. The second comparison is with single-paper reference implementations, the kind released alongside a CQL or IQL paper. Those tend to be tuned to one algorithm, with the training loop and the hyperparameter defaults shaped around that paper's experiments. OfflineRL-Kit's difference is uniformity: one MFPolicyTrainer and one Logger serve all five model-free algorithms, and the policy class is the only thing that changes. That uniformity is exactly what makes cross-algorithm comparison cheaper, and it is also what you give up in fidelity, since a shared trainer cannot reproduce every quirk of an original implementation. A third comparison is with general RL libraries that include offline algorithms as one option among many. OfflineRL-Kit does not attempt online RL, distributed training or a model zoo; it stays inside the offline setting and the D4RL task family.

Maintenance cost and who should pick this up

The maintenance burden here is mostly environmental. Because there are no retrieved releases, upgrading means pulling main and re-reading the example scripts under run_example and tune_example to see whether argument names or policy constructor signatures moved. The hyperparameter surface is wide: CQLPolicy alone takes cql_weight, temperature, max_q_backup, deterministic_backup, with_lagrange, lagrange_threshold, cql_alpha_lr and num_repeart_actions, and those names are the interface you would be coding against. Note that num_repeart_actions is spelled that way in the README, so copy it verbatim rather than correcting it. Adding a new algorithm means writing a policy class that accepts the actor, critic and optimizer objects the rest of the library already produces, then registering its arguments so make_log_dirs and the logger can record them. That is a bounded amount of work, which is the strongest argument for the library. The strongest argument against is that the install path is a chain of three external projects, one of which is a MuJoCo engine you must match by hand. A researcher with a working D4RL environment gets value quickly. A team that needs a supported dependency graph, tagged versions or non-D4RL data should look at whether the algorithm they want exists as a standalone repository they can vendor, because that trades uniformity for a smaller surface to keep alive.

Editorial conclusion

Adopt OfflineRL-Kit if you are a researcher who wants to read and modify algorithm code in PyTorch and you already have a working MuJoCo and D4RL environment. Do not adopt it if you need a supported release cycle, published API stability guarantees or a non-MuJoCo data pipeline, because the repository shows no retrieved releases and the install path is pinned to D4RL. Before committing, verify that your mujoco-py version matches your MuJoCo engine, and check the run_example and tune_example scripts against the current main branch.

Official sources

1. Issues
2. License: MIT
3. README
4. yihaosun1124/OfflineRL-Kit on GitHub