Kaixhin/imitation-learning: Six Off-Policy IL Algorithms Behind One SAC Trainer

The gap between a paper implementation and a comparison harness

Most imitation learning repositories ship one algorithm. Reproducing a comparison across several of them means reconciling different codebases, different logging, and different assumptions about how expert data is loaded. This project takes the opposite route: it fixes SAC as the base reinforcement learning algorithm and implements six imitation methods on top of it, so the only thing that changes between runs is the algorithm config. The README lists AdRIL, DRIL, GAIL, GMMIL, PWIL and RED, and notes that GAIL here is equivalent to DAC or SAM when used with an off-policy algorithm. The intended user is someone who wants to know how these methods behave under a common training loop, not someone who wants the single best implementation of one of them. The repository is also the code behind a paper, A Pragmatic Look at Deep Imitation Learning, published at ACML 2023, and the citation block gives the authors as Arulkumaran and Ogawa Lillrank. That framing matters: the option surface is designed for controlled comparison, which is why so many knobs exist for discriminators and reward functions.

How the Hydra config tree selects an algorithm

The mechanism is configuration composition rather than subclassing at the call site. Running train.py with algorithm=<ALG> and env=<ENV> loads the base settings from conf/config.yaml and the algorithm-specific settings from conf/algorithm/<ALG>.yaml. The README states that hyperparameters live in those two locations, so the effective configuration for a run is the merge of a general file and one algorithm file. Around that core sit shared options that apply across methods: bc_pretraining.iterations for behaviour cloning pretraining with a default of 0, imitation.state-only for state-only imitation, imitation.absorbing for the absorbing state indicator from the DAC line of work with a default of true, imitation.mix_expert_data taking none, mixed_batch or prefill_memory with a default of none, and imitation.bc_aux_loss with a default of false, which the README notes should be true for DRIL. Algorithm-specific options live in the same namespace. AdRIL exposes imitation.balanced for alternating expert and agent batches and imitation.update_freq, which the README says can be set to 0 to obtain SQIL. DRIL adds imitation.quantile_cutoff bounded between 0 and 1. GAIL carries the largest set: reward shaping and subtracting log pi(a|s) for AIRL, a reward_function choice among AIRL, FAIRL and GAIL, a gradient penalty coefficient, spectral normalisation, an entropy bonus, and a loss_function choice among BCE, Mixup and PUGAIL with mixup_alpha, pos_class_prior and nonnegative_margin as the accompanying parameters. PWIL exposes reward_scale and reward_bandwidth_scale, and RED exposes reward_bandwidth_scale as sigma1. This is a flat namespace rather than a per-algorithm schema, so a key that does not apply to the selected algorithm is simply inert.

Getting a first run and a sweep out of the repository

Installation is a single requirements file: pip install -r requirements.txt. The README names PyTorch, OpenAI Gym, D4RL and Hydra as the notable dependencies, with Ax and the Hydra Ax Sweeper plugin needed only for hyperparameter optimisation. A basic run is python train.py algorithm=GAIL env=hopper, where the algorithm is one of AdRIL, BC, DRIL, GAIL, GMMIL, PWIL, RED or SAC, and the environment is one of ant, halfcheetah, hopper or walker2d. SAC is included as a baseline that trains on the real environment reward. Output lands in outputs/<ALGO>_<ENV>/m-d_H-M-S, with the final directory named by the current datetime. To use the tuned hyperparameters from the paper for a specific number of expert trajectories, add optimised_hyperparameters=<ALG>_<NUM_TRAJECTORIES>_trajectories alongside the algorithm flag, for example python train.py algorithm=AdRIL optimised_hyperparameters=AdRIL_5_trajectories env=halfcheetah. Running every environment in parallel is python train_all.py algorithm=<ALG> env=<ENV>, which writes to outputs/<ALGO>_all/m-d_H-M-S with one subdirectory per environment. Sweeps use Hydra's multirun flag with comma-separated values, as in python train.py -m algorithm=PWIL env=walker2d reinforcement.discount=0.97,0.98,0.99, writing to outputs/<ALGO>_<ENV>_sweep/m-d_H-M-S. Seed sweeps combine that with the optimised hyperparameter option, for instance python train.py -m algorithm=RED optimised_hyperparameters=RED_25_trajectories env=halfcheetah seed=1,2,3,4,5,6,7,8,9,10. Bayesian optimisation across all environments is python train_all.py -m algorithm=<ALG>, optionally with imitation.trajectories set, and the results can be inspected with python scripts/print_plot_sweep_results.py --path <PATH> pointed at the sweep output directory.

The MuJoCo and D4RL assumption is not a detail

Every command in the README resolves to one of four locomotion tasks: ant, halfcheetah, hopper, walker2d. The benchmark section states that the algorithms were benchmarked on Gym MuJoCo environments with D4RL expert-v2 data. There is no documented path for a custom environment, a non-MuJoCo simulator, or a dataset in another format. If your expert demonstrations come from a robot, a game, or a proprietary simulator, you are looking at adapting the environment and data loading code yourself, and the README gives no guidance on where that boundary sits. The same applies to the observation and action spaces: locomotion tasks have continuous, relatively low-dimensional observations, and nothing in the material suggests the discriminator or reward architectures were exercised outside that regime. A second constraint is the dependency chain. D4RL and Gym both have version histories that do not always agree, and the README does not pin versions beyond the requirements file. Expect to spend time on dependency resolution before the first successful run, particularly if your environment already has a newer Gym installed.

What the flat option namespace costs you

Because all algorithm-specific keys live under the same imitation prefix, nothing in the configuration itself tells you which options belong to which method. The README compensates with prose: it groups options under DRIL, GAIL, PWIL, RED and AdRIL headings, and notes that GAIL, DRIL and RED include several options for their trained discriminators. But the grouping exists only in the documentation, not in the config schema. Setting imitation.spectral_norm while running PWIL will not raise an error; it will simply have no effect. That is a real failure mode for anyone exploring the option space, because a silent no-op looks identical to a hyperparameter that genuinely does not matter. The same applies to the interaction between optimised_hyperparameters and algorithm: the README explicitly notes that algorithm=<ALG> still needs to be specified when loading the tuned set, which implies the two flags are not cross-validated against each other. Read conf/algorithm/<ALG>.yaml before trusting a flag name. A further gap is that no release was retrieved for this review, so there is no changelog to consult for option renames or removals. The README does point at v1.0 and v2.0 tags, describing v1.0 as containing on-policy IL algorithms and v2.0 as containing off-policy ones, which is the closest thing to a versioning statement in the material.

Where a single-algorithm codebase is the better choice

If you only need GAIL with spectral normalisation and gradient penalty, the original DAC implementation that this repository credits is a narrower target: one algorithm, one set of options, less configuration surface to learn. The trade-off is the reverse of this project's. A single-algorithm repository gives you a shorter path from clone to training run and fewer places for a misconfigured key to hide, but it gives you nothing to compare against, and its choices about the base RL algorithm, data loading and logging are baked in. This repository's value is precisely that those choices are shared, so a difference in final performance between two runs is more plausibly attributable to the imitation method than to the trainer. The cost is that you inherit SAC as the base learner whether or not SAC is what you wanted, and you inherit Hydra's configuration model, including the multirun and sweeper machinery, which is more infrastructure than a single experiment needs. There is also the on-policy question: the README directs readers to v1.0 for on-policy algorithms, so if your setting requires on-policy data collection, the current branch is the wrong artefact and you should be looking at that tag instead.

Maintenance, licensing and what to check before you build on it

The repository is MIT licensed, which permits commercial and closed-source use provided the licence and copyright notice are preserved, but this is a description of the licence text and not legal advice; check LICENSE.md and your own obligations. The last push recorded for the default branch is 2026-08-22, and no releases were retrieved, so there is no tagged version to pin against beyond the v1.0 and v2.0 tags referenced in the README. That means upgrades are effectively branch tracking: you pull master and reconcile any changes to conf/config.yaml and conf/algorithm/<ALG>.yaml yourself. Because the configuration keys are the interface, a rename in one of those files is a breaking change for any script or sweep you have written around them, and there is no changelog in the supplied material to warn you. The practical mitigation is to copy the conf directory into your own project rather than depending on it in place, so your runs stay reproducible even if the upstream defaults move. The maintenance surface you are taking on is therefore twofold: the Python dependencies, which the README does not pin beyond requirements.txt, and the config tree, which is the actual API.

Editorial conclusion

Adopt this if you are already working in Gym MuJoCo with D4RL expert-v2 data and want to compare several off-policy imitation algorithms without rebuilding a trainer for each one. Do not adopt it if your environment is not a Gym MuJoCo task, or if you need an on-policy algorithm, since the README points at the v1.0 tag for those. Before committing, verify three things: that the D4RL expert-v2 dataset for your chosen environment is available, that the Hydra config tree under conf/config.yaml and conf/algorithm/<ALG>.yaml matches the option names you intend to set, and that your PyTorch and Gym versions resolve against requirements.txt rather than a newer stack.

Official sources

1. Issues
2. Kaixhin/imitation-learning on GitHub
3. License: MIT
4. README