# rl-mpc-locomotion: an RL policy that predicts the weights of an MPC controller > This repository trains a reinforcement learning policy to choose the weight parameters of a model predictive controller for a quadruped, rather than to output joint torques directly. The lower half of the stack is a Python rewrite of the MIT Cheetah MPC controller with the sensor and motor interfaces opened up, and the upper half trains in NVIDIA Isaac Gym against a Unitree Aliengo model. **silvery107/rl-mpc-locomotion** — Deep RL for MPC control of Quadruped Robot Locomotion - Repository: https://github.com/silvery107/rl-mpc-locomotion - Website: https://docs.google.com/presentation/d/18bznpYrkCPnhCisySPDz18hvL3Ytere7JiJEbdLvpgU/edit?usp=sharing - Stars: 1,031 · Forks: 92 - Language: Python - License: MIT - Published: 2026-10-19 · Updated: 2026-10-19 · Language: en - Canonical page: https://hysenlabs.com/projects/silvery107-rl-mpc-locomotion ## The policy predicts weights, not torques The repository aims to provide a fast simulation and reinforcement learning training framework for a quadruped locomotion task by dynamically predicting the weight parameters of a model predictive controller. The control framework is a hierarchical controller composed of a higher-level policy network and a lower-level model predictive controller. That hierarchy is the idea worth understanding before any of the code. In a conventional learned-locomotion stack, a neural network receives the robot state and outputs joint torques, and the behaviour it produces is hard to inspect, hard to constrain and hard to reason about when it fails. Here the network does not produce torques. It produces the weights of an existing MPC cost function, which means the controller underneath is still the controller that a robotics engineer wrote, and the learning is about choosing between behaviours rather than about inventing one. The README's own description of the aim, a fast simulation and RL training framework, tells you this is a research artefact rather than a product. The gallery at the bottom shows what that research looks like: a Cheetah trot, a parallel RL run across sixteen environments, a demo of the MPC walking up stairs, and a sim-to-real sequence. The README shows a transfer to a real Aliengo robot, and is careful to note that sim2real is not included in this codebase, so the demonstration is context rather than something you get by cloning. ## The MPC half is Cheetah, in Python The MPC controller refers to the MIT Cheetah Software, but written in Python, and the reason for the rewrite is stated clearly: it completely opens the interface between sensor data and motor commands, so that the controller can be easily ported to any mainstream simulators. Cheetah Software is a fast, well-tested implementation of a convex MPC controller for quadrupeds, written with performance in mind in C++. That performance is exactly what makes it awkward to use inside a research stack, because the interface to it is not a library boundary you can call from Python with a state dictionary. Opening the interface, so that sensor data goes in and motor commands come out, is what makes the controller usable as the lower half of a learning system. The dependency list shows the cost of that choice. The lower level needs a convex solver, and the user notes include installation instructions for OSQP, qpOASES and CVXOPT, with OSQP at version 0.6.2 appearing commented out in the dependency list, which suggests the default path avoids a pinned quadratic programming solver and that the solver is a choice the user makes. The other half of the stack is the simulator. RL training utilizes NVIDIA Isaac Gym in parallel, using the Unitree Robotics Aliengo model. Isaac Gym Preview 4, PyTorch 1.10.0 with CUDA 11.3 and Python 3.8 are the stated dependencies, which dates the project precisely and tells you the graphics driver and CUDA versions you will need. ## The control stack, file by file The roadmap section of the README doubles as a map of the code, and it is the most useful thing in the document for anyone about to read this repository. The MPC Controller side starts with the Quadruped model, then the RobotRunner, which leads to the LegController and the StateEstimator, and then the ControlFSM. The finite state machine has two states, RecoveryStand and Locomotion, and Locomotion leads into the convex MPC locomotion, which in turn uses a foot swing trajectory, a gait definition, and the MPC solver written in C, mpc_osqp.cc. The RL Environment side is smaller. There is a gamepad reader, simulation utilities, a weight policy, and the training script, which sits on top of a vectorised environment and the Aliengo task. Two things are worth noticing in that map. The foot swing trajectory and the gait are separate modules, so the swing profile of a step is a design choice rather than a constant buried in the solver. And the solver itself is a C file compiled as a Python extension, which is the one place where the performance-sensitive inner loop stays in C while everything around it is Python. The finite state machine is the other notable piece, because RecoveryStand and Locomotion are the states a real quadruped needs, and the ability to recover from a fall is not something a learned policy typically provides for free. ## Installation, and the pieces that are pinned The install sequence has five steps, and two of them pin things to a specific version. Clone the repository and initialise the submodules, or use the --recurse option in the clone step to get them at the same time: ```bash git clone git@github.com:silvery107/rl-mpc-locomotion.git git submodule update --init ``` Then create the conda environment from the environment.yml in the root: ```bash conda env create -f environment.yml ``` Then install rsl_rl at a specific commit under the extern folder, which the README links as 2ad79cf, using an editable install from within that directory: ```bash cd extern/rsl_rl pip install -e . ``` And finally compile the Python binding of the MPC solver with an editable install of the project itself: ```bash pip install -e . ``` That rsl_rl commit is the one to pay attention to. rsl_rl is the reinforcement learning library the training code is built on, and pinning it to a commit rather than to a version means the author found a state that works, and the training code will not run against a later head of that library. The submodules matter too, because the MIT Cheetah Software the MPC side is derived from arrives that way, and the user notes include a document on installing it. ## Playing, training and loading a checkpoint The quick start has four scenarios, and the flags are Hydra-style, which is why the same train script covers training, testing and checkpoint loading. To play the MPC controller on the Aliengo: ```bash python RL_MPC_Locomotion.py --robot=Aliengo ``` The supported robot types are Go1, A1 and Aliengo. You need an Xbox-like gamepad plugged in to control it, or you pass --disable-gamepad. The controller mode defaults to Fsm, the finite state machine, and you can also try Min, the minimum MPC controller without the FSM. The gamepad keymap is worth knowing: LB switches gait types between Trot, Walk and Bound, and RB switches the FSM between Locomotion and Recovery Stand. To train a new policy: ```bash cd RL_Environment python train.py task=Aliengo headless=False ``` Press the v key to disable viewer updates and press it again to resume, and set headless=True to train without rendering at all. Tensorboard support is there, run with tensorboard --logdir runs. To load a pretrained checkpoint, the same script takes a checkpoint path, a test flag and a number of environments: ```bash python train.py task=Aliengo checkpoint=runs/Aliengo/nn/Aliengo.pth test=True num_envs=4 ``` Setting test=False continues training from that checkpoint rather than evaluating it. And to run a trained weight policy in front of the MPC controller on the real robot, you set bridge_MPC_to_RL to False in the parameters file under MPC_Controller, and then: ```bash python RL_MPC_Locomotion.py --robot=Aliengo --mode=Policy --checkpoint=path/to/ckpt ``` If no checkpoint is given, it loads the latest run. That switch is the interesting one, because it is the difference between the learned policy driving the controller and the controller running on its own defaults. ## A setup.py that compiles in parallel without killing small machines The setup.py is worth reading on its own, because it solves a problem that most projects ignore. The project compiles a C extension, the MPC solver, and the file monkey-patches distutils' CCompiler to compile several sources in parallel. The default parallelism is twice the CPU count, and then it is reduced. The reduction is the interesting part. On Unix-like platforms the script queries the machine's physical memory and limits the number of parallel compilations to the number of gigabytes of RAM, with a comment explaining that this is to avoid killing smaller platforms like the Pi. On platforms where the memory cannot be queried the limit is skipped, with the reasoning that it is probably a well equipped Windows, Mac OS X or Linux box. There is also a macOS specific path, where extra compile arguments are added for C and C++ sources including a deployment target flag, so that the binary runs on the older macOS versions the project still supports. The result is a setup.py that behaves like a build engineer rather than a configuration file. It knows that a robot developer might be compiling on a Raspberry Pi, and it sizes the build to the machine instead of assuming a workstation. The rest of the repository is a research layout: an MPC_Controller directory, an RL_Environment directory, the top-level RL_MPC_Locomotion.py entry point, an extern directory for the vendored learning library, a docs directory with the user notes, an environment.yml, a test directory, a CITATION.cff and an assets directory. ## Where this repository is the wrong tool The first wrong-tool case is a project that wants to run a quadruped today. The README is explicit that the sim-to-real transfer is not included in the codebase, and that the transfer demonstration in the gallery is context. This is a research framework with a hardware demo, not a deployable stack. The second is a project with a current software stack. Python 3.8, PyTorch 1.10.0, CUDA 11.3 and Isaac Gym Preview 4 are the pinned dependencies, and Isaac Gym in particular is a specific and dated distribution. A team on a modern CUDA and PyTorch combination has to reconcile that, and Isaac Gym has a successor in the same vendor's line. The third is a team that wants to train a policy that outputs joint torques. This design deliberately does not do that. The learned component chooses the weights of an existing cost function, which is a different research question with different failure modes, and the value of the design is precisely that the learned part stays small and the controller stays interpretable. If you need end-to-end torque policies, this repository is the wrong starting point even though the pieces are excellent. A practical note for anyone who tries it: the solver is a choice, and the user notes have separate instructions for OSQP, qpOASES and CVXOPT. The build depends on that decision, so read those before you install. ## Conclusion Adopt this repository as a research base if you are working on learned control for quadruped locomotion and want the lower-level controller to stay interpretable, and if you can afford the Isaac Gym and PyTorch stack it pins. Do not adopt it as a robot stack you can put on hardware, because the sim-to-real transfer is explicitly not included in the codebase, and do not expect a modern setup, since the dependency versions are from the PyTorch 1.10 and CUDA 11.3 era. Before you start, read the user notes on setting up Isaac Gym and on the OSQP, qpOASES and CVXOPT solvers, since the solver choice changes the build, and pin the rsl_rl commit the README names rather than taking the current head of that repository. ## FAQ ### What does rl-mpc-locomotion actually train? A reinforcement learning policy that dynamically predicts the weight parameters of a model predictive controller, in a hierarchical setup with a higher-level policy network and a lower-level MPC controller. The policy does not output joint torques directly, so the underlying controller stays interpretable. ### What are the dependencies for rl-mpc-locomotion? Python 3.8, PyTorch 1.10.0 with CUDA 11.3, and NVIDIA Isaac Gym Preview 4. The MPC side follows the MIT Cheetah Software, rewritten in Python, and the user notes include separate installation instructions for the OSQP, qpOASES and CVXOPT solvers. ### How do I install rl-mpc-locomotion? Clone the repository, run git submodule update --init or use --recurse, create the conda environment with conda env create -f environment.yml, install rsl_rl at the pinned commit with an editable install from extern/rsl_rl, and finally run pip install -e . to compile the Python binding of the MPC solver. ### How do I train a policy and then load a checkpoint? Train with python train.py task=Aliengo headless=False from the RL_Environment directory, and use tensorboard --logdir runs to watch progress. Load a checkpoint with python train.py task=Aliengo checkpoint=runs/Aliengo/nn/Aliengo.pth test=True num_envs=4, where setting test=False continues training from the checkpoint instead of evaluating it. ### What licence is rl-mpc-locomotion released under? MIT, with the LICENSE file at the repository root and a CITATION.cff for citation metadata. The last push was on 2026-04-26, and there are no GitHub releases. ## Sources - [Issues](https://github.com/silvery107/rl-mpc-locomotion/issues) - [License: MIT](https://github.com/silvery107/rl-mpc-locomotion/blob/main/LICENSE) - [Project website](https://docs.google.com/presentation/d/18bznpYrkCPnhCisySPDz18hvL3Ytere7JiJEbdLvpgU/edit?usp=sharing) - [README](https://github.com/silvery107/rl-mpc-locomotion/blob/main/README.md) - [silvery107/rl-mpc-locomotion on GitHub](https://github.com/silvery107/rl-mpc-locomotion) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/silvery107-rl-mpc-locomotion