mpcrl learns a controller by tuning the parameters of an MPC problem

The controller is the policy, and the policy is a solver

The README states the idea in one sentence: the MPC controller, parametrized in its objective, predictive model and constraints, or a subset of these, acts both as policy provider, meaning it provides an action to the environment given the current state, and as function approximation for the state and action value functions, meaning it predicts the expected return following the current control policy from a given state and state-action pair. Concurrently, an RL algorithm is employed to tune this parametrization so that the controller's performance increases. That is the whole architecture, and it is a different shape from the usual one. In a conventional RL setup the policy is a neural network with weights that gradient descent pushes around. Here the policy is a structured optimisation problem, and learning means adjusting the numbers inside that problem, such as the weights in a cost matrix or the entries of a linear predictive model. The result is that a learned controller remains interpretable as a control law, and the README notes the framework was first proposed in a specific paper and has since been shown effective in various applications with different learning algorithms and more sound theory.

Q-learning and DPG are the named learners

The learning side is deliberately small, which is one of the framework's selling points. The README names two algorithms as the most successful: Q-learning and Deterministic Policy Gradient. Both are used here in a setting that is unusual for them, because the function they approximate is not a network but the value of an optimisation problem. Q-learning in this framework estimates the value of a state by running the MPC problem with the current parameters and looking at the resulting predicted return over the prediction horizon, and the policy gradient variant differentiates through the solver to push the parameters in a direction that improves the return. Because the approximation is the controller, the value function inherits the controller's structure, including its constraints, and a constraint that the MPC formulation enforces is a constraint the learned value function respects. That is a meaningful difference from a neural approximation, which can extrapolate into regions the controller would never produce. The project is distributed on PyPI as mpcrl, and the dependency set is listed in the package metadata: csnlp, which is the CasADi based NLP package the README links for symbolic processing, SciPy, Gymnasium and Numba.

Installing from PyPI or working on the source

Installation is a single command for the released package.

pip install mpcrl

The stated requirements are Python 3.11, and the package metadata asks for Python 3.11 or newer, with classifiers for 3.11, 3.12 and 3.13. The dependencies named in the README are csnlp, SciPy, Gymnasium and Numba, and the metadata pins them loosely with lower bounds rather than exact versions, csnlp at 1.6.8 or newer, scipy at 1.11.1 or newer, gymnasium at 0.29.0 or newer and numba at 0.57.1 or newer. If you want to work on the code itself, clone the repository.

git clone https://github.com/FilippoAiraldi/mpc-reinforcement-learning.git

Then install it in editable mode from wherever you cloned it.

pip install -e /path/to/mpc-reinforcement-learning

The branch structure is worth knowing before you start. The main branch contains the main releases of the packages and the occasional post release, and the experimental branch is reserved for the implementation and test of new features, and hosts the release candidates. So a pull request against the wrong branch will not be merged in a hurry, and a project that wants stability should stay on main. The project is MIT licensed, is packaged with setuptools rather than poetry, and its version is read dynamically from the package attribute, so the source tree and the installed version cannot drift apart.

The first application is a linear system wrapped as a Gymnasium environment

The getting started section walks through a simple Linear Time Invariant system, and the point of the example is the shape of the task rather than the system itself. The cost of controlling a state with an action is given by a quadratic form with a state-cost matrix Q and an action-cost matrix R, both positive definite. The text makes the research framing explicit: this is a well known problem in optimal control theory, but in the context of RL these matrices are not known, and only realizations of the cost for each state-action pair the controller visits can be observed. The same applies to the dynamics, which follow the usual state-space model with matrices A and B that could in general be unknown, and the control action is assumed bounded in the interval [-1, 1]. To implement it, the LTI system is written as a gymnasium.Env class, where reset initialises the state and step updates the state from the action and returns the new state and the cost.

from gymnasium import Env
from gymnasium.wrappers import TimeLimit
import numpy as np


class LtiSystem(Env):
    ns = ...  # number of states (must be continuous)
    na = ...  # number of actions (must be continuous)
    A = ...  # state-space matrix A
    B = ...  # state-space matrix B
    Q = ...  # state-cost matrix Q
    R = ...  # action-cost matrix R
    action_space = Box(-1.0, 1.0, (na,), np.float64)  # action space

Because the system is a standard Gymnasium environment, the environment half of your code is ordinary Gymnasium code, and the learning half is where mpcrl takes over.

Three families of algorithm, separated by directory

The examples directory is organised by how the learning signal is obtained, and the split tells you what the library expects of you. There are three example groups, gradient-based-offpolicy, gradient-based-onpolicy and gradient-free, plus an others folder. The gradient-based ones differentiate through the optimisation to update the controller parameters, and the on-policy and off-policy distinction is the usual one about whether the data used for an update comes from the current policy. The gradient-free group is the interesting one for control, because it does not need derivatives of the solver and so can be applied to problems where the optimiser is not differentiable in a usable way, at the cost of a noisier estimate. A fourth file, examples/README.rst, ties them together. The distinction matters for a concrete reason. A gradient through a nonlinear program is only as good as the solver's local solution and its own convergence behaviour, so an on-policy gradient method inherits the risk that the solver returns a slightly different solution between two calls and produces a noisy gradient. The gradient-free route trades that for sample inefficiency. Choosing between them is a judgement about your problem's structure and about how much you trust your optimiser, and the library's job is to give you both rather than to pick for you.

What the framework does not give you

There are limits that come from the design rather than from missing features. First, everything runs through a symbolic optimisation stack, and the dependency on csnlp means a CasADi installation and an NLP solver in the loop. On a control desk that is the point, but for a large scale learning problem the per-step optimisation cost is the thing to measure, and the README does not publish timing numbers, so the cost has to be established on your own hardware. Second, the approach is aimed at problems with a model and a cost you can write down. A task with no state-space structure, or a reward that is not expressible as a cost the controller can optimise, does not fit the parametrization, and the framework will not invent one for you. Third, the documentation is research documentation: the getting started section is a skeleton with ellipses for the parts you must supply, and the value of the library is in the papers it implements rather than in a set of worked examples for your domain. Fourth, the release cadence is uneven. The recent releases are 1.4.1 and 1.4.0, both on 2025-10-17, with 1.3.3 on 2025-07-16, while the repository itself was pushed on 2026-10-19, so work is happening on main that has not been cut as a release. The package is MIT licensed, with the licence file shipped alongside, so embedding it in a research codebase carries no copyleft obligation, though your own review should cover the third party dependencies it pulls in.

Editorial conclusion

mpcrl suits a control researcher who wants an RL algorithm to tune an MPC controller rather than a neural network, and who is comfortable with the CasADi based optimisation stack that comes with it, since the library handles the coupling between the learning algorithm and the solver instead of leaving you to write it. It is the wrong tool for a deep learning team that wants a standard policy gradient loop over a Gymnasium environment, because the learning signal is tied to a structured optimisation problem. Before you build on it, read the examples directory, which separates gradient-based on-policy, gradient-based off-policy and gradient-free approaches, and confirm that the version you pin on PyPI, 1.4.1, matches the documentation you are reading, since the last push was on 2026-10-19 while the newest release on record is from 2025-10-17.

Frequently asked questions

Official sources

1. FilippoAiraldi/mpc-reinforcement-learning on GitHub
2. License: MIT
3. Project website
4. README
5. Releases