CodeRL: teaching a code model to learn from its own unit test results
This is the official code for the paper CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (NeurIPS22).
At a glance
- What is it?
- CodeRL is the official code for a NeurIPS 2022 paper that treats a pretrained code language model as an actor in a reinforcement learning loop, adds a critic that predicts whether a generated program will pass its unit tests, and uses that dense feedback to fine-tune the model. The repository ships the training and inference scripts, the two pretrained CodeT5 checkpoints it builds on, and the example unit test extraction that makes the reward signal possible.
- Who is it for?
- Adopt CodeRL if you are reproducing the NeurIPS 2022 work or if you want a trained critic that scores generated programs by whether they pass their tests, since that critic and the actor fine-tuning loop are the reusable ideas and both are under a permissive BSD-3-Clause licence.
- Can I use it commercially?
- Yes. BSD-3-Clause is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 143 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on October 22, 2026, and from our analysis. They are not legal advice.
Editorial analysis
The idea in two sentences
The repository is the official code for the paper CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning, accepted to NeurIPS 2022, and it comes with a blog post and a poster.
The framing in the README's own overview diagram is the clearest statement of the contribution. An example program synthesis task includes a problem specification in natural language, which often contains example input and output pairs, and the expected output is a program checked for functional correctness against some unit tests. A high-level overview of the framework treats a pretrained language model as a stochastic policy, token predictions as actions, and rewards estimated from the unit test results of output programs.
So the setup is a standard program synthesis problem with an unusual reward. The environment is a unit test suite. The action space is tokens. The policy is a language model that already knows how to write code, and the contribution is a way to use the test results to push that model in a direction the original pretraining never pointed.
The two bullets under the diagram describe what happens in each phase. During training, the code-generating language models are treated as an actor network, and a critic network is introduced to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, there is a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores.
The primary language is Python, the licence is BSD-3-Clause, the default branch is main, and the repository publishes no GitHub releases. The last push was on 2026-06-02 with the project not archived.
Why a critic is needed at all
The dense feedback point deserves a paragraph, because it is the part that distinguishes this from ordinary fine-tuning.
In the plainest version of code generation training, you have a dataset of problems and their solutions, and you train the model to produce the solution. That works, but it tells the model nothing about why a particular generated program failed. If a generated function returns the wrong value for one of five test cases, a pass or fail signal is almost no information. The gradient says only that the whole sequence was wrong, not which token was wrong and in which direction.
A critic trained to predict functional correctness gives a graded signal instead. The critic is another network, trained to estimate, from a sampled program, the probability that it will pass the tests. That estimate becomes a per-sample reward, and the actor is fine-tuned on the samples with the RL objective, factoring in the critic scores.
The wording in the README is careful here. The returns are estimated based on critic scores and then factored into the RL objective to fine-tune the actor network using synthetic samples. That is the whole training loop: sample programs from the actor, score them with the critic, and update the actor on the high-scoring ones. The samples are synthetic, generated by the model itself, which is what lets the method work on problems where no reference solution is available, and it is why the framework is described as applicable to tasks specified only by natural language and example pairs.
At inference the critic earns its keep a second way, in the critical sampling strategy. The overview of the critic sampling approach says that programs are refined and repaired based on their results on example unit tests, and that program candidates are sampled by their critic-predicted scores at the token or sequence level, with dotted lines indicating optional processes that apply during program refining or repairing.
Installation, and the transformers fork you have to build
Installation has two parts, and the second is the one that catches people.
The dependencies are specified in requirements.txt, installed with the standard pip install -r requirements.txt.
The requirements file itself is short and pinned hard:
torch==1.9.1
pyext==0.7
deepspeedA torch pinned to 1.9.1 and a deepspeed dependency tell you the era this code was written for, and deepspeed implies a multi-GPU training setup with ZeRO or a pipeline parallel configuration. If you are reproducing the paper, that is what you need. If you want to run this on a current stack, expect friction on the torch version before you get to anything else.
The second part is the transformers library, which you are told to install from source rather than from PyPI:
cd transformers
pip install -e .The README explains why, saying the current source code is developed from the original code of the transformers version 4.16.1, and the repository tree confirms it with a transformers/ directory and a .gitmodules file. The model code here depends on that specific vintage of the library, so a current pip install of transformers will not satisfy it. The repository also has a scripts/ directory and a utils/ directory, and the .gitmodules file suggests other submodules are involved in the data and model setup.
One more thing to know before you start. The contents list in the README has Generating Programs with Critic Sampling unchecked, while every other item, from the installation through the models, the processes and the example generated programs, is checked. That is the one part of the framework the authors did not ship, which matters if you came for the inference-time procedure rather than the training loop.
The data, and where the reward signal comes from
The dataset section is where the practical work of reproducing this lives.
For pretraining, apart from CodeSearchNet, the authors use the Python Github Code Dataset. They compiled public, non-personal data from GitHub consisting of permissively licensed Python code, and the licence list is given as examples: mit, apache-2, bsd-3-clause, bsd-2-clause, cc0-1.0, unlicense and isc. The paper covers the pretraining data preprocessing in more detail.
After pretraining, the models are fine-tuned and evaluated on two program synthesis benchmarks. APPS, with the downloading and preprocessing instructions from the original repository, and MBPP, from the Google Research repository. On both, the authors follow the same way of preprocessing data and constructing input and output sequences as the original benchmark papers, and files are downloaded and unzipped into the data folder.
The example unit tests are the interesting part, because they are what makes the critic-sampling loop possible at inference time. In addition to the original hidden unit tests on APPS, the authors also use the example tests that are often embedded in problem descriptions. There is a notebook, extract_example_test.ipynb, that extracts and saves the example unit tests of APPS test samples into the corresponding sample folder, for example data/APPS/test/0000/. The extracted tests are released in the data/APPS_test_example_tests/ folder, and the average number of example unit tests per sample is 1.9764.
That number is worth holding on to. Nearly two example tests per problem is a thin reward signal, and it explains a lot about the design: the critic exists partly because the available test signal is too sparse to learn from directly, and the example tests are what let the model repair itself at inference time when the hidden tests are unavailable.
The models, and what the released checkpoints are
The foundation model is CodeT5, a family of encoder-decoder language models for code, and the authors pretrained CodeT5 with a bigger dataset and improved learning objectives. Two large-sized checkpoints are released on Hugging Face, Salesforce/codet5-large and Salesforce/codet5-large-ntp-py, and the model section of the README describes them individually.
CodeT5-large is described as a 770M-CodeT5 model pretrained using the Masked Span Prediction objective on CodeSearchNet, which achieved new state of the art results on several CodeXGLUE benchmarks. The second checkpoint, CodeT5-large-ntp-py, is the Python-flavoured variant, and the README refers to Appendix A.1 of the paper for the details of both.
What the repository adds on top of those checkpoints is the fine-tuning and inference machinery: train.py and generate.py at the root, a trainers/ directory for the training loop, test_one_solution.py for evaluating a single generated program, configs/ for the experiment configuration, a datasets/ directory, an outputs/ directory for results, and an Example Generated Programs section in the README itself.
That last item is worth a look before you start, because it is the cheapest way to see what the method actually produces. The README links a blog post and a poster alongside the paper, and the example generated programs are listed in the contents, so between the paper for the method, the blog for the narrative and the examples for the output, you can assess the work before committing a GPU to reproducing it.
How to evaluate it, and what to be careful about
There is a clear-eyed way to assess this project, and it depends on what you want from it.
If you want to reproduce the paper, the constraints are the torch pin, the deepspeed dependency and the transformers 4.16.1 source install. That is a machine with the right CUDA and driver versions for a 2022 stack, and the data pipeline is the longest part of the work, since the pretraining corpus has to be compiled from permissively licensed GitHub Python and the APPS and MBPP benchmarks have to be downloaded and preprocessed the way their original papers describe. Budget for the data, not the training.
If you want to use the method, the interesting question is whether critic sampling still adds anything on top of a modern model. The approach was designed for a world where a code model could not reliably pass its own tests, and that has changed. The mechanism is still sound in principle, and the example unit test extraction is the reusable piece, since it is a small notebook that turns embedded example tests into a runnable suite you can use to filter or repair generations from any model.
Two cautions. The first is the licence and the data provenance: the pretraining corpus is public, non-personal GitHub Python filtered by licence, and the model checkpoints inherit that filtering, so if you need a code model for a product with strict provenance requirements, that filter is worth understanding rather than assuming. The second is the missing inference procedure, since the README's own contents list leaves generating programs with critic sampling unchecked, so the training side is what you can actually run here.
The BSD-3-Clause licence on the code is permissive and lets you fork the training loop or the critic architecture into your own pipeline without much friction, which is probably how this repository ends up being used.
Editorial conclusion
Adopt CodeRL if you are reproducing the NeurIPS 2022 work or if you want a trained critic that scores generated programs by whether they pass their tests, since that critic and the actor fine-tuning loop are the reusable ideas and both are under a permissive BSD-3-Clause licence. Do not plan on running the inference-time critic sampling procedure from this repository, because the README's own contents list leaves that step unchecked, and do not assume the checkpoints are current, since the foundation models are CodeT5 variants from a specific era and the dependency pins are torch 1.9.1 and a transformers 4.16.1 source build. Before you commit a GPU, assemble the data first, since compiling the permissively licensed GitHub Python corpus and downloading APPS and MBPP is the longest part of the work, and check the model and data provenance against whatever your own use of a code model requires.
Frequently asked questions
What is CodeRL and what is the core idea?
It is the official code for the NeurIPS 2022 paper CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. It treats a pretrained code language model as a stochastic policy whose token predictions are actions, estimates rewards from unit test results, and adds a critic network that predicts functional correctness to give dense feedback for fine-tuning the actor.
How does the CodeRL critic help with code generation?
During training the critic is trained to predict the functional correctness of generated programs and provides dense feedback signals to the actor, so the returns can be estimated from critic scores and factored into the RL objective. At inference, a critical sampling strategy lets the model regenerate programs based on feedback from example unit tests and critic scores.
What are the installation requirements for CodeRL?
Install the dependencies with pip install -r requirements.txt, which pins torch 1.9.1, pyext 0.7 and deepspeed. You also have to install the transformers library from source, developed from version 4.16.1, with a git clone into the transformers directory followed by pip install -e . inside it.
Which datasets and models does CodeRL use?
Pretraining uses CodeSearchNet plus permissively licensed Python from GitHub, and fine-tuning and evaluation use the APPS and MBPP benchmarks. The foundation model is CodeT5, with two released large checkpoints, Salesforce/codet5-large and Salesforce/codet5-large-ntp-py, the first being a 770M-parameter model pretrained with the Masked Span Prediction objective.
Why does CodeRL extract example unit tests from the APPS benchmark?
Beyond the hidden unit tests, the authors use the example tests embedded in problem descriptions, extracted with the extract_example_test.ipynb notebook and released in the data/APPS_test_example_tests/ folder. The average is 1.9764 example tests per sample, and they are what allows the model to regenerate and repair programs at inference time.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/salesforce-coderl)Was this useful?