PG is all you need: policy gradient methods rebuilt as seven Jupyter notebooks

Seven chapters, each pairing derivation with implementation

The repository describes itself as a step-by-step tutorial for policy gradient algorithms from A2C to SAC, including learning acceleration methods using demonstrations for treating real applications with sparse rewards, and it says every chapter contains both theoretical backgrounds and an object-oriented implementation. The contents list is the seven chapters: Advantage Actor-Critic, Proximal Policy Optimization Algorithms, Deep Deterministic Policy Gradient, Twin Delayed Deep Deterministic Policy Gradient, Soft Actor-Critic, DDPG from Demonstration and Behavior Cloning with DDPG. The chapter order is a curriculum rather than an arbitrary list. A2C comes first as the on-policy baseline, PPO follows as the way to stabilise it, then the off-policy continuous control line of DDPG, TD3 and SAC, and finally the two chapters that add demonstrations to deal with sparse reward. Each notebook is linked for both NBViewer and Colab.

One environment keeps the comparison honest

Every chapter runs on the same Pendulum-v0 environment, and the README documents the observation and action spaces precisely. The observation is a Box of three values, the cosine of the angle, the sine of the angle and the angular velocity, with the angle velocity ranging from -8.0 to 8.0. The action is a Box of one value, the joint effort, bounded from -2.0 to 2.0. The reward is the negative of theta squared plus 0.1 times the angular velocity squared plus 0.001 times the action squared, with theta normalised between minus pi and pi, so the lowest cost is -16.2736044 and the highest is 0. The goal is to remain at zero angle with the least rotational velocity and the least effort, with a maximum of 200 steps per episode. The benefit of that choice is that a chapter can be compared against the previous one without a new environment getting in the way.

The last two chapters are the ones that address real sparse reward problems

DDPG from Demonstration and Behavior Cloning are the chapters that make this repository more than a survey of algorithms. DDPGfD takes the DDPG algorithm and adds demonstration data, which is the approach for tasks where the reward signal is too sparse to learn from scratch. Behaviour Cloning with DDPG does the same thing from the supervised side, learning a policy directly from expert trajectories. The README frames the motivation in the first paragraph, saying the tutorial includes learning acceleration methods using demonstrations for treating real applications with sparse rewards. This is the part of the tutorial with the most practical value, because the algorithmic chapters are textbook material available anywhere, whereas showing how demonstrations enter an off-policy update loop is a specific piece of design work.

The implementation style is deliberately object-oriented

The README repeatedly mentions the object-oriented implementation, and the repository root shows a single shared module, segment_tree.py, sitting alongside the seven notebooks, plus a demo.pkl file. A segment tree alongside a policy gradient implementation is a strong hint about the subject: an efficient data structure for prioritised experience replay, which is what DDPG, TD3 and SAC need and what a naive implementation would do with a linear scan. The style choice matters for a tutorial because it means the notebooks read as a set of classes with clear responsibilities rather than a sequence of functions, so the reader can see the actor, the critic and the replay buffer as distinct pieces that a real codebase would also have. The repository also ships a pyproject.toml configuring black with a line length of 100 and isort with the black profile.

Running it is a make target and a conda environment

The installation path is documented in two steps. First you create and activate a conda environment, and the README states the repository is tested on an Anaconda virtual environment with python 3.6.1 or later.

conda create -n pg-is-all-you-need python=3.6.9
conda activate pg-is-all-you-need

Then you clone the repository and install the packages.

git clone https://github.com/MrSyee/pg-is-all-you-need.git
cd pg-is-all-you-need
make dep

The Makefile shows what dep does, which is a pip install of requirements.txt, and the dev target does the same plus requirements-dev.txt, and then installs and enables the Jupyter nbextension for the user. That last step is a slightly dated convention, since nbextensions are the older mechanism and modern Jupyter relies on server extensions, so expect friction there. The pinned requirements are torch 2.5.1, numpy 1.26.4, gymnasium 1.0.0, matplotlib 3.8.0 and jupyter.

The Colab link means you can read it on a phone

The README makes a point that is easy to skim and quite useful: you can execute the notebooks right away with Colab even on your smartphone. Every chapter links to a Colab URL as well as an NBViewer one, so nothing has to be installed to read the code, and for a reader who only wants to understand an update rule, opening a notebook in the browser is a lower barrier than creating a conda environment. For development, the README suggests nbdiff-web from nbdime to see the difference between modified notebooks, which is a sensible workflow for a repository where the notebooks are the source code rather than generated output. The repository publishes no GitHub releases, and the last push was on 2026-07-17.

MIT licensed, with the papers listed and a companion DQN tutorial

The licence is MIT, with a LICENSE file at the root, and the repository lists four contributors through the all-contributors mechanism, with the primary work credited to Kyunghwan Kim and the code and documentation credits to Jinwoo Park and Mincheol Kim among others. The related papers section gives seven references with links, and they are the primary sources for the algorithms covered: the asynchronous advantage actor-critic paper by Babaeizadeh and colleagues, the PPO paper by Schulman and colleagues, the DDPG paper by Lillicrap and colleagues, the function approximation error paper by Fujimoto and colleagues for TD3, the soft actor-critic paper by Haarnoja and colleagues, and the two demonstration papers by Vecerik and Nair. There is also a pointer for readers who want the value-based side, telling them to see Rainbow is All You Need for a tutorial on the DQN series.

Editorial conclusion

pg-is-all-you-need fits a reader who has seen the PPO equations in a paper and wants to see the update rule become a class, because every chapter pairs the derivation with an implementation and runs on one environment you can reason about, which is a better way to build intuition than a large benchmark suite. It is a poor fit as a reference for discrete action spaces, for which the repository points to a separate DQN tutorial, and a poor fit as a production RL library, since nothing here is packaged for reuse. Verify first that your Python and PyTorch versions match the pinned requirements, since the environment list fixes torch 2.5.1, gymnasium 1.0.0 and numpy 1.26.4, and note the repository is on Python 3.6-era conda instructions that are older than the pinned libraries it installs.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. MrSyee/pg-is-all-you-need on GitHub
4. README