Deep Reinforcement Learning Hands-On: What the Companion Repository Actually Contains

What the Deep Reinforcement Learning Hands-On repository is, and what it is not

This is the code repository for the Packt book Deep Reinforcement Learning Hands-On, written by Max Lapan. The README describes it as code samples for the book and states that it contains all the supporting project files necessary to work through the book from start to finish. That framing matters more than it sounds. There is no package to import, no CLI, no service, no configuration file at the root beyond requirements.txt. If you came looking for a library you can add to a project, this is the wrong artifact.

The audience is narrow and clear. You are expected to be reading the book alongside the code, or at least to know which chapter covers which algorithm. The repository is organized as Chapter02 through Chapter18, and the README lists the topic of each: OpenAI Gym, deep learning with PyTorch, cross entropy method, tabular learning and the Bellman equation, DQN, DQN extensions, stocks trading, policy gradients, actor-critic, A3C, chatbots, web navigation, continuous action space, trust regions (TRPO, PPO, ACKTR), black-box optimisation, imagination, and AlphaGo Zero. Someone who wants to understand how a Deep Q-Network is actually assembled in PyTorch, line by line, gets real value here. Someone who wants a trained agent to solve their problem does not.

The repository is not archived, and its last push was on 2026-03-02. The README itself is candid about maintenance: the author writes that he is trying to keep all the examples working under the latest versions of PyTorch and gym, which is not always simple, as software evolves. It also notes that bugs in examples are inevitable, so the exact code might differ from the code in the book text. Treat the repository as a maintained teaching artifact, not as production software.

How the chapter folders and pinned requirements fit together

The mechanism is directory-based. Each chapter is a self-contained folder of Python scripts, and the top-level entries are Chapter02 through Chapter18 plus LICENSE, README.md, requirements.txt, formulas/, download-roboschool.sh and install-roboschool.sh. There is no shared package that all chapters import from, which means each script carries its own model definitions, training loop and environment setup. That redundancy is deliberate for a book: a reader can open Chapter06 and see a complete DQN without chasing imports across the tree.

The dependency story is the part that will decide whether you can run anything. requirements.txt pins exact versions: numpy 1.15.4, atari-py 0.1.6, gym 0.10.9, ptan 0.3, opencv-python 3.4.3.18, scipy 1.1.0, torch 0.4.1, torchvision 0.2.1, tensorboardX 1.6, tensorflow 1.12.2, tensorboard 1.12.0, pybullet 2.3.6 and matplotlib 3.0.2. The README states that examples require Python 3.6. ptan is the author's own library, used across the examples for agent abstraction and experience replay, and it is pinned to 0.3.

The README also describes a versioning scheme that is worth knowing before you check out anything. Tag 01_release marks the code state right after book publication in June 2018. The master branch is described as having the latest version of code updated for PyTorch 0.4.1. A branch for porting to PyTorch 1.0 is mentioned as not yet created. So the branch names reflect an older PyTorch era, and the requirements file confirms it. If you install a modern PyTorch and then run a chapter script, you are outside the configuration the repository documents.

Getting the samples and running your first chapter

There is no install script for the Python side. The README points at requirements.txt as the list of current requirements, and that is the intended entry point. The README does not document a virtual environment workflow, so the environment setup is left to you; the only file the repository names for dependencies is requirements.txt itself.

The pinned contents of requirements.txt are:

numpy==1.15.4
atari-py==0.1.6
gym==0.10.9
ptan==0.3
opencv-python==3.4.3.18
scipy==1.1.0
torch==0.4.1
torchvision==0.2.1
tensorboardX==1.6
tensorflow==1.12.2
tensorboard==1.12.0
pybullet==2.3.6
matplotlib==3.0.2

That is the whole dependency contract. torch 0.4.1, tensorflow 1.12.2 and numpy 1.15.4 are old wheels, and whether they resolve depends on your platform and Python build. If the install fails, that is information about the repository, not about your competence.

Two shell scripts at the root handle a specific environment. install-roboschool.sh and download-roboschool.sh are there because the robotics environments used by some chapters are not part of the core gym install. The README does not describe what these scripts do step by step, so read them before executing.

After the environment builds, pick a chapter folder and run a script from inside it. The chapter layout is the only navigation aid the repository gives; the README lists Chapter02 as OpenAI Gym and Chapter06 as Deep Q-Networks, so a first run in Chapter06 is the natural place to see a training loop. The README does not document expected output, runtime or success criteria for any script, so you are reading the code to learn, not watching for a documented metric.

The pinned stack is the real limitation, not the code style

The most honest thing in this repository is the compatibility section. The author says outright that keeping examples working under the latest PyTorch and gym is not always simple, and gives a concrete casualty: OpenAI Universe, used extensively in Chapter 13, was discontinued by OpenAI. That chapter's environment no longer exists as a supported dependency, and no amount of reading the code will bring it back.

This is the failure mode you should plan for. The examples were written against gym 0.10.9 and torch 0.4.1. Modern gym renamed and restructured environments, and modern PyTorch changed APIs that older training loops rely on. A script that ran in 2018 may fail on import, on environment creation, or silently on a tensor operation that changed semantics. The README's note that exact code might differ from the book text cuts both ways: the repository may be ahead of the printed page in places, and behind your installed stack in others.

There is a second limitation that is easy to miss. Because each chapter is standalone, there is no shared abstraction to fix once and have every example benefit. If you want to modernize the stack, you are editing scripts chapter by chapter. For a reader following the book, that is fine. For anyone hoping to lift a training loop into a current project, the copy-paste cost is real, and the ptan 0.3 pin travels with it. This is a teaching repository, and it behaves like one.

How it compares with Stable-Baselines3 as a starting point

The obvious alternative for someone who wants to train an agent rather than read about training one is Stable-Baselines3, a maintained library with a common interface across algorithms. The difference is architectural, not cosmetic. Stable-Baselines3 gives you a small set of classes you instantiate and call learn() on, with environments supplied through the Gym API; the algorithm internals are hidden behind that interface. This repository does the opposite. Every algorithm is spelled out in the chapter that covers it, with the model, the loss and the update rule visible in the script you are reading.

That makes the two tools answer different questions. If you need a PPO agent running this afternoon, a library is the right shape. If you need to understand why the PPO loss is written the way it is, the Chapter15 material is the more direct route, because there is nothing between you and the code. The trade-off is that the library keeps working as gym and PyTorch move, while the samples are tied to the versions in requirements.txt. Choosing between them is choosing between a maintained abstraction and a readable one; this repository only offers the second.

A second comparison point is the book itself. The repository README links to the Packt page and notes that a DRM-free PDF is available at no cost to readers who already purchased a print or Kindle version. The code and the text are meant to be used together, and the errata section in the README (an entry for page 124, correcting Box to Discrete for the FrozenLake observation and action spaces) shows the author treating the text as something that gets corrected over time.

Licence, upgrade cost and what the repository does not cover

The repository is MIT licensed. That is permissive: you can reuse the code in your own projects, including commercial ones, provided you keep the licence notice. It says nothing about the book text, which is a separate copyrighted work sold by Packt, and nothing about the dependencies, each of which carries its own licence. If you plan to ship anything derived from these samples, check the licences of ptan, gym, PyTorch and the rest independently. This is a description of what the repository states, not legal advice.

Upgrade cost is the thing to budget for. The README's versioning scheme gives you two reference points, tag 01_release for the June 2018 publication state and master for the PyTorch 0.4.1 update, and a torch_1.0 branch that the README describes as not yet created. That means the documented migration path stops at 0.4.1. Anyone moving these examples to a current PyTorch is doing the port themselves, chapter by chapter, with no upstream branch to follow. The last push on 2026-03-02 shows the repository has seen activity, but the requirements file and the README's own compatibility notes are the authoritative picture of what is actually supported.

What the repository does not cover is equally worth stating. There is no CI configuration at the root, no test suite, and no documented expected output for any script. The README does not document rollback, troubleshooting or a support channel. The book is the manual; the repository is the appendix.

Editorial conclusion

Adopt it if you own the book and want runnable chapter code, or if you want to read how DQN, A2C, TRPO and AlphaGo Zero are wired in plain PyTorch. Do not adopt it as a dependency, as a maintained framework, or as an up-to-date reference for current gym and PyTorch APIs. Before you spend a session on it, check requirements.txt and confirm the pinned versions still install on your Python, then open Chapter06 and Chapter15 to see whether the style of code matches how you work.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. PacktPublishing/Deep-Reinforcement-Learning-Hands-On on GitHub
4. README