keon/deep-q-learning: Two Keras DQN Files and What the 2026 Update Changed

The gap this repository fills: a DQN you can read in one pass

Most reinforcement learning repositories are frameworks. They ship replay buffers, runners, logger integrations, and config systems, and by the time you find the line where the target Q value is computed you have lost the thread. This repository goes the other way. The README states the implementations are "Minimal and Simple Deep Q Learning Implemenation in Keras and Gym" and "Under 100 lines of code". The audience is someone who has read about experience replay and target networks and wants to see the smallest working version. The blog article linked from the README carries the explanation for dqn.py, so the code is meant to be read alongside prose rather than as a standalone library. There is no package to install, no CLI, no configuration schema. The repository is a teaching artifact that happens to run.

dqn.py versus ddqn.py: the target update is the whole difference

The two files implement the same loop with one change in how the bootstrap target is built. In DQN, a single network both picks the greedy action and supplies its value, which biases the target upward. Double DQN splits that job. The 2026 changelog states that ddqn.py "now implements actual Double DQN (online net selects, target net evaluates)", which is a correction, not a feature addition. Read that sentence carefully: before this change the file named ddqn.py did not do the thing its name promised. If you copied this repository before May 2026, or if you are reading a fork that predates the fix, your DDQN baseline is a DQN baseline with different hyperparameters. The README is candid about the sibling file: "The training might be unstable for dqn.py. This problem is mitigated in ddqn.py." That is the honest framing. The instability is expected, and the second file exists to show the fix.

Replay memory as a bounded deque, and why that detail is load-bearing

The README notes a deviation from the original tutorial code: "I also made the memory a deque instead of just a list. This is in order to limit the maximum number of elements in the memory." With a plain list, every transition ever seen stays resident, and a long CartPole run grows memory without bound until the process dies. A deque with maxlen drops the oldest transition once the buffer is full, which is the standard behavior and also the reason the buffer size becomes a real hyperparameter instead of an accident. The trade-off is that deque eviction is strictly first-in-first-out. There is no prioritization, no stratification by reward sign, and no guarantee that rare high-value transitions survive. For CartPole that is fine. For an environment with sparse reward, a FIFO buffer can evict the few informative transitions you have.

The 2026 migration: gymnasium, Keras 3, and env-agnostic DDQN

The May 2026 changelog lists four changes that together define the current state of the repository. First, migration "from gym to gymnasium" with the modern reset/step API, which means reset returns a tuple rather than a bare observation, and step returns five values rather than four. Second, Keras 3 compatibility: Adam now takes learning_rate= rather than the older lr=, the Input layer is used, and the loss is keras.losses.Huber. Third, and easiest to miss, "Removed CartPole-specific reward shaping from ddqn.py so it's env-agnostic." Reward shaping is the kind of thing that makes a demo look better and makes a baseline dishonest. Removing it means ddqn.py no longer carries an advantage that does not transfer to your environment. Fourth, save() and load() now persist epsilon alongside the weights, so a resumed run continues the exploration schedule instead of restarting it. The 2018 entry shows dqn_batch.py was split out for batched-update training, so there are more than two files if you count that one.

Running it: what the repository actually gives you

There is no installable package and no entry point script documented in the README, so the workflow is to clone the repository, install the dependencies from requirements.txt, and run a file directly, for example python dqn.py or python ddqn.py. The README does not enumerate the hyperparameters, the buffer size, the epsilon decay schedule, or the network architecture, and it does not state expected episode counts or scores. Those numbers are not in the supplied material, so treat any figure you see quoted elsewhere as unverified. What is documented is the persistence surface: save() and load() persist weights and, since the 2026 update, epsilon. If you write your own training loop around these files, that pair is the interface to preserve. The blog post at keon.kim/writing/deep-q-learning/ is where the dqn.py walkthrough lives, and the README says a DDQN article was planned, though the material does not confirm it was published.

Where this stops being the right tool

Three limits are visible from the repository layout alone. The first is scale. A single-environment, single-network loop with a FIFO buffer will not saturate a GPU, and there is no vectorized environment handling, so wall-clock throughput is bounded by one environment stepping at a time. The second is algorithmic coverage. There is no prioritized replay, no n-step returns, no dueling architecture, and no distributional variant. If your problem needs any of those, you are writing them yourself, and at that point you are maintaining a fork rather than using a reference. The third is evaluation. Nothing in the README describes a separate evaluation pass, a fixed seed protocol, or a way to compare two runs. A DQN that looks like it learned can be a lucky seed, and this repository gives you no machinery to tell the difference. For a CartPole demo that is acceptable. For a claim about a new environment it is not.

Stable-Baselines3 is the alternative, and the difference is scope

The obvious comparison is Stable-Baselines3, which also targets Gymnasium environments and also ships DQN. The difference is not quality, it is what each one is. Stable-Baselines3 is a library with a common interface across algorithms, callbacks, evaluation helpers, and vectorized environment support. This repository is two files you read to understand the update rule. If your goal is a trained agent on a benchmark, Stable-Baselines3 gets you there with less code you own. If your goal is to understand why the target network exists, Stable-Baselines3 hides that line inside an abstraction, and this repository puts it in front of you. A reasonable path is to read dqn.py and ddqn.py first, then move to Stable-Baselines3 for anything you intend to run at length. The two are complements, not substitutes.

Maintenance cost and the MIT licence

The repository is MIT licensed, which permits commercial use, modification, and redistribution provided the copyright notice and permission notice are retained. That is a permissive arrangement and it is one reason a teaching repository like this gets copied into internal codebases. The practical cost is not the licence, it is the migration treadmill. The changelog shows the code has been carried through Keras 2 to Keras 3 and gym to gymnasium across roughly nine years, with the most recent entry in May 2026. Each of those migrations changed API surface: the reset and step signatures, the optimizer keyword, the loss import path. If you vendor these files, you inherit that maintenance. The last push timestamp is recent, so the repository is not abandoned, but there are no retrieved releases, which means there is no versioned artifact to pin. You are tracking the master branch, and that is a real supply-chain consideration for anything beyond a learning exercise. If you need a stable dependency, wrap the parts you use behind your own interface rather than importing the files directly.

Editorial conclusion

Adopt this if you want a readable DQN and DDQN baseline in Keras 3 and Gymnasium, or if you are teaching the target-network update and want two files small enough to read in one sitting. Do not adopt it as a training framework: there is no vectorized environment handling, no prioritization, no n-step returns, and no evaluation harness. Before relying on it, verify the DDQN action selection against the changelog claim, confirm your Gymnasium environment returns the reset tuple the code expects, and check whether load() restoring epsilon changes your intended schedule.

Official sources

1. Issues
2. keon/deep-q-learning on GitHub
3. License: MIT
4. Project website
5. README