keon/seq2seq: A Readable Bahdanau Attention Baseline in PyTorch

The gap keon/seq2seq fills: a working attention model you can actually read

Most seq2seq repositories sit at one of two extremes. Either they are framework-scale codebases with configuration systems, checkpoint managers and distributed launchers, or they are single-file tutorial scripts that drop batching and GPU support to stay short. keon/seq2seq places itself in between. The README lists three stated goals: a modular structure that can be reused in other projects, minimal code for readability, and full utilization of batches and GPU. Those goals are in tension, and the repository resolves them by keeping the model small (a bidirectional GRU encoder and a GRU decoder with attention) while still supporting batched training on CUDA, Apple MPS or CPU. The intended audience is an engineer or student who wants to see Bahdanau attention implemented cleanly, then modify it. The dataset choice reinforces this: Multi30k German to English is small enough that a CPU smoke run is plausible, which the README explicitly acknowledges by suggesting smaller hidden_size and embed_size flags for CPU runs.

Encoder, decoder and the attention mechanism as described

The README states the architecture directly: a bidirectional GRU encoder, a GRU decoder, and attention following the paper Neural Machine Translation by Jointly Learning to Align and Translate. That paper is Bahdanau et al., and the distinguishing feature of that attention formulation is that the alignment score is computed by a small feed-forward network over the decoder's previous hidden state and each encoder annotation, rather than by a dot product. The README does not spell out which variant is implemented in code, so a reader who cares about that distinction should confirm it in the attention module rather than assume it. What the README does establish is the data path: Multi30k DE to EN is loaded through HuggingFace datasets, and tokenization is handled by spaCy. That means the pipeline is dataset load, spaCy tokenize, build vocabulary, batch, encode, decode with attention, compute loss. The repository description repeats the same framing: minimal Seq2Seq with attention for neural machine translation. Nothing in the supplied material describes beam search, length normalization, or any inference-time decoding strategy, so treat the training loop as the documented part and the generation path as something to inspect yourself.

Getting it running: the exact commands and flags from the README

The setup sequence is short and explicit. Requirements are Python 3.9 or newer, PyTorch 2.0 or newer on CPU, CUDA or Apple MPS, HuggingFace datasets, and spaCy 3.7 or newer. Installation is three commands: pip install -r requirements.txt, then python -m spacy download de_core_news_sm, then python -m spacy download en_core_web_sm. Both spaCy models are needed because the source language is German and the target is English. Training is a single invocation: python train.py -epochs 30 -batch_size 32 -lr 3e-4. Note the flag style: single-dash long options, which is argparse's older convention rather than the double-dash GNU style. Device selection is automatic, in the order CUDA, then MPS, then CPU, per the README. The README also names hidden_size and embed_size as the flags to shrink for a CPU smoke run, and gives a concrete example configuration of hidden=128 and embed=64 for that purpose. There is no configuration file, no YAML, and no environment variable mentioned. Everything is passed on the command line.

What the sanity-check table does and does not tell you

The README includes a CPU sanity check over 500 batches at hidden=128 and embed=64. It reports training loss falling from 9.19 at initialization to 4.84 at step 500, with perplexity dropping from 9803 to 127, and a final validation loss of 4.93. The README also notes that the random-initialization prior is log(|V|) which is approximately 9.19, which is a useful anchor: it tells you the initial loss is exactly what an untrained model should produce, so the number is a correctness signal rather than a performance claim. The gap between the final training loss of 4.84 and the validation loss of 4.93 is small, which suggests the run had not yet begun to overfit at that point, but 500 batches is far too short to conclude anything about convergence. What the table does not include is BLEU, any held-out test evaluation, or a comparison against a non-attention baseline. If you need to justify attention to a reviewer, this table will not do it. It is a smoke test, and the README presents it as one.

Where this repository is the wrong tool

Three constraints stand out. First, there is no subword segmentation anywhere in the supplied material. Tokenization is spaCy word-level, and the vocabulary is built from those tokens. For German, which compounds freely, word-level vocabularies produce a long tail of rare tokens and a high unknown-token rate. A production German to English system would use byte-pair encoding or SentencePiece, and adding that here means replacing the tokenization and vocabulary stages, not adjusting a flag. Second, the README documents training but says nothing about inference. There is no described script for translating a sentence, no beam search, and no checkpoint format described. If your goal is to run a translation, you will be writing the generation loop yourself. Third, the model is a single-layer bidirectional GRU with attention. That is the 2015 architecture. It will not match a Transformer, and the README makes no claim that it will. The right reason to use this repository is to understand or modify the mechanism, not to obtain translation quality.

The alternative: OpenNMT-py and what actually differs

The obvious comparison is OpenNMT-py, which implements the same encoder-decoder-with-attention family but as a configurable toolkit. The difference is not quality, it is where the complexity lives. OpenNMT-py expects a YAML configuration file, supports multiple attention variants, subword tokenization through SentencePiece or BPE, beam search at inference, checkpoint management and multi-GPU training. keon/seq2seq has none of that, and that absence is the point: the README's stated goal is minimal code for readability. A second alternative is the PyTorch sequence-to-sequence tutorial, which the README itself lists as a reference. The tutorial is a single script and does not use batching or GPU, so keon/seq2seq is a step up from it in engineering terms while staying close to it in size. If you want a system to train and deploy, OpenNMT-py is the more direct route. If you want to read every line and change the attention scoring function, keon/seq2seq is the smaller surface.

Maintenance, dependencies and the MIT licence

The repository is not archived and its last push is dated 2026-05-18, so it is being touched. There are no retrieved releases, which means there is no tagged version to pin against; you would be tracking the master branch. That matters for reproducibility: a commit hash is the only stable reference. The dependency set is small (PyTorch, HuggingFace datasets, spaCy) but each of those moves quickly, and the README notes that datasets replaced torchtext, which is a migration the project has already absorbed once. Expect similar work when spaCy or datasets change their APIs. The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence with no copyleft obligation, but it also means no warranty and no patent grant. This is a description of the licence text, not legal advice; if the model is going into a product, have counsel review it. The practical maintenance cost is low in absolute terms because the codebase is small, but there is no release cadence to rely on.

Editorial conclusion

Adopt keon/seq2seq if you need a small, readable Bahdanau attention reference you can modify, or a starting point for a course exercise on Multi30k. Do not adopt it if you need a production translation service, subword handling, or a model that has been evaluated beyond the single CPU loss table in the README. Before building on it, verify three things in the code itself: how the attention module masks padding so that padded source positions do not receive attention weight, how the vocabulary and special tokens are constructed from the spaCy tokenization pipeline, and whether the checkpoint saving path is documented at all. If those three checks pass for your use case, the repository is small enough to read end to end in an afternoon.

Official sources

1. Issues
2. keon/seq2seq on GitHub
3. License: MIT
4. README