RETURNN: a config-driven training framework for recurrent and attention sequence models
The RWTH extensible training framework for universal recurrent neural networks
At a glance
- What is it?
- RETURNN is RWTH's PyTorch/TensorFlow training framework for sequence models, built around a declarative config file rather than a Python training script. It suits research groups reproducing ASR and MT setups, and it is a poor fit for anyone expecting a general-purpose deep learning library.
- Who is it for?
- Adopt RETURNN if you are training recurrent or encoder-attention-decoder sequence models and want the model, training loop and decoding defined in a config file rather than hand-written Python, and if you are willing to read the readthedocs pages and work through the demos before writing your own config.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem RETURNN solves: sequence experiments without a hand-written training loop
Most deep learning work starts by writing a training script: data loading, a model class, an optimizer step, a checkpoint saver, a decoder. RETURNN takes the opposite position. The README describes the framework as a PyTorch/TensorFlow-based implementation of modern recurrent neural network architectures, optimized for fast and reliable training of recurrent networks in a multi-GPU environment, and lists simplicity, flexibility and efficiency as its goals, with decoding speed called out as especially important for production. The intended user is a researcher who wants to define an experiment as configuration rather than as code. The framework's own framing of simplicity is about writing, debugging and reading configs, not about minimising the number of concepts. The README points to an Interspeech 2020 tutorial described as an introduction to the core concepts, which is a fair signal that the concepts are the entry cost. The listed feature set is sequence-specific: mini-batch training of feed-forward networks, sequence-chunking based batch training for recurrent networks, LSTMs including a project CUDA kernel, multidimensional LSTM that is GPU-only with no CPU version, memory management for large data sets, work distribution across multiple devices, and encoder-attention-decoder models. Speech recognition and machine translation are the workloads the surrounding material is built around, down to the real-world examples repository containing Switchboard and LibriSpeech setups.
Config as the model definition, and two backends underneath
The mechanism RETURNN is built around is a declarative experiment description. The README's simplicity goal is stated in terms of writing, debugging and reading config and code, and the 2018 release notes describe a TensorFlow-based pre-release with a function for inference with a single string as input, which tells you the config drives both training and inference. The release history also shows the framework changing backends over time: v2.0.1-beta and v2.0.2-beta are labelled TensorFlow-based, v2.1.0-beta adds Transformer architecture support in beta, and the current README describes the project as PyTorch/TensorFlow-based. So the same config vocabulary is meant to sit on top of either backend, with TensorFlow-based setups requiring TensorFlow >= 2.2 and PyTorch-based setups requiring Torch >= 1.0. On top of that sits the sequence-specific machinery: sequence-chunking based batch training for recurrent networks, a project CUDA LSTM kernel, memory management for large data sets, and work distribution across multiple devices. The README does not spell out the config schema in the repository text itself; it defers to the documentation, the basic usage page and the technological overview page on readthedocs. Treat the config format as something you learn from those pages and from the demos, not from the README.
Getting a first experiment running
The README gives two entry points. First, the demos directory at github.com/rwth-i6/returnn/tree/master/demos, described as many example demos which work on artificially generated data and should work as-is. Second, the real-world examples at github.com/rwth-i6/returnn-experiments, which include setups for speech recognition on Switchboard and LibriSpeech. Dependencies are listed in requirements.txt and requirements-dev, with the caveat that some parts of the code may require additional dependencies such as librosa and resampy on demand. Python >= 3.8 is required, and the README says bumps to the minimum Python version are recorded in CHANGELOG.md. Backend minimums are TensorFlow >= 2.2 for TensorFlow-based setups and Torch >= 1.0 for PyTorch-based setups. The README does not print an install command or a run command in the text available here, so the honest sequence is: install the dependencies from requirements.txt, confirm your Python and backend versions against those minimums, then run the demos as-is before adapting a config of your own. If you are evaluating RETURNN for a production decoder, the README points at the TensorFlow LSTM benchmark page on readthedocs for kernel comparisons, and the benchmark setups against other frameworks live in the returnn-benchmarks repository rather than in this one.
Where RETURNN is the wrong tool
The clearest limitation is in the README itself: multidimensional LSTM is GPU only, with no CPU version. If your environment has no GPU, that part of the feature list is simply unavailable. The second limitation is the release cadence. The newest release listed is v2.1.0-beta from November 2018, and the two before it are also beta, so anyone who pins to a tagged release is pinning to something roughly eight years old while the master branch continued to move. That is a real problem for reproducibility across a long project, and it is a reason to decide early whether you track master or a tag. The third is scope. RETURNN is built for sequence models in speech and translation, and its configuration vocabulary reflects that. If your work is image classification or tabular modelling, the sequence-chunking machinery, the decoder tooling and the LSTM kernel are weight you carry without using. The fourth is documentation surface. The README is a signpost collection: it sends you to readthedocs for basic usage and the technological overview, to a wiki, to two papers, to a 2020 tutorial video, and to StackOverflow with the RETURNN tag. That is a lot of places to consult before your first config, and the repository text here does not contain the config schema itself.
RETURNN compared with PyTorch Lightning
The natural alternative for someone who wants less boilerplate is PyTorch Lightning, and the difference is architectural rather than cosmetic. Lightning keeps the model as a Python class and organises the training loop around it, so the experiment is still code, just code with the loop factored out. RETURNN moves the model definition itself into configuration, which is why its stated goals include reading a config and seeing the defined model, the training and the decoding. The trade is expressiveness for uniformity: a config is easier to diff, copy between experiments and hand to a colleague, but anything that does not fit the config vocabulary has to be expressed through the framework's extension points rather than by writing arbitrary Python in the training step. RETURNN also carries sequence-specific components that a general trainer does not, including the project CUDA LSTM kernel, sequence-chunking based batch training, memory management for large data sets, and multi-device work distribution. If your models are recurrent or encoder-attention-decoder and your data is large, those components are the reason to pick RETURNN. If your models are not sequences, Lightning or plain PyTorch will get you there with less to learn.
Licence, maintenance and the cost of tracking master
The repository metadata reports the licence as NOASSERTION, which means no standard licence identifier was detected. The README does not name a licence either. Before you build on RETURNN, read the LICENSE file in the repository and, if the terms matter to your organisation, get them reviewed; this is a factual gap in the metadata, not a judgement about the terms. On maintenance, the signals point in two directions at once. The repository is not archived and the last push is recent, so master is alive. The tagged releases are not: v2.1.0-beta is from 2018 and both earlier tags are beta as well. Practically, that means the upgrade cost is front-loaded. You either track master and accept that the code you depend on can change without a version number to hold onto, or you pin a 2018 beta and accept that you are running an old TensorFlow-era snapshot. The README's own maintenance notes are narrow: dependency lists live in requirements.txt and requirements-dev, minimum Python bumps are recorded in CHANGELOG.md, and backend minimums are TensorFlow >= 2.2 and Torch >= 1.0. There is no stated deprecation policy in the material available here, so plan for the master-tracking path and read CHANGELOG.md as part of your upgrade routine.
Editorial conclusion
Adopt RETURNN if you are training recurrent or encoder-attention-decoder sequence models and want the model, training loop and decoding defined in a config file rather than hand-written Python, and if you are willing to read the readthedocs pages and work through the demos before writing your own config. Do not adopt it as a general-purpose deep learning library or if you need a stable tagged release, since the newest release listed is v2.1.0-beta from 2018 while the master branch is actively developed. Before committing, verify three things: that the demos in the repository run on your machine, that the TensorFlow or PyTorch version your setup needs matches the minimums in the README, and which licence actually applies to your use, because the repository metadata reports NOASSERTION rather than a named licence.
Community notes