RETURNN ships kernels, submodules and no releases since 2018

A research framework whose newest tag is a 2018 beta

RETURNN is the extensible training framework from the RWTH Aachen speech and language processing group, described as a PyTorch and TensorFlow implementation of modern recurrent network architectures, optimised for fast and reliable training in a multi-GPU environment. Two papers describe it, one from 2016 and one from 2018, both on arXiv.

The version story is the first thing to sort out. The newest tags are v2.1.0-beta, described as transformer architecture support beta, published in November 2018, plus v2.0.2-beta and v2.0.1-beta before it. The last push, though, was on 2026-10-01, and the repository is not archived. So the project has taken commits for roughly eight years since its last tag, which means there is no release to pin and no changelog entry that maps a version to a feature.

For a framework whose selling point is reproducibility of research results, that is the central practical fact. Pin a commit hash, not a release, and keep your own record of the configuration that produced a result.

The stated goals are simplicity, flexibility and efficiency, with simplicity broken into writing configuration, debugging and reading code back, and efficiency split between training speed and decoding speed. Decoding speed is called out as especially important for production, which tells you which use case the maintainers had in mind.

Sequence chunking is the batching strategy, and it shapes everything else

The feature list is short and one entry dominates: sequence-chunking based batch training for recurrent networks, alongside ordinary mini-batch training for feed-forward networks.

Chunking is how this framework makes recurrent training fit in memory. Rather than treating a whole utterance as one training example, the sequence is cut into chunks, which is what lets variable-length audio and text be batched together at all. The same concern shows up elsewhere in the list, as memory management for large data sets and work distribution across multiple devices.

The architecture claim is deliberately wide: a flexible and fast architecture allowing all kinds of encoder-attention-decoder models. That is the reason the framework is still used for sequence-to-sequence work rather than only for the acoustic models it started with.

Two things follow for anyone reading the code. First, configuration is the interface, since the goals say explicitly that reading the config back should make the defined model, training and decoding clear, and the documentation splits into a basic usage guide and a technological overview. Second, the project documents its ideas outside the repository: an Interspeech 2020 tutorial on efficient and flexible implementation for ASR and MT with video and slides, plus a 2019 workshop talk with an exercise sheet.

The features that need a GPU are the ones that decide your hardware

Two of the listed features are tied to specific hardware, and one of them has no fallback.

Long short-term memory networks are supported including the group's own fast CUDA kernel, which is benchmarked separately against CuDNN and other TensorFlow LSTM kernels in a benchmark the documentation links. Writing your own kernel next to the vendor library is a research-group choice, and the benchmark page exists precisely because the claim needs evidence.

Multidimensional LSTM is the harder constraint: the documentation states it is GPU only, with no CPU version at all. If your plan involves multidimensional recurrence on CPU hardware, that is not a configuration problem, it is an unavailable feature.

For a framework aimed at multi-GPU training, this is not surprising, but it is worth checking before planning a CPU-only experiment. The environment requirements are modest by comparison: Python 3.8 or newer, TensorFlow 2.2 or newer for TensorFlow-based setups, and Torch 1.0 or newer for PyTorch-based ones, with minimum-version bumps recorded in the changelog.

Demos are synthetic, real recipes are another repository

The project separates what runs out of the box from what needs real data, and that split is documented clearly.

The `demos/` directory works on artificially generated data and is described as working as-is, which makes it the right place to start and the right thing to use for a smoke test. Real-world setups live in a separate repository, with speech recognition on Switchboard or LibriSpeech named as examples. A third repository holds benchmark setups against other frameworks, and the results from those comparisons are in the 2016 paper rather than in the repository.

So the practical shape of an evaluation is three steps: run a demo to confirm the installation works, clone the experiments repository for a realistic recipe, and read the 2016 paper if you want the comparison numbers. That is more work than a library with one example directory, and it is also the difference between a research framework and a product.

Questions have two documented homes, a wiki and a StackOverflow tag carrying the project name, which is useful to know before opening an issue.

Three runtime dependencies, and extras the requirements file omits

The runtime dependency list is short enough to read in one breath:

numpy
h5py
# Nest alternative, provides `tree`: https://github.com/rwth-i6/returnn/issues/1314
dm-tree

`dm-tree` is there with a comment explaining it is an alternative to Nest that provides the `tree` helper, linked to an issue.

The catch is in the sentence after the list. Dependencies are listed in `requirements.txt` and `requirements-dev`, and the documentation concedes that some parts of the code may require additional dependencies on demand, naming `librosa` and `resampy` as examples. So a clean install gives you a framework that imports, and a specific recipe may still need two audio packages installed by hand. That is a small annoyance that shows up at the worst moment, halfway through a preprocessing run.

There is also a legacy shape worth noticing in the tree: `rnn.py` and `__init__.py` sit at the repository root next to the `returnn/` package, alongside `setup.py`, `pytest.ini` and `pylintrc`.

Submodules are vendored, and the formatter steps around them

The repository vendors six external codebases as Git submodules under `returnn/extern/`: blocksparse, KenLM, OpenFST, ParseOggVorbis, a HawkAaronWarpTransducer warp-transducer, and WarpRna's warp-rna.

The build configuration treats that as a hard boundary. Black is configured with a line length of 120 and excludes all six submodule paths, as does Ruff, whose exclude list adds the PyTorch environment collection workflow. The comment above the Black list says these are all the Git submodules, pointing at `.gitmodules`.

Two consequences follow. A plain clone leaves those directories empty, and the reason the configuration mentions submodules at all is that the vendored trees are not the project's code and must not be reformatted when you run the tools. For speech work, KenLM and OpenFST are not optional extras, they are the language model and decoding machinery, so a shallow clone is not a working clone.

Also note where metadata lives. `pyproject.toml` in this repository contains only tool configuration, no project table, and the build backend is setuptools with numpy required at build time. Package metadata is assembled in `setup.py`, which is also where the release instructions live.

Releasing is a manual sequence documented in a docstring

The publishing procedure is written into the docstring at the top of `setup.py` rather than into a release document, and it reads like a working note.

It tells you to create `~/.pypirc` with the PyPI index server details, notes that `python3 setup.py register` is no longer needed, and gives both the old path of `python3 setup.py sdist upload` and a route through `twine` that came out of a specific problem the author had. There is also a debugging path: build the source distribution, install it with `--user`, and add `-v` because without it pip swallows all output, which is a specific and useful piece of advice.

The version string itself is derived by a helper in the package rather than read from a file, and the code comments explain why: the current time is deliberately not used as a fallback, because that would produce a version number larger than what actually exists and therefore useless. That comment is a good summary of how this project treats versions in general, which is to say carefully in the code and not at all in the tag history.

Whether that state suits you depends on what you need. For a research group already inside this ecosystem, the wiki, the tutorial and the two papers are the real documentation. For anyone adopting it fresh, the gap between the 2018 tags and the current tree is the first thing to raise with the maintainers.

Editorial conclusion

RETURNN suits an ASR or machine-translation group that wants encoder-attention-decoder control at the config level, or a research group that wants to compare its own LSTM kernel against CuDNN. It does not suit someone who needs a packaged library with versioned releases, since there are none, or someone on CPU-only hardware who wants multidimensional LSTM, which has no CPU implementation. Verify first that you can pin a commit rather than a release, that you clone with submodules so KenLM and OpenFST are populated, and that the recipe you need has its optional dependencies installed, because they are not in the requirements file.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. rwth-i6/returnn on GitHub