WeNet: a production-oriented end-to-end speech recognition toolkit

What WeNet solves, and who is actually expected to run it

Speech recognition projects tend to split into two halves that do not talk to each other: a research repository that produces checkpoints and a separate engineering effort that turns those checkpoints into something a service can call. WeNet's stated design principle is 'production first and production ready', and the repository layout reflects that. The wenet/ directory holds the Python modeling and decoding code, runtime/ holds a deployment runtime, and examples/ contains per-corpus recipes such as examples/aishell/, examples/librispeech/ and examples/wenetspeech/. The intended user is not someone who wants a transcription API key. It is a team that owns its acoustic models and needs to train, evaluate and ship them.

The Python package covers the lighter end of that. It exposes a console script and a load_model function, so a developer can transcribe a file without touching the training pipeline. The training and deployment install path is heavier and assumes Conda, a specific CUDA version and a torch build to match. Those are two different audiences sharing one repository, and the README treats them as separate installation sections for that reason.

The distinction matters when you estimate effort. A team that only needs inference installs one package and writes three lines of Python. A team that needs a model for its own domain is signing up for data preparation, recipe selection, training runs and a runtime build, and the repository gives no shortcut between those two positions. The pretrained models are the only bridge, and they are a fixed list.

How the toolkit is put together: recipes, models, runtime

The architecture visible in the repository is layered. At the bottom, examples/ holds one directory per dataset. Each recipe carries its own data preparation, training configuration and decoding scripts, which is why the project can claim results on many public datasets without a central data abstraction. Above that, the wenet/ package contains the model definitions and the decoding logic. The topics list names Conformer and Transformer, and the citations point to the two WeNet papers, so the modeling side is the end-to-end family rather than a hybrid DNN-HMM stack.

Decoding is where the project leans on older tooling. The acknowledgements state that WFST-based decoding for language model integration borrows from Kaldi, and that the TLG graph construction follows EESEN. That means a language model integration path exists, but it runs through graph building rather than through a neural rescoring loop. The runtime/ directory is the deployment half: the README describes building it with cmake under runtime/libtorch, and notes that the x86 runtime and language model support are optional. If you skip that build, you stay in Python.

The pretrained model story is separate again. The README links to docs/pretrained_models.md and to a HuggingFace demo space, and the command-line interface accepts model names rather than checkpoint paths. So there are three ways in: run a named pretrained model, train your own from a recipe, or build the runtime and embed it.

The recipe-per-dataset structure has a cost that the README does not discuss. Fifteen example directories means fifteen sets of scripts that were written at different times against different assumptions. Nothing in the README describes a shared data format that would let you move a recipe from one corpus to another, so adapting a recipe to your own audio is a code-reading exercise rather than a configuration change.

Installing the Python package and transcribing a first file

The shortest path is the pip install from the Git URL. It pulls in torch>=1.13.0 with no upper bound, which the README flags explicitly: pip may install a torch build newer than your GPU driver supports. Run the CUDA check immediately after install, because a mismatch does not raise an error.

pip install git+https://github.com/wenet-e2e/wenet.git
python -c "import torch; print(torch.cuda.is_available())"

If that prints False on a machine with a GPU, torch has silently fallen back to CPU. The README's remedy is to reinstall a torch build matching your driver's CUDA version; its example is below.

pip install torch==2.4.0+cu121 torchaudio==2.4.0+cu121 --index-url https://download.pytorch.org/whl/cu121 --force-reinstall

With the environment settled, the command-line entry point takes a model name and an audio file. The README lists paraformer, firered and wenetspeech for Chinese, and whisper-large-v3 or whisper-large-v3-turbo for English. Use -h for the full parameter list.

wenet -m paraformer audio.wav

One detail that catches people: --device defaults to cpu even when a GPU is present. You have to pass --device cuda explicitly. The Python API behaves the same way, with device='cuda' passed to load_model.

import wenet

model = wenet.load_model('paraformer')
result = model.transcribe('audio.wav')
print(result.text)

The README points to docs/python_package.md for GPU and Turing (for example Tesla T4) notes on dtype and attention-backend options. That document, not the README, is where the hardware-specific settings live. Note the two silent-failure modes stacked on top of each other: a torch build that does not match your driver, and a device default that keeps you on CPU. Neither produces an error message, so a first run that is merely slow tells you nothing until you check both.

Training and deployment install: Conda, CUDA 12.1, torch 2.2.2

The training path is a different install with different pins. The README recommends a Conda environment on Python 3.10, sox from conda-forge, CUDA 12.1, and torch 2.2.2+cu121 with the matching torchaudio. The sox dependency is a recurring failure point: the README documents a RuntimeError about set_buffer_size requiring the sox extension, with separate fixes for Ubuntu, CentOS and Conda.

conda create -n wenet python=3.10
conda activate wenet
conda install conda-forge::sox
pip install torch==2.2.2+cu121 torchaudio==2.2.2+cu121 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
pre-commit install

The requirements.txt file adds deepspeed>=0.14.0, tensorboard and tensorboardX for training runs, and pins openai-whisper==20231117. That last pin is exact, while the torch entries in the same file use lower bounds only. The mixed pinning style is worth reading before you assume the file defines a reproducible environment.

If you need the x86 runtime or language model support, the README describes a cmake build under runtime/libtorch that requires cmake 3.14 or above. Otherwise it says you can ignore that step. For Ascend NPU users there is a separate extras target, pip install -e .[torch-npu], with a version table listing minimum and recommended CANN, torch, torch-npu, torchaudio and deepspeed versions. The NPU table is the only place in the README where a full version matrix is spelled out; the CUDA path relies on prose recommendations instead.

Where WeNet is the wrong tool

The clearest limitation is operational, not algorithmic. The README's own GPU note says the CLI runs on CPU by default and that you must opt into CUDA. A user who installs the package, runs a transcription and finds it slow may conclude the model is slow, when the actual cause is a default they did not override. That is a design choice that trades safety for surprise.

The second limitation is dependency control. The package metadata pins torch>=1.13.0 and torchaudio>=0.13.0 with no ceiling, while the training instructions recommend 2.2.2+cu121 and the NPU extras pin 2.2.0. Three different torch expectations live in one repository. If your deployment depends on a frozen environment, you are the one who reconciles them.

Third, the language model path is graph-based. The acknowledgements credit Kaldi for WFST decoding and EESEN for TLG graph construction. That is a mature approach, but it is not the shallow-fusion or neural rescoring workflow that teams coming from Whisper-style tooling may expect. If your plan is to bolt on a large external language model at decode time, check the runtime documentation before assuming the path exists.

Finally, the pretrained model list is the boundary of what you get for free. The README names five model identifiers and points to docs/pretrained_models.md for the rest. Nothing in the README suggests an automatic fallback if your language is not on that list, and nothing describes how much labeled audio a fine-tune would need. If your language is absent, you are back to the recipe path with no stated starting point.

WeNet compared with Whisper

The comparison is unavoidable because WeNet ships openai-whisper as a dependency and exposes whisper-large-v3 as a model choice. The difference is in what each side owns. Whisper is a fixed set of pretrained multilingual models: you install it, pick a size, and transcribe. There is no training recipe in the loop and no runtime to build. WeNet's structure assumes you may want to train on your own corpus, which is why examples/ contains fifteen dataset recipes and why the runtime exists as a separate cmake project.

That difference cuts both ways. If your audio matches Whisper's training distribution and you have no labeled data of your own, WeNet's training machinery is overhead you will never exercise. If you have domain audio, a labeled set and a latency budget, the ability to train a Conformer on your own data and then serve it through a libtorch runtime is the reason to be here.

Note also that using Whisper inside WeNet does not give you Whisper's ecosystem; it gives you WeNet's loading and inference path around a Whisper checkpoint. The model names are the same, but the surrounding tooling, the device default and the runtime story are WeNet's, not OpenAI's. Teams that expect the two to behave identically will be surprised by the CPU default alone.

Maintenance, licensing and what an upgrade costs

The repository is not archived, and the last push was on 2026-09-07. The most recent tagged release listed in the repository is v3.1.0 from 2024-05-23, following v3.0.1 and v3.0.0 in early 2024. That gap between the last release tag and the last push is worth noting: if you depend on tagged releases, you are depending on something that has not been tagged in over two years, even though the default branch has moved.

Upgrade cost concentrates in two places. The first is torch. Because the package allows any torch at or above 1.13.0, a fresh install months from now can resolve to a very different build than the one you validated. Pin torch and torchaudio in your own environment file rather than trusting the package metadata. The second is the runtime. The libtorch build is a cmake project under runtime/libtorch; if you ship it, you own that build pipeline and the CUDA version it links against.

On licensing, the repository is Apache-2.0 and the badge in the README states the same. Apache-2.0 permits commercial use and modification and includes a patent grant. The README does not describe the licences of the individual pretrained checkpoints, and the project acknowledges borrowing code from ESPnet, Kaldi, EESEN and OpenTransformer. If you redistribute a model or a runtime binary, verify the provenance and terms of the specific artifacts you ship; this article is not legal advice.

Trying it against your own audio

The fastest honest evaluation is to install the package, force the device, and run one of your own files through two of the named models to see which fits your language and audio. The README lists paraformer, firered and wenetspeech for Chinese and whisper-large-v3 or whisper-large-v3-turbo for English, so a Chinese-language evaluation has three candidates and an English one has two.

wenet -m wenetspeech --device cuda audio.wav

If that output is good enough, you may never need the training half of the repository. If it is not, the next step is not to tune the CLI; it is to pick the recipe under examples/ closest to your data and read its run scripts. The distance between those two paths is the real decision this project asks you to make.

A practical way to frame the trial: run the same file through the CLI and through the Python API, with --device cuda set in both, and compare the text. If the two agree, your environment is consistent and the remaining question is model quality. If they diverge, you have a device or dtype problem, and docs/python_package.md is the place to look before you touch any training configuration.

Editorial conclusion

Adopt WeNet if you need to train or fine-tune your own Conformer or Transformer acoustic models, or if you need a libtorch runtime to embed recognition in a service, and you are willing to manage CUDA and torch versions yourself. Do not adopt it if you only want a hosted transcription endpoint or if your team cannot maintain a Python training environment. Before committing, verify three things on your own hardware: that torch.cuda.is_available() returns True after install, that your target language is covered by the models listed in docs/pretrained_models.md, and that the Apache-2.0 terms fit how you plan to redistribute the runtime.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. wenet-e2e/wenet on GitHub