OpenAI Whisper: a Python speech recognition model you run yourself

What Whisper replaces in a speech pipeline

A traditional speech stack chains separate components: a voice activity detector to find speech, a language identifier, an acoustic model, a language model for rescoring, and a translation step if the audio is not English. Whisper collapses those into one Transformer sequence-to-sequence model. The README describes the training tasks as multilingual speech recognition, speech translation, spoken language identification, and voice activity detection, all jointly represented as a sequence of tokens the decoder predicts. Special tokens act as task specifiers or classification targets, which is why a single checkpoint can be asked to transcribe, translate, or name the language.

The project is aimed at developers who want a working recognizer without assembling that pipeline. It is not aimed at people who want a hosted API with a dashboard; the repository ships a Python package and a CLI, and the homepage field points back at the GitHub repository. If your product needs real-time captions with partial results as someone speaks, this design, which reads a whole file and processes it in 30-second windows, is a mismatch.

How the sliding 30-second window works

The README states that transcribe() reads the entire file and processes the audio with a sliding 30-second window, performing autoregressive sequence-to-sequence predictions on each window. That sentence explains most of the operational behaviour you will observe. Audio is not streamed; the file is loaded first. Each window is decoded token by token, and the special tokens steer whether the decoder is transcribing, translating, or identifying language.

Two consequences follow directly from that architecture. First, memory scales with the model you load, not with the length of the audio, because the window is fixed at 30 seconds. Second, latency for a long file is roughly proportional to its duration divided by 30 seconds, multiplied by decode time per window. The README's speed table is measured transcribing English speech on an A100 and explicitly warns that real-world speed may vary significantly with language, speaking speed, and hardware. Treat those relative numbers as a ranking, not a promise.

The lower-level API exposes detect_language() and decode() for callers who want to skip transcribe()'s orchestration. The README shows that example but the excerpt here is truncated, so the exact signature of decode() is something to read in the repository rather than assume.

Installing Whisper and transcribing your first file

The README gives two install paths. The released package comes from PyPI:

pip install -U openai-whisper

If you want the latest commit instead, the README offers a direct Git install:

pip install git+https://github.com/openai/whisper.git

Whisper also needs ffmpeg on the system, installed through whatever package manager you use. The README lists apt, pacman, Homebrew, Chocolatey, and Scoop commands. On Ubuntu or Debian that is:

sudo apt update && sudo apt install ffmpeg

The README warns that tiktoken may lack a pre-built wheel for your platform, in which case you need Rust installed. If pip fails with No module named 'setuptools_rust', the documented fix is:

pip install setuptools-rust

With the package and ffmpeg in place, the CLI is the fastest first use. The README's example passes several files and selects the turbo model:

whisper audio.flac audio.mp3 audio.wav --model turbo

You should see transcription output per file. To transcribe non-English speech, name the language, as in the README's Japanese example:

whisper japanese.wav --language Japanese

The same work in Python is three lines, and the README prints result["text"]:

import whisper

model = whisper.load_model("turbo")
result = model.transcribe("audio.mp3")
print(result["text"])

Run whisper --help for the full option list. The README points at tokenizer.py for the list of available languages, which is the authoritative place to check a language code before passing it.

The turbo model cannot translate, and other limits

The sharpest limitation in the README concerns translation. The turbo model is the default and is described as an optimized version of large-v3 with faster transcription and minimal accuracy degradation, but the README states plainly that turbo is not trained for translation tasks. A note adds that turbo will return the original language even if --task translate is specified. If you need non-English speech rendered into English, the documented path is a multilingual model such as medium or large:

whisper japanese.wav --model medium --language Japanese --task translate

Accuracy is uneven across languages. The README says performance varies widely depending on the language and points to WER and CER figures for large-v3 and large-v2 on Common Voice 15 and Fleurs, with more metrics in the paper's appendices. It does not claim uniform quality, and the language-breakdown.svg in the repository is the visual version of that caveat. If your deployment language is not one of the well-resourced ones, measure before you commit.

The English-only models are another trade-off worth naming. The README says the .en variants tend to perform better for English-only applications, especially tiny.en and base.en, and that the gap narrows for small.en and medium.en. Choosing a multilingual model for an English-only workload is therefore a deliberate cost, not a neutral default.

Finally, the README does not document streaming, chunked input, or rollback of a partial transcription. Those are absences in the documentation, not features you should assume exist.

Choosing between Whisper and a hosted speech API

The real alternative for most teams is a hosted transcription service, and the difference is architectural rather than a matter of accuracy. A hosted API keeps the model on someone else's hardware, bills per minute, and typically returns results over HTTP, which suits intermittent workloads and removes the VRAM question entirely. Whisper runs locally: you install PyTorch and ffmpeg, download a checkpoint, and pay in GPU memory and decode time instead of per-request fees. The README's VRAM table runs from roughly 1 GB for tiny and base through about 2 GB for small, 5 GB for medium, 6 GB for turbo, and 10 GB for large.

That table is the deciding document. If you have no GPU, the smaller models are the only realistic path, and the README's own note that tiny.en and base.en benefit most from the English-only variants suggests those are the intended low-resource options. If you need translation into English, the hosted route may be simpler than running medium or large locally, since turbo, the fast default, is explicitly not trained for that task.

Self-hosting also buys you something a hosted API does not: the MIT licence means you can inspect, modify, and redistribute the code, and the model weights and code stay on your infrastructure. Whether that matters depends on your data handling requirements, which the repository does not address.

Maintenance, versions, and the MIT licence

The repository is not archived. Its last push was on 2025-06-26, which is more than six months before today, so the project should not be described as actively developed on the strength of that date alone. The release history shows v20250625 on 2025-06-26, preceded by v20240930 and v20240927. That cadence, roughly annual releases in the recent record, is the upgrade cost you are signing up for: the package is stable enough that updating is infrequent, and the README's own update instruction for a Git install is a force reinstall without dependencies:

pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git

The pyproject.toml declares requires-python >=3.8 with classifiers through Python 3.13, and names more-itertools, numba, numpy, tiktoken, torch, and tqdm as dependencies, plus triton on x86_64 Linux. That triton marker is platform-specific, which is a reminder that the dependency set is not identical everywhere. The README says the code was trained and tested on Python 3.9.9 and PyTorch 1.10.1 but expects compatibility with Python 3.8 through 3.11 and recent PyTorch versions, so the tested configuration and the supported configuration are not the same thing.

The licence is MIT, declared both in the LICENSE file and in pyproject.toml as license = { text = "MIT" }. MIT is permissive, but this is a description of the licence text, not legal advice; if you redistribute the package or its weights, read the LICENSE file and the model card yourself.

Editorial conclusion

Adopt Whisper if you can install ffmpeg and PyTorch and want a local, MIT-licensed transcription model with a one-line Python API. Do not adopt it if you need streaming, word-level timestamps as a documented feature, or translation from non-English speech using the turbo model, which the README says returns the original language even with --task translate. Before committing, verify which model size fits your VRAM (the README lists roughly 1 GB for tiny up to 10 GB for large), check whether tiktoken has a pre-built wheel for your platform, and confirm that the last push, on 2025-06-26, is recent enough for your dependency policy.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. Release notes