WhisperX: word-level timestamps and diarization on top of Whisper

The timestamp problem WhisperX was built to fix

Whisper's own timestamps are utterance-level. The WhisperX README states plainly that they "can be inaccurate by several seconds", which is fine for reading a transcript and useless for burning subtitles, cutting clips, or aligning a transcript to a video timeline. WhisperX keeps the Whisper transcription quality and replaces the timing layer: it runs phoneme-level forced alignment with wav2vec2 to place each word in the audio. The audience is narrow and practical. Subtitle and caption pipelines, podcast and interview tooling, meeting transcription where you need to know who spoke, and any downstream system that indexes audio by timestamp. If you only want a text dump, this is a heavier tool than the problem requires.

How the pipeline is wired: VAD, batched ASR, alignment, diarization

The README describes the stages. Voice activity detection runs first and is on by default; the README says it "reduces hallucination & batching with no WER degradation". The detected speech regions are then batched into a faster-whisper backend, which is where the speed claim comes from: the README quotes 70x realtime with large-v2. Transcription output is then passed to wav2vec2 for forced alignment, which produces the word-level timings. Separately, pyannote-audio supplies speaker diarization, and WhisperX merges those speaker segments onto the transcript so each line carries a speaker identity. The v3 release note adds a sentence-level segmentation step using nltk sent_tokenize, which the README says improves subtitling and diarization. Two consequences follow from that ordering. Alignment can only be as good as the transcript it is aligning, so a mis-transcribed word gets a confident-looking timestamp on the wrong word. And diarization is a second model with its own failure modes, so speaker labels are an additional source of error rather than a free upgrade.

Installing WhisperX and running your first file

The README recommends the PyPI route. If you want GPU acceleration, install CUDA toolkit 12.8 before WhisperX; skip that step for CPU only.

pip install whisperx

There is also a uvx path, which runs the tool without a persistent install:

uvx whisperx

For a development checkout, the README gives a clone plus uv sync with all extras and dev dependencies:

uv sync --all-extras --dev

Once installed, the command-line entry point is whisperx. The README's example runs the default model on a single file:

whisperx path/to/audio.wav

The README notes you can add --highlight_words True to visualise word timings in the .srt output. Expect a subtitle file whose word timings come from the alignment stage rather than from Whisper's own segment boundaries. The README also warns that the development version may contain experimental features and bugs, and points production users at the stable PyPI release. Note the version in pyproject.toml is 3.8.7rc1, a release candidate, so pin deliberately rather than letting an installer resolve to whatever is newest.

Turning on diarization, and the token you have to bring

Diarization is not on by default because it needs an external credential. The README says to pass your Hugging Face access token after the --hf_token argument, and to accept the user agreement for the pyannote/speaker-diarization-community-1 model on Hugging Face first. If you skip the agreement, the model download will not work regardless of how valid your token is. This is the step most first-time users hit, and it is a licensing gate rather than a bug. It also means diarization cannot run fully offline on a fresh machine until the model has been fetched once.

Where WhisperX is the wrong tool

The dependency list in pyproject.toml is the clearest statement of the constraints. Python is pinned to >=3.10, <3.14, so a 3.9 environment cannot install it at all. torch is pinned at 2.8.0 and torchcodec at >=0.6.0,<0.8.0, and the project's own uv override notes that torchcodec has no wheels for Linux aarch64. On that architecture the install path is not merely awkward, it is excluded by the project's own configuration. The README also cites requiring under 8GB of GPU memory for large-v2 with beam_size=5, which is a real ceiling on smaller cards. Beyond hardware: if your audio is clean single-speaker narration and you only need text, the alignment and diarization stages add models, download time and failure surface for output you will not use. And if you need a hosted API rather than a local process, WhisperX itself is a CLI and library, not a service; the README does not document a server mode.

WhisperX compared with faster-whisper and plain Whisper

WhisperX is not a reimplementation of Whisper. It depends on faster-whisper for the transcription stage, so the raw recognition is largely the same engine underneath. The difference is everything after: forced alignment for word timings, VAD-gated batching, and pyannote diarization. Plain Whisper gives you segment-level timestamps and no speaker labels. faster-whisper gives you the faster inference backend and its own timestamp behaviour, but no wav2vec2 alignment pass and no speaker attribution. So the honest framing is not "WhisperX versus faster-whisper" as competing engines; it is whether you want the alignment and diarization layers stacked on top. If you do, WhisperX is the assembly. If you do not, faster-whisper alone is fewer moving parts and one fewer model download.

Maintenance, licence and upgrade cost

The repository is not archived and the last push was on 2026-08-30, so it is being worked on. Recent releases run 3.8.5 in April 2026, 3.8.6 in May 2026, and 3.8.7rc1 in June 2026, which tells you the current head is a release candidate rather than a stable tag. The licence is BSD-2-Clause, declared in both the repository metadata and pyproject.toml. That is permissive: it does not impose copyleft obligations on your own code. It says nothing about the models WhisperX downloads. The Whisper weights, the wav2vec2 alignment model and the pyannote diarization model each carry their own terms, and the pyannote model in particular sits behind a Hugging Face user agreement you have to accept. Check those separately; this is a factual boundary, not legal advice. Upgrade cost is dominated by the pinned scientific stack. torch, torchaudio, torchvision and torchcodec move together, and the aarch64 exclusion means a platform change can turn a working install into an unsupported one.

Editorial conclusion

Adopt WhisperX if you need word-level timestamps or speaker labels and can run Python 3.10 to 3.13 with a CUDA 12.8 GPU, or accept CPU speed. Do not adopt it if you only need plain transcripts, or if your platform is Linux aarch64, where the project's own pyproject.toml excludes torchcodec. Before committing, verify three things: that pyannote/speaker-diarization-community-1 loads with your Hugging Face token, that your GPU has the under-8GB headroom the README cites for large-v2 at beam_size=5, and that the 3.8.7rc1 release candidate is not what your installer picks up by default.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-2-Clause
3. m-bain/whisperX on GitHub
4. README
5. Releases