whisper-timestamped: word-level timestamps from Whisper cross-attention weights

The gap whisper-timestamped fills: Whisper segments are not words

Whisper models were trained to predict approximate timestamps on speech segments, most of the time with 1-second accuracy, and they cannot originally predict word timestamps. That sentence from the README is the whole premise of this project. If you are generating subtitles, aligning a transcript to a recording for search, or cutting audio at word boundaries, a segment-level timestamp rounded to a second is not usable. You get a block of text and a rough idea of where it starts. The project's stated goal is to predict word timestamps and provide a more accurate estimation of speech segments when transcribing with Whisper models, and to attach a confidence score to each word and each segment. The intended user is someone who has already chosen Whisper for its multilingual coverage and now needs finer temporal resolution without switching recognition engines. The README also carries an explicit disclaimer: the extension is intended for experimental purposes and may significantly impact performance. Treat that as the project's own framing of its maturity, not as a marketing caveat.

Dynamic Time Warping over cross-attention weights, and why memory stays flat

The mechanism is Dynamic Time Warping applied to cross-attention weights, an approach the README credits to a notebook by Jong Wook Kim in the openai/whisper repository. Whisper's decoder attends over encoder frames; those attention weights carry a soft alignment between output tokens and input audio frames. DTW turns that soft alignment into a monotonic path, and the path gives you start and end positions per word. The project lists three additions over the original notebook: more accurate start/end estimation, confidence scores per word, and, if possible without beam search, no additional inference steps, because word alignment is done on the fly after each speech segment is decoded. The fourth claim is about memory: special care has been taken so that whisper-timestamped can process long files with little additional memory compared to regular Whisper use. That matters because the naive alternative, running a second acoustic model over the audio, scales memory with the length of the file and the size of that second model. Here the alignment is derived from tensors the decoder already produced. The conditional in the README is the part to read twice: the no-extra-inference property is tied to not using beam search. Turn beam search on and the alignment cost changes.

Installation, optional dependencies and the VAD path

The base install is a single pip command: pip3 install whisper-timestamped. Requirements are python3 version 3.7 or higher, with at least 3.9 recommended, plus ffmpeg, for which the README points at the openai-whisper installation instructions. Cloning the repository and running python3 setup.py install is the alternative path. Three optional extras change what you can do. matplotlib is needed for the word alignment plot. onnxruntime and torchaudio are needed for the VAD option, installed with pip3 install onnxruntime torchaudio. transformers is needed if you want to use finetuned Whisper models from the Hugging Face Hub. The VAD feature is not cosmetic: the README describes it as a way to avoid hallucinations caused by errors in the training data, giving the example of predicting "Thanks you for watching!" on pure silence. Several VAD methods are available, with silero as default and auditok and auditok:v3.1 as alternatives. For CPU-only machines the README documents a light installation path, though the supplied excerpt cuts off before the details. A Dockerfile is present and the README states the resulting image is about 9GB, built with docker build -t whisper_timestamped:latest . after cloning.

Where the cross-attention approach breaks down

The README is unusually direct about failure modes, which is a point in its favour. The first is beam search: the on-the-fly alignment advantage is stated as conditional on not using beam search, so a configuration tuned for maximum transcription accuracy may pay extra inference for timestamps. The second is the disclaimer itself, that the extension is experimental and may significantly impact performance, with no quantification given in the supplied material. The third concerns the VAD feature: it exists precisely because Whisper hallucinates on silence, but enabling it pulls in onnxruntime and torchaudio, and the default silero model is one more component in the stack. The README's critique of the timestamp-token approach is also a warning about the underlying model: Whisper tends to predict timestamps only after a certain number of words, typically at the end of a sentence, and its timestamp precision tends to be rounded to 1 second. Any downstream logic that assumes sub-second ground truth from Whisper itself is building on sand. Finally, the README notes that disfluencies such as fillers, hesitations and repeated words are usually removed by Whisper, which is a limitation for verbatim transcription regardless of which alignment method you use.

whisperX and the wav2vec alternative: one model versus many

The README names whisperX as the relevant alternative and explains the difference in approach rather than just gesturing at it. whisperX recovers word-level timestamps using wav2vec models that predict characters. The trade-off is structural: you need to find one wav2vec model per language you support, which the README argues does not scale well against Whisper's multilingual capability, and you need to hold at least one additional neural network in memory. There is also a normalization problem. Whisper's output has to be mapped onto the wav2vec character set, which means language-dependent conversions such as turning "2" into "two" or "%" into "percent" and "€" into "euro(s)". The README further claims wav2vec-based alignment is less robust around disfluencies, since Whisper tends to strip them before alignment ever happens. The second alternative it discusses is the timestamp-token approach used in whisper.cpp and stable-ts: read the probabilities of timestamp tokens emitted after each subword. The README rejects this on robustness grounds, arguing Whisper was not trained to emit meaningful timestamps after every word, and that results can be totally out-of-sync on some periods, with jingle music called out specifically. That is a claim from the project about its competitors, not an independent measurement, and the supplied material offers no numbers behind it.

Licence, upgrade surface and what to check before you depend on it

The licence is AGPL-3.0. For a library that runs inside a transcription pipeline, that is a distribution question rather than a technical one, and it is the kind of question to put to someone qualified to answer it for your product; the README does not discuss licensing at all in the supplied text. On maintenance, the release history shows v1.15.9 in September 2025, preceded by v1.15.8 and v1.15.7 in November 2024, and the repository's last push is dated August 2026 with the project not archived. The version numbering suggests incremental changes rather than a rewrite, and the README states the package is meant to be compatible with any version of openai-whisper, which is a commitment that costs maintenance effort every time upstream Whisper changes its internals. Two practical checks are worth running before you build on it. First, confirm that the on-the-fly alignment path is the one you actually exercise, meaning your decoding configuration does not enable beam search. Second, since the README explicitly flags jingle music as a case where timestamp-token methods desynchronize, test on audio that contains music beds or stingers if your material has them, and compare the alignment plot (matplotlib required) against the audio rather than trusting the JSON alone.

Who should adopt it, and the one configuration to settle first

Adopt whisper-timestamped when you are already committed to openai-whisper, you need word-level timestamps and per-word confidence, and you want to avoid maintaining a wav2vec checkpoint for every language in your catalogue. The confidence scores per word and per segment are the feature that distinguishes it from simply post-processing segment timestamps, and the on-the-fly alignment means long files do not balloon memory the way a second model would. Do not adopt it if your decoding depends on beam search, since the README ties the no-additional-inference property to its absence, or if you need verbatim transcripts including fillers and hesitations, which Whisper removes before alignment happens. The single configuration decision to settle first is whether VAD is on. Turning it on adds onnxruntime and torchaudio as dependencies and changes which segments reach Whisper at all; leaving it off means accepting the hallucination behaviour the README describes on silence. That choice, plus the beam search setting, determines both your dependency list and your timestamp quality more than any other option in the project.

Editorial conclusion

Adopt whisper-timestamped if you are already running openai-whisper and need word-level timestamps and per-word confidence across many languages without adding a wav2vec checkpoint per language. Do not adopt it if your pipeline depends on beam search, because the README states that on-the-fly alignment without additional inference steps only holds when beam search is not used, or if AGPL-3.0 is incompatible with how you distribute your product. Before committing, verify two things on your own audio: that the VAD default (silero, requiring onnxruntime and torchaudio) removes the hallucinated segments you actually see, and that word timings hold up on any stretch containing music, which the README names as a case where timestamp-token methods drift out of sync.

Official sources

1. Issues
2. License: AGPL-3.0
3. linto-ai/whisper-timestamped on GitHub
4. README
5. Releases