What is Speech-to-text (ASR)?

How audio becomes text

The signal goes through three stages. First it becomes features: a waveform is sliced into short overlapping windows, and each window is turned into a spectrogram, usually on a mel scale, which is a compact description of how much energy sits in each frequency band. A hundred frames per second is a typical rate, and the model sees a sequence of those rather than raw samples.

An encoder then turns that sequence into a representation. Conformer and transformer encoders are both common, the first because convolution helps with local timing and the second because attention helps with long-range structure.

The decoder is where two families diverge. Connectionist temporal classification emits a label for every frame and then collapses repeats, which is fast because nothing is generated step by step, and weaker on rare words because there is no language model correcting toward plausible sentences. An attention decoder generates text tokens autoregressively, conditioned on the encoder output, which is slower and more accurate and can be paired with an external language model. Most systems let you choose, because the two answers to the question are the two answers to a different question about latency.

What changed the field was training data rather than architecture. openai/whisper is a single model trained on a very large, weakly supervised corpus, with a special token at the start of the audio telling the model which task to do. Our analysis of Whisper describes it as a general-purpose recognition model distributed as a Python package with a command line entry point, handling transcription in many languages plus translation into English and language identification, and it notes the turbo model is English-only for translation, which is the sort of detail that matters when you pick a checkpoint.

The three things that are not the recogniser

A recogniser returns a string. Almost every feature people want around it is a different model, and understanding the split is most of what there is to know.

Word-level timestamps. Frame-level alignments are not accurate enough to put a subtitle on, and one fix is to take a pretrained acoustic model trained for alignment, run it over the audio, and force the recogniser's text onto its boundaries. m-bain/whisperX is that arrangement, and our analysis describes it as wrapping faster-whisper, wav2vec2 forced alignment and pyannote-audio into one command line tool that emits word-level timings and speaker labels. The same analysis says it is worth adopting when subtitle sync or speaker attribution matters more than a single pip install, which is the right framing: it is a stack, not a drop-in.

Speaker diarization. Knowing what was said is a different problem from knowing who said it, and it is a clustering problem over speaker embeddings rather than a recognition problem at all. The overlap case, two people talking at once, is where it degrades, and some systems still require the number of speakers in advance.

Streaming. An offline recogniser can look at the whole utterance before deciding what was said, and that is why it is more accurate. A live caption needs a decision at every chunk boundary, which is why streaming models are causal and slightly worse, and why the latency of a caption is measured in chunk size rather than in model speed. modelscope/FunASR is a toolkit that treats this as one component among several: our analysis describes it as an MIT-licensed Python toolkit where you pick a task, a checkpoint and a runtime separately, from a small model on CPU to a larger one on GPU behind vLLM, with the trade-off that support in one component does not carry over to another.

When you need it, and when you do not

The case is strongest where the audio is yours and the volume is real.

It fits privacy-sensitive audio. Interviews, medical or legal recordings, internal meetings, anything where uploading is not an option. cjpais/Handy is the small version of that argument: our analysis describes it as a Tauri desktop app that turns a keyboard shortcut into transcribed text in any field, running Whisper or Parakeet models locally, and it is deliberately small and forkable.

It fits bulk transcription of an archive you already hold, where a hosted API is billed per audio minute and a local recogniser is a machine you already own.

It fits domain vocabulary. A model that has never heard your product names will spell them wrongly every time, and the fix is a fine-tune, a biasing list or a language model over the output.

It fits live captions, with the streaming caveat above.

It does not fit when a hosted endpoint is acceptable, because a local model is a machine to buy, a dependency to upgrade and a GPU to schedule. It does not fit when the audio is not speech. Music, traffic, a room with no talking: a recogniser does not return nothing, it returns plausible sentences, and that behaviour is the subject of the next section. And it does not fit where the answer must be exact and constrained: a voice interface with a fixed grammar of a few hundred commands is better served by a small recogniser plus a parser than by a general model trying to be accurate at everything.

The failure modes

Five, and the second is the one that surprises people most.

Word error rate hides what matters. It counts substitutions, deletions and insertions, so a system that drops all the ums and the you knows scores better while being less usable for a transcript, and a system that paraphrases is punished. Worse, measuring it needs a reference transcript, which means a human labelling a sample before you can compare anything.

It does not know when nobody is speaking. Hand a general recogniser a few seconds of silence, music or a keyboard, and it will produce text, because the model has learned that audio maps to tokens and it has no concept of there being nothing to say. This is why voice activity detection is a stage in any serious pipeline, and it is why PaddlePaddle/PaddleSpeech bundles voice activity detection and punctuation restoration alongside recognition rather than leaving you to add them.

The language you actually have. A model trained mostly on one language is weaker on a language with different phonology and little training data, and the error is confident rather than obvious. PaddleSpeech is a good illustration of the general shape: our analysis says it is strongest for Chinese and English pipelines that accept the PaddlePaddle runtime, and weakest where you need a small dependency footprint or a non-Paddle deployment.

Numbers, names and identifiers. The top error class in real transcripts. A post-correction step that looks for patterns in the output is the usual answer, and a model fine-tuned on your domain is the better one.

Chunk boundaries in streaming. A streaming recogniser makes a decision at every chunk edge, and the words that straddle an edge are where the errors cluster.

How the projects divide

Six groups, and the group tells you what is being run.

The general model. openai/whisper is the one most stacks are built on, either directly or through a wrapper, because one model covering many languages and three tasks is simpler than maintaining several.

The alignment and diarization layer. m-bain/whisperX is the reference arrangement, and it is the piece to add when timestamps or speakers are the actual requirement.

Modular toolkits. modelscope/FunASR lets you pick task, checkpoint and runtime independently, and its analysis notes the design is modular with the trade-off that support in one component does not carry over to another. PaddlePaddle/PaddleSpeech puts recognition, synthesis, translation, punctuation and speaker verification behind one Python API, which our analysis says is strongest for Chinese and English pipelines and weakest where the dependency footprint or the runtime is a constraint.

Research frameworks. NVIDIA-NeMo/Speech is the one to read if you are training rather than deploying: our analysis describes a Python and PyTorch framework with training infrastructure, pretrained checkpoints on HuggingFace and NVIDIA GPU containers, and it records that the project narrowed its scope to audio and speech in 2026 after splitting from the broader NeMo framework, which is a real consideration for anyone starting a dependency on it.

Offline and edge. k2-fsa/sherpa-onnx is a C++ library built on next-gen Kaldi and ONNX Runtime that runs speech-to-text along with diarization, voice activity detection and source separation entirely offline, targeting embedded hardware and mobile platforms with bindings for twelve languages.

The applications. chidiwilliams/buzz is a desktop app and Python package running Whisper locally with GPU backends for CUDA, Apple Silicon and Vulkan, and our analysis describes it as heavier than a hosted API for people who want captions without uploading their recordings. Zackriya-Solutions/meetily is the meeting case, transcribing and summarising on the device with Whisper or Parakeet plus a configurable local backend, and our analysis notes that no audio or transcript data leaves the machine. bradautomates/claude-video is the agent case: our analysis describes its watch skill downloading a video, extracting frames, transcribing the audio and handing both to a model, with the frame budget deciding whether the answer is useful.

In practice

Speech-to-text is a well-understood model wrapped in three separate systems, and picking the wrong wrapper is the usual mistake. Start with the recogniser that matches your languages and your accuracy needs, add forced alignment and diarization only if you need word timings or speaker labels, and gate the audio with voice activity detection so the model is never asked to transcribe nothing. For local privacy, cjpais/Handy and chidiwilliams/buzz are the small desktop answers and Zackriya-Solutions/meetily is the meeting one; for training and edge deployment, NVIDIA-NeMo/Speech, modelscope/FunASR, PaddlePaddle/PaddleSpeech and k2-fsa/sherpa-onnx are the four to compare. And measure with something better than word error rate on a sample you have labelled, because the metric will not tell you about the dropped filler words.