Picovoice speech-to-text-benchmark: a harness for comparing ASR engines on the same audio
speech to text benchmark framework
At a glance
- What is it?
- Picovoice's benchmark repository wraps ten speech-to-text engines behind one Python entry point and reports WER, punctuation error rate, core-hours, word emission latency and model size. It is a comparison harness, not a production transcription library, and its value depends on you supplying the datasets and credentials yourself.
- Who is it for?
- Adopt this if you are choosing between offline and cloud ASR engines and want WER, punctuation error rate, core-hours, word emission latency and model size computed by one script on datasets you control. Do not adopt it as a transcription component; it is a measurement harness and the repository states it was developed and tested on Ubuntu 22.04, so treat other platforms as unverified.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 69 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The comparison problem this repository is built around
Anyone picking a speech-to-text engine faces the same mess: vendor pages quote word error rate on undisclosed audio, open model cards quote numbers on splits that may or may not overlap, and nobody reports how many CPU hours one hour of audio costs. Picovoice's repository exists to remove that asymmetry. It puts ten engines (Amazon Transcribe, Azure Speech-to-Text, Google Speech-to-Text, IBM Watson, OpenAI Whisper, Whisper.cpp, Vosk, Moonshine, Picovoice Cheetah and Picovoice Leopard) behind a single benchmark.py, so the same audio and the same reference transcripts produce directly comparable output. The audience is engineers and technical decision makers who need to justify a choice, not end users who want a transcript. Two of the ten engines belong to Picovoice itself, which is worth noting: the harness is maintained by a vendor that also competes in the category it measures. That does not invalidate the numbers, because the datasets and metrics are named, but it does mean you should read the published results with the source of the harness in mind and, ideally, rerun the comparison on your own audio.
What the five metrics actually measure, and what they leave out
The README defines five metrics and, importantly, states when each is omitted. Word error rate is the edit distance between reference and hypothesis words divided by the number of reference words. Punctuation Error Rate follows Meister et al. (arXiv 2310.02943) and counts punctuation-specific errors against the number of punctuation operations in the reference; the harness reports PER for periods and question marks. Core-Hour is CPU hours per hour of audio, and the README says it is omitted for cloud engines, which makes sense because you do not control the hardware. Word emission latency is the average delay between a word finishing and the engine emitting its transcription, and it is measured only for streaming engines. Model size is the combined acoustic and language model footprint in MB, again omitted for cloud engines. The gaps matter as much as the metrics. Three of the five numbers do not exist for cloud engines, so a cloud-versus-offline table is inherently incomplete. Latency exists only for streaming variants, so batch engines cannot be compared on responsiveness at all. And WER is computed against text that has already been normalized by whatever the dataset provides; the README does not describe a text normalizer, so casing, numerals and filler words are handled however the reference and the engine happen to handle them.
Datasets and languages the harness accepts
The supported dataset identifiers are COMMON_VOICE, LIBRI_SPEECH_TEST_CLEAN, LIBRI_SPEECH_TEST_OTHER, TED_LIUM, MLS, VOX_POPULI and FLEURS, drawn from LibriSpeech, TED-LIUM, Common Voice, Multilingual LibriSpeech, VoxPopuli and Fleurs. The supported languages are EN, FR, DE, ES, IT, PT_BR and PT_PT. The harness does not download audio for you. You fetch each corpus yourself, and the repository points to script/README.md for the Fleurs download instructions specifically. That split is a deliberate design choice: it keeps the repository small and avoids redistributing corpora with their own terms, but it also means the first hour of using this project is spent on dataset plumbing rather than benchmarking. Note the language list is narrower than the dataset list. Multilingual LibriSpeech covers far more than seven languages, yet only those seven values are accepted, so a French or German run is easy while a Dutch or Polish run is not supported by the harness as documented. IBM Watson is called out as English-only, so pairing it with any other --language value is a configuration the README does not claim works.
Running benchmark.py: the flags that matter
Every engine shares the same skeleton: python3 benchmark.py with --dataset, --dataset-folder, --language and --engine. What changes is the credential block. Amazon Transcribe takes --aws-profile and --aws-location, and switching --engine to AMAZON_TRANSCRIBE_STREAMING selects streaming mode. Azure takes --azure-speech-key and --azure-speech-location, with AZURE_SPEECH_TO_TEXT_REAL_TIME for the streaming path. Google takes --google-application-credentials pointing at a downloaded credential file, with GOOGLE_SPEECH_TO_TEXT_STREAMING as the streaming variant. IBM Watson takes --watson-speech-to-text-api-key and --watson-speech-to-text-url. The local models are simpler: Whisper is selected by passing the model as the engine value itself, so --engine WHISPER_LARGE_V3 or WHISPER_LARGE_TURBO, with WHISPER_TINY, WHISPER_BASE, WHISPER_SMALL, WHISPER_MEDIUM, WHISPER_LARGE_V1 and WHISPER_LARGE_V2 also listed. Whisper.cpp streaming uses WHISPER_CPP_STREAMING_TINY through WHISPER_CPP_STREAMING_LARGE_TURBO. Two global flags cut across all of them: --punctuation enables Punctuation Error Rate, and --punctuation-set selects which marks to score against, one or more of period, question mark and comma, defaulting to period and question mark. Setup is FFmpeg, the datasets, then pip3 install -r requirements.txt. One caveat on the README itself: the Azure and IBM example blocks are missing the line-continuation backslash before their second credential flag, so copying those blocks verbatim will not run. Add the backslash or put the flags on one line.
Where the harness breaks down
The most obvious limitation is platform. The README says the benchmark has been developed and tested on Ubuntu 22.04 and says nothing about macOS or Windows, so a Windows engineer is on untested ground with FFmpeg, Python packaging and whatever native dependencies the local engines pull in. The second limitation is that this is a batch-oriented measurement tool. There is no incremental scoring API, no way to feed a live microphone and get a running WER, and no partial-result handling beyond what the streaming engine adapters do internally. If your goal is to evaluate an engine inside a real pipeline with your own audio characteristics (telephone bandwidth, overlapping speakers, domain vocabulary), this harness will not do it; it scores whole files against reference transcripts. Third, the metric set silently excludes cloud engines from core-hours and model size, so any summary table you build will have holes. Fourth, the punctuation machinery is bounded: PER is reported for periods and question marks per the README, and the --punctuation-set flag accepts period, question mark and comma, which means other marks are outside the framework's scoring. Finally, the repository lists no releases, so there is no versioned artifact to pin against; you are tracking the master branch.
How this differs from a general ASR evaluation toolkit
The natural comparison is with a general-purpose evaluation library such as SpeechBrain or the evaluation utilities that ship alongside individual model repositories. Those tend to be scoring libraries: you bring your own hypotheses and references, and they compute WER or CER. This repository is the opposite shape. It owns the engine adapters, so it will call Amazon Transcribe or run Whisper for you, and it owns the metric definitions, so WER, PER, core-hours, latency and model size come out of one run. The trade-off is flexibility. A scoring library will accept any hypothesis file you produce, including output from a system this repository has never heard of. Picovoice's harness only measures the ten engines listed, and adding an eleventh means writing an adapter against the project's own interface, which the README does not document. If your question is which of these ten engines performs best on LibriSpeech test-clean in English, the harness answers it in one command. If your question is how your in-house model compares to Whisper, you are better served by a scoring library and a reference transcript.
Licence and the cost of keeping it current
The repository is Apache-2.0, which permits commercial use, modification and redistribution provided you keep the licence and notice files intact and state significant changes. That covers the harness code. It does not cover the corpora, which carry their own terms (Common Voice in particular has its own usage conditions), and it does not cover the cloud engines, whose pricing applies per minute of audio transcribed. The real maintenance cost here is not the Python. It is the engine adapters. Cloud vendors deprecate API versions and rename parameters, local model families add sizes (the Whisper list already runs to eight variants including LARGE_TURBO), and each change can break an adapter until someone patches it. Because there are no tagged releases, there is no stable version to freeze; you either follow master or fork. Budget for periodically rerunning your comparison rather than treating any single result set as permanent. This is general information about the licence text, not legal advice; check the Apache-2.0 terms and each corpus licence against your own use case.
Editorial conclusion
Adopt this if you are choosing between offline and cloud ASR engines and want WER, punctuation error rate, core-hours, word emission latency and model size computed by one script on datasets you control. Do not adopt it as a transcription component; it is a measurement harness and the repository states it was developed and tested on Ubuntu 22.04, so treat other platforms as unverified. Before trusting any number, confirm the exact dataset split behind LIBRI_SPEECH_TEST_CLEAN or LIBRI_SPEECH_TEST_OTHER, check whether --punctuation-set matches the punctuation you actually care about, and verify that your cloud credentials can reach the region you pass to --aws-location or --azure-speech-location.
Community notes