Whisper-Finetune: fine-tuned models you cannot download, and a dependency list without torch
Fine-tune the Whisper speech recognition model to support training without timestamp data, training with timestamp data, and training without speech data. Accelerate inference and support Web deployment, Windows desktop deployment, and Android deployment
At a glance
- What is it?
- A Chinese-language Whisper fine-tuning project built around LoRA, with three training modes, two accelerated inference paths and deployment targets for Android, Windows and the web. The benchmark tables are unusually detailed for a project like this, which is what makes two things in them easy to notice: every model sits behind a paid community, and the turbo numbers do not hold up.
- Who is it for?
- The engineering here is worth reading: three training modes including text-only, LoRA merging, two accelerated runtimes and a converter that lets you skip fine-tuning entirely and speed up the stock model. What you cannot do is download any of the trained checkpoints without joining the author's paid community, so the tables cannot be reproduced from the repository.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Activity is slowing. The repository last received commits 6 months ago.
- What is it written in?
- Mainly C, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- yeyupiaoling/Whisper-Finetune1,223 ★ · +2
| Repository | 2026-09-21 | 2026-11-15 |
|---|---|---|
| yeyupiaoling/Whisper-Finetune | 1.2k | 1.2k |
2 Stars in the last 55 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
Every model in every table has to be joined for
The benchmark section is three tables of word error rates, and every one of them ends with the same column.
That column reads, in every row of every table, that the model is available by joining the author's paid knowledge community. It is there for the rows using OpenAI's original checkpoints and for the rows using the project's own fine-tuned checkpoints alike.
That second part is the problem. The tables are the project's argument, and they measure artefacts that cannot be obtained from the repository, from Hugging Face or from anywhere else named in the document. There is no download link, no hash and no dataset recipe sufficient to reproduce a fine-tuned row, so the fine-tuned numbers are unverifiable by anyone outside the community.
The introduction also points somewhere else. It says several models have been open-sourced and that they can be seen on Hugging Face under the openai organisation. That is true of the original checkpoints, and it is the only route to a downloadable model that the document describes.
The turbo rows do not hold up in the author's own numbers
The first table measures the original models with the language fixed to Chinese, across three test sets plus a Cantonese set.
On that table, large-v3 is the best model on all three Chinese sets: 0.08086 on the AIShell test set, 0.11452 on the network test set and 0.19878 on the meeting test set. The turbo model scores worse than large-v3 on every one of them, at 0.08647, 0.21225 and 0.20390, and it is also worse on the Cantonese set, 0.38069 against 0.35301.
The fine-tuned tables repeat the pattern more sharply. On the AIShell fine-tune, large-v3 reaches 0.09835 on the network test set while the turbo of the same size sits at 0.23038, more than double. On the WenetSpeech fine-tune, large-v3 scores 0.17429 on the meeting test set against the turbo's 0.35697.
And that last figure appears twice in the same row. The turbo Cantonese fine-tune lists 0.35697 in the meeting test set column and 0.35697 in the Cantonese test set column, where large-v3 scores 0.33464. A duplicated cell is the kind of error that survives publication when the tables are maintained by hand, and it is the single clearest sign that these numbers need rechecking.
The dependency list does not contain torch
The requirements file is twenty packages, and nineteen of them are pinned to an exact version: numpy 1.26.4, soundfile 0.12.1, librosa 0.10.0, transformers 4.51.3, bitsandbytes 0.45.5, datasets 3.5.0, evaluate 0.4.3, ctranslate2 4.6.0, faster-whisper 1.1.1, jiwer 3.1.0, peft 0.15.2, accelerate 1.6.0, soundcard 0.4.2, uvicorn 0.21.1, fastapi 0.95.1, starlette 0.26.1, tensorboardX 2.6.2.2 and zhconv 1.4.2. The two exceptions are click and tqdm, which carry no version at all.
PyTorch is not among them. Neither is torchaudio, which a Whisper training setup needs.
So the file is meticulous about everything except the dependency the project is about, and the README handles it separately by listing the environment as Anaconda 3, Python 3.11, PyTorch 2.4.0, Ubuntu 18.04 and one A100 PCIe with 40GB of memory. A reader who installs the requirements file and then runs the fine-tuning script gets an import error rather than a version conflict.
The two unpinned entries are the other half of the story: an otherwise exact lock with two loose packages in it.
Three training modes, and one converter that skips fine-tuning entirely
The stated purpose of the project is LoRA fine-tuning of Whisper in three situations, and the three are named in the introduction: training with no timestamp data, training with timestamp data, and training with no speech data at all.
That third mode is the interesting one, because it is the case where a model is trained on text that was never spoken aloud, which is how the original Whisper was built and how most of its languages are covered at all.
The pipeline around it is standard and well described by its file names. aishell.py builds AIShell training data, finetune.py does the training, merge_lora.py merges the LoRA adapter back into the Whisper weights, and evaluation.py scores either a fine-tuned model or the original.
The inference side is where this project adds something beyond the usual fine-tuning recipe. CTranslate2 and GGML are both supported, and the introduction makes a point that is easy to miss: accelerated inference can use a converted version of the original Whisper model directly and does not require a fine-tune at all. convert-ggml.py produces the GGML format that the Android and Windows applications consume, so a user with no training data can still get the faster runtime.
Eleven programs are listed, and twelve scripts are in the tree
The project describes itself with a numbered list of eleven items, and it maps cleanly onto the shape of the work.
Inference comes in four variants: infer.py for a script, infer_ct2.py for a converted CTranslate2 model, infer_gui.py with a graphical interface, and infer_server.py to deploy the model on a server for clients to call. Around those sit the data preparation, training, merging and evaluation scripts, plus convert-ggml.py and two application directories, AndroidDemo for the Android deployment source and WhisperDesktop for the Windows desktop application.
The tree has a twelfth script that the list does not mention. infer_thirteen.py sits beside infer_ct2.py and infer_gui.py in the top level, with no entry in the description of what it does.
The rest of the repository follows the same division: configs/, dataset/, docs/, metrics/, static/, templates/, tools/ and utils/, plus a run.sh. The two application directories are the reason the GGML converter exists, since those runtimes cannot load a CTranslate2 model.
The reference environment is Ubuntu 18.04 and one A100
The environment section is five lines and is worth reading as a claim about the project's era as much as about its requirements.
Anaconda 3, Python 3.11, PyTorch 2.4.0, Ubuntu 18.04, and a single A100 PCIe with 40GB of memory. The 40GB figure is the binding constraint for anyone reproducing the large and large-v2 rows of the tables, since those are the checkpoints the fine-tuning tables cover.
The demonstration material is hosted on the author's own domains rather than on a public demo service. There is an online transcription demo at one subdomain using the whisper-small fine-tuned model, and a separate web deployment demonstration on port 8082 at another subdomain, with an interface document linked from the table of contents. A WeChat mini program built on the same fine-tuned model is shown as well.
Three of those artefacts depend on the community-gated checkpoints from the first section, so the most visible parts of the project are also the parts that cannot be examined from outside it. The README is written in Simplified Chinese with an English version linked beside it, and it invites readers to join a paid knowledge community or a QQ group for discussion, where the model files are said to be available.
The Uyghur rows specify Chinese, and the README explains why
The third table covers a different language, and it has a column that looks like an error until the notes are read.
Six rows, from whisper-tiny to large-v2 and the turbo, all for Uyghur audio, all with the specified language given as Chinese rather than as a language code for Uyghur. The word error rates run from 0.06798 for the tiny model down to 0.01480 for large-v2, with the turbo at 0.01744.
The note under the table resolves it. The author tested specifying the audio as Chinese and as uzbek, and got 0.06798 and 0.0685 for the tiny model, found the difference negligible, and therefore used Chinese throughout. That is a legitimate finding and a good one to publish, since it says the model's language token barely matters for this pair.
The second note describes the split, and it is the more consequential of the two. The CommonVoice-Uyghur test set is used as this project's test set, and everything else, including the whole of THUYG20, is training data.
So the Uyghur numbers measure generalisation to one held-out corpus rather than to a standard benchmark, which makes them hard to compare against anything else and means the reported error rate depends on a split the author chose.
Editorial conclusion
The engineering here is worth reading: three training modes including text-only, LoRA merging, two accelerated runtimes and a converter that lets you skip fine-tuning entirely and speed up the stock model. What you cannot do is download any of the trained checkpoints without joining the author's paid community, so the tables cannot be reproduced from the repository. Treat the numbers as marketing for that community rather than as measurements, install the training requirements yourself with torch added by hand, and reproduce the base-model row from OpenAI's own weights if you need a comparable figure.
Frequently asked questions
What does the Whisper-Finetune project do?
It fine-tunes OpenAI's Whisper models with LoRA in three situations: training without timestamp data, training with timestamp data, and training with no speech data at all. It also supports CTranslate2 and GGML accelerated inference, and states that accelerated inference can use a converted copy of the original Whisper model without any fine-tuning.
Can I download the fine-tuned Whisper models from this repository?
No. Every row of every benchmark table carries a column saying the model is obtained by joining the author's paid knowledge community, and no download link, hash or dataset recipe is given for the fine-tuned checkpoints. The introduction does point to OpenAI's original models on Hugging Face.
What do I need to install to run the fine-tuning scripts?
The requirements file pins nineteen packages exactly, including transformers 4.51.3, peft 0.15.2, datasets 3.5.0 and ctranslate2 4.6.0, with click and tqdm left unpinned. PyTorch and torchaudio are not in that file at all; the README lists the environment separately as Anaconda 3, Python 3.11, PyTorch 2.4.0, Ubuntu 18.04 and one 40GB A100.
Which Whisper models does Whisper-Finetune cover?
Eight sizes, from whisper-tiny and whisper-base through small, medium, large, large-v2, large-v3 and large-v3-turbo. Word error rate tables are published for the original models in Chinese, for fine-tunes on AIShell and WenetSpeech, for Cantonese fine-tunes, and for Uyghur.
How does Whisper-Finetune deploy to Android and Windows?
convert-ggml.py converts a model into GGML format, and two application directories consume it: AndroidDemo holds the Android deployment source and WhisperDesktop holds the Windows desktop application. Inference also comes as a script, a CTranslate2 variant, a GUI variant, and infer_server.py for serving a model to clients.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/yeyupiaoling-whisper-finetune)Star history chart for your README
[](https://hysenlabs.com/projects/yeyupiaoling-whisper-finetune)Was this useful?