Whisper-Finetune: LoRA fine-tuning for Whisper, and where it stops being the right tool
Fine-tune the Whisper speech recognition model to support training without timestamp data, training with timestamp data, and training without speech data. Accelerate inference and support Web deployment, Windows desktop deployment, and Android deployment
At a glance
- What is it?
- The repository wraps LoRA fine-tuning, LoRA merging, CTranslate2 and GGML conversion, and Web, Windows and Android deployment of Whisper models. The training side is the interesting part; the deployment side is the part that ages fastest.
- Who is it for?
- Adopt Whisper-Finetune if you have Chinese or Cantonese audio, a GPU, and you want a LoRA adapter you can merge and convert to CTranslate2 or GGML rather than a full fine-tune. Do not adopt it if you need a maintained Python library with a stable API, if your data has no transcripts at all, or if you expect the published error-rate tables to transfer to your domain.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 131 days ago.
- What is it written in?
- Mainly C, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Whisper-Finetune is for, and who it is not for
Whisper-Finetune is a training and export pipeline around OpenAI's Whisper checkpoints. The README states the project's purpose plainly: fine-tune Whisper with LoRA, and it claims support for training without timestamp data, training with timestamp data, and training without speech data. The last of those is the unusual one. Most fine-tuning recipes assume you have audio and transcripts; this repository is built for the case where one side of that pair is missing or partial.
The target user is someone with a GPU who wants a Chinese or Cantonese recognizer that beats the stock checkpoint on their own audio, and who then wants to ship it somewhere other than a Python process. The README's test tables are all Chinese-language: aishell_test, test_net, test_meeting, and a Cantonese set. That is a strong signal about who the project was written for. English speakers can use it, but the documentation, the data-preparation scripts and the published numbers all assume Chinese corpora.
It is not a library. There is no package to import, no versioned API, no release history in the repository. It is a set of scripts you run in order, and the scripts are named after the steps: finetune.py, merge_lora.py, evaluation.py, infer.py. If you want a dependency you can pin and upgrade, this is the wrong shape of project.
The LoRA path and why merging is a separate step
The mechanism is LoRA on top of a frozen Whisper checkpoint. finetune.py trains the adapter, merge_lora.py folds the adapter weights back into the base model. Keeping those as two scripts is not an accident of packaging. An unmerged adapter is small and easy to keep, but it only loads inside a process that also knows how to apply it. A merged model is a normal Whisper checkpoint that every downstream tool understands, including the CTranslate2 and GGML converters.
That is why the deployment story works at all. infer_ct2.py runs a CTranslate2 model, convert-ggml.py produces a GGML file for the Android and Windows applications. Neither converter knows about LoRA. They consume a merged model. If you skip merge_lora.py, you have a training artifact, not a deployable one.
The repository also separates inference paths by backend rather than by task. infer.py and infer_gui.py run the transformers model. infer_ct2.py runs the converted model, and the README points at it as the reference for CTranslate2 usage. infer_server.py exposes the model over HTTP for clients. The GGML path feeds AndroidDemo and WhisperDesktop. Four inference entry points for one model is a lot of surface area, and it means a bug fix in one path does not reach the others.
The README also notes that the acceleration paths accept a stock Whisper model. You do not have to fine-tune to use CTranslate2 or GGML conversion. That is worth knowing, because it separates two decisions that are easy to conflate: whether you need a domain-adapted model, and whether you need faster inference.
Installing the environment and running a first fine-tune
The README specifies the environment it was developed against: Anaconda 3, Python 3.11, PyTorch 2.4.0, Ubuntu 18.04, and a single A100-PCIE-40GB. The requirements.txt pins the rest. Note that the pins are exact, including transformers==4.51.3 and peft==0.15.2, so a fresh install is reproducible but not flexible.
Install the pinned dependencies first:
pip install -r requirements.txtThat pulls in the training stack (transformers, peft, accelerate, bitsandbytes, datasets), the evaluation stack (evaluate, jiwer), and the serving stack (fastapi, uvicorn, starlette). ctranslate2 and faster-whisper come along for the accelerated inference path.
Before training you need data. The repository ships two preparation scripts, and the README names aishell.py as the one that builds AIShell training data. The WenetSpeech path is in tools/create_wenetspeech_data.py. Run the one matching your corpus; the output is what finetune.py consumes.
python aishell.pyTraining is then a single script invocation, launched through the provided run.sh or directly:
python finetune.pyAfter training, merge the adapter into the base model:
python merge_lora.pyThe README does not document the command-line flags for finetune.py. Configuration lives under configs/, so read the config file you intend to use before launching, rather than assuming defaults match your data layout. If the script starts and immediately fails on a missing label column, the config is the first place to look.
The published error rates, and what they do not tell you
The README includes two tables. The first is the stock models, the second is the same models after fine-tuning on AIShell. The numbers move a lot. whisper-large-v3 on aishell_test goes from 0.08086 to 0.03660 after fine-tuning on AIShell. whisper-tiny goes from 0.31898 to 0.13043.
Read that carefully. aishell_test is the test split of AIShell, and the fine-tuning data is AIShell. The improvement on the in-domain test set is partly a measurement of how well the model memorized the corpus. The out-of-domain columns are more informative. whisper-large-v3 on test_net goes from 0.11452 to 0.09835, a much smaller gain. On test_meeting it goes from 0.19878 to 0.13706. For whisper-large-v3-turbo, test_net actually gets worse after fine-tuning: 0.21225 before, 0.23038 after. On test_meeting it goes from 0.20390 to 0.35697, which is a large regression.
That is the honest limitation of this kind of recipe, and the tables show it rather than hide it. Fine-tuning on one domain can degrade another. If your deployment audio looks more like meetings than like read speech, the turbo variant is a poor choice here. The README does not explain the turbo regression, but the number is there to read.
The other limitation is data. The project claims support for training without speech data and without timestamp data, which sounds like it removes the hard part. It does not. Whisper is a sequence-to-sequence model trained on paired audio and text. If you have no audio, you are not fine-tuning Whisper in any conventional sense; you are working with a much narrower setup, and the README does not spell out what that pipeline actually does. Treat that claim as a pointer to read the code, not as a guarantee.
CTranslate2 and GGML are different answers to different problems
The repository supports two acceleration formats, and they are not interchangeable. CTranslate2 is the server and desktop path. infer_ct2.py is the reference implementation, and the README says to use it as the main guide for converted-model inference. CTranslate2 runs on CPU and GPU with quantized weights, and it is the format you would put behind infer_server.py for a Web deployment.
GGML is the edge path. convert-ggml.py produces the file that AndroidDemo and WhisperDesktop consume. Those two directories are in the repository root as source trees, not as release artifacts. If you want to ship an Android app, you are building it from that source.
The split matters because it decides your deployment target early. A merged model can go either way, but the tooling after that point diverges completely. There is no single artifact that serves both a FastAPI server and an Android APK. Plan for two conversion steps if you need both.
The README also mentions a Web deployment demo at whisper.yeyupiaoling.cn:8082 and an online demo for a whisper-small fine-tune. Those are the author's hosted instances, not something you install. The Web deployment section in the README covers the interface documentation for the server, which is what you would actually run yourself.
How Whisper-Finetune compares to faster-whisper and plain transformers
faster-whisper is in requirements.txt, which is the clearest statement of the relationship. faster-whisper is a CTranslate2-based inference library for Whisper. It is a runtime, not a trainer. You install it, load a model, and transcribe. It will not fine-tune anything, and it has no notion of LoRA adapters.
Whisper-Finetune uses that runtime for its accelerated path but wraps training around it. So the difference is not which is faster at inference; if you convert the same merged model to CTranslate2, you are running the same engine. The difference is that Whisper-Finetune gives you a way to produce a domain-adapted model in the first place, and faster-whisper assumes you already have one.
The other alternative is the transformers training stack directly. Everything Whisper-Finetune does with LoRA is built on transformers, peft and accelerate. You could assemble the same pipeline yourself: load the processor, build a dataset, attach a LoRA config, train, merge. What you would not get for free is the data preparation scripts, the four inference entry points, the GGML conversion, and the two application source trees. Those are the actual content of this repository. The training loop is the least original part of it.
If you only need faster inference on a stock model, use faster-whisper directly and skip this project. If you need a model adapted to your own audio, the training pipeline is the reason to be here.
Licence, maintenance and the cost of the pinned stack
The repository is Apache-2.0. That is a permissive licence with an explicit patent grant, and it is compatible with commercial use. The Whisper checkpoints you fine-tune are separate artifacts under their own terms, and the README lists the supported ones as openai/whisper-tiny through openai/whisper-large-v3-turbo. Check the licence on the checkpoint you pick, not just on this repository. Nothing here is legal advice.
One practical wrinkle: the README repeatedly points to a knowledge community for model files, and the model-download column in the test tables says to join that community to obtain the fine-tuned checkpoints. The code is Apache-2.0, but the published fine-tuned weights are not distributed through the repository. If you want the author's trained models rather than your own, that is the route the README describes.
The last push was on 2026-05-08. That is recent enough that the project is not abandoned, but there are no releases, so there is no version to pin against. Your upgrade unit is a git commit, and the dependency pins in requirements.txt are exact. Upgrading transformers means editing the pin and retesting the whole chain: training, merging, CTranslate2 conversion, GGML conversion, and both application trees. That is a real cost, and it is the main reason to vendor the repository into your own build rather than track it live.
Editorial conclusion
Adopt Whisper-Finetune if you have Chinese or Cantonese audio, a GPU, and you want a LoRA adapter you can merge and convert to CTranslate2 or GGML rather than a full fine-tune. Do not adopt it if you need a maintained Python library with a stable API, if your data has no transcripts at all, or if you expect the published error-rate tables to transfer to your domain. Before training, verify the label format your corpus produces with aishell.py or tools/create_wenetspeech_data.py, and confirm that finetune.py accepts the config you point it at.
Frequently asked questions
Do I need timestamped transcripts to fine-tune Whisper with Whisper-Finetune?
No. The README states the project supports training without timestamp data, training with timestamp data, and training without speech data. The exact behaviour of each mode is not documented in the README, so read the code before relying on the no-speech path.
Can Whisper-Finetune run on Windows or without a GPU?
The README lists the development environment as Ubuntu 18.04 with an A100-PCIE-40GB, and the project badges say Win/Linux/MAC are supported systems. The deployment side includes a Windows desktop application under WhisperDesktop, but the README does not document CPU-only training.
How do I deploy a Whisper-Finetune model to Android?
Run convert-ggml.py to produce a GGML model, then build the Android application from the AndroidDemo directory in the repository. The README also notes that the acceleration path works with a stock Whisper model, so fine-tuning is not a prerequisite for the Android deployment.
Community notes