jt-live-whisper: a voice toolbox that never calls the cloud

The premise, and where it came from

The project describes itself as a fully local AI voice toolbox: live transcription, live translation, batch processing of recorded files, speaker identification, meeting summaries, and reading aloud in Taiwanese Mandarin, with every model running on your own device and no data passing through any cloud service.

The origin story is in the readme, and it is the kind of story that explains a design well. The author attended an online technical course taught entirely in English, could not follow it, and built the tool to translate it live for himself. The features then multiplied, and the readme says so with some self-deprecation: the features grew and it became what it is now.

The six reasons the project gives for the local approach are worth reading closely, because they are the arguments a procurement conversation actually turns on. Fully local execution means the recognition, the translation, the speaker identification and the summaries all run on your own hardware, with no cloud API key and nothing uploaded. Privacy follows from that, and the readme points at a separate document covering data protection and compliance for the things the tool does and the things an organisation still has to do about personal data law and information security standards. Zero monthly cost is the third reason, and it is stated bluntly: no paid cloud services are needed, because every model used is free and open source.

The fourth reason is application independence. Audio is captured at the system device level rather than from a specific program: on macOS through the built-in screen capture framework with no driver to install, on Windows through the loopback mode of the audio API, and on Linux through the monitor source of PipeWire or PulseAudio. The consequence is that in principle any software's audio output can be processed, including online meetings, video sites, and podcasts.

The last two reasons are completeness and a one-click install, where the install script downloads and builds the models and the dependencies for you.

The model table, where the real decisions are

The readme's largest table is the list of models, and each row carries four things: the purpose, the model, an explanation, and the licence. Writing the licence next to every component is unusual and is the most useful thing in the document.

For recognition there are four options. A general multilingual model from the Whisper family is the main one, with several sizes available. A Taiwanese-accent model from a semiconductor research group is specialised for the Taiwanese language, with an option for Mandarin mode when a meeting mixes the two, and it outputs Chinese characters directly. An English-only model with very low streaming latency is offered, with the note that it does not support Intel Macs. And an experimental Chinese and English model was added later, with measured numbers: on twenty real Chinese meetings the character error rate drops from about twenty-nine percent to about sixteen percent, and when Chinese and English are mixed it recovers two to three times more of the minority language. It is limited to Chinese, English and Korean, and it is not offered for Japanese, Taiwanese, bidirectional, or live subtitle work, and the menu hides it in those cases and explains why on the command line.

Translation is done by a self-hosted language model server, with a large default model and a smaller fallback when the server does not have it. The advice is to use fourteen billion parameters or more, and to pick a model that does not think, or whose thinking can be turned off, because the program switches thinking off automatically for the families that allow it, while one model family is architecturally always reasoning and cannot be disabled, which makes it noticeably slower for live translation.

And then the licence caveat that a careful reader will notice first: the offline translation model from a large research lab is licensed for non-commercial use only. The readme says explicitly that an organisation should check its own terms and should not assume internal use counts as non-commercial, and that the alternative is the language model server route. That is a licence limitation stated in the place where somebody will read it before deploying it, which is exactly where it belongs.

Speech synthesis is newer, with a model that reads text aloud using Taiwanese pronunciation for Chinese, handles mixed English, and supports about thirty languages in the model itself, though the reading is tuned for Taiwanese Mandarin. It ships with eight generated Chinese voices and two English ones, none of them recordings of real people, and you can import your own recording. A second synthesis model with a Taiwanese accent is offered as an option, with the warning that it is slow, unsuitable for live use, and only runs on a GPU server.

One model, several engines, and the platform quirks

The second table lists the inference engines, and the point of it is that the same model runs on different backends depending on the platform and the audio source. That is a more interesting design than it first appears.

The native core of the Whisper family is used for live recognition on macOS when the audio comes from one particular device path. A faster compiled variant is used for live recognition on Windows and Linux, for offline processing everywhere, and for a GPU server. An Apple Silicon backend provides Metal acceleration for both live and offline work, including the Taiwanese model. The low-latency English model is its own engine. And a serving engine plus a local Apple backend plus the standard transformer library cover the experimental Chinese model, on a GPU server, on an Apple Silicon machine, and on a Windows or Linux machine with a CUDA device or with only a CPU.

So the user does not choose an engine. The program picks one based on the platform and the audio source, and the table documents which combination is available where. That is a much better user experience than exposing seven backends, and it also means the project has to maintain the awkward cases: an Intel Mac has no Metal, so the Apple backend does not apply; a machine with only a CPU can still run the experimental model, with a warning that it may be slower than the recording it is processing.

The speaker identification section is where the project is most candid about its own design history. The obvious choice would be the well known diarization toolkit, and the readme explains why it is not used: its open models are permissively licensed, but downloading them requires signing in, filling in a user data form, and configuring a token locally. That breaks the design principle of the project, which is no accounts and no registration.

So since a later version the default is a diarization model from a graphics hardware vendor, licensed for commercial use, with no account or token, downloaded automatically by the installer at about seven hundred megabytes. On an Intel Mac, or with more than eight speakers, or without a recent enough version of the library, the program falls back to the older method of voice print features with a spectral clustering algorithm.

That fallback chain is the whole story of the project in miniature: a preference for tools that need no account, a documented exception where that preference cannot be met, and an automatic downgrade when the exception applies.

Three deployment shapes

The deployment section describes three arrangements, and the distinctions between them are the practical part of the design.

The single machine mode needs nothing but a Mac, a Windows PC, or a Linux desktop. On Apple Silicon the recognition runs through the Metal backend at roughly one to three seconds; on Windows and Linux with an NVIDIA device, the installer detects the card and enables CUDA acceleration at roughly half a second to a second, with no separate server needed; and on a machine with no graphics card, or an Intel Mac, recognition runs on the processor and a small model is the practical choice.

The second arrangement splits the work. The local machine handles audio capture and the interface, and a GPU server on the local network does the recognition and the speaker identification, with both the system audio and the microphone able to be sent to the remote machine. The readme gives the numbers: offline processing is five to ten times faster, and live recognition takes about three to five tenths of a second. It names the hardware as being either a dedicated compact machine or a Linux host with an NVIDIA card, a consumer card being enough if CUDA is already installed.

The third is a headless server, installed on a Linux machine with no desktop, running the web interface as a service so that machines on the network can upload recordings through a browser. Operating it from another computer requires a password, and the installer generates an administration password and a read-only one. The resource figures are strikingly modest: with a GPU server doing the recognition, two virtual processors, four gigabytes of memory and thirty gigabytes of disk are enough, and a thirty-seven minute meeting peaked at about four hundred megabytes of memory.

And the modes can be switched at any time, with automatic fallback to local processing when the server is unreachable, so an interrupted network does not stop the work. The server's recognition service is a separate program that does not update with the local installation, so the local side compares versions on connection and warns when they differ without interrupting anything. There is also an optional update mechanism in which the local side pushes a new version to the server and the server verifies it before replacing the old one, and that mechanism is off by default.

The repository, and the interface

The top level of the repository is a good picture of the shape of the project, and a few of the file names are worth explaining.

There are three install scripts, one for macOS and Linux, one for Windows in PowerShell, and one for Linux again, and two start scripts in the same two forms. There is a Python file for the subtitle overlay and another for the meeting translation, a Swift file for the macOS audio capture, and a remote recognition server script. The web interface is a single HTML file with a Python server behind it. And there are three directories that are clearly the separate programs: one for the meeting features, one for the API, and one for the text to speech.

Alongside the code are the documents a project of this kind needs: a benchmarks file, a compliance note, an operations manual, a change log, and a documentation directory.

The operations manual being a separate document with the update mechanism described in the readme as a detail to be found there is a good sign. A tool with a GPU server, a local client, a server mode, and an update path has moving parts that can get out of step, and writing the procedure down is what keeps that from becoming a support problem.

The readme also has a table of contents that covers the core features, the other characteristics, the system requirements, a quick start, the usage, an interactive menu with a summary of what each menu entry does, the command line arguments, the technical architecture, hardware recommendations, and the upgrade path. The presence of a full interactive menu alongside the command line arguments is worth noting, because it means the project is usable by somebody who will never read the command line documentation and by somebody who will never use the menu.

The system requirements section is where the models become concrete, and the readme is honest that this is a heavy application: the audio models range from a very small English model to the largest multilingual one, the language model server wants a machine that can host twenty-six billion parameters, and the speech synthesis on an Apple Silicon machine wants sixteen gigabytes of memory.

Editorial conclusion

jt-live-whisper fits a person or a team whose material must not leave the machine, who wants live subtitles for an English meeting or a video, and who is willing to download several gigabytes of models and learn which one to choose for which job. It does not fit somebody who wants a service with no setup, and it does not fit a commercial deployment that cannot accept a licence with a non-commercial clause on one of the optional translation models. Verify first that the licence of every model you plan to switch on permits your use, that the machine you will run it on can hold the models you intend to use, and that the pronunciation dictionary and the accent-specific models are relevant to the language your users actually speak.

Frequently asked questions

Official sources

1. Issues
2. jasoncheng7115/jt-live-whisper on GitHub
3. License: Apache-2.0
4. Project website
5. README