SmartSub: local Whisper subtitles, translation, dubbing and burn-in in one desktop app

Who SmartSub is built for, and the workflow it replaces

The README frames the product as one pipeline: speech to text, subtitle translation, proofreading, TTS dubbing, and burn-in, all inside a desktop application. The list of jobs it names is concrete. Watching a foreign-language video or lecture and getting bilingual subtitles. Adding subtitles to a Bilibili or YouTube link by pasting the URL rather than reaching for a separate downloader. Producing multi-language versions of a video by translating subtitles and then generating a foreign-language audio track. Cloning your own voice from one reference clip. Turning podcasts, online courses or meeting recordings into SRT files in batch. Burning subtitles into a finished cut after checking each line.

The audience follows from that list: individual editors, translators and content producers who work with video files sitting on their own disk. Every step is described as usable on its own or as part of a batch pipeline, which matters if you only want transcription and already have a translation workflow. The project is a desktop app, not a library, but the README also documents 111 MCP tools with matching CLI commands through a smartsub command line, so external AI agents, shell scripts and CI can run the pipeline.

How the pipeline actually runs: local models, optional cloud services

Transcription is the core. The README says transcription is done with local models such as whisper.cpp and sherpa-onnx, and that files do not leave the machine. Eight transcription engines can be switched per task: whisper.cpp, faster-whisper, FunASR, Qwen3-ASR, FireRedASR, NVIDIA Parakeet, a local Whisper CLI, and cloud dictation from nine providers. The language guidance is specific: FunASR or FireRedASR for Chinese, Parakeet models for English, European languages and Japanese.

On top of raw transcription sits an optional AI refinement step. The README describes it as semantic sentence splitting plus batch correction: sentence breaks are reorganized by meaning while the timeline stays word-accurate, conjunctions are not left dangling at line ends, and numbers are not split by pauses. Correction fixes homophones, removes filler words and normalizes punctuation. The provider defaults to the AI translation configuration, so a local Ollama model makes it free, and the README states that a failure falls back to rule-based sentence splitting.

Translation offers 20 services, from built-in free endpoints (Bing and Google free interfaces, with automatic fallback and rate limiting) to Baidu, Aliyun, Tencent, iFlytek, Volcengine, Doubao, DeepLX, Azure, Google, and LLM services including Ollama, DeepSeek, Gemini, Qwen, SiliconFlow, Azure OpenAI and DeerAPI. Any OpenAI-style API is accepted, and per-service request parameters are configurable in the UI and can be exported and imported.

Dubbing is a separate workbench: one subtitle file plus an optional video, synthesizing line by line and aligning to the timeline. Local engines are Kokoro (103 voices, multilingual) and VITS (174 Chinese voices). Voice cloning uses local ZipVoice zero-shot cloning from one reference audio clip, or cloud options from Volcengine and ElevenLabs. Timeline alignment starts with speech-rate pre-control.

Burn-in uses local ffmpeg: after checking each line in the proofreading view, subtitles can be hard-burned or soft-muxed, with what-you-see-is-what-you-get styling.

Installing SmartSub and running a first transcription

Releases are published on GitHub, and the app runs on Windows, macOS and Linux.

The README lists hardware acceleration for NVIDIA, AMD, Intel and Apple Silicon. Online video download uses two engines, yt-dlp for YouTube and 1800+ sites and lux for Chinese platforms such as Bilibili, Douyin and Xiaohongshu, matched by platform and installed or updated from inside the app. If a platform provides official subtitles, including auto-generated ones, the task wizard pairs them automatically and transcription can be skipped.

Where SmartSub stops being the right tool

SmartSub is an Electron desktop app whose main entry is app/background.js and whose build scripts run through nextron. Automation runs through the same app: the README describes 111 MCP tools with matching smartsub CLI commands and a windowless background daemon that starts on first call and shares task state, configuration and models with the desktop client.

The free path has a cost that is easy to miss. The README states that the whole chain can run without an API key, but that path depends on downloading local models, and model quality varies by language and by engine. The README itself gives language-specific recommendations rather than a single default, which is a signal that no one engine covers everything. The built-in free translation endpoints rely on Bing and Google free interfaces with automatic fallback and rate limiting, so throughput depends on an external service you do not control.

Voice cloning deserves the same caution. ZipVoice cloning is described as zero-shot from one reference clip, and the README says the reference audio is used to build the voice on the spot. That is convenient, but consent and provenance for a cloned voice remain your responsibility.

Privacy claims need to be read precisely. The README says local processing keeps files on the machine and that cloud services are optional add-ons. That is accurate for the local engines. It is not a blanket statement about every configuration, because choosing a cloud dictation provider, a cloud translation service or a cloud TTS service sends the corresponding data out.

How SmartSub differs from running Whisper yourself

The obvious alternative is a command-line Whisper setup: whisper.cpp or faster-whisper installed directly, driven by shell scripts, with ffmpeg for burn-in and a separate translation step. The difference in approach is not the model, because SmartSub uses those same runtimes. The difference is everything around them.

With a CLI you get composability and automation. You can run it in a container, schedule it, version the exact arguments, and pipe output anywhere. You also get no GUI, no model manager, no proofreading view, no dubbing workbench and no voice cloning, and you assemble translation and burn-in yourself. SmartSub wraps those runtimes in a managed path: engines switchable per task, a line-by-line proofreading view, a dubbing workbench with timeline alignment, and burn-in with what-you-see-is-what-you-get styling, while its smartsub CLI and MCP tools keep scripted batch runs possible.

If your work is one-off or a few dozen files with human review at the end, the GUI saves real time. If your work is thousands of files a week with no human in the loop, the CLI wins on every axis that matters. A middle option the README mentions is worth noting: the local Whisper CLI is itself one of the eight selectable engines, so you are not forced to abandon an existing whisper.cpp binary if you only want SmartSub's translation, proofreading or burn-in stages.

Maintenance, releases and what the MIT licence means here

The repository is not archived, and the last push was on 2026-10-30. Releases are frequent: v3.12.1 on 2026-10-30 is an emergency fix for Windows long-path transcription; v3.12.0 on 2026-10-29 made tasks more reliable and added browser cookie import for online downloads; v3.11.0 on 2026-10-22 reorganised the "More settings" area, made font selection easier and made translation output more reliable. The package version in package.json is 3.12.1, matching the latest release.

The upgrade cost is low for the app itself. The larger ongoing cost is not the binary but the surrounding assets. Models are downloaded separately, and the yt-dlp and lux download engines are installed and updated from inside the application. Those are the pieces that drift when a platform changes its site or a model is replaced, and they are also the pieces that make the first run heavier than a typical desktop install.

The licence is MIT, which is permissive and places few conditions on use, modification or redistribution. Two things sit outside that grant and are worth checking before commercial use. First, the bundled or downloaded components have their own terms: model weights, ffmpeg builds, yt-dlp, lux and the various cloud service SDKs are separate from the MIT licence on this repository. Second, if you enable a cloud provider, that provider's terms govern the data you send. This is a description of what the repository states, not legal advice; read LICENSE and the individual component licences yourself.

Editorial conclusion

Adopt SmartSub if you work on a desktop with video files and want the whole chain (transcribe, translate, dub, burn) without sending audio to a cloud service, and you accept that the free path means downloading models yourself. If you need scripted or CI runs, the README documents a smartsub CLI and MCP tools backed by a headless background daemon. Before committing, verify three things on your own machine: that hardware acceleration works on your machine, that your chosen transcription engine produces usable output for your language, and that the free translation source is reachable from your network.

Frequently asked questions

Official sources

1. buxuku/SmartSub on GitHub
2. License: MIT
3. Project website
4. README
5. Releases