SmartSub: local Whisper subtitles, translation, dubbing and burn-in in one desktop app

Who SmartSub is built for, and the workflow it replaces

The README frames the product as one pipeline: speech to text, subtitle translation, proofreading, TTS dubbing, and burn-in, all inside a desktop application. The list of jobs it names is concrete. Watching a foreign-language video or lecture and getting bilingual subtitles. Adding subtitles to a Bilibili or YouTube link by pasting the URL rather than reaching for a separate downloader. Producing multi-language versions of a video by translating subtitles and then generating a foreign-language audio track. Cloning your own voice from one reference clip. Turning podcasts, online courses or meeting recordings into SRT files in batch. Burning subtitles into a finished cut after checking each line.

The audience follows from that list: individual editors, translators and content producers who work with video files sitting on their own disk. Every step is described as usable on its own or as part of a batch pipeline, which matters if you only want transcription and already have a translation workflow. The project is a desktop GUI, not a library. No HTTP API or CLI entry point is documented, so anything that needs to run unattended on a server is out of scope.

How the pipeline actually runs: local models, optional cloud services

Transcription is the core. The README says transcription is done with local models such as whisper.cpp and sherpa-onnx, and that files do not leave the machine. Eight transcription engines can be switched per task: whisper.cpp, faster-whisper, FunASR, Qwen3-ASR, FireRedASR, NVIDIA Parakeet, a local Whisper CLI, and cloud dictation from nine providers. Engine and model management is centralized on an "engines and models" page. The language guidance is specific: FunASR or FireRedASR for Chinese, Parakeet models for English, European languages and Japanese.

On top of raw transcription sits an optional AI refinement step. The README describes it as semantic sentence splitting plus batch correction: sentence breaks are reorganized by meaning while the timeline stays word-accurate, conjunctions are not left dangling at line ends, and numbers are not split by pauses. Correction fixes homophones, removes filler words and normalizes punctuation. The provider defaults to the AI translation configuration, so a local Ollama model makes it free, and the README states that a failure falls back to rule-based sentence splitting.

Translation offers 20 services, from built-in free endpoints (Bing and Google free interfaces, with automatic fallback and rate limiting) to Baidu, Aliyun, Tencent, iFlytek, Volcengine, Doubao, DeepLX, Azure, Google, and LLM services including Ollama, DeepSeek, Gemini, Qwen, SiliconFlow, Azure OpenAI and DeerAPI. Any OpenAI-style API is accepted, and per-service request parameters are configurable in the UI and can be exported and imported.

Dubbing is a separate workbench: one subtitle file plus an optional video, synthesizing line by line and aligning to the timeline. Local engines are Kokoro (103 voices, multilingual) and VITS (174 Chinese voices). Voice cloning uses local ZipVoice zero-shot cloning from one reference audio clip, or cloud options from Volcengine and ElevenLabs. Timeline alignment uses speed pre-control, measured verification, and borrowing silence gaps; lines that still exceed their slot go to a manual list where you can edit the text, regenerate a single line, or accept a speed change. Output can be audio only, a replaced track, a mixed video, or an MKV with dual audio tracks.

Burn-in is ffmpeg under the hood: hard subtitles rendered permanently, or soft subtitles muxed losslessly as a switchable track, with font, size, color, outline, shadow and nine-grid position controls plus live preview.

Installing SmartSub and running a first transcription

The README points to GitHub Releases and a Quark netdisk link for downloads, with packages named by platform and chip: windows-x64, mac-arm64, mac-x64, linux-x64. GPU acceleration packs are not chosen at download time; they are fetched inside the app. The README notes that the Intel macOS build is CPU-only and does not support GPU acceleration. On macOS, Homebrew is the recommended route because it matches the chip automatically:

brew tap buxuku/tap          # only once
brew install --cask smartsub # install
brew upgrade --cask smartsub # upgrade

After installation the README gives a three-step start. First, follow the onboarding to download a speech model; if you have no GPU or do not want to download a model, you can configure cloud dictation instead. Second, pick a task on the launch screen and drag in audio, video or subtitle files, or paste a link to download an online video, then set source language, target language and other parameters. Third, start processing. When it finishes you can continue to proofread the subtitles, dub them, or burn them in.

For acceleration, the README states that the CUDA, Vulkan and Core ML packs are downloaded inside the app and that no manual CUDA Toolkit installation is needed, with an automatic fallback to CPU if loading fails. Online video download uses two engines, yt-dlp for YouTube and 1800+ sites and lux for Chinese platforms such as Bilibili, Douyin and Xiaohongshu, matched by platform and installed or updated from inside the app. If a platform provides official subtitles, including auto-generated ones, the task wizard pairs them automatically and transcription can be skipped.

Where SmartSub stops being the right tool

The most important constraint is packaging. SmartSub is an Electron desktop app whose main entry is app/background.js and whose build scripts run through nextron, so the natural unit of use is one person at one machine. There is no documented server mode, no CLI for batch jobs, and no API surface described in the README. If your pipeline is a scheduled job on a Linux box, or a service several people submit to, this is the wrong shape regardless of transcription quality.

The free path has a cost that is easy to miss. The README states that the whole chain can run without an API key, but that path depends on downloading local models, and model quality varies by language and by engine. The README itself gives language-specific recommendations rather than a single default, which is a signal that no one engine covers everything. The built-in free translation endpoints rely on Bing and Google free interfaces with automatic fallback and rate limiting, so throughput depends on an external service you do not control.

Voice cloning deserves the same caution. ZipVoice cloning is described as zero-shot from one reference clip, and the README says the reference audio is used to build the voice on the spot. That is convenient, but the README gives no guidance on consent, provenance or how to handle a cloned voice once an audio track is exported. The README also notes that over-long dubbing lines are pushed to a manual list rather than silently stretched, which is honest but means dubbing is not a fully unattended step.

Privacy claims need to be read precisely. The README says local processing keeps files on the machine and that cloud services are optional with a first-use privacy confirmation. That is accurate for the local engines. It is not a blanket statement about every configuration, because choosing a cloud dictation provider, a cloud translation service or a cloud TTS service sends the corresponding data out.

How SmartSub differs from running Whisper yourself

The obvious alternative is a command-line Whisper setup: whisper.cpp or faster-whisper installed directly, driven by shell scripts, with ffmpeg for burn-in and a separate translation step. The difference in approach is not the model, because SmartSub uses those same runtimes. The difference is everything around them.

With a CLI you get composability and automation. You can run it in a container, schedule it, version the exact arguments, and pipe output anywhere. You also get no GUI, no model manager, no proofreading view, no dubbing workbench and no voice cloning, and you assemble translation and burn-in yourself. SmartSub trades that composability for a managed path: engines and models installed through the app, GPU packs downloaded in-app rather than via a CUDA Toolkit install, a per-line proofreading table, a dubbing timeline alignment step, and burn-in with live preview.

If your work is one-off or a few dozen files with human review at the end, the GUI saves real time. If your work is thousands of files a week with no human in the loop, the CLI wins on every axis that matters. A middle option the README mentions is worth noting: the local Whisper CLI is itself one of the eight selectable engines, so you are not forced to abandon an existing whisper.cpp binary if you only want SmartSub's translation, proofreading or burn-in stages.

Maintenance, releases and what the MIT licence means here

The repository is not archived, and the last push was on 2026-09-11, three days before this writing. Releases are frequent and described in detail: v3.8.0 on 2026-09-10 added multi-format subtitle export, backup service failover and expanded audio engines; v3.7.0 on 2026-08-06 added multi-role dubbing and speaker recognition plus more ASR engines; v3.6.0 on 2026-07-30 introduced AI subtitle refinement with more control over language and transcription. The package version in package.json is 3.8.0, matching the latest release.

The upgrade cost is low for the app itself. Homebrew users run one command, and release downloads replace the previous install. The larger ongoing cost is not the binary but the surrounding assets. Models are downloaded separately, GPU acceleration packs are fetched in-app, and the yt-dlp and lux download engines are installed and updated from inside the application. Those are the pieces that drift when a platform changes its site or a model is replaced, and they are also the pieces that make the first run heavier than a typical desktop install.

The licence is MIT, which is permissive and places few conditions on use, modification or redistribution. Two things sit outside that grant and are worth checking before commercial use. First, the bundled or downloaded components have their own terms: model weights, ffmpeg builds, yt-dlp, lux and the various cloud service SDKs are separate from the MIT licence on this repository. Second, if you enable a cloud provider, that provider's terms govern the data you send. This is a description of what the repository states, not legal advice; read LICENSE and the individual component licences yourself.

Editorial conclusion

Adopt SmartSub if you work on a desktop with video files and want the whole chain (transcribe, translate, dub, burn) without sending audio to a cloud service, and you accept that the free path means downloading models yourself. Do not adopt it if you need a headless server pipeline, a CLI for CI, or a web service that several people share. Before committing, verify three things on your own machine: that the GPU acceleration pack for your chip installs and loads, that your chosen transcription engine produces usable output for your language, and that the free translation source is reachable from your network.

Frequently asked questions

Official sources

1. buxuku/SmartSub on GitHub
2. License: MIT
3. Project website
4. README
5. Releases