SoniTranslate: synchronized video dubbing through a Gradio web UI
Synchronized Translation for Videos. Video dubbing
At a glance
- What is it?
- SoniTranslate wraps WhisperX, Pyannote diarization and several TTS backends into one browser interface for translating a video and re-voicing it in the target language. Here is what the repository documents, and where it stops short.
- Who is it for?
- Adopt SoniTranslate if you need a self-hosted, scriptable pipeline that turns a video plus a target language code into a dubbed track, and you are willing to accept a Linux and CUDA 11.8 target with a Hugging Face token as a prerequisite. Do not adopt it if you need Windows or macOS support, a stable release cadence, or an API you can call without a GPU.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 21 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What SoniTranslate actually does with a video file
SoniTranslate is a Gradio web application that takes a video, extracts its audio, transcribes it, translates the transcript, and then generates a new voice track in the target language. The repository describes itself as "Synchronized Translation for Videos" and lists topics including asr, diarization, dubbing, speech-to-text, subtitle-to-speech and text-to-speech, so the pipeline is meant to cover the whole chain rather than one stage of it.
The audience is narrow but real. Someone who wants a one-click web service will find the Hugging Face Space demo easier. SoniTranslate is for the person who wants the pipeline on their own machine, with the model choices exposed in a browser form, and who is prepared to install CUDA drivers and accept model licenses to get there. The README points to a Colab notebook as the low-setup path and reserves the local install instructions for Linux, with the explicit note that installation was tested there.
One thing worth noticing before you invest time: the README opens with a sponsorship block for Recall.ai, a meeting transcription API. That is an ad, not a dependency. Nothing in requirements.txt pulls it in.
The pipeline behind the Gradio form: WhisperX, Pyannote, and a choice of TTS engines
The dependency list in requirements.txt is the clearest description of the architecture. Transcription comes from whisperX, installed straight from a git commit (git+https://github.com/m-bain/whisperX.git@a5dca2c) rather than a released version. Speaker separation comes from Pyannote, which is why the README tells you to accept the license on pyannote/speaker-diarization and pyannote/segmentation before starting. Translation is handled by deep_translator, with gTTS and edge_tts as the simpler speech backends.
Voice cloning and higher-quality synthesis pull in a heavier stack: praat-parselmouth, pyworld, torchcrepe and faiss-cpu for pitch and feature work, plus a fork of OpenVoice installed as git+https://github.com/R3gm/openvoice_package.git@lite. The repository also ships app_rvc.py and vci_pipeline.py at the top level, which points to an RVC-based voice conversion path alongside the default synthesis route.
That is a lot of moving parts pinned to specific commits and CUDA builds. The upside is that the pipeline is reproducible in the sense that it always resolves to the same whisperX and OpenVoice revisions. The downside is that upgrading any one component means editing a git URL by hand, and the README does not document a version matrix or a rollback procedure for a dependency that breaks. There are also separate requirements_base.txt, requirements_extra.txt and requirements_xtts.txt files, which suggests the install is meant to be layered, but the README excerpt does not explain which layer covers which feature.
Installing SoniTranslate locally and running a first translation
The README states that the local installation was tested on Linux and lists three prerequisites before any pip command: NVIDIA drivers for CUDA 11.8.0, an accepted Pyannote license on Hugging Face, and a Hugging Face token. The token requirement is documented right after the license step, which means the diarization models are gated and the app cannot fetch them anonymously.
The repository provides a requirements.txt that pins the CUDA 11.8 wheel index directly. Installing it is a single pip call, but it will pull torch, torchvision and torchaudio from the PyTorch cu118 index rather than from PyPI:
pip install -r requirements.txtAfter that line, the file carries a trailing comment instructing you to install omegaconf from a pull-request ref:
pip install git+https://github.com/omry/omegaconf.git@refs/pull/1137/mergeThat second command is easy to miss because it sits below a comment marker in the file rather than in the README's numbered steps. If you skip it, expect an import failure rather than a clear error message about a missing dependency.
The fast path the README recommends before any local install is the Colab notebook, linked as SoniTranslate_Colab.ipynb in the repository root. There is also a second notebook, SoniTranslate_Colab_embedded.ipynb, and a public demo at huggingface.co/spaces/r3gm/video_dubbing_sonitr_demo. For a first real use, the Colab route avoids the driver and license setup entirely and is the honest recommendation for anyone who just wants to see whether the output quality is acceptable for their footage. The README also links a video tutorial by DEV-MalletteS on YouTube, which is the only walkthrough the documentation offers for the UI itself.
Where SoniTranslate breaks down: languages, hardware, and the release gap
The language tables are the most detailed part of the README, and they also contain the sharpest limitation. SoniTranslate splits its supported languages into two groups. The first table covers languages that go through the full transcription and translation path. The second, labelled Non-transcription, lists codes such as ay, bm, ceb, ny, dv, doi, ee, gn, ilo, rw, kri, ku, ky, lg, mai, or, om, qu, sm, ti, ts, ak and ug. The README does not explain what happens to a video when you pick one of those codes, but the label itself is the warning: the pipeline treats them differently, and anyone whose target language sits in that second list should test the output before assuming parity with the first table.
Hardware is the second constraint. The requirements file targets cu118 wheels specifically, and the README's first prerequisite is an NVIDIA driver install. There is no documented CPU-only path, no macOS or Windows install section, and no statement about how much VRAM the diarization plus TTS stack needs. If you do not have an NVIDIA GPU, the Colab notebook is the only documented option.
The third issue is maintenance cadence. The most recent release listed is 0.5.0, published on 2024-05-18, with 0.4.0 the same day. The last push to the repository was on 2026-08-29, so work is happening, but it is not landing as tagged releases. Anyone pinning to a version number is pinning to something from 2024 and will be running code that predates two years of commits. There is no changelog in the repository, and the README does not describe what changed after 0.5.0.
How SoniTranslate differs from a hosted dubbing API
The obvious alternative for the same job is a hosted dubbing or transcription API, and Recall.ai is the one this repository itself surfaces at the top of the README as a sponsor. The difference is not quality, it is where the work happens. A hosted API takes an upload or a meeting link and returns a transcript or a dubbed file; you pay per minute and you never touch a driver. SoniTranslate runs the models on your hardware, which means no per-minute cost and no upload of your footage to a third party, in exchange for the CUDA 11.8 setup, the gated Pyannote licenses, and however long the GPU takes to chew through the file.
A closer comparison is a general-purpose ASR and TTS toolchain assembled by hand: WhisperX for alignment, Pyannote for diarization, edge_tts or gTTS for speech. SoniTranslate is essentially that assembly with a Gradio front end and a translation step wired in. If you already have a working WhisperX setup, the marginal value here is the UI, the translation stage, and the voice conversion scripts (app_rvc.py, vci_pipeline.py) rather than the transcription itself. If you do not, SoniTranslate saves you the wiring work but hands you a dependency list with three git refs in it, including a fairseq pull-request merge and an OpenVoice fork, which is a heavier maintenance surface than a plain pip install of whisperx.
Licence and the cost of keeping SoniTranslate current
The repository is Apache-2.0, which covers the code in this project. It does not automatically cover the models the code downloads. Pyannote's diarization and segmentation models are gated behind a Hugging Face license acceptance, as the README's prerequisites make explicit, and those terms are separate from Apache-2.0. The same caution applies to any TTS or voice-conversion model pulled in through voice_main.py or the OpenVoice fork. This is a description of where the boundaries sit, not legal advice; if you plan to dub content commercially, the model licences are the thing to read, not the repository's LICENSE file alone.
Upgrade cost is the practical concern. Three dependencies are pinned to git refs rather than versions: whisperX at commit a5dca2c, fairseq at refs/pull/5359/merge, and the OpenVoice fork at the lite tag. None of those can be bumped by editing a version number. The requirements file also pins gradio==4.19.2 and gradio_client==0.10.1, which are old enough that a Gradio upgrade would be a migration rather than a version bump. With the last tagged release at 0.5.0 from 2024-05-18 and the last push on 2026-08-29, the practical upgrade path is tracking main and re-resolving those git refs yourself. Budget for that before you build a workflow on top of it.
Editorial conclusion
Adopt SoniTranslate if you need a self-hosted, scriptable pipeline that turns a video plus a target language code into a dubbed track, and you are willing to accept a Linux and CUDA 11.8 target with a Hugging Face token as a prerequisite. Do not adopt it if you need Windows or macOS support, a stable release cadence, or an API you can call without a GPU. Before committing, verify that your card and driver match the cu118 wheels pinned in requirements.txt, that you have accepted the pyannote/speaker-diarization and pyannote/segmentation licenses on Hugging Face, and that the language pair you need appears in the README's transcription table rather than only in the non-transcription list.
Frequently asked questions
How do I install SoniTranslate locally?
The README documents a Linux install with three prerequisites first: NVIDIA drivers for CUDA 11.8.0, an accepted Pyannote license on Hugging Face, and a Hugging Face token. Then install requirements.txt with pip, and afterwards install omegaconf from the pull-request ref noted at the bottom of that file. A Colab notebook is offered as the alternative if you would rather skip the driver setup.
Is there a free AI video translator I can run myself?
The code is Apache-2.0 and the README points to both a Colab notebook and a public Hugging Face Space demo, so there is no licence fee to run it. The gated Pyannote models require a Hugging Face account and licence acceptance, and running it locally needs an NVIDIA GPU with CUDA 11.8 drivers.
Is there a free AI audio translator available?
The repository topics include translate-audio and audio-processing alongside video-dubbing, and the pipeline transcribes and re-voices the audio track of a video. The README's examples show original and translated audio clips rather than standalone audio files, so the documented workflow is video-in, dubbed video-out.
What are some good voice recognition translation apps like SoniTranslate?
SoniTranslate combines WhisperX for transcription, Pyannote for speaker diarization and deep_translator for translation, with gTTS and edge_tts among the speech backends listed in requirements.txt. The README also links a public Hugging Face Space demo and a Colab notebook, which are the two hosted ways to try the same pipeline without installing it.
Community notes