pyVideoTrans: a local pipeline for translating videos and dubbing them

What pyVideoTrans actually replaces

Translating a video by hand is four separate jobs: transcribe the audio, translate the transcript with timing intact, generate speech in the target language, and mux it back onto the picture. Each step has its own tool, and each handoff is where timing drifts. pyVideoTrans exists to collapse those four steps into one program. The README describes the flow as "Speech Recognition (ASR) -> Subtitle Translation -> Speech Synthesis (TTS) -> Video Synthesis", which is the whole product in one line.

The intended user is not a casual viewer. It is someone who already has FFmpeg installed, is comfortable running Python, and wants to point the tool at a folder of clips rather than paste a link into a website. The repository ships a CLI (cli.py), a desktop GUI (sp.py) and a browser interface (webui.py), plus a Dockerfile, which tells you the maintainer expects it to run on a server as well as a laptop. The GPL-3.0 licence matters here: if you embed this in a product you distribute, the obligations follow the code.

The four-stage pipeline and where your data goes

Each stage is swappable. For speech recognition the README lists Faster-Whisper as the local option and describes it as "Recommended, fast speed, high accuracy", with WhisperX and Parakeet for timestamp alignment and speaker diarization, and hosted options from Alibaba Qwen3-ASR and ByteDance Volcano. Translation can run through DeepSeek, ChatGPT, Claude, Gemini, MiniMax, Ollama locally, or Alibaba Bailian. Speech synthesis covers Edge-TTS, OpenAI, Azure, Minimaxi, ChatTTS and ChatterBox.

That menu is the architecture. There is no single model doing the work; the program orchestrates whichever channel you select at each stage. The practical consequence is that the privacy boundary is your choice, not the project's. Pick Faster-Whisper plus Ollama plus a local TTS model and nothing leaves the machine. Pick a hosted ASR and a hosted LLM and both your audio and your transcript are sent to third parties. The README does not present this as a decision point, but it is the first configuration choice anyone with sensitive footage has to make.

The README also points to docs/architecture.md for the technical detail and mentions an interactive editing mode that lets you pause and proofread between recognition, translation and dubbing. That staging matters more than it sounds: machine translation of subtitles frequently mangles names and idioms, and a checkpoint before synthesis saves regenerating audio you will throw away.

Installing pyVideoTrans from source with uv

The README recommends uv for package management and gives the commands below. Python 3.10 is the recommended version, and pyproject.toml pins requires-python to ">=3.10, <3.11", so a 3.11 or 3.12 interpreter will not satisfy the constraint. FFmpeg must be installed and on the environment variables; on Ubuntu or Debian the README gives sudo apt-get install ffmpeg libsndfile1-dev.

git clone https://github.com/jianchang512/pyvideotrans.git
cd pyvideotrans
uv sync

After uv sync finishes, the GUI starts with the command below. The README notes that whisper.net and the WebUI are not installed by default; uv sync --all-extras pulls in the optional channels, uv sync --extra webui adds the browser interface.

uv run sp.py

The first real use is transcription, which is the cheapest way to find out whether your audio quality and your chosen model agree. This writes an SRT from an audio file using the large-v3 model:

uv run cli.py --task stt --name "./audio.wav" --model_name large-v3

Once the SRT looks right, the full video translation task takes the same file path plus a target language and a voice role. The README's own example for Chinese to English is:

uv run cli.py --task vtv --name "./video.mp4" --source_language_code zh-cn --target_language_code en --voice_role "en-US-GuyNeural"

For server deployment, the README builds a WebUI image and exposes port 7860:

docker build -t pyvideotrans-webui .
docker run -d -p 7860:7860 --name pyvideotrans pyvideotrans-webui

The Dockerfile sets GRADIO_SERVER_PORT to 7860 and installs FFmpeg from a static build, so the container does not depend on the host having it. A GPU image is built by passing --build-arg USE_CUDA=true, which switches the base image to nvidia/cuda:12.8.0-cudnn-runtime-ubuntu22.04.

GPU setup is the part most likely to break

The README's optional acceleration section is explicit that the default install is CPU-only and that you have to replace PyTorch yourself. It gives these commands, with the caveat that the index URL is an example for CUDA 12.x:

uv remove torch torchaudio
uv add torch==2.7 torchaudio==2.7 --index-url https://download.pytorch.org/whl/cu128
uv add nvidia-cublas-cu12 nvidia-cudnn-cu12

The Windows pre-packaged build has its own requirement: CUDA 12.8 and cuDNN 9.11 installed on the machine before GPU acceleration works. Note the version mismatch between the two paths (torch 2.7 wheels from the cu128 index versus a stated CUDA 12.8 and cuDNN 9.11 requirement), and note that the Dockerfile pins torch==2.7.1 inside the CUDA branch. Three different version statements across three files is a maintenance smell, and it is the most likely source of a failed first install. The README does not document a rollback path if the torch swap breaks the environment, so keep the working CPU environment recoverable before you touch it.

Where pyVideoTrans is the wrong tool

Version pinning is the first constraint. requires-python is ">=3.10, <3.11", so this cannot share an environment with a project that needs a newer interpreter. The dependency list in pyproject.toml is long and fully pinned, including accelerate==1.12.0, ctranslate2==4.8 and av==16.0.1, which makes conflict resolution with an existing environment unlikely to succeed. Use the uv-managed environment the README describes rather than installing into a shared one.

Speaker handling is the second. The README lists speaker diarization and multi-role dubbing as features, and voice cloning through F5-TTS, CosyVoice and GPT-SoVITS. What it does not document is behaviour on overlapping speech, crosstalk, or recordings where the diarization model merges two speakers into one. If your source is a panel discussion with people talking over each other, the pipeline's assumption of clean turn-taking is a poor fit, and you will spend the time you saved on manual correction.

The third case is scale with a service-level expectation. This is a local program that runs FFmpeg and neural models on your hardware. The README documents no queueing, no retry semantics for hosted API failures, and no concurrency controls. For a one-off documentary or a batch of internal clips it is fine. As the compute layer behind a product with an uptime target, it is not the shape of thing you want.

How it differs from KrillinAI and hosted dubbing services

KrillinAI appears in the same search space and solves a similar problem, but the README here does not compare the two and neither should be judged on the other's documentation. The honest distinction you can draw from this repository alone is about deployment shape: pyVideoTrans ships a desktop GUI, a CLI, a WebUI and a Dockerfile in one tree, and its channel list spans local models and hosted APIs at every stage. A tool that only wraps a hosted API cannot offer the offline path; a tool that only runs local models cannot offer the hosted quality ceiling. That flexibility is the actual differentiator, and it is also why the install is heavier than a single-purpose wrapper.

Against hosted services such as Vozo AI or Rask AI, the difference is not quality, it is control and cost structure. A hosted service bills per minute and holds your media. pyVideoTrans bills you in GPU time and disk, and holds your media only if you choose hosted channels. The README does not publish benchmark comparisons, so any claim that local Faster-Whisper beats a commercial ASR on your footage is something you have to measure yourself on a representative clip.

Licence, maintenance and upgrade cost

The project is GPL-3.0. If you distribute software that links this code, the copyleft terms apply to the combined work. Running it internally to produce translated videos is a different situation from shipping it inside a closed product, and the LICENSE file in the repository is the authoritative text. This is not legal advice; read the licence and, if you are embedding it, talk to someone qualified.

Maintenance looks current. The last push was on 2026-09-14, and the most recent release is v4.12 from 2026-09-06, following v4.11 on 2026-08-23 and v4.10 on 2026-08-17. That is a release roughly every two to three weeks. The upgrade cost is where the pinned dependency list bites: pyproject.toml and uv.lock pin nearly everything, so upgrading the application means re-resolving that lock file, and the torch pins mean a CUDA upgrade and an application upgrade are the same operation. Budget for testing the GPU path after every version bump, not just the translation quality.

Editorial conclusion

Adopt pyVideoTrans if you want the whole speech recognition to dubbing chain in one Python program and are willing to keep Python 3.10, FFmpeg and a GPU stack in working order. Skip it if you need a hosted service with an uptime commitment, or if your video has overlapping speakers, because the README documents speaker diarization as a feature but does not describe how it behaves on crosstalk. Before committing, run the CLI on a short clip with --task stt, confirm the SRT timings, then repeat with --task vtv and check the dubbed audio alignment.

Frequently asked questions

Official sources

1. jianchang512/pyvideotrans on GitHub
2. License: GPL-3.0
3. Project website
4. README
5. Releases