Bernini (bytedance/Bernini): a semantic planner plus a diffusion renderer for video editing
Bernini is a unified framework for video generation and editing that combines an MLLM-based semantic planner with a DiT-based renderer.
At a glance
- What is it?
- ByteDance's Apache-2.0 repository splits video generation into an MLLM planner and a DiT renderer, and also ships a renderer-only variant. Here is what each family does, how the install actually goes, and where the pinned CUDA stack will bite.
- Who is it for?
- Adopt Bernini if you have a Hopper-class GPU, can hold Python 3.11.2 and CUDA 12.6, and need instruction-driven video editing rather than plain text-to-video. Skip it if you are on CPU, on Python 3.12, or unwilling to install Open-VeOmni with --no-deps, since the README states all inference paths import it.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 37 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Bernini targets: instruction-driven video editing, not plain text-to-video
Most open video diffusion models take a prompt and produce a clip. Editing an existing clip is a different job: the model has to work out what in the frame the instruction refers to, change that part, and leave the rest alone. Bernini's answer is to split that into two stages. An MLLM-based semantic planner interprets the instruction and plans the semantic change; a DiT-based renderer then produces the pixels. The repository describes the combination as "a unified framework for video generation and editing", and the shared task interface covers both sides: t2i, i2i, t2v, v2v, rv2v and r2v. If your work is style transfer, subtitle or watermark removal, or local edits on existing footage, that interface is aimed at you. If you only ever need prompt-to-clip generation, the planner stage is overhead you are paying for.
Two model families, one task interface: Bernini versus Bernini-R
The README's model table is the most consequential page in the repository, because picking the wrong family changes your hardware bill. Bernini is the full pipeline: planner plus renderer, with checkpoints at ByteDance/Bernini-Diffusers and ByteDance/Bernini-Diffusers-v2. Bernini-R is renderer-only, fine-tuned from the Wan diffusion renderer, with checkpoints at ByteDance/Bernini-R-Diffusers (14B), ByteDance/Bernini-R-1.3B-Diffusers and a separate ByteDance/Bernini-R set. The trade-off is stated plainly: Bernini decomposes complex instructions and plans semantic changes before rendering, which the README credits with stronger instruction following; Bernini-R has "fewer moving parts" and a simpler setup. The 1.3B release note is the most useful honesty in the file: fine-tuned from Wan2.1-1.3B, it performs close to the 14B variant on simple tasks such as style transfer, subtitle or watermark removal, and local editing, while lagging behind on more complex tasks such as human generation. That is a workload boundary, not a marketing line, and it should drive your choice more than the benchmark table does. The benchmark numbers (EditVerse, OpenVE, OpenS2V, VBench, Bernini-v2v, Bernini-rv2v) come from the project's own arena platform, where the README says human annotators blindly vote on paired edits. Self-built arenas measure what their authors chose to measure; treat the table as the project's claim, not as an independent result.
Installing Bernini for inference: pinned torch, and Open-VeOmni with --no-deps
The README's inference path is short but has one step that people will get wrong. Clone the repository, install the pinned requirements, then install Open-VeOmni separately with --no-deps. The reason is in the requirements file: a plain install would pull a different torch build and override the pinned torch==2.7.1+cu126.
git clone https://github.com/bytedance/Bernini.git bernini && cd bernini
pip install -r requirements.txt
pip install --no-deps git+https://github.com/ByteDance-Seed/VeOmni.git@v0.1.11The README is explicit that Open-VeOmni (Apache-2.0, Python 3.11) is required and that all inference paths import it, including single-GPU. Skipping it is not an option even on one card. The environment is pinned tightly: Python 3.11.2, CUDA toolkit 12.6, torch==2.7.1+cu126, diffusers==0.35.2, accelerate==0.34.2, transformers==4.57.3. The reference machine is an H100 with CUDA 12.6. For training, the README recommends uv instead:
uv sync
uv sync --extra all
uv pip install --no-build-isolation "flash-attn==2.8.3"uv sync creates and uses .venv, which you activate with source .venv/bin/activate or bypass with uv run. The pyproject.toml routes torch and torchvision to the correct CUDA index automatically, which is why the training path avoids the manual index flag that requirements.txt needs. Note the discrepancy between the two files: requirements.txt lists flash-attn==2.8.3 as a plain requirement, while pyproject.toml deliberately leaves it out because it needs torch present at build time and --no-build-isolation. If you follow the uv path, install flash-attn yourself.
A first run: pick a guide, download weights, choose an entry point
The repository root gives you the entry points directly: infer_single_gpu.py, infer_multi_gpu.py and gradio_demo.py, alongside bernini/, configs/, tasks/, tools/ and scripts/. The README does not put a full inference command in the top-level file; it points to docs/bernini.md for the full pipeline and docs/bernini_r.md for the renderer-only model, and says to follow the chosen guide for weight download, inference commands and ready-to-run scripts. That is where the actual flags live, and you should read the matching guide before running anything. The optional extras are worth knowing before your first run. The demo extra installs gradio==6.15.0 to power gradio_demo.py, the in-browser UI. The pe extra installs openai>=1.0 to power --use_pe, which rewrites the prompt through an OpenAI-compatible endpoint; the requirements file calls it strongly recommended for best generation quality and notes the v1 SDK API (from openai import OpenAI) is required. If you want the browser UI, install the demo extra rather than hand-rolling a Gradio version:
pip install "gradio==6.15.0"
python gradio_demo.pyThe README does not document what the demo binds to by default, so check the script before exposing it beyond localhost. For multi-GPU sequence parallel (Ulysses), the pyproject.toml says you need Open-VeOmni and lists veomni==0.1.11 as the extra; single-GPU inference does not need that extra, though it still needs the --no-deps Open-VeOmni install described above.
Where Bernini will fail you: CUDA pinning, attention backends and the 1.3B ceiling
The tight version pinning is the first real limitation. torch==2.7.1+cu126 and CUDA 12.6 are not suggestions; the requirements file resolves torch from the cu126 wheel index. If your driver or cluster image is built around a different CUDA line, you are rebuilding the environment, not adjusting a flag. The second is attention. A Hopper GPU (H100/H800/H200) is recommended so FlashAttention-3 can be used; other CUDA GPUs fall back to FlashAttention-2 or PyTorch SDPA. FlashAttention-3 is the awkward one: the README states flash_attn_interface is not on PyPI and must be built from the flash-attention repository's hopper/ directory at tag v2.8.3, and that requires CUDA 12.3 or newer, PyTorch 2.4 or newer, and SM 90a hardware. FlashAttention-2 is the easier path and covers general CUDA GPUs including A100/A800. Third, there is no CPU path documented anywhere in the README; the requirements are CUDA GPU, full stop. Fourth, the 1.3B model is a genuine ceiling, not a smaller version of the same thing: the release note says it lags on complex tasks such as human generation. If your edits involve people, plan for 14B. Finally, the top-level README does not document rollback, checkpoint conversion, or what happens when the planner and renderer disagree, and the repository has no releases listed, so there is no versioned artifact to pin your own builds against.
Alternatives: Wan, and what the renderer-only model gives up
The most direct comparison is the model Bernini-R is built on. The README says Bernini-R is fine-tuned from the Wan diffusion renderer, and the 1.3B variant specifically from Wan2.1-1.3B. The difference in approach is the planner. Wan-style pipelines render from a prompt or conditioning signal directly; Bernini inserts an MLLM stage that decomposes the instruction and plans semantic changes before the renderer runs. That extra stage is what the README credits for stronger instruction following on complex edits, and it is also what you remove by choosing Bernini-R. So the honest framing is not "Bernini versus Wan" but "how much instruction complexity do you have". For style transfer, subtitle or watermark removal, and local editing, the README says the 1.3B renderer-only model is close to the 14B variant, which makes the full planner pipeline hard to justify on those tasks. For multi-step edits where the instruction has to be interpreted before anything is rendered, the planner is the point of the project. A second alternative is diffusers itself: Bernini ships on diffusers==0.35.2 and the checkpoints are published in Diffusers format, so a team already running Wan through diffusers has a shorter migration than a team starting from a bespoke inference stack.
Licence, maintenance and the cost of staying on this stack
The repository is Apache-2.0, and the README notes that Open-VeOmni is also Apache-2.0. The model weights live on Hugging Face under separate repositories (ByteDance/Bernini-Diffusers, ByteDance/Bernini-Diffusers-v2, ByteDance/Bernini-R-Diffusers, ByteDance/Bernini-R-1.3B-Diffusers, ByteDance/Bernini-R), and the README does not state a licence for the weights themselves. That gap matters if you plan to redistribute or ship a product, and it is worth resolving before you build on the checkpoints rather than after. This is not legal advice; the point is that the repository licence and the weight licence are separate questions the README leaves open. On maintenance, the last push was on 2026-08-13, and the repository is not archived. The news entries show a steady cadence through mid-2026: the paper in May, renderer weights in June, the full pipeline in June, and the Bernini-R training code in July. No releases were retrieved, so upgrades are git-based: you track main and re-resolve the pinned stack yourself. The upgrade cost is dominated by the pins, not by API churn. Moving to a newer torch means re-checking diffusers==0.35.2, accelerate==0.34.2 and transformers==4.57.3 together, and rebuilding flash-attn with --no-build-isolation because it needs torch present at build time. Budget for that as a recurring task, not a one-off.
Editorial conclusion
Adopt Bernini if you have a Hopper-class GPU, can hold Python 3.11.2 and CUDA 12.6, and need instruction-driven video editing rather than plain text-to-video. Skip it if you are on CPU, on Python 3.12, or unwilling to install Open-VeOmni with --no-deps, since the README states all inference paths import it. Before committing, verify that pip install -r requirements.txt resolves torch==2.7.1+cu126 against the cu126 wheel index, and decide between Bernini (docs/bernini.md) and Bernini-R 1.3B (docs/bernini_r.md) on your own footage, because the README itself notes the 1.3B variant lags on complex tasks such as human generation.
Frequently asked questions
What is Bernini AI from ByteDance?
It is a unified framework for video generation and editing that combines an MLLM-based semantic planner with a DiT-based renderer, released under Apache-2.0 with checkpoints on Hugging Face.
What is Bernini-R?
Bernini-R is the renderer-only model fine-tuned from the Wan diffusion renderer, offered at 14B and 1.3B. The README says the 1.3B variant performs close to the 14B variant on simple tasks such as style transfer, subtitle or watermark removal, and local editing, but lags on more complex tasks such as human generation.
What is Bernini?
In this repository, Bernini is the full pipeline: an MLLM-based semantic planner plus a DiT-based renderer, with checkpoints at ByteDance/Bernini-Diffusers and ByteDance/Bernini-Diffusers-v2. It shares the t2i, i2i, t2v, v2v, rv2v and r2v task interface with Bernini-R.
Community notes