VBench: How to Use the Video Generation Benchmark
[CVPR2024 Highlight] VBench - We Evaluate Video Generation
At a glance
- What is it?
- VBench is a Python evaluation suite that scores text-to-video models across sixteen dimensions rather than one. It assumes CUDA, a specific PyTorch build and a pile of pretrained weights, so the install is the first real test.
- Who is it for?
- Adopt VBench if you are choosing between video generation models and want per-dimension numbers instead of a single score, and if you have a CUDA machine and the patience to fetch the pretrained weights. Do not adopt it if you have no GPU, or if you want to evaluate image-to-video, long video or trustworthiness today: those live under separate beta entry points.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 28 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What VBench measures that a single score hides
Video generation quality is not one number. A model can produce sharp frames that ignore the prompt, or follow the prompt while melting objects between frames. VBench splits that into a hierarchical set of evaluation dimensions, each with its own prompt set and its own automatic scoring method, and the README describes the result as aligned with human preference annotation. The audience is anyone comparing video generation models: research groups, model teams deciding what to train next, and people maintaining a leaderboard. The repository is not a model. It is the measuring instrument, plus the prompt suite and the sampled videos that make a comparison reproducible. The README also states that the repository now holds the whole VBench series in one place, so the same checkout covers the original benchmark, VBench++ and the VBench-2.0 beta directories.
How the dimension, prompt and method suites fit together
The architecture is a three-part pipeline. A dimension suite decomposes quality into named dimensions. A prompt suite supplies test cases per dimension and per content category. An evaluation method suite scores the generated videos for each dimension. The repository layout mirrors that split: prompts/ holds the prompt data, vbench/ holds the core evaluation code, pretrained/ holds the weights the methods load, and dimension_to_folder.json maps a dimension name to the folder that implements it. Entry points sit at the top level: evaluate.py for the main run, evaluate_i2v.py for image-to-video, and evaluate_trustworthy.py for the trustworthiness track, with matching shell scripts evaluate.sh and the vbench2_beta_i2v/, vbench2_beta_long/ and vbench2_beta_trustworthiness/ directories holding the newer work. That mapping file matters in practice: when a dimension fails to load, the folder it points at is the first place to look.
Installing VBench and running a first evaluation
setup.py performs a CUDA check before anything else. It imports torch, requires torch.cuda.is_available() to be true, and rejects any CUDA version outside 11.6, 11.7, 11.8 and 12.1. If the import fails it prints the exact commands to run, including the cu118 and cu121 index URLs. Install PyTorch first, from the index that matches your driver, then the package itself.
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install vbenchThe first command installs a CUDA 11.8 build of PyTorch. The second installs the vbench package from PyPI, whose latest published version is 0.1.5. Running the install without a usable CUDA build will not fail silently: the check raises.
Dependencies are pinned loosely in requirements.txt, with two exceptions worth noting: numpy is capped below 2.0.0 and transformers is pinned to exactly 4.33.2. The detectron2 line is commented out, so anything depending on it is not installed by default.
Evaluation runs through evaluate.py. The README points to evaluate.sh for the invocation pattern, and the repository ships scripts/ alongside it for the per-dimension runs.
bash evaluate.shThe README does not document a rollback path or a resume flag for an interrupted run, so plan for the full pass rather than assuming you can restart midway.
The CUDA gate and the pretrained weights are the real cost
The CUDA restriction is the sharpest limitation. A machine with an AMD GPU, an Apple Silicon laptop, or a CUDA 12.4 driver stack will not get through setup.py, and the failure is a raised exception rather than a warning. That rules out the most convenient development machines for anyone who just wants to sanity-check a model. The second cost is the pretrained/ directory. Several evaluation methods load their own weights, and the repository also links a separate dataset download for the sampled videos. Neither is small, and the README does not state a total size, so budget disk and download time before starting. This is also the wrong tool for a quick aesthetic opinion. If you want to know whether one clip looks better than another, watching them is faster than installing a benchmark. VBench earns its cost when you need the same measurement applied across many models and many prompts.
VBench against a general video quality metric
The obvious alternative is a single perceptual metric applied directly to generated frames, the kind of score that ships inside pyiqa, which VBench itself lists as a dependency. The difference is what gets measured. A general metric answers whether the pixels look plausible. VBench asks whether the specific dimension under test holds, using prompts designed for that dimension and a method chosen for it, then aggregates into a dimension-level score. The trade-off runs the other way too. A general metric needs no prompt suite, no dimension mapping and no per-dimension weights, so it runs on a laptop in seconds. VBench needs a CUDA machine, a large download and a full evaluation pass. If your question is narrow, the lighter metric answers it. If your question is which model to ship, the dimension breakdown is the thing a single score cannot give you.
Licence, releases and what upgrading costs
VBench is Apache-2.0, which permits commercial use and modification, but the licence covers the code, not the pretrained weights or the sampled video dataset that the evaluation methods load. Those come from separate download links and may carry their own terms, so check them before shipping anything derived from a score. On releases, the published versions are v0.1.1 and v0.1.2 from June 2024, v0.1.4 from September 2024, and 0.1.5 as the version string in setup.py. The last push to the repository was on 2026-08-21. Upgrading is not a drop-in affair: the transformers pin at 4.33.2 and the numpy cap below 2.0.0 mean a VBench environment will conflict with a project that tracks current releases of either library. Expect a separate virtual environment rather than a shared one, and expect the VBench-2.0 beta directories to move faster than the stable vbench/ package.
Editorial conclusion
Adopt VBench if you are choosing between video generation models and want per-dimension numbers instead of a single score, and if you have a CUDA machine and the patience to fetch the pretrained weights. Do not adopt it if you have no GPU, or if you want to evaluate image-to-video, long video or trustworthiness today: those live under separate beta entry points. Before committing, check that your PyTorch reports CUDA 11.6 through 12.1, since setup.py refuses anything else, and confirm which of the sixteen dimensions your prompt files actually cover.
Frequently asked questions
How do I use VBench?
Install PyTorch with a supported CUDA build, then install the vbench package from PyPI, and run the evaluation through evaluate.py using the pattern in evaluate.sh. The README points to the Usage section and the scripts/ directory for the per-dimension invocations.
Which CUDA versions does VBench support?
setup.py accepts CUDA 11.6, 11.7, 11.8 and 12.1 only, and it raises an error if torch.cuda.is_available() is false. For anything else it prints the PyTorch install commands for the cu118 or cu121 index URLs.
Does VBench need a GPU to run?
Yes. The setup script requires CUDA to be available before the package installs, so a CPU-only machine cannot complete the install. The README does not describe a CPU fallback.
Can VBench evaluate image-to-video and long video models?
The repository includes evaluate_i2v.py and the vbench2_beta_i2v/, vbench2_beta_long/ and vbench2_beta_trustworthiness/ directories for those tracks. The README describes them as part of the VBench++ and VBench-2.0 work, so treat them as separate entry points from the main evaluate.py run.
Community notes