# The vbench installer refuses to run without a GPU, and the package is ahead of its releases

> VBench is the evaluation suite for video generation models, published to PyPI and holding three benchmark generations in one repository. Installing it runs a CUDA check before setup, and the declared version sits one step above the newest published release.

**Vchitect/VBench** — [CVPR2024 Highlight] VBench - We Evaluate Video Generation

- Repository: https://github.com/Vchitect/VBench
- Website: https://vchitect.github.io/VBench-project/
- Stars: 1,805 · Forks: 135
- Language: Python
- License: Apache-2.0
- Published: 2026-09-16 · Updated: 2026-09-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/vchitect-vbench

## Installation runs a CUDA check before setup is called

`setup.py` does not go straight to `setup()`. It calls `check_torch_version()` first, and that function imports torch, requires `torch.cuda.is_available()`, and accepts exactly four CUDA versions:
```python
if cuda_version not in ["11.6", "11.7", "11.8", "12.1"]:
    raise RuntimeError(f"\033[91mUnsupported CUDA version: {cuda_version}. Please install PyTorch with 11.6<=CUDA<=12.1.\033[0m")
```
Anything else raises, and a bare `except` catches the failure, prints instructions and re-raises. Three things follow. A machine without a usable NVIDIA GPU cannot install the package at all, which rules out CPU-only evaluation hardware and most CI runners. A machine with a newer CUDA, which is where the ecosystem has gone, is rejected by name. And the instructions it prints do not cover the whole accepted range: the text offers commands for CUDA 11.8, for CUDA 12.1, and for CUDA 11 with PyTorch below 2.0 pinned to `torch==1.13.1` and `torchvision==0.14.1`, while 11.6 and 11.7 pass the check without a matching instruction.

## The package says 0.1.5 and the newest release is from 2024

The version in `setup.py` is `0.1.5`. The published GitHub releases are v0.1.1, v0.1.2 and v0.1.4, the newest of them from September 3, 2024, so the declared package version is one step ahead of anything tagged, which means the code on the branch is not the code in a release artifact. The release list has two smaller oddities. The v0.1.2 tag was published on June 4, 2024 and v0.1.1 on June 5, 2024, so the older version was released a day after the newer one. And the release titles are not uniform: v0.1.4 is titled v0.1.4 release, v0.1.2 is titled VBench v0.1.2, and v0.1.1 is titled v0.1.1 release. The default branch's most recent commit is dated August 21, 2026, so there has been commit activity well after the last release, and PyPI carries the package separately under the name vbench.

## Git-URL dependencies are filtered out before the install list is built

`install_requires` is not read straight from `requirements.txt`. `fetch_requirements()` opens the file and keeps only the lines that do not contain an `@` character, which is a filter aimed at direct references. Everything else is passed through verbatim, so the constraint mix you inherit is whatever that file says: `numpy<2.0.0` carries a ceiling, `transformers==4.33.2` is pinned exactly, `timm>=0.9,<=1.0.12` is a range, `fairscale>=0.4.4` is a floor, `Pillow` has no constraint at all, and `easydict` is listed twice. The filtering has a visible consequence. The detectron2 line in that file is a git URL and is commented out, and the filter would drop it even uncommented, so detectron2 is not installed with the package. A dimension that needs it will fail at runtime with a missing import rather than at install time, and the file gives no hint that this is the arrangement.

## Three benchmark generations live in one distribution

The top level carries four module trees and five entry scripts. `vbench/` is the original suite. The three VBench++ components sit in `vbench2_beta_i2v/`, `vbench2_beta_long/` and `vbench2_beta_trustworthiness/`, covering image-to-video, long video, and trustworthiness across fairness, bias and safety. A separate `VBench-2.0/` directory carries a third generation, which means two naming schemes for related work: a capitalised, hyphenated directory and three lowercase underscore directories with a beta marker in the name. The page says so directly, noting that those three modules belong to VBench++ rather than to VBench or VBench-2.0, and that they remain in this repository to maintain backward compatibility for people who already installed it. Running them means picking the right script: `evaluate.py`, `evaluate_i2v.py`, `evaluate_trustworthy.py`, `evaluate.sh` and `static_filter.py` all sit at the top level, so the benchmark you run is chosen by which file you invoke.

## Dimensions are wired to their code through one JSON file

The benchmark's structure is data before it is code. Quality is decomposed into a hierarchical set of evaluation dimensions, and for each dimension and each content category the authors wrote a prompt suite to act as test cases, sampled generated videos from a set of models, and designed a separate evaluation method suite per dimension for automatic scoring, then added human preference annotation for the same videos and used the agreement between the two as the justification for the scores. `dimension_to_folder.json` at the top level is the mapping that ties a dimension name to the code that scores it, which means adding a dimension is an edit to that file plus a folder, not a change to a registry in code. Around it sit the directories the run depends on: `prompts/`, `pretrained/`, `sampled_videos/`, `integrity_check/`, `submodules/` and `competitions/`. Weights to download, sample videos to score, and something called submodules are all part of getting a number out of this suite.

## Support is a mail address and the docs are three files

Two documentation habits are worth knowing before you file anything. The page carries an explicit instruction to contact the maintainer directly, at an address written in the anti-spam form with at and dot spelled out, if a question is not answered there. That is a request for email rather than an issue, which tells you what to expect back and where. The repository also keeps three readme files at the top: the main `README.md`, a `README-FAQ.md` and a `README-pypi.md`. The last one is not decoration, because `setup.py` reads it through `fetch_readme()` and hands it to the packaging call as the long description, so the page a user sees on the package index is that file and not the one on the repository. Anything you find in the main readme may not be what the installed package describes.

## Conclusion

Use VBench if you are reproducing or extending published numbers on a machine with an NVIDIA GPU, since the install path hard-requires CUDA and the dimension scores are only comparable against the same prompt suite and sampled videos the authors used. Before you rely on it, note three things. A CPU-only host cannot install the package at all, because the check runs before setup and re-raises. Anything installed from a git URL is dropped from the dependency list, so a dimension that needs detectron2 will fail later rather than at install time. And the declared version, 0.1.5, is ahead of the newest published release, v0.1.4 from September 3, 2024, so the branch's later work, whose last commit is dated August 21, 2026, is not in a tagged artifact.

## FAQ

### how to use vbench

Pick the entry script for the generation you want: `evaluate.py` for VBench, `evaluate_i2v.py` for the image-to-video component, `evaluate_trustworthy.py` for trustworthiness, with `evaluate.sh` and `static_filter.py` also at the top level. The prompt suite, sampled videos and evaluation methods live in `prompts/`, `sampled_videos/` and the per-dimension modules.

### What are the differences between VBench, VBench++ and VBench-2.0?

VBench++ extends VBench with VBench-I2V for image-to-video, VBench-Long for long videos, and VBench-Trustworthiness covering fairness, bias and safety, and those three live in `vbench2_beta_i2v/`, `vbench2_beta_long/` and `vbench2_beta_trustworthiness/`. A separate `VBench-2.0/` directory holds a third generation, and the page says the three VBench++ modules are not part of VBench or VBench-2.0 but stay in the repository for backward compatibility.

### Why does installing vbench fail on my machine?

`setup.py` calls `check_torch_version()` before packaging, and it requires CUDA to be available and rejects any version outside 11.6, 11.7, 11.8 and 12.1. A host without a usable NVIDIA GPU cannot install the package at all, and a newer CUDA is rejected by name.

### Which dependencies does vbench pin?

A mixed set: `numpy<2.0.0` has a ceiling, `transformers==4.33.2` is pinned exactly, `timm>=0.9,<=1.0.12` is a range, `fairscale>=0.4.4` is a floor, and `Pillow` carries no constraint. `fetch_requirements()` drops every line containing an `@`, so the commented detectron2 git line is not installed with the package.

### What is the current version of vbench?

`setup.py` declares version 0.1.5, while the newest published GitHub release is v0.1.4 from September 3, 2024, and the package is also distributed on PyPI as vbench. The default branch's last commit is dated August 21, 2026.

## Sources

- [License: Apache-2.0](https://github.com/Vchitect/VBench/blob/master/LICENSE)
- [Project website](https://vchitect.github.io/VBench-project/)
- [README](https://github.com/Vchitect/VBench/blob/master/README.md)
- [Releases](https://github.com/Vchitect/VBench/releases)
- [Vchitect/VBench on GitHub](https://github.com/Vchitect/VBench)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/vchitect-vbench