t2v_metrics: VQAScore for Scoring Text-to-Image, Video and 3D Outputs
Evaluating text-to-image/video/3D models with VQAScore
At a glance
- What is it?
- VQAScore turns a vision-language model into a scoring function for generated media, and t2v_metrics is the Python package that wraps roughly two dozen such models behind one interface. The value is real, but the package is version-split and some model families are currently unavailable.
- Who is it for?
- Adopt t2v_metrics if you are comparing generative models on prompt adherence and want a single Python call instead of maintaining your own VLM scoring harness. Do not adopt it if you need to reproduce the original VQAScore or GenAI-Bench paper numbers, because that path requires the separate t2v-metrics==3.0 release, not the current version.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 102 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The scoring gap VQAScore fills between CLIPScore and human raters
CLIPScore and similar embedding-similarity metrics compare a prompt and a generated image in a shared vector space. That works for coarse alignment but degrades on compositional prompts, where the ordering of objects, attributes and spatial relations matters more than overall semantic overlap. VQAScore takes a different route: it asks an image-to-text model a yes/no question about whether the image matches the text, and uses the model's token probabilities as the score. The README describes this as evaluating text-to-visual generation with image-to-text generation, and cites the ECCV 2024 paper for the method.
The audience is narrow and specific. This is a tool for people who train or benchmark text-to-image, text-to-video and text-to-3D models and need an automatic number that correlates better with human judgement than CLIPScore does. The README states that VQAScore is highlighted in Google's Imagen3 report as a replacement for CLIPScore in automated evaluation, and that GenAI-Bench has been adopted by Google DeepMind, Bytedance Seed and NVIDIA. Those are adoption claims from the project itself, not independently verified here. If you are building a product that generates images and you want a quality gate, this is heavier machinery than you probably need.
How VQAScore turns a VLM into a score: the yes/no probability path
The mechanism is a forced-choice question. For a given image and text pair, the vision-language model is prompted to answer whether the text describes the image, and the score is derived from the probability the model assigns to the affirmative token. This is why the package cares so much about logprobs: without token-level probabilities from the model, there is no continuous score, only a discrete yes or no.
That design choice explains several constraints in the model table. Gemini VQAScore requires a Vertex AI project because, per the README, the standard Gemini Developer API key does not support logprobs and therefore cannot be used for scoring. It also explains why the open-source Qwen families are the ones that carry forward_with_trace, described as full token-level scoring transparency: if you control the model weights, you can inspect exactly which tokens contributed to the score.
The package spans image, video and audio inputs. Qwen3-Omni-30B is noted as the first audio-enabled model in the framework, and CameraBench integration brought camera-motion evaluation into the same interface. The model table marks PaliGemma as image-only, with video inputs unsupported, so the input modality is a property of the chosen backbone rather than of the package as a whole.
Installing t2v_metrics from source or pip, and the v3.0 fork
The README gives two installation paths. From source, it clones the repository, creates a conda environment on Python 3.10, installs ffmpeg from conda-forge, and runs pip install -e . inside the checkout. The ffmpeg step is not incidental: video inputs need decoding, and the README places it before the package install rather than leaving it to a transitive dependency.
The pip path is a single install, though the README excerpt cuts off before naming the exact command for the current version. What it does state clearly is the legacy path: pip install t2v-metrics==3.0 reproduces the original VQAScore and GenAI-Bench paper results, including CLIP-FlanT5, InstructBLIP and LLaVA-1.5. Full documentation for that branch lives in V_3.0_README.md.
This split is the single most important thing to understand before you start. v3.1 targets torch>=2.7.0 and transformers>=5.0.0 to support newer frontier models. The legacy models that the papers were built on are preserved only in v3.0. If you install the current release expecting CLIP-FlanT5, you will not find it there.
Model coverage in v3.1 and what the asterisks actually restrict
The v3.1 table lists GPT-4 (gpt-4o, gpt-4.1), Gemini 2.5 Flash and Pro, Gemma 3 at 4B, 12B and 27B, PaliGemma at three resolutions, Qwen2.5-VL from 3B to 72B, a long Qwen3-VL list including thinking variants and a 235B MoE, Qwen3.5 at 4B, 9B and 27B, and Qwen3-Omni in three configurations.
The footnotes carry more decision weight than the model names. Gemini needs a project_id and Vertex AI, which means cloud billing and project setup rather than a dropped-in API key. PaliGemma cannot take video. The star marking forward_with_trace appears on Qwen2.5-VL, Qwen3-VL, Qwen3.5 and Qwen3-Omni, so token-level transparency is a Qwen-family feature, not a package-wide one. If your evaluation pipeline depends on explaining why a particular generation scored low, that narrows your backbone choice considerably.
The README also notes that models using trust_remote_code, specifically InternVL and Molmo2, are temporarily unavailable because of breaking changes in transformers 5.x, with a return promised in a future release. Treat that as a live gap: those model families are named in the documentation but cannot be used today.
Where t2v_metrics is the wrong tool
The clearest failure mode is version drift. The package moved to a newer torch and transformers baseline to chase frontier models, and in doing so lost two trust_remote_code families and moved the paper-reproduction models into a separate release. Anyone whose results need to be comparable to published VQAScore or GenAI-Bench numbers is working with a frozen v3.0 install, not the version the README leads with. That is a maintenance cost paid in environment pinning.
A second limit is dependency on logprobs. Any model or API that will not expose token probabilities cannot produce a VQAScore, and the Gemini case shows how that plays out in practice: a standard API key is insufficient, and you need Vertex AI. If your organisation has standardized on a provider that does not surface logprobs, this method is simply unavailable to you.
Third, the scoring is only as good as the backbone. A 3B model answering a yes/no question about a complex compositional prompt will be less reliable than a 72B one, and the package does not hide that. Choosing a small model to save memory means accepting noisier scores, and there is no calibration step in the documented interface to correct for it. For a single image sanity check, a human reviewer is faster and more accurate than setting up a VLM scoring harness.
Against CLIPScore: what changes when the metric is a question
CLIPScore is a similarity computation. You embed the text and the image, take a cosine similarity, and you are done. It is cheap, deterministic, and runs on a single encoder. Its weakness is that it measures distributional overlap, not whether a specific claim about the image is true.
VQAScore replaces similarity with a question-answering probability. The model must decide whether the text describes the image, and the answer's probability becomes the score. This is more expensive per sample and it inherits every bias of the underlying VLM, but it can distinguish an image that contains a cat and a dog from one that contains a dog and a cat, which a global embedding often cannot. The README frames this as VQAScore being positioned as a stronger replacement for CLIPScore, and the Imagen3 report citation is the project's evidence for that claim.
The trade-off is operational. CLIPScore needs one model download and no logprob plumbing. VQAScore needs a VLM that exposes token probabilities, plus ffmpeg for video, plus in the Gemini case a cloud project. If your evaluation set is small and your prompts are simple, the extra machinery buys you little.
Licence, maintenance and upgrade exposure
The repository is Apache-2.0, which permits commercial use and modification with the usual attribution and notice requirements. That covers the package code. It does not cover the model weights you point it at: Gemma, Qwen, PaliGemma and the GPT and Gemini endpoints each carry their own terms, and several of those are not Apache-2.0. Running t2v_metrics against a hosted model also means your prompts and images leave your infrastructure. Check the terms of the specific backbone rather than assuming the package licence settles the question. This is not legal advice.
Maintenance cost is concentrated in the dependency floor. v3.1 requires torch>=2.7.0 and transformers>=5.0.0, which is a recent and moving target. The InternVL and Molmo2 breakage is the concrete consequence: a transformers major version change disabled two model families, and the README says they will return in a future release without committing to a date. If your environment is pinned to an older transformers for other reasons, v3.1 may not install cleanly alongside your existing stack, and v3.0 becomes the practical choice.
Upgrades between minor versions are not free either. Moving from v3.0 to v3.1 changes which models exist, which changes your scores, which means any prior evaluation numbers are not directly comparable. Budget for re-running baselines whenever you bump the version.
Editorial conclusion
Adopt t2v_metrics if you are comparing generative models on prompt adherence and want a single Python call instead of maintaining your own VLM scoring harness. Do not adopt it if you need to reproduce the original VQAScore or GenAI-Bench paper numbers, because that path requires the separate t2v-metrics==3.0 release, not the current version. Before committing, check which model family you intend to score: PaliGemma is image-only, Gemini needs a Vertex AI project_id rather than a standard API key, and InternVL and Molmo2 are unavailable in v3.1 because of transformers 5.x changes.
Community notes