VLMEvalKit: One-Command Benchmarking for 220+ Vision-Language Models
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
At a glance
- What is it?
- VLMEvalKit is an open-source Python toolkit that standardizes evaluation of large vision-language models across 80+ benchmarks, supporting 220+ models with a single command. It trades some flexibility for consistency, and its generation-based approach with LLM answer extraction is its core design bet.
- Who is it for?
- Adopt VLMEvalKit if you need to compare many vision-language models on standard benchmarks without building your own evaluation pipeline, especially if you work with open models like InternVL or QwenVL. Do not use it if you need fine-grained control over prompt formats, custom metrics, or if your model generates responses longer than 32k tokens without enabling TSV output.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What It Solves and Who It Serves
Evaluating a large vision-language model (LVLM) is not a single command in most toolkits. You download a benchmark, write a prompt template, parse model outputs, and then match answers against ground truth. VLMEvalKit collapses that workflow into one command. The project supports over 220 models and 80+ benchmarks, which means a researcher can take a new model like InternVL3 or Qwen2.5-Omni and immediately run it against MMMU, Video-MME-v2, or a dozen other tests without writing glue code. The intended user is someone who needs comparative numbers across many models, not someone who wants to design a novel evaluation protocol. The README is explicit about this: it promises 'one-command evaluation' and removes the 'heavy workload of data preparation under multiple repositories.' This is a tool for benchmarking at scale, not for deep-diving into a single model's failure modes.
Generation-Based Evaluation and Answer Extraction
The core mechanism is generation-based evaluation. Every model, regardless of architecture, is asked to produce free-form text responses to benchmark prompts. That is a deliberate choice. Some toolkits use log-likelihood scoring or multiple-choice classification, but VLMEvalKit standardizes on generation. The trade-off is that raw model outputs are messy, so the toolkit applies two extraction strategies: exact matching and LLM-based answer extraction. Exact matching works when the model outputs a clean option letter like 'A' or 'B'. For everything else, the toolkit routes to an LLM-based extractor that reads the response and decides which option the model chose. The README notes that recent changes to the can_infer_option and can_infer_text functions have increasingly pushed more cases toward the LLM extractor, empirically improving MCQ performance. This means the evaluation result depends not only on the model being tested but also on the quality of the extraction LLM. If that extractor misreads a verbose answer, the score drops even when the model was correct. Users should treat leaderboard numbers as a joint measure of model capability and extraction robustness.
Thinking Mode and Long Responses: Two Recent Fixes
Two changes from September 2025 address real failure modes. First, models with a 'thinking mode' often wrap their reasoning in <think> tags. If the evaluator counts that reasoning as part of the final answer, it can distort results. VLMEvalKit now supports a split_thinking function that parses content inside those tags and stores it separately in a 'thinking' key. You enable this with the environment variable SPLIT_THINK=True. Second, prediction files were previously saved as .xlsx, which truncates any cell beyond 32,767 characters. Models that generate long responses, over 16k or 32k tokens, would silently lose data. The fix is to set PRED_FORMAT=tsv, which writes tab-separated values without that cell limit. These are not cosmetic updates. They change what data you get back from an evaluation run. If you test a reasoning-heavy model like QVQ-72B and do not enable SPLIT_THINK, your accuracy numbers may be wrong. If you test a long-context model and keep the default xlsx output, you may be scoring truncated answers. The README strongly recommends both flags for affected models, and that recommendation should be treated as mandatory.
Getting It Running: Commands and Configuration
The QuickStart guide is the entry point, but the README gives enough to understand the shape of usage. Installation is via pip or from source, with the package name vlmeval. After installation, you run evaluation by specifying a model and a benchmark. The exact command syntax lives in the docs, but the repository layout shows a config.py file where custom models are registered. To use distributed inference, you add the use_lmdeploy or use_vllm flag to your model configuration. LMDeploy supports InternVL, QwenVL, and LLaMA4 series; vLLM supports QwenVL and LLaMA4. This is a meaningful option because running a 72B model on a single GPU is slow. The environment variable VLMEVALKIT_USE_MODELSCOPE switches video benchmark downloads to ModelScope, which is useful if you are in a region where Hugging Face is slow or blocked. The configuration is Python-based, not a separate YAML file, so you edit config.py directly. That is convenient for developers but means a typo in Python syntax breaks the whole run.
Where It Can Be the Wrong Tool
VLMEvalKit is not a general-purpose evaluation framework. It is benchmark-specific. If you have a proprietary benchmark or a custom metric, you must write a new dataset adapter and possibly a new evaluator. The toolkit's strength is breadth, not depth. Another limitation is the reliance on external model downloads. To evaluate a model, you need its weights, which can be tens of gigabytes. The README does not describe an offline mode for models you already have. Also, the extraction logic is tuned for multiple-choice benchmarks. For open-ended generation tasks that require semantic similarity or human judgment, exact matching is meaningless and LLM extraction may be arbitrary. The README's own news about refining can_infer functions shows that the extraction heuristics are still evolving. If you need stable, reproducible results over time, know that the toolkit's scoring logic can change between releases, which can shift leaderboard numbers even for the same model. Finally, the project supports 80+ benchmarks, but not all benchmarks are equal. Some are recent additions like Video-MME-v2 from April 2026, which may have less community validation than older ones.
Alternatives and How They Differ
The most direct alternative is LMMS-Eval, a fork of lm-evaluation-harness that targets vision-language models. LMMS-Eval uses a similar generation-based approach but has a different model registration system and a smaller built-in model list. Another alternative is the official evaluation code that individual benchmark authors release, such as the MMMU repository's own scripts. Those scripts are often more precise for that single benchmark because they are written by the benchmark creators, but they do not generalize to other tests. The key difference is scope. VLMEvalKit aims to be a single entry point for dozens of benchmarks and hundreds of models, which means it sacrifices the fine-tuned prompt details that benchmark-specific code might have. If your goal is to reproduce a leaderboard number exactly, use the benchmark's official code. If your goal is to compare a new model against 50 existing ones, VLMEvalKit is the more practical route. The trade-off is that you accept the toolkit's extraction logic as a black box.
Maintenance, Licensing, and Upgrade Cost
The repository is under active development, with a release v0.3rc1 from June 2025 and a default branch that receives pushes into September 2026. The license is Apache-2.0, which permits commercial use and modification with attribution. That is a permissive license, so you can integrate it into internal tooling without open-sourcing your changes. The upgrade cost is real. The README lists major changes to extraction functions and output formats. If you have scripts that parse the .xlsx prediction files, switching to TSV output will break them. The addition of SPLIT_THINK changes the structure of output records, adding a 'thinking' key. Any downstream analysis that expects only a final answer field must be updated. The project also has a rapid benchmark adoption cycle. New benchmarks are added frequently, which is good for coverage but means the codebase changes often. A version you pin today may not produce identical results to the main branch six months later. For production use, pin a specific release and read the changelog before upgrading. The documentation is spread across a Feishu wiki and docs folders, which is less centralized than some projects, so finding the exact behavior of a flag may require digging.
Editorial conclusion
Adopt VLMEvalKit if you need to compare many vision-language models on standard benchmarks without building your own evaluation pipeline, especially if you work with open models like InternVL or QwenVL. Do not use it if you need fine-grained control over prompt formats, custom metrics, or if your model generates responses longer than 32k tokens without enabling TSV output. Before adopting, verify that your target model is in the supported list, check the latest release notes for breaking changes, and test the SPLIT_THINK=True flag if your model uses thinking tags, as the default extraction may otherwise misjudge answers.
Community notes