VLM2Vec and MMEB-V3: a benchmark harness for omni-modality embeddings

The problem MMEB-V3 was built to expose

Retrieval evaluation has a modality problem. A text-only embedding benchmark tells you nothing about whether the same model can return the right video clip when the query asks for it, and an image-text benchmark says nothing about audio or GUI elements. MMEB-V3 states its purpose directly: it aims to diagnose whether current embedding models can reliably follow modality-specific instructions, such as retrieving an image, video, audio clip, document, tool, GUI element, or memory item under explicit task constraints. That framing matters. The benchmark is not measuring whether a model can embed things. It is measuring whether the model respects the instruction that says which kind of thing to retrieve. MMEB-V3 extends MMEB-V2 by adding 111 new tasks for a total of 190, and it groups them into three new categories: audio tasks (audio classification, cross-modal audio retrieval, audio temporal grounding), text retrieval (instruction-following, reasoning, long-context, multi-condition, and general text retrieval), and agent tasks (tool retrieval, GUI control, agent memory retrieval). The audience is narrow but real: researchers and engineers comparing omni-modality embedding models, and teams that need a defensible number before swapping an embedding model inside a retrieval pipeline.

OmniSET and the instruction-conditioned retrieval claim

The diagnostic component is OmniSET, short for Omni-modality Semantic Equivalence Tuples. The README describes it as grouping semantically equivalent instances across text, image, video, and audio, and says it is designed for controlled analysis of modality effects and instruction-conditioned cross-modal retrieval behavior. The mechanism implied by that description is a tuple structure: the same underlying meaning expressed in several modalities, so a model can be queried with one modality and asked to return another. That is a harder test than ordinary cross-modal retrieval, because a model can score well on image-to-text pairs by memorizing caption style while failing to align an audio clip with the matching paragraph. The repository does not spell out the tuple schema in the material available here; the only concrete artifact is the materialized path, MMEB-V3/omniset/omniset.jsonl alongside catalog.jsonl, with val2014/, videos/, audios/ and frames_omni/ holding the referenced media. Anyone planning to build on OmniSET rather than just run the aggregate scores should read the JSONL files themselves before assuming a field layout.

Getting the data in place: archives in, task directories out

The dataset is distributed as compressed assets plus lightweight metadata, and the setup step is the part of this repository most likely to consume your afternoon. Download first:

export MMEB_V3_ROOT=/path/to/MMEB-V3 hf download VLM2Vec/MMEB-V3 --repo-type dataset --local-dir $MMEB_V3_ROOT

Then materialize the evaluation-ready layout:

python experiments/public/data/dataset_setup_v3.py --root $MMEB_V3_ROOT python experiments/public/data/dataset_setup_v3.py --root $MMEB_V3_ROOT --check-only

The script is described as idempotent: it skips directories marked with .done, extracts tar and zip files, and checks the expected final layout. The naming convention is the trap. Downloaded archives live in _tasks directories such as image_tasks/ and video_tasks/, while the evaluation code reads -tasks directories such as image-tasks/ and video-tasks/. The setup script performs that translation. It also means you must not upload both forms: the README states plainly that if the corresponding archives are uploaded, you should not also upload already materialized directories like video-tasks/frames/video_cls/ or visdoc-tasks/data/. One documented escape hatch exists: if image-query is missing from your local download, pass --image-query-source /path/to/image-query. OmniSET can also be extracted by hand with tar -xzf $MMEB_V3_ROOT/omniset.tar.gz -C $MMEB_V3_ROOT, which is useful when only that component failed.

Running an evaluation with eval.py

The README gives one worked invocation, for image tasks against Omni-Embed-Nemotron:

CUDA_VISIBLE_DEVICES=0 python eval.py \ --pooling mean \ --normalize true \ --per_device_eval_batch_size 8 \ --dataloader_num_workers 1 \ --model_backbone nvomniembed \ --model_name /path/to/omni-embed-nemotron-3b \ --dataset_config experiments/public/eval/image.yaml \ --encode_output_path exps/vlm2vec/omni-embed-nemotron-3b/image \ --data_basedir $MMEB_V3_ROOT

Several things are worth reading off that command. The dataset is selected by a YAML config (experiments/public/eval/image.yaml), not by a task name, so other categories presumably have sibling YAML files, though the README excerpt does not list them. Pooling and normalization are explicit flags rather than model defaults, which means scores are comparable only when those two settings match. The model is identified by a backbone key plus a path, and the README states that the same entrypoint supports adapted omni-modality baselines such as e5_omni and lco_omni by changing --model_backbone, --model_name, pooling and other arguments. Encoding output goes to --encode_output_path, so embeddings are written to disk rather than scored in memory, which is what makes re-scoring with different metrics feasible without re-encoding.

Where this repository stops being the right tool

Two boundaries are visible in the material. First, VLM2Vec is a benchmark and data interface, not a serving stack. There is no mention of an inference server, an index build step, or a query API. If your goal is a production retriever, the useful artifact here is the score, not the code path. Second, the setup assumes you can hold the full corpus locally. The expected layout includes image-tasks, audio-tasks, video-tasks with separate data/ and frames/ trees, visdoc-tasks with both data/ and images/, plus text, tool, memory, gui and omniset directories. Video frames are extracted into per-task subdirectories (video_cls, video_ret, video_mret, video_qa), which multiplies the storage footprint beyond the source archives. The README gives no size figures, so the honest position is that you should measure the download before committing to it. There is also a versioning hazard: the repository now spans three benchmark generations (V1, V2, V3) with separate Hugging Face datasets, and the README instructs that all V1, V2 and V3 evaluation data should live under one MMEB-V3 root so scripts run with a single --data_basedir. Mixing generations across roots is the kind of mistake the --check-only pass is meant to catch.

How it differs from MTEB-style text benchmarks

The obvious comparison is MTEB, the widely used text embedding benchmark. The difference is not scale, it is the unit of evaluation. MTEB tasks are overwhelmingly text-in, text-out, with the modality fixed by construction. MMEB-V3 makes the modality part of the instruction, so a single model is asked to retrieve an image, then a video, then an audio clip, then a GUI element, under task constraints stated in the prompt. That changes what a good score means. A model that tops an image-text benchmark by learning caption distribution can still fail MMEB-V3's agent tasks, because tool retrieval and agent memory retrieval have no caption distribution to exploit. The trade-off runs the other way too: MTEB's task set is stable and its harness is widely reproduced, while MMEB-V3 is on its third generation with the dataset split across VLM2Vec/MMEB-V3, TIGER-Lab/MMEB-V2 and TIGER-Lab/MMEB-eval. If you need a number that other teams already have, MTEB is the safer currency. If you need to know whether one model can serve several retrieval surfaces, MMEB-V3 is asking the question that matters.

Licence, maintenance and what upgrading costs

The repository is Apache-2.0, which permits commercial use and modification, but the licence covers the code, not the datasets or the model weights it evaluates. Those are hosted separately (VLM2Vec/MMEB-V3, TIGER-Lab/MMEB-V2, TIGER-Lab/MMEB-eval, and the VLM2Vec model organization), and each carries its own terms that this repository does not restate. Check them individually before shipping anything derived from them; this is a factual pointer, not legal advice. On maintenance, the release history is thin and clustered: v1.0 in June 2025 and v2.0.1 later that same month, with the last push to main in August 2026. The README is written for MMEB-V3 while the releases stop at v2.0.1, so the tag scheme and the benchmark generation are not in step. The practical upgrade cost is the data layout, not the Python. Moving from V2 to V3 means another 111 tasks of archives, another extraction pass, and a re-run of dataset_setup_v3.py --check-only to confirm the merged root still matches the expected tree. Budget for that before you promise a comparison table.

Editorial conclusion

Adopt VLM2Vec if you are building or selecting a retrieval embedding model and need a single harness that scores text, image, video, audio, visual document, GUI and memory retrieval under one --data_basedir. Do not adopt it if you want a trained model you can drop into a product: this repository is the benchmark and data interface, and the models it evaluates live on Hugging Face under separate names such as omni-embed-nemotron-3b. Verify your disk budget and archive layout before anything else, since the setup script expects compressed assets under image_tasks, audio_tasks, video_tasks, visdoc_tasks, gui_tasks, memory_tasks, text_tasks, tool_tasks and omniset.tar.gz, and it will not reconstruct a materialized directory you deleted. Run dataset_setup_v3.py --check-only first and treat its output as the gate for the rest of your evaluation run.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. TIGER-AI-Lab/VLM2Vec on GitHub