WeMM-Embedding: Tencent's Multimodal Embedding Family for Text, Image, Video and Visual Documents
WeMM-Embedding is a family of universal multimodal embedding models by the WeChat Vision Team at Tencent, supporting multimodal understanding and retrieval.
At a glance
- What is it?
- WeMM-Embedding is a family of universal multimodal embedding models from the WeChat Vision team, with 2B, 4B and 9B checkpoints and Matryoshka dimensions. It is worth adopting when you need one vector space for images, video and visual documents, and the documentation is clear about where it stops.
- Who is it for?
- Adopt WeMM-Embedding if your retrieval corpus mixes images, video and visual documents and you want one embedding space instead of two pipelines; the 2B checkpoint is the sensible starting point, and MRL lets you cut the vector to 256 dimensions and keep most of the quality. Do not adopt it if your corpus contains audio, since the README states audio is not currently supported and the MMEB-v3 audio column is zero for every WeMM-Embedding row.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The retrieval problem WeMM-Embedding is built for
Most retrieval stacks are still two stacks. A text encoder handles queries and documents, a separate vision encoder handles images, and anything that mixes the two, such as a screenshot with a caption or a slide deck with charts, falls between them. WeMM-Embedding targets that gap directly. The README describes it as a family of universal multimodal embedding models that provides unified representations for text, images, videos, visual documents and interleaved multimodal inputs. The audience is teams building search or recommendation over heterogeneous content: a media archive, a product catalogue with photos and spec sheets, a documentation site with diagrams, or an agent that needs to retrieve a video clip from a natural-language request. The models are published in three sizes, 2B, 4B and 9B, which maps onto the usual trade-off between a single-GPU service and a quality ceiling. The claim of unified representation is the whole point: one index, one distance metric, one query path, no score fusion between modalities. Whether that claim holds for your data is a separate question, and the evaluation tables are the only evidence the repository offers.
How the embeddings are produced and what the Matryoshka dimensions do
The mechanism is stated plainly in the README: embeddings are taken from the last-layer hidden state at a dedicated <embedding> token position, then L2-normalized. There is no pooling head to configure and no separate projection to train. That single-token design is what lets the same checkpoint consume text, an image, a video or an interleaved mixture without modality-specific branches in your code. The second mechanism is Matryoshka representation learning. The full dimension differs per model (2048 for 2B, 2560 for 4B, 4096 for 9B), and each model advertises a supported set of truncation points, for example 64, 128, 256, 512, 1024, 2048 for the 2B checkpoint. Truncation is not a magic property of the vector; the README tells you to truncate and normalize again:
embedding = torch.nn.functional.normalize(embedding[..., :d], dim=-1)The payoff is quantified for one case: on MMEB-v2, the README states the 2B model at 256 dimensions retains 98.7% of its full-dimensional image and video performance. That is the number to plan a vector index around, because 256 dimensions is a fraction of the storage and memory bandwidth of 2048. Note what is not claimed: no equivalent retention figure is given for the 4B or 9B checkpoints, and none is given for the visual-document or text tasks. If you pick a non-2B model and truncate hard, you are extrapolating.
Installing WeMM-Embedding and running a first inference
The repository ships no package on PyPI; installation is from the repository itself. The README gives a single install line, and requirements.txt pins the rest of the stack, including transformers==5.2.0, sentence-transformers==5.7.0, qwen-vl-utils[decord]==0.0.14 and accelerate>=1.1.0.
pip install -r requirements.txtThe version pins matter more than usual here. The README recommends transformers==5.2.0 for inference and reproducibility and warns that newer versions may differ in preprocessing behavior, so an unpinned environment is a reproducibility risk rather than a convenience. Weights are not in the repository; they live on Hugging Face under tencent/WeMM-Embedding-2B, tencent/WeMM-Embedding-4B and tencent/WeMM-Embedding-9B, and the examples accept either a local directory or a model id.
The quickest first run is the Transformers example. It produces independent text, image and video embeddings, and the same script accepts a --dimension flag for the Matryoshka truncation:
python examples/transformers_inference.py \
--model /path/to/WeMM-Embedding-2B \
--image /path/to/image.jpg \
--video /path/to/video.mp4 \
--dimension 2048Omit --dimension and you get the full embedding dimension. If you would rather stay inside an existing Sentence Transformers pipeline, examples/sentence_transformers_inference.py takes the same arguments, and the README notes that SentenceTransformer loads the model directly, so a Hugging Face model id such as tencent/WeMM-Embedding-2B works in place of a local path. Text, image and video all go through SentenceTransformer.encode(), which is the part that makes migration from a text-only index cheap. For serving, the README lists tested versions of vLLM 0.27.0 and SGLang 0.5.9, with one-command wrappers in scripts/serve_vllm.sh and scripts/serve_sglang.sh. The SGLang path is not a plain launch: it requires running scripts/patch_sglang_video.py first, which is a signal that video handling needed a workaround in that serving stack.
What the benchmarks cover, and the audio gap
The evaluation tables are unusually specific, and they are also where the limits show. On MMEB-v2, across 78 datasets, the 2B checkpoint reports an average of 77.9 against 73.2 for Qwen3-VL-Embedding 2B and 74.8 for a closed-source leaderboard entry the README marks with a dagger, noting it has no public weights or endpoint. The 9B checkpoint reports 80.6. On MMEB-v3, which spans 190 tasks including 53 text tasks, 47 agent tasks, 11 audio tasks and MCMR, the 9B checkpoint reports 59.3 overall. The README states that unsupported tasks are assigned a score of zero, and the audio column is 0.0 for every WeMM-Embedding row while competitors such as E5-Omni and LCO-Embedding-Omni post 43.0 and 43.2. The README confirms this directly: audio input is not currently supported. That is not a rounding error in a leaderboard, it is a scope boundary. If your corpus includes podcasts, call recordings or any audio track, this family cannot embed them, and a mixed audio-visual index would need a second model whose vectors do not live in the same space. There is also a correction note: PeerQA and DeepPlanning scores and affected aggregates were updated on 2026-09-16 using task-wide global retrieval, referencing issue #9. Numbers in this family have moved, so pin the report revision you cite.
Serving WeMM-Embedding with vLLM or SGLang
For production, the README documents two serving paths and names the versions it tested. The vLLM path uses the pooling runner plus the model's own chat template:
MODEL_PATH=/path/to/WeMM-Embedding-2B
vllm serve "$MODEL_PATH" \
--runner pooling \
--chat-template "$MODEL_PATH/embedding_chat_template.jinja"The chat template file ships with the model weights, not with this repository, so a local path is the safe choice here. The SGLang path is the one to scrutinize:
MODEL_PATH=/path/to/WeMM-Embedding-2B
python scripts/patch_sglang_video.py
python -m sglang.launch_server \
--model-path "$MODEL_PATH" \
--is-embedding \
--enable-precise-embedding-interpolationThe patch script runs against your SGLang installation before the server starts, and the flag enable-precise-embedding-interpolation is not something a stock embedding deployment would set. Neither the README nor the repository notes explain what the patch changes or whether it survives an SGLang upgrade, so treat the tested version 0.5.9 as the version you must hold. vLLM 0.27.0 carries the same pinning implication. This is the least documented part of the project and the part most likely to break on a routine dependency bump.
The evaluation harness and what it costs to reproduce
mmeb_v3_eval/ contains the MMEB-v3 evaluation code used to produce the reported numbers. The README describes it as the official TIGER-AI-Lab/VLM2Vec pipeline with a minimal diff: multi-node multi-GPU inference via torchrun --nnodes=N, a wemm_embedding backbone implementing the project's preprocessing and batched inference, dataset instructions aligned with the released model, and 64-frame video sampling. The two commands are data download and evaluation:
cd mmeb_v3_eval
DATA_ROOT=/path/to/MMEB-V3 bash scripts/download_data.sh
MODEL_PATH=/path/to/WeMM-Embedding-2B DATA_BASEDIR=/path/to/MMEB-V3 \
OUTPUT_DIR=exps/wemm_embedding bash scripts/run_eval.shTwo things follow. First, reproducing the published table is a multi-node GPU job, not a laptop task, which is worth knowing before you promise a stakeholder an independent verification. Second, the 64-frame video sampling rate is a preprocessing choice baked into the harness; if your own pipeline samples video differently, your retrieval quality will not match the reported video numbers, and that gap has nothing to do with the model. The README points to mmeb_v3_eval/README.md for the full command set, and that file is where the node count and data layout are actually specified.
Alternatives and when to pick something else
The comparison the README itself invites is Qwen3-VL-Embedding, which is the closest published model in both tables and is also available at 2B and 8B. The difference is not a single score. On MMEB-v3, WeMM-Embedding 9B reports 59.3 overall with 50.1 on agent tasks and 49.3 on MCMR, while Qwen3-VL-Embedding 8B reports 53.5 overall with 38.4 on agent tasks and 38.0 on MCMR. If your workload is interleaved multimodal retrieval, the gap is meaningful. If your workload is audio, the comparison inverts entirely: Qwen3-VL-Embedding also scores 0.0 on audio, so neither is the answer, and E5-Omni or LCO-Embedding-Omni is where you would look instead, accepting that their visual-document numbers are lower. The other axis is deployment shape. WeMM-Embedding wants transformers==5.2.0, a chat template shipped with the weights, and a patch script for SGLang; a text-only stack such as a plain sentence-transformer model has none of that surface area. If your corpus is text and nothing else, this family is more machinery than the task requires, and the MMEB-v3 text column of 48.8 for the 9B checkpoint is not a reason to switch on its own.
Licence, maintenance and upgrade cost
There is a discrepancy to resolve before adoption. The README badge links to LICENSE and reads License Apache 2.0, but the repository metadata reports NOASSERTION, which means an automated classifier could not match the file to a known licence. Open the LICENSE file and read it; if the terms of the weights on Hugging Face differ from the repository code, that matters for redistribution. Nothing here is legal advice, and the two sources disagreeing is exactly the case where you read the text rather than the badge. On maintenance, the last push to the default branch was on 2026-09-16, and the repository is not archived. That date is also the date the README says PeerQA and DeepPlanning aggregates were updated, so the project is moving, but the absence of any retrieved releases means there is no tagged version to pin against. You are tracking main. The practical upgrade cost sits in three places: the transformers==5.2.0 pin, the SGLang patch script whose behaviour is undocumented, and the fact that preprocessing changes between library versions are exactly what the README warns about. A dependency bump is a re-validation of your retrieval quality, not a routine merge.
Editorial conclusion
Adopt WeMM-Embedding if your retrieval corpus mixes images, video and visual documents and you want one embedding space instead of two pipelines; the 2B checkpoint is the sensible starting point, and MRL lets you cut the vector to 256 dimensions and keep most of the quality. Do not adopt it if your corpus contains audio, since the README states audio is not currently supported and the MMEB-v3 audio column is zero for every WeMM-Embedding row. Before committing, verify the licence text in the LICENSE file, since the repository reports NOASSERTION while the README badge says Apache 2.0, and confirm that transformers==5.2.0 is the version your serving stack can pin.
Frequently asked questions
What does embedding mean in the context of WeMM-Embedding?
An embedding is the fixed-length vector the model returns for an input; WeMM-Embedding takes it from the last-layer hidden state at a dedicated <embedding> token position and L2-normalizes it. The same procedure applies whether the input is text, an image, a video, a visual document or an interleaved mixture.
Can I use an LLM as an embedding model, as WeMM-Embedding does?
WeMM-Embedding is itself built on a multimodal backbone and exposes embeddings rather than generated text, so the pattern is viable. What makes it usable as an embedding model is the dedicated <embedding> token and the L2 normalization step described in the README, not the language modelling head.
Is GPT an embedding model, and how does WeMM-Embedding differ?
The repository does not describe GPT at all, so no comparison can be made from this material. What the README does state is that WeMM-Embedding is a family of universal multimodal embedding models covering text, images, videos, visual documents and interleaved inputs, with Matryoshka dimensions and no audio support.
What is an example of an embedding model like WeMM-Embedding?
The README's own comparison tables list VLM2Vec, GME, Qwen3-VL-Embedding, Omni-Embed-Nemotron, E5-Omni and LCO-Embedding-Omni as other embedding models evaluated on the same benchmarks. WeMM-Embedding is published in 2B, 4B and 9B sizes on Hugging Face.
Community notes