CoVT spends twenty visual tokens inside a VLM's reasoning chain, then throws them away

Twenty tokens, split eight-four-four-four across four vision experts

The framing is a complaint about representation. Rather than restricting a model's reasoning to a discrete language space with limited representational capacity, CoVT forms a visual thought chain so the model can reason in continuous visual space, using tokens that encode perceptual cues such as segmentation, depth, instance, and edge structure. The result is described as a chain of textual and visual thoughts that links semantic reasoning to perceptual grounding.

The budget is small and it is itemised. Within roughly twenty tokens, knowledge is distilled from four lightweight vision experts, each aligned on its own task during training. SAM takes eight tokens used as mask prompts. DepthAnything takes four tokens to reconstruct depth. PIDINet takes four to reconstruct edges. DINO takes four to match patch-level features. Eight plus four plus four plus four is the twenty, and the split is the design: the four experts contribute complementary properties, described as two-dimensional appearance, three-dimensional geometry, spatial layout, and edge structure.

Mechanically, the tokens are learned by reconstruction. A VLM equipped with CoVT autoregressively predicts the visual tokens in order to rebuild dense supervision signals, and the ones named are depth, segmentation, edges, and DINO features. So the tokens are not free parameters; they are a fixed-width bottleneck that has to carry enough information to satisfy four different perception heads.

The tokens are a training scaffold that disappears at inference

The most consequential line in the method description is the one about inference, and it is flagged as a note rather than stated as a feature. During inference, dense predictions are decoded only when interpretability is desired. Otherwise, reasoning occurs entirely in the latent visual space.

That means the twenty tokens are not a side channel you can read. The model's internal representation is continuous and there is nothing in the default path that turns it into a picture, which is why the paper describes reasoning as efficient in that mode. The interpretable path exists and is genuinely useful for a human looking at why the model answered what it answered: the tokens can be decoded into dense predictions that visualise the reasoning process, and the cues they encode are named as object localization, spatial structure, and scene semantics.

The framing claim worth noting is that all of this happens without relying on external tools. Many systems solve fine-grained perception by calling a separate segmentation or depth model mid-inference. CoVT folds that perception into the VLM's own reasoning, so the counting, depth ordering, and scene understanding the authors highlight are done by the same forward pass that produces the answer. That is the efficiency argument, and it is also the reason the token budget has to be so tight.

It is a LoRA finetune, not a new model family

The training setup is one sentence: the VLM is finetuned with LoRA and all projection layers are trainable. That single line tells you most of what you need to know about what CoVT is. There is no new base architecture, no from-scratch training run, and no family of checkpoints to host. There is an adapter, plus the projection layers that connect it to the backbone, trained on top of a model you already have.

The base models named are Qwen2.5-VL and LLaVA, which covers the two common lineages: a modern instruction-tuned vision-language model and the earlier open family that a lot of comparison work still uses. The results section frames its comparisons in two directions. One is against the baseline and against closed-source models, where the claim is consistent improvement across all the vision-centric benchmarks. The other is a specific head-to-head with Aurora on LLaVA-v1.5-13B.

That second comparison carries a methodological detail worth pausing on. A symbol in its table indicates results the authors reproduced themselves from the provided checkpoints, as distinct from figures reported by the other paper. Reproducing someone else's numbers on their released weights is a meaningful amount of work and it is the difference between a comparison and an assertion, so the fact that they marked which is which is a point in the project's favour.

Evaluation runs through a fork that also went upstream

The evaluation story has two halves and the project is explicit about both. For consistency and reproducibility, the authors use VLMEvalKit as the framework for evaluating models, and a forked copy of it sits in the repository as a top-level directory. The quick start is a document in the docs directory rather than a code sample in the readme.

Vendoring the harness is a deliberate choice. Benchmark harnesses change, and a paper's numbers are only reproducible if the harness that produced them still exists in the state it was used. Keeping a fork inside the repository means the tables can be regenerated without depending on whatever the upstream project does next.

The second half is a news item dated 2026-02-18 recording that the evaluation code is now supported in VLMEvalKit itself. So the relationship runs in both directions: a local fork for reproducibility of the paper's numbers, and an upstream contribution so that the wider community can evaluate the released weights without vendoring anything. Those are different goals and it is unusual to see a research repository do both.

The practical consequence for a user is that there are two ways in, and they are not equivalent. Following the local instruction gives you the configuration the authors used. Installing the upstream support gives you a maintained path, which may not match the paper exactly.

The reported gain runs from three to sixteen percent, and that range is the finding

The headline number is a range, not a value. Integrating CoVT into strong models such as Qwen2.5-VL and LLaVA is described as consistently improving performance by three to sixteen percent, evaluated across more than ten diverse perception benchmarks. Six are named in the visible text: CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench.

A range that wide is the interesting part, not a weakness in the claim. A method that added a fixed amount everywhere would suggest it is doing something generic. A method that varies by more than a factor of five across benchmarks suggests the gain is task-dependent, and the file offers the mechanism for that: each type of visual token contributes most effectively to the tasks that align with the perceptual information it encodes. Depth tokens should matter more where depth ordering is the difficulty, and mask tokens more where counting and localization are.

Two things are not in the visible text and both matter for planning. The per-benchmark breakdown is not written out in prose, so you cannot tell from the readme which end of the range belongs to which benchmark. And no independent reproduction is named, so the figures are the authors' own evaluation of their own method. That is normal for a paper repository and it is still the number to treat as a claim.

Everything shipped on a single day in November 2025

The news section is short and one date is crowded. On 2025-11-24 four things are announced together: the training data, released as a dataset on the model hub; the code; the evaluation and a Gradio demo; and the finetuned weights, as a collection on the same hub. One day for data, code, eval, demo, and weights is either very good project management or a deadline, and either way it means nobody has to wait for a second release to get started.

Two entries follow. On 2026-02-18 the evaluation code is recorded as supported in the upstream evaluation harness. On 2026-07-28 the work is recorded as selected for an oral presentation at ECCV 2026, which is the top category for a paper at that venue and explains why the repository has a project page and a polished readme with no code samples in it.

The distribution shape is worth noting. There are no GitHub releases, so the version story lives on the model hub as a collection of weight checkpoints and a dataset, while the code lives on GitHub at whatever the main branch says today. There is no tag to pin a paper reproduction to. The paper itself is on the arXiv under an identifier from the same November 2025 window as the code drop.

The author list runs to seven names across four institutions, with the corresponding author carrying two affiliations, one of them the group that appears first on the paper.

The file stops at the Gradio demo and promises four more sections

The table of contents promises six sections and the visible text delivers two of them. Overview and Quick Start are present. Model Zoo, the project's own to-do list, the licence section, and the contact section are named in the contents and then not readable, because the file ends partway through the sentence introducing the Gradio demo.

The to-do entry is the one that would have been most useful to see. A research repository that puts a to-do list in its own table of contents is telling you the work is not finished, and which items are open is exactly the information a person reproducing the results needs. Its absence here is a real gap rather than a stylistic one.

The repository tree is small and tells you what is actually shipped: a gitignore, a licence, the readme, the forked evaluation harness as a directory, an assets directory for images, a docs directory, a gradio directory for the demo, and a train directory for the training code. Notably there is no packaging file at the root, no configuration file, and no inference-only entry point, so there is no installable package described in the visible material. If you want to use the released weights, the model hub and the demo are the documented paths; if you want to retrain, the train directory is the thing to read.

Editorial conclusion

CoVT is worth reading if you build on vision-language models and keep hitting the ceiling that comes from reasoning about perception in words, because the design is specific and the arithmetic is small: twenty tokens, four experts, one LoRA adapter per base model, and the weights and training data are already published. It is a poor fit if you need a drop-in inference package, since the repository has no packaging file at its root and the training code is the thing that is actually there. Before you build on it, check three things yourself: that the reported three to sixteen percent range is broken down per benchmark for your task, because a range that wide across ten-plus benchmarks is not a single number you can plan around; whether you need the decoded dense predictions at inference, since by default the reasoning happens entirely in latent space and nothing human-readable is produced; and that the comparison against a competing method is the authors' own reproduction from published checkpoints rather than a reported figure, which the results section marks with a symbol.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. Wakals/CoVT on GitHub