Vision-Language-Models-Overview: a dated survey of VLM architectures, benchmarks and applications

What the Vision-Language-Models-Overview repository actually contains

The repository is a survey, not a toolkit. Its README describes it as a "Frontend Collection and survey of vision-language model papers, and models GitHub repository", and the file listing confirms that shape: README.md, README_zh.md, assets/, index.html, build_site.py, a __pycache__/ directory, and a progressive reports/ folder. There is no package manifest, no setup.py, no pyproject.toml at the top level. The primary language is HTML, which tells you the deliverable is a rendered page rather than an importable module.

The audience is therefore people who need orientation rather than inference. If you are choosing a backbone for a multimodal product, writing a literature review, or trying to place a new model release into a lineage, the repository gives you a taxonomy and a set of links. If you want to run a model, this is the wrong artifact entirely; you would follow the links out to the model repositories instead.

The four architectural eras and where the split happens

The core contribution is a periodisation. The README states that VLM design has gone through "four distinct architectural eras in just six years", and that the third era splits into two parallel branches.

Era 1 keeps frozen vision and language towers, aligned either contrastively (CLIP, which the README notes has no generative decoder) or through a learnable bridge such as a Q-Former into a frozen language model, as in BLIP-2 and Flamingo. Output is text only. Era 2 makes a pretrained LLM the trunk and treats vision as a bolt-on adapter, projecting visual tokens into the LLM's vocabulary space through an MLP or Resampler; LLaVA, Qwen2.5-VL and GPT-4V sit here. Era 3a drops the bridge entirely: image, video, audio and text share one tokenizer and embedding space and flow through a single early-fused transformer, but generation remains autoregressive text. Era 3b keeps that fused trunk and adds decoder heads (image and video DiT, VAE, audio codec) so the model can emit images, video or speech. Era 4 adds an action token stream, persistent state and a policy head, closing the observation to action to next-observation loop.

The useful part is the axis the README uses to separate 3a from 3b. It is not parameter count or training data; it is output modality. The README puts the question plainly: the choice is essentially "how much do you want the model to generate, and to act?" That framing is more actionable than a flat list of model names, because it tells you which branch to look at once you know whether your product needs to produce pixels or only describe them.

VLM Trends and the dated progressive reports

Two mechanisms keep the survey from going stale, and they do different jobs.

The first is VLM Trends, a live dashboard at zli12321.github.io/VLM_Survey. The README draws the boundary explicitly: this README records what exists, while VLM Trends tracks what changed today. According to the README, the dashboard pulls new model releases, papers, benchmarks and datasets daily from arXiv, Hugging Face, GitHub and Semantic Scholar, scores them on a published rubric, and groups them by topic and model family. It also charts how each research direction moves over time. That rubric is the part worth inspecting before you trust the ranking, since a scoring scheme is only as good as its published criteria.

The second is the progressive reports folder. These are dated mini-surveys covering new VLMs, benchmarks and post-training methods that have not yet been folded into the main tables. The README lists ten of them, the latest dated 2026-09-13, described as adding 50 new entries. The cadence matters for how you read the main tables: anything in a table has survived at least one consolidation pass, while anything only in a report is fresh and unreconciled. If you are scanning for recent work, start with the reports and treat the tables as the settled layer.

Installing and building the survey site locally

There is no package to install. The repository is a static site plus a build script, so the only thing you can set up locally is the rendered page.

Clone the repository and move into it. The default branch is main.

git clone https://github.com/zli12321/Vision-Language-Models-Overview.git
cd Vision-Language-Models-Overview

The top-level listing includes build_site.py and index.html. You can open index.html directly in a browser to read the survey without running anything. If you want to regenerate the page from the markdown sources, run the build script; the README does not document its arguments or its dependencies, so check the file itself before running it.

python build_site.py

If you prefer to read the source rather than the rendered output, the README is the entry point and README_zh.md is the Simplified Chinese version. The progressive reports are markdown files under the progressive reports/ directory, named by date, so the 2026-09-13 report is the most recent one to read first.

What the survey does not give you

The first limitation is that nothing here is runnable. There are no model weights, no inference code, no evaluation harness. The README says the repository compiles papers, models and GitHub repositories, and the benchmark section is described as "Evaluate VLM benchmarks and corresponding link to the works". A link to a benchmark is not the benchmark. If you need to reproduce a number, you leave this repository immediately and go to the source paper or the benchmark's own code.

The second is that the survey is prose-driven and hand-curated, with the dashboard as the only automated layer. The README does not document how entries are added to the main tables, who reviews them, or how a model is retired when a newer one supersedes it. The progressive reports are dated, which is good, but the tables they feed into are not visibly versioned, so you cannot tell from a table row when it was last touched.

The third is licensing. No licence file appears in the top-level listing, and the README does not state one. For a survey that aggregates links and quotes from other people's work, the terms under which you can reuse the compiled tables and diagrams are simply not stated. Treat the arXiv paper and the individual linked projects as the authoritative sources for anything you intend to redistribute.

How this differs from an awesome list or a model hub

The obvious alternative is an awesome-style link list, of which there are many for multimodal models. The difference in approach is that this repository imposes a taxonomy before it imposes a list. An awesome list typically appends new entries chronologically or by category, and the reader has to infer the relationships. Here the README argues for a specific periodisation and places each model inside it, which means a new release arrives with a position already implied: is it a bridged two-tower model, an LLM-backbone adapter, a natively fused input model, a unified I/O model, or a world-action model? That is a stronger editorial stance, and it is also a liability, because if you disagree with the era boundaries the whole organisation becomes harder to use.

The second alternative is a model hub or a leaderboard site. Those give you sortable numbers across a fixed evaluation set. This repository gives you context around the numbers and points at the benchmarks rather than hosting results. If your question is "which model wins on benchmark X today", a leaderboard answers it faster. If your question is "why did the field move from connectors to early fusion", this survey is the better starting point.

Maintenance cadence and what it costs to keep up

The last push to the repository was on 2026-09-14, and the repository is not archived, so the content is current as of that date. The README states that updates are continuous and that new models and benchmarks will keep being added, and the dated progressive reports give that claim a concrete form: the latest report is 2026-09-13, one day before the last push.

That cadence is also the cost. A survey that tracks daily releases from arXiv, Hugging Face, GitHub and Semantic Scholar requires someone to keep scoring and consolidating, and the README does not describe a contribution process for outside reviewers. If you build a workflow on top of these tables, plan to re-read the progressive reports rather than assuming the main tables are frozen. On the licence side, no licence file appears in the top-level listing, so you cannot assume a permissive grant for the compiled tables, the SVG diagrams under assets/, or the site itself; the README also does not state terms for the linked arXiv paper, which carries its own.

Editorial conclusion

Use this repository if you need a structured, dated map of vision-language model research: the four-era taxonomy, the benchmark tables and the progressive reports give you entry points that a plain arXiv feed does not. Do not use it if you need runnable code, a maintained Python package, or a leaderboard whose numbers you can cite without checking the original papers, because the repository is a collection of links and prose, and no licence file appears in the top-level listing. Before relying on it, open the progressive report dated 2026-09-13 and confirm that the models you care about are still labelled with the era you expect, then follow the arXiv link for any number you intend to quote.

Frequently asked questions

Official sources

1. Issues
2. README
3. zli12321/Vision-Language-Models-Overview on GitHub