Open-source project
OpenBMB/VisRAG avatar
OpenBMB/VisRAG

VisRAG's setup.py still belongs to a different project

Parsing-free RAG supported by VLMs

991 stars79 forksPythonApache-2.0

At a glance

What is it?
A parsing-free retrieval pipeline for visually rich documents, now in its second generation, with two model families, two script trees and two requirement files in one repository. The packaging metadata names another project's package, a different author, an MIT classifier and a Python floor two versions below the documented environment.
Who is it for?
The research is the substance here and the two generations are both worth reading, but this is a research repository rather than a maintained library, and the packaging should be treated as an inconvenience rather than a contract. Three things to know before you build on it.
Can I use it commercially?
Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 13 days ago.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • OpenBMB/VisRAG991 ★ · +10
Star history
Repository2026-09-112026-11-15
OpenBMB/VisRAG981991

10 Stars in the last 65 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

The setup block changes into a directory the clone never creates

There are two documented environments, one per generation of the model, and the first one cannot work as printed. It clones the repository, which by default creates a directory named after the repository, creates a conda environment with a given name and Python version, activates it, and then changes into a directory with a different name. Nothing in those steps creates it, so the command fails on a fresh clone. The second block, for the original pipeline, does the same sequence and changes into the directory the clone actually made, so it works. The same first block also writes a version constraint of 3.10 while the second writes 3.10.8, and each then installs from its own requirements file. Two generations, two environments, two requirement files, and a path that only resolves for one of them.

The package metadata belongs to another research project

The setup file at the root does not describe this project. It names the distribution after a different retrieval package, gives it version 0.0.1, credits a different author with a different email, and describes itself as a package for research on information retrieval. The only content it takes from this repository is the readme, which it reads as the long description. The licence field is a trove classifier for MIT, while the repository itself is licensed under Apache 2.0, and the Python floor is declared as 3.7 or newer while both documented environments pin 3.10. So the one command the page gives for installing this project in editable mode produces a package with another project's identity. Its dependency list is also thin: three packages, none of which is the deep learning stack.

The retriever learned mostly from synthetic queries

The retriever was trained on 362,110 query and document pairs, and the composition is stated precisely. Thirty-four percent comes from the training splits of openly available academic datasets. The other sixty-six percent is synthetic: pages taken from web-crawled PDF documents, paired with pseudo-queries written by a proprietary model. So two thirds of the supervision for the component that actually gets deployed as an index is generated text rather than a real query, and the page gives no count of how the two halves are weighted inside the loss or any ablation showing what happens without the synthetic half. The training and test data are published on a model hub, so the split is checkable, and the generative model behind the pseudo-queries is named rather than left vague.

A training script invoked with thirteen bare arguments

The retriever training command is one line with thirteen positional values and no flags:

bash
bash visrag_scripts/train_retriever/train.sh 2048 16 8 0.02 1 true false config/deepspeed.json 1e-5 false wmean causal 1 true 2 false <model_dir> <repo_name_or_path>

A reader can guess that the first three are sizes and that a deepspeed config and a learning rate are in there, and nothing more. There is no flag naming, no argument list, and the only documentation offered is a separate reference page. Compare that with the second generation, whose two training stages are each a single named script with no arguments, and the difference in usability is stark. Both stages also depend on external training frameworks cloned separately, and the first stage requires you to edit the model path inside the shell script before it will run.

The dependency stack is pinned to 2023 releases

The requirements file pins exact versions for the whole stack: a deep learning framework and its vision counterpart at 2.1.2 and 0.16.2, the transformers library at 4.40.2, numpy at 1.26.4, an image library at 10.1.0, a protobuf runtime at 4.25.0, plus a training accelerator, a distributed training library, an evaluation toolkit, a video decoding library and a PDF library. Those are all releases from the middle of 2023. The second generation's model card is dated October 2025 and the first generation's paper October 2024, so the code being served is two years newer than the versions it is pinned to. That is a coherent choice for a reproducibility artifact, and an awkward one for anyone who needs a current version of any of those packages for their own reasons.

Two generations share one repository and one branch

The top level carries two script trees named for the two generations, two requirements files, an examples directory holding training data, a configuration directory, an assets directory, and a source directory. The news list runs from October 2024 to July 2026 and mixes both: model releases for the first generation in October 2024, a pipeline release in November 2024, the second generation's paper, code and model in October 2025, benchmarks in December 2025, and a conference acceptance in July 2026. Nothing in the page says which parts are deprecated, and the default branch is a year-neutral name holding both. One small inconsistency in the news list itself: the newest entry is written in a different date format from all the others.

A forked copy of a vision library sits at the repository root

One of the top level entries is a directory holding an enhanced copy of a widely used vision library, installed with its own editable install step in the original pipeline's setup block. The page says what was changed: it adds support for gradient checkpointing, which the authors use during training to reduce memory use. That is a small, well-motivated change, and vendoring it this way is a common way to keep a training recipe reproducible. The cost is that anyone who wants the same memory saving in their own codebase has to take the fork, and anyone auditing this repository has three installable Python trees in it rather than one, including the setup file that names a different project.

Editorial conclusion

The research is the substance here and the two generations are both worth reading, but this is a research repository rather than a maintained library, and the packaging should be treated as an inconvenience rather than a contract. Three things to know before you build on it. The editable install publishes a package under another project's name, so anything you depend on from it carries that name and its metadata, including a Python floor and a licence classifier that do not match this repository. The pinned dependency stack is several releases behind, so expect to unpick versions if you need a current torch. And the retriever, which is the component you would actually reuse, was trained on a set that is two thirds synthetic pseudo-queries generated by a proprietary model, which is the first thing to establish before you build an index on top of it.

Frequently asked questions

What does VisRAG do differently from text-based RAG?

Instead of parsing a document into text and embedding the text, it embeds the document directly as an image using a vision-language model, then retrieves those embeddings to help a generator answer. The page argues this maximises retention of the original information and removes the loss that parsing introduces.

What is the difference between VisRAG and EVisRAG?

The original pipeline uses a retrieval model built on a small vision-language model to embed pages as images, with a swappable generator. The second generation is an end-to-end model that observes retrieved images to collect per-image evidence and then reasons over those cues for multi-image questions, trained with a reward-scoped reinforcement method, and released at 7B and 3B on top of the Qwen2.5 vision-language models.

How is the VisRAG retriever trained, and on what data?

On 362,110 query-document pairs. Thirty-four percent of the pairs come from training splits of openly available academic datasets, and sixty-six percent are synthetic, built from pages of web-crawled PDF documents paired with pseudo-queries written by a proprietary model. The training and test data are published on a model hub.

What do I install to set up VisRAG?

Clone the repository, create and activate a conda environment, and pip install from a requirements file, with a second editable install of a vendored copy of a vision library in the original pipeline's instructions. The two documented environments name different directories and different Python versions, and the second generation's block changes into a directory name the clone does not create.

Can I use a different generator model with VisRAG?

Yes. The page says the paper uses two versions of one small vision-language model plus a proprietary model as generators, and then says plainly that you can use any vision-language model you like. The retriever is not swappable in the same way, since it is a specific composition of a vision encoder and a small language model.

Official sources

  1. Issues
  2. License: Apache-2.0
  3. OpenBMB/VisRAG on GitHub
  4. README
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/openbmb-visrag.svg)](https://hysenlabs.com/projects/openbmb-visrag)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/openbmb-visrag.svg)](https://hysenlabs.com/projects/openbmb-visrag)

Was this useful?