VisRAG and EVisRAG: parsing-free retrieval over document images

The parsing step VisRAG removes, and what that costs

Text-based RAG starts with a parser: PDF goes in, plain text comes out, chunks get embedded. The README states the motivation plainly, that VisRAG maximizes the retention and utilization of the data information in the original documents, eliminating the information loss introduced during the parsing process. That loss is real and easy to name: a two-column academic PDF read in the wrong order, a table whose row labels detach from their cells, a chart whose meaning lives entirely in the axis labels. None of those survive a naive text extraction, and no amount of chunking strategy repairs them afterward.

VisRAG's answer is to skip the parser. Pages become images, images are embedded by a vision-language model, and the same kind of model reads the retrieved images to produce an answer. The audience is anyone whose documents are visual artifacts first and text second: scanned contracts, financial statements, product manuals, slide decks, scientific figures. If your source material is already clean Markdown or well-formed HTML, the parsing loss VisRAG targets is close to zero, and you are paying a large compute bill to solve a problem you do not have.

VisRAG-Ret, VisRAG-Gen, and the split between them

The pipeline has two named components. VisRAG-Ret is the document embedding model, built on MiniCPM-V 2.0, which the README describes as integrating SigLIP as the vision encoder and MiniCPM-2B as the language model. That is the retrieval half: a page image goes in, a dense vector comes out, and a vector index does the rest.

VisRAG-Gen is the generation half, and here the repository is deliberately loose. The README says the paper uses MiniCPM-V 2.0, MiniCPM-V 2.6, and GPT-4o as generators, then adds that you can use any VLMs you like. Read that as a design statement rather than a missing feature: retrieval and generation are decoupled, so you can pair a locally hosted retriever with a hosted generator, or keep both on-premises. The trade-off is that the repository cannot give you one canonical end-to-end configuration. Whatever quality you get depends on which generator you bolt on, and the README does not enumerate per-generator behaviour.

The concrete artifact for the retriever is the VisRAG-Ret checkpoint on Hugging Face, and the file2img conversion scripts live under visrag_scripts/file2img. Those scripts are the front door of the whole pipeline, since nothing downstream works until your PDFs are page images.

EVisRAG: evidence first, then reasoning over multiple images

EVisRAG, labelled VisRAG 2.0 in the README, targets a specific weakness of plain visual RAG: a question whose answer is spread across several retrieved pages. The README describes the mechanism as first linguistically observing retrieved images to collect per-image evidence, then reasoning over those cues to answer. So the model produces intermediate evidence text per image before it commits to a final answer, rather than fusing everything in one pass.

Training uses what the README calls Reward-Scoped Group Relative Policy Optimization (RS-GRPO), which binds fine-grained rewards to scope-specific tokens to jointly optimize visual perception and reasoning abilities. The released checkpoints are EVisRAG-7B and EVisRAG-3B, both built on Qwen2.5-VL-Instruct. The two-stage recipe is explicit: Stage 1 is SFT and is based on LLaMA-Factory, Stage 2 is RS-GRPO and is based on Easy-R, with the README's training section truncated mid-name.

Two things are worth flagging. First, the 3B and 7B sizes are the only ones released, so there is no small model for a constrained GPU. Second, the evidence stage is generated text, not a pointer into the source image. Nothing in the README describes span-level or bounding-box citations back to the page region that produced a given evidence sentence.

Installing VisRAG and EVisRAG: two environments, two Python pins

The README gives separate setup paths, and they do not share an environment. For EVisRAG:

git clone https://github.com/OpenBMB/VisRAG.git conda create --name EVisRAG python==3.10 conda activate EVisRAG cd EVisRAG pip install -r EVisRAG_requirements.txt

For VisRAG the pin is different and there is an extra editable install:

conda create --name VisRAG python==3.10.8 conda activate VisRAG conda install nvidia/label/cuda-11.8.0::cuda-toolkit cd VisRAG pip install -r requirements.txt pip install -e . cd timm_modified pip install -e .

That last step matters. The README explains that timm_modified is an enhanced version of the timm library that supports gradient checkpointing, used during training to reduce memory usage. In other words, the training path depends on a patched fork of a widely used library, installed as an editable package. If you already have a project that imports timm, this is the first place an environment conflict will show up.

Note also that the CUDA toolkit is pinned to 11.8.0 in the VisRAG setup, while EVisRAG's requirements file is not described in the README at all. Training EVisRAG starts by cloning LLaMA-Factory and running evisrag_scripts/full_sft.sh, which means the EVisRAG environment inherits LLaMA-Factory's own dependency surface on top of whatever EVisRAG_requirements.txt pins.

Where this approach breaks down

The clearest failure mode is cost per document. A text chunk is a few hundred tokens. A page image at a resolution a VLM can actually read is far more expensive, both in embedding compute and in the context window of the generator, which now has to look at several images at once. EVisRAG's multi-image reasoning stage multiplies that again, because the model generates per-image evidence before answering. At 7B parameters this is not a laptop workload.

Retrieval granularity is the second constraint. VisRAG-Ret embeds documents, and the README's own framing is page-level: the pipeline is described as supporting visual understanding across multiple PDF documents, and the conversion scripts turn files into images. If your queries need sentence-level precision inside a 40-page report, a page-level embedding is a coarse instrument, and the generation stage has to compensate by reading more content than a text pipeline would.

The third issue is auditability. Regulated workflows often need to point at the exact clause that produced an answer. VisRAG can show you which page image was retrieved, which is genuinely useful, but the README documents no mechanism for finer provenance than that. If your compliance requirement is character-level citation, this is the wrong tool, and a text pipeline with offset tracking will serve you better.

Finally, the README is a research README. There are no releases listed, no versioning scheme, no upgrade notes. The last push date is recent, but the interface you depend on is a set of scripts and Hugging Face checkpoints rather than a versioned library.

What to compare it against, and how the approaches differ

The obvious alternative is a conventional text RAG stack: a PDF parser such as PyMuPDF or a layout-aware extractor, a text embedding model, and a text LLM. The difference is not quality in the abstract, it is where the information loss happens. Text RAG loses structure at parse time and then operates on cheap tokens. VisRAG loses nothing at parse time because there is no parse, and instead pays in image tokens and GPU memory. Choose based on which cost you can absorb.

A second comparison is a hybrid: parse the document, but keep the page image alongside the extracted text and route visually complex pages to a VLM. The README does not describe such a mode, so this would be your own engineering on top of file2img. It is worth considering precisely because VisRAG's retriever and generator are decoupled, which makes partial adoption technically feasible even though the repository does not ship it as a supported path.

A third option is the paper's own generator list. Because VisRAG-Gen is explicitly swappable, the honest comparison for many teams is not VisRAG versus text RAG but VisRAG-Ret plus a hosted VLM versus VisRAG-Ret plus a local one. The retriever is the part you cannot easily substitute, since it is the component trained for image-page embeddings.

Maintenance, licence, and what the repository actually commits to

The project is Apache-2.0, which is permissive and includes a patent grant. That applies to the code in the repository. The model weights are distributed separately on Hugging Face, and the README does not state a licence for the checkpoints themselves; VisRAG-Ret is built on MiniCPM-V 2.0 and EVisRAG on Qwen2.5-VL, so the upstream model terms are worth checking before commercial deployment. This is not legal advice, just a pointer to where the questions are.

Maintenance cost is dominated by that dependency chain rather than by the VisRAG code itself. The VisRAG environment pins CUDA 11.8.0 and installs an editable fork of timm. The EVisRAG environment pins Python 3.10 and builds on LLaMA-Factory for SFT and Easy-R for RS-GRPO. Each of those is a moving target with its own release cadence. Upgrading any one of them means re-validating the others.

On the positive side, the README notes that both EVisRAG and VisRAG can be reproduced within UltraRAG v2, which gives you a second, more structured entry point if you would rather not maintain the raw scripts. Evaluation benchmarks are published in the VisRAG collection on Hugging Face, so you can measure the pipeline against the paper's own test sets before trusting it on your corpus.

Editorial conclusion

Adopt VisRAG when your corpus is layout-heavy (scanned forms, multi-column PDFs, charts, tables) and a text parser is visibly destroying structure. Do not adopt it if you only have clean digital text, a CPU-only serving budget, or a need for token-level citations, since retrieval happens over page images and the README shows no span-level provenance. Before committing, verify three things from the repository itself: that VisRAG-Ret's embedding dimension and index format fit your vector store, that your GPU can hold the VLM you pick for generation, and that the EVisRAG_requirements.txt stack resolves under Python 3.10, because the two pipelines install into separate environments with different Python pins.

Official sources

1. Issues
2. License: Apache-2.0
3. OpenBMB/VisRAG on GitHub
4. README