Vision is All You Need: a V-RAG demo that embeds PDF pages instead of chunks

The chunking problem this demo removes

Retrieval augmented generation over PDFs usually starts with a parsing step: extract text, decide where one chunk ends and the next begins, keep a page number and bounding box as metadata, then hope the answer was not split across a boundary. The V-RAG architecture in this repository skips that. The README states that the approach uses a vision language model to embed pages of PDF files as vectors directly, without the chunking process. A page is the unit of retrieval. That choice matters most for documents where layout carries meaning: tables, forms, scanned pages, anything where the reading order is not a single column of text. If your corpus is plain prose in a clean single-column layout, page-level embedding throws away precision you already had, and you should say so before adopting it.

Ingestion path: pypdfium, ColPali, Qdrant

The README lays out eight steps. Pages are converted to images with pypdfium, and the README notes that in theory the images can be anything, but the current demo uses PDF files because the underlying model was trained on them. Those images go through a VLM to produce embeddings, and ColPali is the model used in this demo. The vectors land in Qdrant. On the query side, the same VLM embeds the query, Qdrant returns the nearest page embeddings, and the query plus the images of the best matches are sent to a model that can read images. GPT-4o or GPT-4o-mini generate the answer. Two details are worth flagging. First, the retrieval step compares a text query embedding against image page embeddings, which is the whole point of ColPali-style late interaction, and it is why the query must go through the same model family. Second, the generation step re-sends page images, not extracted text, so every answer costs vision tokens. The README does not state how many pages are retrieved per query or how images are sized before they reach OpenAI, and those two numbers dominate both latency and cost.

Running it: Modal, a dotenv file, and two API endpoints

The setup assumes a Hugging Face account and an OpenAI API key. The README instructs you to log in with transformers-cli login and to put OPENAI_API_KEY and HF_TOKEN in a dotenv file. Then: install Python 3.11 or higher, pip install modal, modal setup, modal serve main.py. Note the backslash in the README's modal serve command; on a Unix shell the correct form is modal serve main.py. Once Modal prints a URL, append /docs to reach the FastAPI schema. Indexing is done through POST /collections: click Try it out, upload a PDF, execute. The README says indexing takes time depending on PDF size and the GPU, and that the demo uses an A10G. Retrieval is POST /search, which sends page images and the query to OpenAI and returns the response. The frontend lives in frontend/: set VITE_BACKEND_URL in .env.development, run npm install and npm run dev, and it serves on http://localhost:5173. Deployment is npm run build followed by modal deploy main.py, with VITE_BACKEND_URL set again in .env.production.

The in-memory vector store is the real constraint

The README says indexing a PDF puts it into an in-memory vector database. That single sentence defines the demo's boundary. Qdrant is named as the vector database, but as configured here the collection lives with the running Modal container, so a redeploy or a cold start can leave you with an empty index and a frontend that answers nothing. For a demo that is fine. For anything with a document set you care about, you would need to point the same code at a persistent Qdrant instance, and the README does not describe that configuration. The second constraint is hardware. The demo runs on an A10G, and ColPali inference plus image embedding is the expensive half of the pipeline; the README gives no throughput figure, so you cannot estimate indexing time for a large corpus from this material alone. Third, the pipeline depends on two external services at once, Modal for compute and OpenAI for generation, so a failure in either produces a broken answer rather than a degraded one.

Where a text pipeline still wins

The obvious alternative is a conventional text RAG stack: parse the PDF, split into chunks, embed the chunks with a text embedding model, store them in a vector database, and generate from retrieved text. The difference in approach is what gets embedded. A text pipeline embeds strings and can return a passage that starts mid-sentence, which is often exactly what a language model wants. This demo embeds whole page images and returns whole pages, which preserves tables and figures but spends more context per retrieved item and gives the generator no clean way to cite a passage. Text pipelines are also cheaper to index, since a text embedding model is far smaller than a vision language model, and they can run without a GPU. The trade-off is not about which is smarter. It is about whether your documents lose meaning when they are converted to plain text. If they do not, the text route is simpler and cheaper.

What the repository does not tell you

Several things a reader would want are absent from the README. There are no benchmark numbers, no accuracy comparison against a text baseline, and no statement of how many pages are passed to GPT-4o per query. There is no description of how the Qdrant collection is created, what distance metric is used, or how page images are encoded before storage. The README points to a blog post for background, which is where the design rationale presumably lives, but the repository text itself stops at the eight-step description. The licence is MIT, which is permissive and places few obligations on reuse, but the model weights are a separate matter: ColPali is loaded from Hugging Face and the README's login step implies gated or authenticated access, so check the model's own licence before shipping anything built on it. None of this is a criticism of a demo. It is a reason to read main.py rather than the README before you decide.

Maintenance surface and who this is for

The maintenance cost sits in three places: the ColPali model and its Hugging Face access, the Modal deployment configuration including the GPU type, and the OpenAI model choice, which the README leaves open between GPT-4o and GPT-4o-mini. Any of those changing means editing the backend and redeploying with modal deploy main.py. The frontend is a separate build with its own VITE_BACKEND_URL per environment, so a backend URL change is a two-file edit and a rebuild. This is a reasonable amount of surface for a demo and a real amount of work for a team that wants to keep it running. The people who should read this repository are engineers who have already tried chunking on layout-heavy PDFs and want to see the alternative implemented end to end: FastAPI endpoints, a Modal deployment, a React frontend, and a Qdrant lookup in one place. The people who should not start here are teams with clean text corpora, or teams that need persistence and cost predictability on day one.

Editorial conclusion

Adopt this if you want a working reference for page-level retrieval and you are comfortable running Modal and paying for OpenAI vision calls. Do not adopt it as an application: the README describes an in-memory vector database, so indexed PDFs do not survive a restart, and the demo is tied to a single A10G GPU on Modal. Before committing, verify that the embedding model named in main.py is the one you intend to use, and check whether the ColPali weights are gated on Hugging Face, because the login step in the README exists for a reason.

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. Softlandia-Ltd/vision-is-all-you-need on GitHub