colpali-cookbooks: Notebook Recipes for ColVision Retrieval, Fine-Tuning and Adapter Swapping
Recipes for learning, fine-tuning, and adapting ColPali to your multimodal RAG use cases. 👨🏻🍳
At a glance
- What is it?
- This repository is a set of Jupyter notebooks, not a library. It shows how to run ColPali and ColQwen2 inference, fine-tune with LoRA, generate similarity maps, and hold a whole RAG pipeline on one vision-language model. The value is in the recipes; the cost is that nothing here is packaged.
- Who is it for?
- Adopt colpali-cookbooks if you need to learn the ColVision family or prototype a visual document retrieval pipeline before committing to an engine: open the notebooks from the examples directory in Colab or clone the repository and run them in Jupyter.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 13 days ago.
- What is it written in?
- GitHub does not report a main language for this repository.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Problem the Cookbooks Solve, and for Whom
ColPali retrieves documents by their visual features. The README is explicit about the contrast: unlike systems that depend on text extraction and OCR, ColPali treats each page as an image, using Paligemma-3B to capture text along with layout, tables and charts, producing multi-vector embeddings that are scored with pairwise late interaction similarity. That is a different retrieval stack from a text-chunk-and-embed pipeline, and it is not obvious how to stand one up.
The repository exists to close that gap. It is a set of notebooks for learning about the ColVision family of models, fine-tuning them for a specific use case, creating similarity maps to interpret predictions, and more. The audience is an engineer or researcher who already knows what retrieval is and needs working code for a model family whose tooling is still moving. The homepage points at the Vidore organisation on Hugging Face, so the models themselves live outside this repository; what you get here is the procedure for using them.
If your documents are born-digital text and OCR works well, this is not the shortest path. The notebooks matter when layout, tables and charts carry retrieval signal that text extraction flattens.
The Notebook Lineup and What Each One Actually Does
The examples directory holds six notebooks, listed newest first. Two cover inference and interpretability for the transformers-native implementations of ColQwen2 and ColPali. Two generate similarity maps, one per model, described as a way to interpret the model's predictions. One fine-tunes ColPali with LoRA and optional 4bit or 8bit quantization. The last is the most interesting: ColQwen2, one model for your whole RAG pipeline with adapter hot-swapping.
That last recipe is the one with a concrete engineering claim. The README says it saves VRAM by using a unique VLM for your entire RAG pipeline, and states it works even on Colab's free T4 GPU. The mechanism implied by the name is adapters swapped in and out of a single loaded model rather than separate models held in memory at once. The notebook is the only place the details live; there is no library code in the repository to read instead.
The release history tracks the transformers-native work. v0.4.0 in December 2024 covers the transformers-native ColPali cookbook, and v0.5.0 in June 2025 adds the transformers-native ColQwen2 cookbook. The fine-tuning and similarity-map notebooks do not appear in the release titles, so treat the release notes as a record of the transformers-native track rather than a full changelog.
Running the Notebooks: Colab, Clone, or Neither
There are two documented paths. The first is Colab. Open the notebook you want from the examples directory and click the Colab button, which opens it in Google Colab where you can run the code and experiment with the models. The second is local: clone the repository and open the notebooks in Jupyter Notebook or in your IDE.
That is the whole of the setup documentation. There is no requirements file, no environment specification and no install command in the material. Dependencies are whatever each notebook installs in its own cells, which is normal for this format and also the reason two notebooks opened a month apart can disagree about a transformers version.
The Colab path is the one the README pushes, and the adapter hot-swapping notebook is written with Colab's free T4 in mind. Running locally means you supply the GPU and the Python environment yourself. The repository does not tell you which CUDA version, which transformers version, or how much VRAM the fine-tuning notebook needs at 4bit versus 8bit. You find that out by opening the notebook and reading its first cells, or by running it and watching what fails.
Where a Notebook Collection Stops Being Enough
The honest limitation is the format. A notebook is a script with prose attached, not an interface. If you want to call ColPali retrieval from a service, you will copy cells out of these notebooks into your own code, and from that moment the repository no longer tracks your version. There is no package to pin, no changelog entry that tells you a function signature moved, and no test suite in the material to tell you a notebook still runs against the current model weights.
The release cadence reinforces this. Between v0.4.0 and v0.5.0 the project moved from a ColPali cookbook to a ColQwen2 cookbook, six months apart. A team that copied the v0.4.0 cells and never revisited them is running code the repository has already superseded. Nothing in the repository detects that for you.
The second limitation is scope. The fine-tuning notebook targets ColPali specifically, with LoRA and optional quantization. If your work is on ColQwen2, the material gives you inference, interpretability and the RAG recipe, but the fine-tuning path shown is for the other model. Whether the same recipe transfers is not stated, so you would be adapting it yourself.
Neither of these is a defect in a cookbook. It is the boundary of the genre, and it is worth knowing before you treat the notebooks as a dependency.
How This Differs from the ColPali Engine
The README links to the ColPali Engine at illuin-tech/colpali and to the ViDoRe Benchmark at illuin-tech/vidore-benchmark. The split is clean. The engine is the library: installable code that wraps the models behind an interface. The benchmark is the evaluation harness. The cookbooks are neither. They are the worked examples that sit between the paper and the engine.
That difference changes how you use them. With an engine you add a dependency, call it from your application, and upgrade it deliberately. With the cookbooks you read a notebook, understand the shape of the pipeline (encode pages as images, produce multi-vector embeddings, score with late interaction), and then write that pipeline into your own service. The cookbooks teach the mechanism; the engine implements it.
For the adapter hot-swapping idea the distinction matters more than usual. A notebook can demonstrate that one VLM serves the whole RAG pipeline on a free T4. Turning that into a serving path with adapter selection per request is application work, and the repository does not provide it. The ColVision material here is the reference for what the models do, not the runtime you deploy.
Licence, Maintenance and What You Inherit
The repository is MIT licensed. That is permissive: you can copy notebook cells into your own codebase, including commercial code, provided you keep the licence notice. The MIT grant covers the notebooks in this repository. It does not automatically cover the model weights you download from the Vidore organisation on Hugging Face, the datasets the notebooks use, or the third-party libraries the cells install. Check those separately. This is a description of the licence text, not legal advice.
Maintenance is the part to think about. The last push recorded is September 2025, and the most recent release is v0.5.0 from June 2025. That is an actively touched repository, but its unit of maintenance is the notebook. When transformers changes, someone has to open each notebook and fix the cells. When a new ColVision model appears, a new notebook appears rather than a new version of an existing one.
The cost you inherit is re-validation. Every time you refresh your environment, the notebooks you copied from are a snapshot of a working configuration at a point in time, and you are responsible for confirming they still describe how the model behaves. The citation block in the README points at the ColPali paper on arXiv, which is the stable reference for the method; the notebooks are the moving part.
Editorial conclusion
Adopt colpali-cookbooks if you need to learn the ColVision family or prototype a visual document retrieval pipeline before committing to an engine: open the notebooks from the examples directory in Colab or clone the repository and run them in Jupyter. Do not adopt it if you need a supported library with an API you can pin, test and upgrade; the MIT-licensed repository is a notebook collection, and the notebooks carry the model, dataset and library versions they were written against. Before you build on it, verify the environment the notebook assumes, the VRAM headroom the ColQwen2 adapter hot-swapping recipe requires on your hardware, and whether the transformers-native paths in v0.5.0 match the checkpoint you intend to serve.
Community notes