# smol-vision: a notebook collection for fine-tuning and compressing vision models

> smol-vision is a GitHub repository of Jupyter notebooks and Python scripts covering fine-tuning, quantization, knowledge distillation, and multimodal RAG for vision and VLM models. GitHub stopped rendering its notebooks with rich outputs, so the readable version lives at huggingface.co/merve/smol-vision.

**merveenoyan/smol-vision** — Recipes for shrinking, optimizing, customizing cutting edge vision models. 💜 

- Repository: https://github.com/merveenoyan/smol-vision
- Stars: 2,002 · Forks: 154
- Language: Jupyter Notebook
- License: Apache-2.0
- Published: 2026-10-11 · Updated: 2026-10-11 · Language: en
- Canonical page: https://hysenlabs.com/projects/merveenoyan-smol-vision

## A collection of standalone notebooks, not an installable library

smol-vision is a GitHub repository of Jupyter notebooks and Python scripts. There is no pip package, no shared utility layer, and no common interface across notebooks. Each notebook addresses a specific technique applied to a specific model, and adapting one to a different architecture or dataset requires opening it and editing the cells directly.

The repository holds more than 30 top-level files, including notebooks named after the models they target: PaliGemma, Florence-2, IDEFICS3, ColPali, Qwen2-VL, Gemma-3n, OWLv2, and others. 3 Python scripts sit alongside the notebooks: smolvlm.py, paligemma.py, and gemma3n_fine_tuning_on_all_modalities.py. These scripts provide training loops as plain Python files rather than notebook cells, which avoids the risk of a kernel interruption during a long training run.

The README table categorises the notebooks into technique families: VLM fine-tuning, quantization, model shrinking, speed-up, and multimodal RAG. The table is not a complete inventory. Several notebooks at the top level, including Fine_tune_SmolVLM2_on_Video.ipynb, PaliGemma_DPO.ipynb, Idefics_FT.ipynb, and Qwen3VL_Multimodal_RAG.ipynb, do not appear in the table rows.

## VLM fine-tuning: QLoRA on VQAv2, DocVQA, and grounded bounding box tasks

4 notebooks cover VLM fine-tuning across different model families and output types. Smol_VLM_FT.ipynb targets IDEFICS3 and SmolVLM on the VQAv2 dataset, and the companion script smolvlm.py provides the same recipe as a plain Python training script. Both support QLoRA and full fine-tuning. QLoRA reduces GPU memory at the cost of some precision; full fine-tuning uses the entire weight space but requires more VRAM.

Fine_tune_Florence_2.ipynb targets the DocVQA dataset, which is document question answering on scanned pages rather than natural photos. This is a distinct input distribution from general visual question answering, and its notebook is categorised separately in the README.

Grounded_Fine_tuning GH.ipynb applies to Kosmos2.5 on OCR tasks with bounding boxes. Grounded fine-tuning produces outputs that include spatial coordinates alongside text, which is a different output schema from standard VQA. PaliGemma also has a DPO notebook (PaliGemma_DPO.ipynb) among the top-level files, showing alignment rather than supervised fine-tuning, though this entry does not appear in the README table.

## Quantization paths: Quanto for hardware fit, ONNX for OWLv2, and torch.compile for latency

3 distinct quantization approaches appear across 4 notebooks. Fit_in_vision_models_using_quanto.ipynb uses the Quanto library to fit vision models onto constrained hardware through post-training weight quantization. The framing is hardware fit rather than export, which is Quanto's primary scenario.

Faster_Zero_shot_Object_Detection_with_Optimum.ipynb takes a different path: it exports OWLv2, a zero-shot object detection model, to ONNX using Optimum ONNXRuntime tools before quantizing. OWLv2 is substantially larger than classification models, so the notebook covers a case where runtime-level optimization matters alongside weight reduction.

Reduce_any_model_to_fp16_using_Optimum_DETR.ipynb introduces ONNX export and quantization more gently, using DETR as the example. It is billed as a soft introduction to exporting vision models to ONNX and quantizing them.

A fourth notebook, Faster_foundation_models_with_torch_compile.ipynb, improves latency using torch.compile rather than changing model weights. This is complementary to quantization: torch.compile optimizes the computational graph at the PyTorch level, while quantization changes the numerical precision of the weights.

## ColPali and OmniEmbed: retrieval without document preprocessing

2 ColPali notebooks address multimodal document retrieval as part of a RAG pipeline. ColPali_+_Qwen2_VL.ipynb retrieves documents using ColPali through the Byaldi library, then passes them to Qwen2-VL for answer generation. This setup performs RAG without hefty document processing, meaning pages are retrieved as images and not parsed for text first. This matters for PDFs with complex layouts, tables, or diagrams where text extraction loses structure.

Finetune_ColPali.ipynb extends the retrieval setup by applying contrastive fine-tuning to ColPali, adapting the retriever for a custom document collection rather than using the pre-trained weights.

Any_to_Any_RAG.ipynb moves beyond documents to video, using OmniEmbed alongside Qwen for retrieval and generation across modalities including video. A separate notebook, Qwen3VL_Multimodal_RAG.ipynb, is in the top-level repository but does not appear in the README table, so its contents are not described there.

## GitHub stopped rendering these notebooks; HuggingFace hosts the readable versions

The README flags a practical limitation: GitHub stopped rendering notebooks with rich output, and the author points readers to huggingface.co/merve/smol-vision for the version with actual cell outputs. The author still updates the repository but notes it is inconvenient to read there. Opening a notebook in the GitHub interface shows raw JSON or a partial render without executed outputs, which makes it difficult to evaluate what a technique produces without running the cells.

The README table has 15 completed entries and 2 placeholder rows marked SOON: the Speed-up/Memory Optimization row for a TGI serving notebook, and the Quantization/Optimum/ORT row for all levels of quantization and graph optimization for image segmentation. Both rows have descriptions but no linked notebook.

One inconsistency between the README's 2 listing sections: DINOv3_FT.ipynb is called 'fine-tune DINOv3 on image classification' in the Latest examples block at the top but 'fine-tune DINOv3 for vision tasks' in the table. These are slightly different scopes for the same file.

## Gemma-3n notebooks: fine-tuning across audio, image, and text in one pass

Gemma3n_Fine_tuning_on_All_Modalities.ipynb and its script counterpart gemma3n_fine_tuning_on_all_modalities.py fine-tune Gemma-3n to handle audio, text, and image input together. The description calls this any modality, meaning the same checkpoint can accept any combination of those input types after fine-tuning. This is a different class of notebook from the single-modality VLM fine-tuning recipes: Gemma-3n is positioned as an any-to-any model, and the notebook exercises all three input types rather than focusing on image plus text alone.

2 additional Gemma notebooks are in the top-level files but do not appear in the table: Gemma_3_for_Video_Understanding.ipynb and Gemma_3n_Video_Vibe_Tests.ipynb. Fine_tune_SmolVLM2_on_Video.ipynb also covers video but for a different model. These 3 notebooks expand the collection beyond what the README table covers, and their actual contents are not summarised there.

## Safety, image search, DINOv3, and the knowledge distillation reference

ShieldGemma_2_for_Vision_LM_Safety.ipynb covers safety evaluation for vision language models. This sits outside the fine-tuning and quantization categories and is more about filtering or evaluating model outputs for harmful content signals. It is not listed in the README table.

Image_Search_with_MetaCLIP2.ipynb demonstrates image similarity search using MetaCLIP2. This is a retrieval-by-image technique rather than a RAG pipeline with a generative component, and it does not appear in the table either.

Knowledge distillation is handled through knowledge_distillation.md, which links to the official transformers documentation rather than containing a notebook directly. It is the only technique in the repository that is not self-contained: a reader must follow the external link to find the actual code. Instance segmentation has a notebook at the top level (instance_segmentation.ipynb) that the table does not mention, and a rf-detr/ subdirectory holds additional experiments organized as a directory rather than a single top-level file.

The last push was 2026-08-12, and no GitHub releases have been tagged.

## Conclusion

smol-vision is for practitioners who want a working starting point for fine-tuning or compressing a specific vision model from the HuggingFace ecosystem, and who are comfortable adapting a notebook to their own dataset and hardware. Skip it if you need a library with a stable API, documented function signatures, automated tests, or a consistent interface across notebooks. Before running any notebook, open huggingface.co/merve/smol-vision to read the rendered version with actual outputs, because GitHub stopped rendering these notebooks with rich content, and confirm that the model name in the notebook matches the checkpoint you intend to use.

## FAQ

### What is smol-vision?

smol-vision is a GitHub repository of Jupyter notebooks and Python scripts that provide recipes for fine-tuning, quantizing, and building multimodal RAG pipelines with vision and VLM models. There is no pip package; each notebook is a standalone demonstration for a specific model.

### How do I run smol-vision notebooks?

Open the desired notebook from the GitHub repository or go to huggingface.co/merve/smol-vision, which hosts the versions with rendered cell outputs. GitHub no longer renders these notebooks with rich output, so the HuggingFace copy is the readable version.

### Does smol-vision include video model notebooks?

Gemma_3_for_Video_Understanding.ipynb, Gemma_3n_Video_Vibe_Tests.ipynb, and Fine_tune_SmolVLM2_on_Video.ipynb are in the top-level repository files. Any_to_Any_RAG.ipynb also covers video retrieval using OmniEmbed and Qwen, and is listed in the README table.

### Does smol-vision support ColPali fine-tuning?

Finetune_ColPali.ipynb applies contrastive fine-tuning to ColPali for a custom document retrieval use case. ColPali_+_Qwen2_VL.ipynb shows how to use a pre-trained ColPali for document RAG without fine-tuning the retriever.

### What is the difference between the Quanto and ONNX quantization notebooks?

Fit_in_vision_models_using_quanto.ipynb uses Quanto for post-training weight quantization, targeting hardware where model size is the constraint. Faster_Zero_shot_Object_Detection_with_Optimum.ipynb exports OWLv2 to ONNX using Optimum ONNXRuntime tools, focusing on inference runtime optimization rather than pure weight reduction.

## Sources

- [Issues](https://github.com/merveenoyan/smol-vision/issues)
- [License: Apache-2.0](https://github.com/merveenoyan/smol-vision/blob/main/LICENSE)
- [merveenoyan/smol-vision on GitHub](https://github.com/merveenoyan/smol-vision)
- [README](https://github.com/merveenoyan/smol-vision/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/merveenoyan-smol-vision