MiniMind-V: training a 65M vision language model from scratch

What MiniMind-V is and who it is for

Most vision language models arrive as a checkpoint and a paper. MiniMind-V arrives as a pipeline. The repository contains the vision encoder wiring, the projector, the language model, the dataset conversion, the pretrain and SFT loops, and an inference script. The stated goal is to train a 65M parameter multimodal model from zero in about two hours on rented GPU time, at a cost the README puts at three RMB.

The audience is narrow and specific. This is for engineers who want to understand how image tokens reach a transformer, and who would rather read 800 lines of Python than a 40 page paper. It is also for people who need a model small enough to run on modest hardware: the README notes that the smallest release is roughly 1/2600 the size of GPT-3. It is not aimed at teams that want a multimodal model to solve hard visual reasoning tomorrow.

The project is one branch of a family. MiniMind-V extends the pure language MiniMind with vision, and the README points to MiniMind-O for an omni variant. The release table lists six checkpoints, from a 27M v1-small up to a 200M-A65M mixture-of-experts model.

How the SigLIP2 encoder and MLP projector fit together

The architecture changed noticeably in the 2026-04-20 update, and the changelog is the best description of the current design. The vision tower is now SiglipVisionModel at P32, with images fixed at 256x256. The projector is an MLP with LayerNorm, and the reshape-based token merging was removed because P32 already produces 64 tokens natively, so no downsampling is needed.

That gives a clean data flow. An image is resized to 256x256, the SigLIP2 encoder turns it into 64 patch tokens, the MLP projector maps those into the language model's embedding space, and the text prompt carries an <|image_pad|> placeholder where the visual tokens are spliced in. The language model is the 768-dimension MiniMind backbone, and the same code supports both dense and mixture-of-experts modes.

Freezing is the part worth understanding before you train. The default for SFT is freeze_llm=1, which the README describes as training the vision_proj plus the first and last LLM layers while leaving the middle layers untouched. The intent is to keep the existing language ability intact while teaching the model to read images. max_seq_len moved from 360 to 450 in the same release, which is a hard ceiling on how much text and how many image tokens fit in one sample.

Installing MiniMind-V and running your first inference

The README puts environment setup first, and it is two commands. The clone is shallow, and the dependency install points at a Tsinghua PyPI mirror.

git clone --depth 1 https://github.com/jingyaogong/minimind-v
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

Before anything runs, you need two artifacts. The SigLIP2 vision encoder goes into ./model/siglip2-base-p32-256-ve, and the base MiniMind language checkpoint llm_768.pth goes into ./out. Both are pulled with the modelscope CLI.

modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out

For inference, download the released weights into ./out and run the eval script. The --load_from flag selects the loading path: model means native PyTorch weights, any other path means a transformers-format directory.

modelscope download --model gongjy/minimind-3v-pytorch --local_dir ./out
python eval_vlm.py --load_from model --weight sft_vlm

If you prefer the transformers format, clone the HuggingFace repo and point --load_from at that directory instead. The optional WebUI has a constraint the README flags with a warning: the model folder must sit under ./scripts/, because web_demo_vlm.py scans that directory for subfolders containing weight files and errors out if it finds none.

Training your own checkpoint, and the Parquet dataset change

The README recommends skipping pretrain and going straight to SFT, because the SFT file now contains the pretrain caption subset merged in. One file, sft_i2t.parquet, goes into ./dataset. The training command takes epochs and a starting weight.

python train_sft_vlm.py --epochs 2 --from_weight llm

If you want the projector to align images and text before SFT, run pretrain first and then start SFT from the pretrain output.

python train_pretrain_vlm.py --epochs 2 --from_weight llm
python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm

The dataset format is worth calling out. The README states that as of 2025-12-27 the data is stored as Parquet, image and text together, replacing roughly 500,000 loose image files that were slow to unpack. The SFT file holds 2.9 million pairs, with the pretrain captions folded in as a subset after global dictionary encoding deduplication, which the README says adds only about 10 percent over the original SFT size. You can inspect the first five image-text pairs by running python lm_dataset.py from the dataset/ directory.

Resuming is supported through --from_resume 1. The README describes atomic saving with a temporary file and replace, and says each save writes both a .pth weight file and a checkpoints/**_resume.pth training state file. It also claims the step count converts automatically if the GPU count changes between runs.

Where MiniMind-V will disappoint you

The 65M parameter figure is the whole point and also the main constraint. A model this small has a limited capacity for visual reasoning, and nothing in the README suggests otherwise. The fixed 256x256 input is a second ceiling: fine text in a screenshot, small objects, and high-resolution diagrams are all squeezed into a P32 grid that produces 64 tokens. If your task depends on reading dense UI text, this is the wrong tool.

The training story is also narrower than the headline suggests. The two-hour, three-RMB claim is explicitly scoped in the README to one epoch of the SFT stage on a single NVIDIA 3090, with the cost being the GPU rental for that window. It is not a claim about pretraining from random initialization, and it is not a claim about training the vision encoder, which stays frozen in the default configuration. If you want to adapt SigLIP2 itself to a new image domain, this repository does not document that path.

Finally, the transformers dependency is pinned at 4.57.6 in requirements.txt, and the torch lines are commented out, so the README tells you to install torch yourself. That is a real setup step, not a footnote: the README includes a small snippet to check torch.cuda.is_available() and points at the PyTorch wheel index if it returns False.

MiniMind-V compared with LLaVA-style recipes

The obvious reference point is the LLaVA family, which established the encoder-plus-projector-plus-LLM pattern that MiniMind-V follows. The difference is scale and intent. LLaVA recipes pair a CLIP or SigLIP tower with a 7B or 13B language model and expect multi-GPU training over days. MiniMind-V pairs the same kind of tower with a 768-dimension MiniMind backbone and targets a single 3090.

There is a second difference in the projector. Earlier MiniMind-V versions used a QFormer, and the 2026-04-01 changelog records the switch to MLP projection plus reshape compression, followed by the 2026-04-20 removal of the reshape step once P32 made it unnecessary. LLaVA's standard design is a two-layer MLP, so the current MiniMind-V projector is closer to LLaVA than its own earlier versions were. The interesting divergence is the freeze policy: MiniMind-V defaults to training the projector and only the first and last LLM layers, which LLaVA-style full fine-tuning does not do.

If you want a small VLM without writing training code, a distilled or quantized checkpoint from a larger model will beat a 65M model trained from scratch on most benchmarks. MiniMind-V is not competing there. It is competing with reading the LLaVA paper and reimplementing it.

Maintenance, licence and what the repository does not cover

The repository is not archived, and the last push was on 2026-08-06. The most recent tagged release is v2, dated 2025-10-21, while the changelog records checkpoint and architecture updates through 2026-04-20. That gap between the tag and the changelog means the release page is not the best signal of what the code currently does; the changelog entries are.

The licence is Apache-2.0, which permits commercial use and modification, and the README states the project is completely free. That covers the code. It does not automatically settle the terms of the SigLIP2 encoder weights or the training data, which are distributed separately through modelscope and HuggingFace collections. If you plan to ship a model, check the licence attached to each downloaded artifact rather than assuming the repository licence covers the whole stack. This is not legal advice.

Upgrade cost is mostly tied to the pinned dependencies. transformers==4.57.6 and the commented-out torch lines mean a version bump can ripple through the model code, and the changelog shows the project has already absorbed two structural changes in 2026 alone: the encoder swap and the projector rework. Anyone who trained a checkpoint before 2026-04-20 should expect the new code to be incompatible with those weights.

Editorial conclusion

Adopt MiniMind-V if you want to read and modify every line of a VLM pipeline, or if you need a tiny model for a constrained device and can accept limited visual reasoning. Do not adopt it as a drop-in replacement for a production multimodal API. Before training, verify that the SigLIP2 encoder lands in ./model/siglip2-base-p32-256-ve and that llm_768.pth is in ./out, because train_sft_vlm.py expects both paths.

Frequently asked questions

Official sources

1. jingyaogong/minimind-v on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases