TinyLLaVA Factory: a modular codebase for small multimodal models

What TinyLLaVA Factory is for, and who ends up using it

Training a vision-language model usually means picking a reference implementation and editing it until it stops resembling the original. TinyLLaVA Factory takes the opposite position: the parts that vary between models are configuration, and the parts that stay fixed are shared code. The README describes it as an open-source modular codebase for small-scale large multimodal models, implemented in PyTorch and HuggingFace, with a focus on simplicity of code implementations, extensibility of new features, and reproducibility of training results.

The intended user is someone who wants a model under roughly 4B parameters and needs to control what is inside it. The repository ships trained checkpoints including TinyLLaVA-Phi-2-SigLIP-3.1B, TinyLLaVA-Gemma-SigLIP-2.4B, TinyLLaVA-OpenELM-450M-SigLIP-0.89B and TinyLLaVA-Qwen2-0.5B-SigLIP, so the framework is also a source of ready weights. But the codebase is organised around the training pipeline, not around serving. If your problem is inference only, the README's own path is to pull a checkpoint from HuggingFace or use the demo, not to install this repository.

The audience is therefore narrow in a useful way: research groups and small teams who need to run pretraining plus finetuning on their own data and want the chat template, connector and vision encoder to be choices rather than forks.

The three swappable parts: LLM, vision tower, connector

The architecture is the standard three-stage assembly for this class of model, and the value is in which stages are exposed. An image goes through a vision tower, the resulting visual features pass through a connector that maps them into the language model's embedding space, and the LLM generates text conditioned on the combined sequence.

The README lists the supported options per stage. The LLM side covers OpenELM, TinyLlama, StableLM, Qwen, Gemma and Phi. The vision tower side covers CLIP, SigLIP, Dino, and a combination of CLIP and Dino. The connector side covers MLP, Qformer and Resampler. Training recipes cover frozen, fully and partially tuned parameters, plus LoRA and QLoRA.

That menu is the actual product. Swapping a connector from MLP to Qformer changes how many visual tokens reach the LLM, which changes memory use during training, and the framework treats that as a config change rather than a rewrite. The same applies to the LLM: the changelog notes that the old codebase, TinyLLaVABench, was moved to a separate branch, so the current main branch is the modular rewrite rather than an accumulation of benchmark scripts.

One detail worth noticing is conv_version. The README describes it as a hyperparameter for choosing chat templates per LLM, and it is not uniform across models: pretraining uses pretrain for all LLMs, while finetuning uses phi for Phi-2, StableLM and Qwen-1.5, llama for TinyLlama and OpenELM, and gemma for Gemma. This is a real piece of domain knowledge encoded as a config key, and getting it wrong produces a model that trains without error and then answers badly.

Installing TinyLLaVA Factory and running a first training job

The README is explicit that the environment requirements differ from LLaVA's and recommends building the environment from scratch. The install is a clone, a conda environment on Python 3.10, and an editable install:

git clone https://github.com/TinyLLaVA/TinyLLaVA_Factory.git
cd TinyLLaVA_Factory
conda create -n tinyllava_factory python=3.10 -y
conda activate tinyllava_factory
pip install --upgrade pip
pip install -e .

The pip install --upgrade pip step is there to enable PEP 660 support, which the editable install needs. After this, the README installs one more package separately, outside the dependency list, with build isolation disabled:

pip install flash-attn==2.5.7 --no-build-isolation

Expect that second step to be the slow and fragile one. It compiles against your local CUDA toolkit, and the README offers no CPU or non-flash-attention alternative, so a machine without a working CUDA toolchain has no documented path forward here.

For a first training run the README uses Phi-2 as the worked example, and the sequence is: replace data paths in scripts/train/train_phi.sh, replace output_dir in scripts/train/pretrain.sh, replace pretrained_model_path and output_dir in scripts/train/finetune.sh, and adjust the GPU ids and per_device_train_batch_size in the two stage scripts. Then:

bash scripts/train/train_phi.sh

The README states the hyperparameters to keep: global batch size 256 with learning rate 1e-3 and conv_version pretrain for pretraining, then global batch size 128 with learning rate 2e-5 and conv_version phi for finetuning. It defines global batch size as GPUs times per_device_train_batch_size times gradient_accumulation_steps, and advises holding the global batch size and learning rate at those values except when doing LoRA tuning. That formula is the one number you should check before launching, because it is the difference between reproducing the published recipe and quietly training something else.

Where the framework gets in your way

The dependency pins are tight and they are the first real constraint. The pyproject.toml fixes torch==2.0.1, torchvision==0.15.2, transformers==4.40.1, tokenizers==0.19.0, accelerate==0.27.2, deepspeed==0.14.0, peft==0.10.0, bitsandbytes==0.41.0 and numpy==1.26.4. That is a coherent set for the training recipe it was validated against, and an obstacle everywhere else. If another project in your environment needs a newer transformers, you are choosing between them.

The second constraint is that the README delegates the parts you will actually spend time on. Data preparation points to a Data Preparation page in the documentation, and evaluation points to an Evaluation page, with no inline description of the expected data format in the README itself. The README also does not document rollback, checkpoint resumption, or how to recover an interrupted pretraining run. For a multi-day job on a shared cluster, that silence matters more than the model menu.

The third is scope. The README states the best model, TinyLLaVA-Phi-2-SigLIP-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL, but that is a claim about a specific checkpoint on the benchmarks the project chose, and it is not a general statement that a small model replaces a large one on your task. If your inputs are long video, high-resolution documents, or anything outside the image-text setting the framework is built around, this is the wrong tool. The project's own follow-up work, TinyLLaVA-Video, lives in a different repository.

How it compares to editing LLaVA directly

The obvious alternative is the LLaVA codebase itself, which is the reference many of these models descend from. The difference is structural rather than a matter of features. LLaVA's code tends to be organised around one model configuration, so changing the vision encoder or the connector means editing model code and retracing which parts of the training script depend on it. TinyLLaVA Factory puts the LLM, vision tower and connector behind named options and keeps the training scripts as shell entry points over shared code.

A second alternative is to skip the framework and assemble the pieces yourself from transformers, peft and deepspeed. You would get exactly the components you want and no pinned versions you did not choose. What you would lose is the parts that are easy to get wrong and hard to notice: the conv_version mapping to the right chat template per LLM, and the global batch size and learning rate pairing the project reports for its two training stages.

The README also notes that the older TinyLLaVABench codebase was moved to the tinyllava_bench branch. If you are looking for the benchmark-oriented code from the earlier paper, that is where it went, and the main branch is not a superset of it.

Licence, maintenance and the cost of upgrading

The repository is Apache-2.0, and the pyproject.toml carries the matching classifier. Apache-2.0 is permissive and includes an explicit patent grant, which matters if you intend to ship a model trained with this code. The licence covers the code in this repository. It does not automatically cover the weights you download from HuggingFace, and it does not cover the base LLMs or vision towers you plug in, which carry their own terms. Checking those separately is your responsibility, and this is not legal advice.

The last push to the repository was on 2026-09-08, so the project is not dormant. There are no releases retrieved for it, which means the practical upgrade path is the one the README documents: pull the branch and reinstall.

git pull
pip install -e .

Because the install is editable, that second command is what re-resolves the dependency pins in pyproject.toml. The cost of upgrading is therefore not the git pull, it is the chance that a pin moves and the flash-attn build has to be redone. Budget for that on any environment you care about, and pin your own lockfile rather than tracking main if the training run is expensive.

Editorial conclusion

Adopt TinyLLaVA Factory if you need to train or fine-tune a sub-4B vision-language model and want the LLM, vision tower and connector to be independently replaceable without rewriting the training loop. Do not adopt it if you only need inference from a released checkpoint, since the README points to the HuggingFace collection and the Gradio demo for that case, and the training stack is heavier than the task requires. Before committing, verify that torch==2.0.1 and transformers==4.40.1 resolve against your CUDA and driver combination, and confirm that the flash-attn==2.5.7 wheel exists for your platform, because the README installs it with --no-build-isolation and gives no fallback path if the build fails.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. TinyLLaVA/TinyLLaVA_Factory on GitHub