# LLaVA-pp: putting Phi-3 and LLaMA-3 behind the LLaVA vision encoder

> LLaVA-pp is a research fork from MBZUAI that swaps the language model inside LLaVA 1.5 for Phi-3 Mini Instruct 3.8B or LLaMA-3 Instruct 8B, and publishes the resulting checkpoints and training recipes. The model zoo is the draw, the license is the problem, because the repository itself declares no license at all.

**mbzuai-oryx/LLaVA-pp** — 🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)

- Repository: https://github.com/mbzuai-oryx/LLaVA-pp
- Stars: 843 · Forks: 58
- Language: Python
- License: not declared
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/mbzuai-oryx-llava-pp

## A fork that swaps the language model under the vision encoder

LLaVA-pp is described as extending visual capabilities with LLaMA-3 and Phi-3, and the README puts it more precisely: it enhances the capabilities of the LLaVA 1.5 model by incorporating the latest LLMs released at the time, which it names as Phi-3 Mini Instruct 3.8B and LLaMA-3 Instruct 8B.

The problem it addresses is narrow and specific. LLaVA 1.5 is a well known vision language architecture, a vision encoder feeding a language model through a projection layer, and by the time this work was done the available language models had moved on to LLaMA-3 and Phi-3 while the LLaVA 1.5 recipe still pointed at an older backbone. Rather than design a new architecture, this project keeps the LLaVA recipe and changes the language model underneath it, then retrains the whole thing so the vision and language halves are compatible again.

That framing matters for evaluation. Because the architecture is unchanged, the comparison the project is interested in is between backbones rather than between architectures, and the results section presents benchmark comparisons for instruction following on LMMS and academic task oriented datasets, with a radar plot and a results table, and a note that the average excludes MME with the second best entries underlined.

The authors are Hanoona Rasheed and Muhammad Maaz as equal contributors, with Salman Khan and Fahad Khan, and the affiliation is Mohamed bin Zayed University of AI, MBZUAI. The name Oryx appears in the image alt text and in the repository path, which is the name of the antelope the university uses in its branding.

This is a research artifact from an academic group, published so that others can build on it, and the documentation is written in that register: a model zoo, a set of training scripts and a pointer to a hosted demo.

## The model zoo and what each checkpoint actually is

The model zoo is the most useful part of the repository, and it is worth reading carefully because the naming encodes the training recipe rather than just a model name.

For the Phi-3 line there are four checkpoints. The pretrain variant is pretrained on LCS-558K, the LLaVA-Pretrain dataset. The lora variant holds LoRA weights fine-tuned on LLaVA-Instruct-665K, and the plain instruct variant is the merged LoRA weights in HuggingFace format. The FT variant is the fully fine-tuned weights in the same format.

The LLaMA-3 line follows the same pattern with an extra step. There is the pretrain checkpoint on LCS-558K, the lora checkpoint with LoRA weights fine-tuned on the instruct dataset, the merged instruct checkpoint, the FT checkpoint with fully fine-tuned weights, and an FT-S2 variant described as fully fine-tuned S2 model weights, referring to the S2 fine-tuning approach whose own repository is linked in the updates section.

The distinction between the three finished forms is the practical part. The lora checkpoint is small and only holds the adapter, so you need the base language model as well. The merged instruct checkpoint is the adapter applied, in HuggingFace format, and the FT checkpoint is the result of training every weight rather than a subset. If you are picking a checkpoint to deploy, the merged or FT variants are the ones that do not need the original LLaVA-3 or Phi-3 weights on the side.

Every model has a Hugging Face page under the MBZUAI organisation, and there is a collection for the llava models. The Phi-3 checkpoints are 3.8B parameter models and the LLaMA-3 ones are 8B, which is the main practical difference between the two families, and it decides what hardware you need and how long inference takes.

## Cloning the fork and pinning transformers

The installation is a clone plus a submodule update, because LLaVA itself comes in as a submodule:

```bash
git clone https://github.com/mbzuai-oryx/LLaVA-pp.git
cd LLaVA-pp
git submodule update --init --recursive
```

After that, the README names one package you need to update from LLaVA, and it pins it to a specific commit rather than a version:

```bash
pip install git+https://github.com/huggingface/transformers@a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3
```

That pin is the single most consequential line in the setup. Installing a fork of transformers at a specific commit is what makes the Phi-3 vision support work with the LLaVA training code, and it also means your environment is now coupled to a point in transformers history. When that commit is far enough in the past, other packages in the same environment will start to conflict with it, and resolving those conflicts becomes your problem rather than the project's.

The repository layout shows the structure clearly. At the top level there are the Phi-3-V directory, the LLaMA-3-V directory, the LLaVA directory for the upstream code, a scripts directory, an images directory for the figures in the README, and a .gitmodules file.

The README also points to several ways to see the models working without installing anything: a Google Colab notebook for chatting with the Phi-3-V 3.8B model, an online demo behind an ngrok address, and two Hugging Face Spaces, one for LLaMA-3-V and one for Phi-3-V. The Colab route is the fastest way to confirm the models behave as expected before committing a GPU to a local run.

## Phi-3-V is a set of file copies, not a package

The integration instructions for Phi-3-V are the part of this project that tells you most about how it was made.

Rather than a patch series or a plugin, the approach is to copy specific files from the Phi-3-V directory into the LLaVA tree:

```bash
# Copy necessary files
cp Phi-3-V/train.py LLaVA/llava/train/train.py
cp Phi-3-V/llava_phi3.py LLaVA/llava/model/language_model/llava_phi3.py
cp Phi-3-V/builder.py LLaVA/llava/model/builder.py
cp Phi-3-V/model__init__.py LLaVA/llava/model/__init__.py
cp Phi-3-V/main__init__.py LLaVA/llava/__init__.py
cp Phi-3-V/conversation.py LLaVA/llava/conversation.py

# Training commands
cp scripts/Phi3-V_pretrain.sh LLaVA/Vi-phi3_pretrain.sh
cp scripts/Phi3-V_finetune_lora.sh LLaVA/Vi-phi3_finetune_lora.sh
```

Read as a whole, the copy list is a description of the changes needed to make LLaVA speak a different language model. The training loop is replaced because the Phi-3 integration needs its own training code. A new language model implementation, llava_phi3.py, is dropped into the language_model directory. The builder, which is what assembles the model from a config, is replaced. The model and package init files are replaced so the new class is registered. And the conversation module is replaced, which is where the prompt templates and the chat structure live, and that matters because Phi-3 has its own chat format.

The two shell scripts are the training commands, a pretrain script and a LoRA finetune script, copied into the LLaVA directory under new names. The pretrain step is the LCS-558K stage and the finetune is the instruct stage with LoRA.

This is a fork-and-overwrite approach, and it has a clear consequence. There is no way to switch between LLaMA-3 and Phi-3 backends in one checkout without copying files over each other, and there is no merge with upstream LLaVA, so a future LLaVA release does not arrive in your tree. For a one-off research reproduction that is a fair trade. For a maintained fork it is not, and it is the main reason the project is best treated as a snapshot of a specific set of experiments.

## The licensing gap and the unversioned repository

Two facts about this repository should stop an evaluator before the feature list does.

First, the licence is unknown. The project metadata records the license field as unknown, a custom licence GitHub cannot classify, and the README itself carries no licence section, unlike most projects of this kind. LLaVA upstream carries a permissive licence, and the Phi-3 and LLaMA-3 models each carry their own model licences, but the combination published here has no stated terms. If you intend to use the checkpoints in a product or a service, that is a question to answer in writing with the authors before you build anything on top of them, because the default position when a licence is absent is that you have no permission to do anything.

Second, there are no releases. The repository has no GitHub releases, so there is no versioned artifact to pin to. The updates section is a dated list of announcements running from 2024-04-26 through 2024-04-30, covering the release of the Phi-3-V and LLaVA-3-V integration, the Colab notebook, the online demo, the addition of the LoRA, fully fine-tuned and S2 fine-tuned models, and the Hugging Face Spaces.

The repository itself is not archived, and the last push was on 2026-09-05, so the code has been touched more recently than those announcements suggest. The gap between the last announcement and the last push suggests commits without a corresponding public update, and since there are no tags, the commit history is the only version reference you have.

The topic list on the repository shows the intended audiences and the naming conventions, with entries for llava, llama-3-vision, phi-3-vision, lmms, conversation and the model name variants, and it also shows that the project is aimed at the evaluation community as much as the deployment community.

## When this fork is the wrong answer

There are clear cases where the general LLaVA codebase or a different checkpoint is a better choice.

If you want to train your own vision language model, LLaVA-pp is a set of finished checkpoints plus two working training scripts for one specific pairing. Upstream LLaVA gives you the general recipe, the model zoo and the tooling, and you point it at whatever backbone you want. This fork optimises for two, and does it by overwriting the files that make the general recipe general.

If you need a model with a permissive licence, the unknown status of this repository is disqualifying on its own, and the upstream LLaVA weights or the base Phi-3-V and LLaMA-3-V releases are the path.

If you want long term maintenance, the file copy integration has no merge path with upstream. Every LLaVA release that changes the builder, the init files or the conversation module will need to be reapplied by hand to this tree, and the people who know how are the authors.

The benchmark comparison is also narrower than it looks. The results are presented for instruction following on LMMS and academic task oriented datasets, with the average computed excluding MME, and the two figures are a radar plot and a table of results. That tells you how the two backbones compare against LLaVA 1.5 on those suites. It does not tell you how they compare against every other vision language model, and the April 2024 vintage of the comparison matters, since the field moves quickly and Phi-3-V and LLaMA-3-V were new releases at the time.

What the project is genuinely good for is a specific and common need: you want a vision language model with a LLaMA-3 or Phi-3 language backbone, you would rather not assemble the training yourself, and you want the checkpoints in a form you can load directly. For that, the model zoo is the product, and it is a good one.

## Conclusion

Adopt LLaVA-pp if you want a vision language model with a Phi-3 or LLaMA-3 language backbone and prefer the published checkpoints to training one from scratch, and if your use is research or internal evaluation. Do not adopt it for a commercial product until the licensing question is settled, because the repository declares no license, and skip it if you need an actively maintained general purpose VLM framework rather than a set of fixed checkpoints. Verify first which checkpoint matches your hardware, since the LLaMA-3 variant is an 8B model, and check the transformers commit the README pins, because that pin is the part of this project most likely to age badly.

## FAQ

### What is LLaVA-pp and what does it add to LLaVA?

It enhances the LLaVA 1.5 model by swapping the language model for Phi-3 Mini Instruct 3.8B or LLaMA-3 Instruct 8B, and publishes the resulting checkpoints and training scripts. The architecture is otherwise unchanged, so the comparison of interest is between backbones.

### How do I set up the LLaVA-pp repository?

Clone it, then run `git submodule update --init --recursive` inside the directory so the LLaVA submodule is present, and install the pinned transformers commit with `pip install git+https://github.com/huggingface/transformers@a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3`.

### What is the difference between the LoRA, instruct and FT checkpoints in LLaVA-pp?

The lora checkpoint holds LoRA weights fine-tuned on the instruct dataset, the instruct checkpoint is the merged LoRA weights in HuggingFace format, and the FT checkpoint is the fully fine-tuned weights. The pretrain variant is the one trained on LCS-558K before any instruction tuning.

### How is the Phi-3-V integration applied to the LLaVA codebase?

By copying files. The README copies Phi-3-V/train.py, llava_phi3.py, builder.py, the model and package init files and conversation.py into the LLaVA tree, and copies the pretrain and LoRA finetune shell scripts into the LLaVA directory.

### Can I try LLaVA-pp without installing it?

Yes. The README links a Google Colab notebook for chatting with the Phi-3-V 3.8B model, an online demo, and Hugging Face Spaces for both LLaMA-3-V and Phi-3-V under the MBZUAI organisation.

## Sources

- [Issues](https://github.com/mbzuai-oryx/LLaVA-pp/issues)
- [mbzuai-oryx/LLaVA-pp on GitHub](https://github.com/mbzuai-oryx/LLaVA-pp)
- [README](https://github.com/mbzuai-oryx/LLaVA-pp/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mbzuai-oryx-llava-pp