LlamaFactory: One Codebase for Fine-Tuning 100+ LLMs and VLMs

What LlamaFactory Solves and Who It Serves

Fine-tuning a large language model involves many moving parts: loading a checkpoint, applying a parameter-efficient method, choosing a quantization scheme, formatting datasets, and logging experiments. Most open-source projects address one model family or one training technique. LlamaFactory bundles all of this into a single Python package. The README claims support for over 100 models, including LLaMA, LLaVA, Mistral, Qwen3, DeepSeek, Gemma, GLM, and Phi. It also lists multiple training methods: continuous pre-training, supervised fine-tuning, reward modeling, PPO, DPO, KTO, and ORPO. This makes it attractive for researchers and engineers who need to compare fine-tuning approaches across different model architectures without rewriting code for each. The target user is someone comfortable with command-line tools but who does not want to maintain a custom training pipeline. The project also provides a Gradio-based web UI called LLaMA Board, which lowers the barrier for users who prefer point-and-click configuration.

The Architecture: How Training Methods Are Unified

The core design is a unified interface over Hugging Face Transformers and PEFT. LlamaFactory does not reimplement training from scratch; it wraps existing libraries and adds a configuration layer that translates a single set of arguments into the appropriate model loading and training setup. The README lists advanced algorithms like GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, LoRA+, LoftQ, and PiSSA. These are integrated as pluggable optimizers or adapters. For quantization, it supports 2/3/4/5/6/8-bit QLoRA via AQLM, AWQ, GPTQ, LLM.int8, HQQ, and EETQ. The data flow is straightforward: you provide a dataset in a predefined format, specify a model name and a training method in a YAML or JSON config, and the CLI launches training. The framework also handles logging to TensorBoard, W&B, MLflow, and SwanLab, so experiment tracking is consistent across runs. The key architectural choice is to abstract away the differences between model families, which is both its strength and its risk: any model that does not fit the abstraction may require custom handling.

Getting Started: Installation and First Run

The README provides a quickstart that begins with installing the package. The recommended route is to install from PyPI: pip install llamafactory. Alternatively, you can build a Docker image, and the README links to a Docker Hub repository with prebuilt tags. After installation, you need to prepare data. The project provides a set of built-in datasets, but you can also supply your own in a format that matches the expected schema, typically a JSON or JSONL file with instruction, input, and output fields. The CLI entry point is llamafactory-cli, and you pass a configuration file that specifies the model name, dataset, training method, and hyperparameters. For example, a LoRA run might look like: llamafactory-cli train config.yaml, where config.yaml contains model_name_or_path, dataset, and lora_rank. The README also shows a zero-code CLI, meaning you can run training without writing a Python script. For those who prefer a GUI, the LLaMA Board is launched with llamafactory-cli webui, which opens a Gradio interface in the browser. The quickstart also mentions a free Colab notebook, so you can test the tool without local GPU resources. One caveat: the README notes that the documentation is a work in progress, so expect some rough edges when looking for detailed configuration options.

A Genuine Limitation: The Abstraction Layer Can Leak

The main limitation is that LlamaFactory's promise of '100+ models' does not mean every model works flawlessly out of the box. The abstraction layer sits on top of Transformers and PEFT, and each model family has its own quirks. For instance, multimodal models like LLaVA or Qwen3-VL require special preprocessing for images and video, and the README lists these as supported, but the underlying implementation may not cover every possible use case. The README also mentions 'Day-N Support' for cutting-edge models, with Day 0 for Qwen3 and Gemma 3, and Day 1 for Llama 4. This implies that brand-new models may not be supported immediately, and there is a delay between a model's release and its integration. Another failure mode is quantization: while the framework supports many methods, not every quantization method works with every model. For example, AQLM may not be available for a specific architecture. If you hit such a case, you cannot simply tweak a flag; you may need to modify the source or wait for a fix. Additionally, the README warns that all websites other than the official links are unauthorized, which suggests that third-party tutorials may be outdated or incorrect. So, the wrong tool for you is if you need to fine-tune a model that is not in the supported list, or if you need to use a training method that is not integrated. In those cases, you are better off with a custom script.

The Real Alternative: Hugging Face Trainer with PEFT

The most direct alternative is to use Hugging Face Transformers Trainer directly, combined with the PEFT library for LoRA and QLoRA. This is the underlying technology that LlamaFactory wraps. The difference is that with Trainer, you write your own training loop, handle dataset preprocessing yourself, and manually configure quantization. You gain full control and can adapt to any model that Transformers supports, even if it is not in LlamaFactory's list. You also avoid the risk of the abstraction layer hiding important details. The cost is time: you must write and debug code for each new model or training method. LlamaFactory's value is that it removes that boilerplate for the common cases. For someone who fine-tunes a handful of models a year, Trainer might be overkill in setup but more predictable. For someone who experiments with many models weekly, LlamaFactory saves days of work. The choice depends on whether you value convenience over control. The README also mentions that LlamaFactory uses vLLM or SGLang for inference, which is another layer on top of the standard Hugging Face pipeline. If you already have a vLLM deployment, that integration is a plus. But if you prefer to keep your stack minimal, Trainer plus vLLM separately is simpler.

Maintenance and Upgrade Cost

The project is actively maintained, with a recent release v0.9.5 on 2026-05-30, and previous versions on 2025-12-31 and 2025-06-16. This cadence of roughly two releases per year means you should expect breaking changes when new models are added. The v0.9.5 release mentions support for Qwen3.5/3.6, Gemma 4, and Transformers v5, which is a major dependency upgrade. Upgrading Transformers can affect your custom code if you rely on specific behaviors. The README does not provide a changelog in the provided text, but the release titles indicate that each version adds new models and sometimes changes the underlying framework. The license is Apache-2.0, which is permissive and allows commercial use, modification, and distribution, with the requirement to retain copyright notices. There is no mention of a CLA requirement, so contributions are standard open-source. For maintenance, you should budget time to test your training configurations after each upgrade. The documentation is marked as WIP, so you may need to read the source code to understand new features. The project also has a community Discord and a blog, which are good resources for troubleshooting, but the README explicitly warns that only the official links are authorized, so be cautious with community advice.

Deployment and Inference: Beyond Training

LlamaFactory is not just a training tool; it also provides deployment options. The README mentions that you can deploy a fine-tuned model with an OpenAI-style API and vLLM or SGLang worker. This is a significant feature because it bridges the gap between training and serving. After training, you can start a server that exposes a compatible endpoint, which is useful for testing and integration. The CLI and Gradio UI are also available for inference, so you can chat with your model without writing a separate app. The integration with vLLM is noteworthy because vLLM is a popular high-throughput inference engine, and having it built-in saves you from writing a custom wrapper. However, this also means you are tied to the versions of vLLM and SGLang that LlamaFactory supports. The README also mentions the ability to download models from ModelScope Hub and Modelers Hub, which are alternatives to Hugging Face, useful for users in regions where HF is slow or blocked. This is a practical touch. The broader point is that LlamaFactory covers the full lifecycle from data to deployment, which is rare among fine-tuning libraries. But each added feature increases the surface area for bugs, so you should verify that the inference path works for your specific model, especially if you use quantization, as some methods may not be compatible with vLLM.

Editorial conclusion

Adopt LlamaFactory if you need to fine-tune a wide range of modern LLMs and VLMs without writing custom training loops, and if you value a zero-code CLI and web UI. Avoid it if you require bleeding-edge training methods not yet integrated, or if you prefer to stay close to raw PyTorch and Hugging Face Trainer internals. Before adopting, verify that your exact model and quantization method (AQLM, AWQ, GPTQ, etc.) are listed in the supported models table, and check the documentation for any recent API changes, especially after the v0.9.5 release. The project is actively maintained, with a release cadence of roughly every six months, so plan for regular updates.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes