DoRA: magnitude and direction split for LoRA fine-tuning

The problem DoRA targets: LoRA's single update rule for two different quantities

Low-rank adaptation updates a frozen pre-trained weight by adding a low-rank product to it. The pre-trained weight itself is never restructured. DoRA takes the position that this conflates two things that behave differently during fine-tuning: how large a weight vector is, and which direction it points. The README states that DoRA decomposes the pre-trained weight into magnitude and direction, and applies LoRA only to the directional update. Magnitude becomes a separate trainable parameter. The stated goal is to improve both learning capacity and training stability relative to LoRA without adding inference overhead, since the decomposition can be folded back into a single weight at serving time. The intended audience is narrow: people fine-tuning LLaMA, LLaVA or VL-BART who already understand LoRA and want a different parameterisation of the same budget. The repository is the official implementation behind an ICML 2024 paper, not a general-purpose tuning framework.

What the decomposition actually changes in the forward pass

In LoRA, the effective weight is W plus BA, where W is frozen and B and A are the low-rank factors. DoRA instead treats the columns of W as vectors, keeps a trainable magnitude scalar per column, and lets the LoRA factors supply only the directional component. The README describes this as decomposing the pre-trained weight into magnitude and direction, with LoRA used for directional updates to minimise the number of trainable parameters. Two consequences follow. The trainable parameter count grows slightly relative to LoRA because of the magnitude terms, but the README's claim is that this is offset at inference: the decomposition can be merged, so no additional inference overhead is introduced. The repository layout reflects the paper's evaluation surface rather than a single library. There are four directories. ./commonsense_reasoning holds LLaMA-7B/13B fine-tuning code adapted from LLM-Adapter. ./instruction_tuning_dvora holds LLaMA-7B and LLaMA2-7B instruction tuning with DoRA and DVoRA (DoRA combined with VeRA), adapted from VeRA. ./image_video_text_understanding holds VL-BART fine-tuning. A separate /QDoRA directory carries step-by-step instructions for reproducing the QDoRA and FSDP results. Each directory descends from a different upstream codebase, so the integration surface is per-directory, not uniform.

Getting it running through PEFT versus through the reproduction scripts

There are two entry paths and they are not equivalent. The short path is HuggingFace PEFT, which the README says supports DoRA as of v0.10.0, with Linear, Conv1d and Conv2d layers plus bitsandbytes-quantized linear layers. The install command given is pip install git+https://github.com/huggingface/peft.git -q, and the configuration change is the use_dora argument on LoraConfig set to True. The README's example is abbreviated and shows config = ( use_dora=True, ... ), which is illustrative rather than runnable as written. The long path is cloning this repository and working inside one of the four directories, each of which has its own upstream heritage and therefore its own dependency set. The README does not list a single top-level install command, a requirements file, or a unified entry point across those directories, so treat each one as a separate environment. For diffusion models, the README points outward to the HuggingFace Diffusers advanced_diffusion_training example and a Colab notebook rather than shipping that path here.

The hyperparameter gap the README is unusually direct about

This is the most practically useful part of the documentation and the part most likely to be skimmed. A note in the README states that using an existing LoRA configuration with DoRA will usually already beat LoRA, but that reaching optimal performance still requires adjusting hyperparameters. Three concrete suggestions follow. Start with a slightly lower learning rate than the LoRA setting. Experiment with LoRA dropout ratios. And try starting at half the LoRA rank, which the README says often yields comparable or superior accuracy. That last point matters for anyone budgeting GPU memory, because it implies the comparison is not rank-for-rank. The README also relays two observations from diffusion fine-tuning: LoRA appears to converge faster than DoRA, meaning a parameter set that overfits under LoRA may train cleanly under DoRA, and the quality gap over LoRA appears larger at low rank, such as rank 8, than at rank 32 or 64. Those diffusion observations are attributed to community reports in the README, not to a controlled study in this repository.

Where DoRA is the wrong choice

The README marks diffusion fine-tuning with DoRA as still experimental and says it is likely to need different hyperparameter values than LoRA to perform best. If your workflow is text-to-image and you want a configuration you can copy from a LoRA recipe, DoRA is not that yet. The convergence difference is the second constraint. If your training budget is tight in wall-clock terms rather than in final accuracy, a method that the README says converges more slowly than LoRA is a poor fit, and the recommended compensation is a hyperparameter search you may not have time to run. Third, the repository is a set of research reproduction directories rather than a maintained library. It was last pushed in March 2026, but there are no releases retrieved and the licence is reported as NOASSERTION, which means the terms are not machine-readable from the repository metadata. The README directs business inquiries to an NVIDIA Research Licensing form. If you need a clear permissive licence before shipping, that ambiguity is a reason to check the actual licence file rather than assume. Finally, if your models are not in the LLaMA, LLaVA or VL-BART families and you are not going through PEFT, this repository gives you code adapted from three different upstream projects, and you will be porting rather than installing.

The alternative to weigh against it, and the real difference

The obvious alternative is LoRA itself, and the difference is not a quality claim but a structural one. LoRA adds a low-rank product to a frozen weight and leaves the weight's scale untouched. DoRA adds a per-column magnitude parameter and restricts the low-rank product to the direction. That means DoRA has a slightly larger trainable set and one extra concept to tune, in exchange for a parameterisation the README argues improves learning capacity and training stability at no inference cost after merging. The practical decision rule follows from the README's own hyperparameter note: if you are at rank 32 or 64, the reported advantage over LoRA narrows, so the extra tuning may not pay for itself. If you are at rank 8 and accuracy is the binding constraint, the low-rank regime is where the README says the gap is widest. QDoRA, the quantized variant documented in the /QDoRA directory, is the third option and the one aimed at consumer-level GPUs, since it combines the decomposition with quantized base weights and FSDP. The README links an external write-up claiming QDoRA outperforms QLoRA and edges out full fine-tuning, but that result lives outside this repository and is not reproduced by the code shown in the README.

Maintenance, licence and what the repository does not tell you

There are no retrieved releases, so there is no versioned upgrade path to reason about. The last push is dated March 2026, which suggests the repository is still being touched, but a push date is not a support commitment. The four directories track three upstream projects (LLM-Adapter, VeRA, and the Diffusers example), so when those upstreams move, the reproduction code here can drift without any signal in this repository. Budget for pinning your own dependency versions per directory. On licensing, the identifier reported for this repository is NOASSERTION, which means no standard licence could be detected from the metadata. The README's only licensing statement is a pointer to NVIDIA Research Licensing for business inquiries. That is not a substitute for reading the licence file, and it is not legal advice either way. The concrete thing to check before you build on this: open the licence file in the repository root and confirm the terms yourself, then confirm that the PEFT version you install actually accepts use_dora, since the README's install line pulls from git rather than from a pinned release.

Editorial conclusion

Adopt DoRA if you are already running LoRA through HuggingFace PEFT and want to test whether a magnitude/direction split helps at your rank, or if you need the paper's LLaMA, LLaVA and VL-BART reproduction scripts. Do not adopt it as a drop-in speedup: the README states that LoRA converges faster and that diffusion fine-tuning with DoRA is still experimental. Verify three things before committing. First, that your PEFT version accepts use_dora in LoraConfig, since the README points at v0.10.0 as the release that added support. Second, whether a rank at roughly half your current LoRA rank reaches comparable accuracy, which is the README's own suggested starting point. Third, what the NOASSERTION licence identifier on this repository actually resolves to, because the README routes business inquiries to an NVIDIA licensing form rather than to an open licence.

Official sources

1. Issues
2. NVlabs/DoRA on GitHub
3. Project website
4. README