NVlabs DoRA: the reference implementation, with a README example that cannot run

The README's own PEFT example cannot run

The quickstart recommends installing PEFT from git rather than from a release:

pip install git+https://github.com/huggingface/peft.git -q

That is an unpinned install of a moving branch, on a repository whose own documentation points at release v0.10.0 as the version that added DoRA support. Nothing forces you onto that version, and nothing stops a later PEFT change from moving the behaviour underneath you.

The configuration example that follows is the problem. As printed it reads:

from peft import LoraConfig

# Initialize DoRA configuration
config = (
    use_dora=True, ...
)

That is not valid Python. A parenthesised expression cannot hold a keyword argument, the ellipsis is not a placeholder the interpreter accepts here, and the LoraConfig call that was imported at the top is never made. The instruction in the surrounding prose is the accurate one: set the use_dora argument of LoraConfig() to True. Treat the snippet as a diagram of intent and copy the argument name, not the code.

Support, as of the earliest news entry on 2024-03-20, covers Linear, Conv1d and Conv2d layers plus linear layers quantized with bitsandbytes.

The method splits a weight into magnitude and direction

The mechanism is a decomposition of the pre-trained weight into a magnitude component and a direction component, with LoRA doing the work on the direction side. The stated purpose of the directional update is to keep the number of trainable parameters small, and the claim is that this raises learning capacity and training stability relative to LoRA while adding no inference overhead, because the decomposition is folded away once merged.

The evaluation claim is equally specific about where it was measured: fine-tuning LLaMA, LLaVA and VL-BART, across commonsense reasoning, visual instruction tuning, and image and video to text understanding. The paper is arXiv 2402.09353, presented at ICML 2024 as an Oral with a stated acceptance rate of 1.5%.

Seven authors are listed, with the note that the work was done during an internship at NVIDIA Research. Business inquiries are routed to an NVIDIA Research Licensing form rather than to the repository, which matters more than it usually would: the code is a research artifact with commercial licensing sitting next to it, and the person asking about redistribution is pointed at a form, not a file.

Four directories, each forked from someone else's trainer

There is no packaging at the root. No pyproject, no setup.py, no requirements file; the tree holds .gitattributes, .gitignore, LICENSE, README.md, imgs/, four experiment directories and QDoRA/. Each experiment directory is a modification of an existing codebase rather than a fresh implementation:

./commonsense_reasoning fine-tunes LLaMA-7B and 13B and is modified from LLM-Adapter. ./instruction_tuning_dvora fine-tunes LLaMA-7B and LLaMA2-7B with DoRA and with DVoRA, which is DoRA combined with VeRA, on a cleaned Alpaca instruction tuning dataset, and is modified from VeRA. ./image_video_text_understanding fine-tunes VL-BART for image and video to text work and is modified from VL-Adapter. ./visual_instruction_tuning fine-tunes LLaVA-1.5-7B on visual instruction tuning, and that sentence stops mid-word after Do.

QDoRA/ sits at the root separately, holding step by step instructions for reproducing the QDoRA with FSDP results.

The practical consequence is that there is no install step to run. Each directory carries the dependency expectations of the project it was forked from, so the environment has to be assembled from those upstreams.

Hyperparameters have to move away from the LoRA settings

The most practically useful part of the documentation is a set of cautions, and they cut against the instinct to copy a working LoRA configuration.

The first note says that using the LoRA configuration while fine-tuning with DoRA already produces better results most of the time, and that reaching optimal performance against LoRA still requires hyperparameter changes. The second gives three concrete starting points: a slightly lower learning rate than LoRA's, experimentation with different LoRA dropout ratios, and starting at half the rank of the LoRA configuration, which the note says can often already reach comparable or superior accuracy.

Half the rank is the detail worth pausing on. If a quarter of the adapter parameters matches a full-rank LoRA, then any comparison that held rank fixed across methods was measuring DoRA under a handicapped setting. Rank is one of the two axes people tune first, so a fair comparison has to state whether rank was matched.

These notes are framed as suggestions for users rather than as results from a sweep, so treat them as a starting grid rather than as a tuned recipe.

Diffusion models are labelled experimental, with two differences noted

DoRA also works through HuggingFace Diffusers, with a training example in that project and a Colab notebook from Linoy Tsaban. The status is stated plainly: fine-tuning diffusion models with DoRA is still experimental and is likely to need different hyperparameter values to do its best compared with LoRA.

Two differences are called out for diffusion training. First, LoRA appears to converge faster than DoRA, which means a parameter set that overfits a LoRA run may be the right one for DoRA, so a training curve that looks worse early can still be the better run. Second, DoRA's quality advantage over LoRA is more visible at low ranks: the difference between rank 8 DoRA and rank 8 LoRA looks bigger than the difference at rank 32 or 64.

The visual evidence offered is a pair of image comparisons, one from Linoy Tsaban and one from merve, where DoRA output sits on the left and LoRA on the right. That is a useful smoke test and nothing more, since it gives you no numbers and no seeds.

Two third-party resources are also linked: a from-scratch implementation tutorial by Sebastian Raschka, and an Answer.AI post on QDoRA with FSDP claiming results that edge out full fine-tuning.

Released weights cover LLaMA2-7B and LLaMA3-8B, and the news stops in July 2024

The news section is a dated log and it ends. Reading it in order gives the shape of the project: PEFT support on 2024-03-20, source code plus DoRA weights on 2024-04-18, weights for fine-tuning LLaMA2-7B and LLaMA3-8B on commonsense reasoning on 2024-04-27, ICML acceptance on 2024-05-02, QDoRA and FSDP reproduction instructions on 2024-05-24, the Oral decision on 2024-06-02, and the NVIDIA technical blog on 2024-07-02.

Two things stand out. Released weights exist for two model sizes on one task family, commonsense reasoning, so anyone reproducing other rows of the paper's tables is training from scratch. And every entry is from 2024, with nothing appended afterwards, which is consistent with a reference implementation whose job was done rather than one that tracks new base models.

The last commit on the default branch main is dated 2026-03-24, and there are no GitHub releases, so there is no tag to install and no changelog to diff. For a paper reproduction that is workable. For anything that needs a maintained dependency, it is not.

The license field is unresolved while a LICENSE file sits at the root

Two signals about licensing do not agree. The repository's license metadata carries no recognised identifier, and a LICENSE file does exist at the root of the tree. Nothing in the repository reconciles the two, so this article will not guess which terms apply or which file governs.

What is clear is where the project points people who want clarity. Business inquiries are directed to an NVIDIA Research Licensing form linked from the README, which is the route for anything involving redistribution or commercial use. A research code release with a paper, a project page and an NVIDIA blog post is normally unambiguous about this; here the metadata field is the weak link rather than the file.

The rest of the repository is small and self-describing: an arXiv link, a project page hosted on a personal domain rather than an institutional one, the author list, and a link to the third-party licensing form. Commercial use questions, contribution questions and reuse of the trained DoRA weights are three different things, and only the first has a documented route.

Editorial conclusion

DoRA suits researchers who want the reference implementation behind the ICML 2024 paper and who intend to run it inside HuggingFace PEFT, where the change is a single argument. It does not suit someone looking for a maintained library: the last commit on the default branch is dated 2026-03-24, every news entry stops at 2024-07-02, there are no GitHub releases, and the four experiment directories are forks of other people's trainers with no package metadata at the root. Verify four things first. Fix the quickstart snippet before you copy it, because the configuration example printed in the README is not runnable Python. Read the hyperparameter guidance rather than reusing a LoRA config, since the project suggests a lower learning rate, a different dropout and possibly half the rank. Decide whether you want the PEFT route or the research code, because the recommended install pulls PEFT from git rather than pinning the release that first shipped DoRA. And settle the licensing question before you build on it: the repository's license metadata carries no recognised identifier while a LICENSE file does exist at the root, and the two are not reconciled anywhere in the repository.

Frequently asked questions

Official sources

1. Issues
2. NVlabs/DoRA on GitHub
3. Project website
4. README