Doc-to-LoRA: Turning a Document into a LoRA Without Gradient Steps
Hypernetworks that update LLMs to remember factual information
At a glance
- What is it?
- Sakana AI's reference implementation trains a hypernetwork that converts context into LoRA weights in a single forward pass, so a base model can absorb a document without any per-document fine-tuning. The repository ships the training scripts, an interactive demo, and a Python API that only handles one document at a time.
- Who is it for?
- Adopt Doc-to-LoRA if you need to internalize a fixed document into a Gemma-class base model and want to avoid a per-document fine-tuning job, and if you can accept the single-input constraint of the documented Python API. Do not adopt it if you need batched inference out of the box, if you cannot run the training pipeline yourself, or if you expect a released checkpoint to cover a base model outside the demo set.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 93 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Problem D2L Targets: Context That Will Not Fit or Will Not Persist
Long-context prompting and retrieval both pay a cost on every call. A document either occupies the context window each time it is needed, or it is chunked and retrieved, and retrieval decides what the model sees before the model gets a chance to reason. Doc-to-LoRA takes a third route. The README describes it as a reference implementation of a method that learns to internalize contexts, and the repository topics list memory and llm-agent alongside hypernetworks and lora. The intended user is someone who has a body of factual text and wants the model to answer from it without re-supplying the text at every turn. The README's own example makes the failure mode explicit: with the document internalized, the model answers about Sakana AI; the commented-out block shows that without internalization the model will hallucinate. That contrast is the whole pitch. This is not a retrieval layer and not a prompt template. It is a weight-space write.
How the Hypernetwork Writes a Document into LoRA Weights
The mechanism lives in src/ctx_to_lora/modeling/hypernet.py. The class the README exposes is ModulatedPretrainedModel, constructed from a saved state dict with ModulatedPretrainedModel.from_state_dict(state_dict, train=False, use_sequence_packing=False). The state dict belongs to a hypernetwork trained to map context into LoRA parameters for a frozen base model, which the code reaches through model.base_model.name_or_path. The data flow in the README example is short: read a document into a string, call model.internalize(doc), then call model.generate on tokenized chat input. Between those two calls the hypernetwork has run and produced the LoRA weights that modulate the base model. model.reset() removes them, returning the model to its unmodified state. That reset is the interesting part of the design. The internalized knowledge is not baked into the checkpoint on disk; it is a transient set of weights that can be discarded. The README also notes that the documented Python interface only supports non-batched inputs, and points at hypernet.py for batched inference. So the simple path and the fast path are not the same code path, and anyone moving past a single document should expect to work against the modeling module directly rather than the wrapper.
Installation and the Checkpoint Download
The README gives two installation steps. First, install uv with curl -LsSf https://astral.sh/uv/install.sh | sh. Then run ./install.sh from the repository root. The pre-trained models are hosted on Hugging Face and pulled with the CLI: uv run huggingface-cli login, then uv run huggingface-cli download SakanaAI/doc-to-lora --local-dir trained_d2l --include "*/". Note the trailing slash in the include pattern, which the README uses to select directories. The example checkpoint path is trained_d2l/gemma_demo/checkpoint-80000/pytorch_model.bin, which tells you the released demo is built on a Gemma base model and that the checkpoint is a training step artifact rather than a single exported file. The README does not list which other base models have released checkpoints, so treat gemma_demo as the confirmed case and verify the directory contents after download before writing code against another name. The tokenizer comes from get_tokenizer(model.base_model.name_or_path), so it follows the base model rather than the hypernetwork checkpoint.
Training, Evaluation and the Script Layout
The repository is organized around two experiment tracks. The main experiment lives under scripts/main_exp/ with a data download step (0-download_data.sh), a training step (1-train.sh) and evaluation scripts under eval/. The README notes that downloading data is faster and that regeneration is only needed if you want fresh synthetic data, which implies the training data is partly synthetic and that the download is the default path. The second track is scripts/niah/, covering needle-in-a-haystack style evaluation, with 0-gen_data.sh, 1-train.sh and 2-eval.sh meant to run in order. The README states that the evaluation scripts reproduce the main paper metrics, so the repository is positioned as a reproduction kit as much as a drop-in library. Two auxiliary tools ship alongside: uv run demo/app.py for the interactive demo, and uv run webui/self_gen_viewer.py to inspect generated data, documented further in webui/SELF_GEN_VIEWER.md. Everything is invoked through uv run from the project root, which keeps the environment pinned to the lockfile rather than to whatever is active in your shell.
Where D2L Breaks Down
The clearest constraint is stated in the README itself: the Python API supports non-batched inputs only. If your workload internalizes many documents and serves many queries, the wrapper is the wrong entry point and you are expected to work in hypernet.py. The second constraint is the checkpoint situation. The README shows one demo checkpoint path, and no release notes were retrieved for this repository, so there is no published list of supported base models or versioned artifacts to pin against. You are cloning main and downloading whatever the Hugging Face repository currently holds. Third, internalization is per-document state on a live model object. The README's reset comment implies single-document semantics: internalize, generate, reset. It does not describe what happens if you internalize a second document without resetting, and nothing in the supplied material addresses composition or eviction. If your use case is many documents with selective recall, that gap matters, and the honest answer is that this material does not tell you how the model behaves there. Finally, the README's hallucination note cuts both ways: it demonstrates that internalization changes behavior, but it also means an incorrect or stale document produces confident wrong answers with no retrieval trace to inspect.
How This Differs from RAG and from Per-Document Fine-Tuning
Retrieval-augmented generation keeps the document outside the model. A retriever selects passages at query time and the prompt carries them, so you can cite sources, swap documents between requests, and update the corpus without touching weights. Doc-to-LoRA moves the document inside the model, which removes the retrieval step from the serving path and removes the context window as a hard limit on how much text can influence an answer. The trade is auditability: there is no retrieved passage to show a user, only a set of LoRA weights produced by a hypernetwork. Standard LoRA fine-tuning sits at the other extreme. It also writes weights, but it does so by running gradient descent on your document, which takes minutes to hours per document. D2L's claim is that the hypernetwork amortizes that cost, so internalization is a forward pass rather than a training run. That is the difference worth evaluating: not whether weight-space memory is better than retrieval, but whether one forward pass through a trained hypernetwork is accurate enough to replace a per-document training job for your text.
Licence, Maintenance and Upgrade Surface
The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the repository's licence; the README does not state the licence of the released checkpoints or of the base model they modulate, and Gemma-family weights carry their own terms, so check those separately rather than assuming MIT covers the whole stack. This is not legal advice. On maintenance, the last push recorded is 2026-06-15 and the repository is not archived, so it is active, but no releases were retrieved, which means there are no tagged versions to pin. Upgrades arrive as commits on main. The practical consequence is that your install is tied to a commit hash rather than a version number, and the checkpoint path in the README (checkpoint-80000) is an artifact of a specific training run, not a stable interface. Budget for reading diffs in src/ctx_to_lora/modeling/hypernet.py before pulling, because the documented API is a thin wrapper over that module and changes there will surface in your code.
Editorial conclusion
Adopt Doc-to-LoRA if you need to internalize a fixed document into a Gemma-class base model and want to avoid a per-document fine-tuning job, and if you can accept the single-input constraint of the documented Python API. Do not adopt it if you need batched inference out of the box, if you cannot run the training pipeline yourself, or if you expect a released checkpoint to cover a base model outside the demo set. Before committing, download the checkpoint with the huggingface-cli command in the README, run demo/app.py, and confirm that model.internalize(doc) followed by model.reset() produces the behavior change the README describes on your own text.
Community notes