RAG-FiT: A Four-Script Pipeline for Training and Scoring Retrieval-Augmented LLMs
Framework for enhancing LLMs for RAG tasks using fine-tuning.
At a glance
- What is it?
- IntelLabs RAG-FiT (formerly RAG Foundry) is a configuration-driven Python framework that turns a retrieval pipeline into a persisted training set, fine-tunes on it with PEFT and TRL, then scores the output with RAG-specific metrics. It is aimed at teams who already have a retriever and want to measure whether fine-tuning actually helps.
- Who is it for?
- Adopt RAG-FiT if you already run a retriever and want a reproducible record of what the model saw before answering, because the processing stage persists retrieval results and metadata next to the prompt and completion. Skip it if you need a served retrieval API or a hosted evaluation dashboard, since the repository ships four CLI scripts and Hydra config files, not a service.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 99 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap RAG-FiT fills: retrieval results that survive as training data
Most retrieval-augmented generation work is evaluated at the end of the pipeline. You change a prompt, swap a ranker, rerun the whole thing, and compare two aggregate scores. The intermediate artifact, the exact set of retrieved passages that produced a given answer, is usually thrown away. RAG-FiT treats that artifact as the product. The processing module, per the README, creates datasets that persist RAG interactions: dataset loading, column normalization, few-shot aggregation, retrieval through external tools, API calls, and template-based prompt creation. The result is saved in what the README calls a consistent, model-independent, input-output format, along with all other fields and metadata. That phrasing matters more than it looks. If retrieval results, reasoning traces, citations and attributions stay in the dataset, then the evaluation module can score on any of them, not just on the input and output strings. The README states this directly: metrics can utilize any feature in the dataset, like retrieval results, reasoning, citations and attributions. So the audience is narrow but real: engineers who already have a retriever, who suspect that fine-tuning a smaller model on their own retrieval distribution beats prompting a larger one, and who need evidence rather than intuition. It is not for someone who wants a retrieval system. RAG-FiT consumes retrieval output; it does not provide retrieval.
Four scripts, one Hydra config tree, and a target keyword that does the wiring
The architecture is unusually flat. Four top-level scripts, processing.py, training.py, inference.py and evaluation.py, correspond to the four modules, and every invocation follows the form python SCRIPT options. There is no daemon and no server. Configuration is Hydra, which the README describes as a configuration-as-code approach, with the _target_ keyword naming the Python class to instantiate in a given context. That is the mechanism that makes the modularity work. A processing config does not call a hardcoded retriever; it names a class, and Hydra builds it. Swapping a retriever, a prompt template or a metric means editing or overriding a config value, not editing library code. Hydra also brings hierarchical configs, CLI overrides, and the ability to run multiple jobs remotely with SLURM and Ray integrations. The data flow runs in one direction. Processing writes an augmented dataset. Training reads it and fine-tunes on the completions, using PEFT for parameter-efficient training and TRL for supervised fine-tuning, with the option to push models to the Hugging Face Hub. Inference generates predictions against the same augmented dataset, with or without a trained model. Evaluation consumes those predictions. Each stage is a file on disk, which means you can rerun evaluation without retraining, or retrain without re-retrieving. That separation is the strongest design decision in the repository and the reason the persisted format exists.
Getting a first run: clone, extras, and the two override flags
Installation is a clone followed by pip install -e . in the repository root. Two optional extras are documented: pip install -e .[haystack] and pip install -e .[deepeval]. Those correspond to the retrieval integration and one of the evaluation metric families, so a minimal install will not have every metric listed in the README. The README points to a PubmedQA tutorial under docs/pubmed.md for a simple end-to-end example, and that is the right entry point before touching the paper configs. The running pattern is worth internalizing because it is the whole interface. A config file is selected with -cp for the config path and -cn for the config name: python processing -cp configs/paper -cn processing-asqa-retrieval. Individual values are overridden as trailing key=value pairs, and the README shows output_path=/store/data/here and cache=true on the same command line. So the two flags you will use constantly are -cp/-cn to pick a config group and bare key=value to change one field without forking the YAML. The configs/paper folder holds the configurations for reproducing the ASQA experiments from the accompanying paper, which is the closest thing to a reference implementation in the repository. If you are new to Hydra, the _target_ indirection is the part to read up on first, because a typo in a class path fails at instantiation time rather than at config parse time.
Evaluation breadth is the selling point, and also where the dependencies pile up
The evaluation module accepts a list of metrics and the README enumerates them: EM, F1, ROUGE, BERTScore, Deepeval, RAGAS, the Hugging Face evaluate library, and classification. It also draws a distinction that many evaluation harnesses blur. Metrics can be local, run on each example, or global, run on the entire dataset, with recall given as the global example. That split matters because recall over a whole retrieval set is not computable per example, and a framework that only supports per-example scoring forces you to write the aggregation yourself. Custom metrics are described as easy to implement, though the README does not specify the interface, so that claim is unverified from the supplied material. The cost of this breadth is dependency surface. Deepeval arrives through an extra, Haystack through another, and RAGAS, BERTScore and the HF evaluate library each carry their own model downloads and transitive requirements. A metric list is therefore not free configuration. Each entry you add is a package you must pin and a model weight you may have to fetch. For a team that only needs EM and F1, the honest move is to leave the extras uninstalled and keep the environment small.
Where RAG-FiT is the wrong tool
The framework assumes a batch mindset. Processing persists a dataset, training consumes it, inference generates over it, evaluation scores the result. Nothing in the README describes an online path, a request handler, or incremental index updates. If your requirement is a service that answers questions against a live corpus, RAG-FiT is upstream of that service, not a replacement for it. A second limitation is configuration weight. Hydra with _target_ indirection is powerful and it is also a learning curve: you need to understand config groups, overrides and class instantiation before you can change a retriever. Teams without prior Hydra experience will spend their first day reading configs rather than running experiments. Third, the repository is research-shaped. The reference configurations reproduce a paper on one dataset, ASQA, and the tutorial covers PubmedQA. There is no evidence in the README of a broad catalog of ready-made configs for arbitrary corpora, so expect to write your own processing config for your data. Finally, the fine-tuning path is parameter-efficient by design, through PEFT and TRL. If your plan is full-parameter fine-tuning of a large model, the documented training route is not aimed at you. None of these are defects. They are the boundaries of a batch experimentation framework, and knowing them saves a wasted week.
How it differs from an evaluation library like RAGAS used on its own
RAGAS appears in RAG-FiT's metric list, which makes the comparison concrete rather than hypothetical. Used directly, RAGAS scores a set of question, answer, context and ground-truth rows that you assemble yourself. It is a scoring library, and the burden of producing faithful inputs falls on your pipeline code. RAG-FiT inverts that. The processing stage is what produces the rows, and it produces them by actually running retrieval and prompt construction, then persisting the full interaction including retrieval results and metadata. Evaluation is the last of four stages rather than the whole product. The practical difference shows up when a score moves. With a standalone metric library you see a number change and then go hunting through your own pipeline code for what differed. With RAG-FiT the retrieved passages are in the dataset file, so you can diff two processing runs and see whether the retriever returned different context or the model answered differently given the same context. That is a debugging property, not a scoring property, and it is the reason to accept the Hydra overhead. The trade is that you take on a training framework and a dataset format to get an evaluation workflow. If you never intend to fine-tune, the configuration cost is harder to justify.
Maintenance, releases and the Apache-2.0 terms
The repository is not archived and shows a push in June 2026, but the release cadence visible in the material is uneven: v1.1.2 in September 2024, v1.2.0 in October 2024, v1.5.0 in November 2024, and no tagged release listed after that. A user installing today is therefore likely installing v1.5.0 and pinning around it. The upgrade cost concentrates in the training stack. PEFT and TRL are fast-moving dependencies, and a framework that delegates fine-tuning to them inherits their breaking changes; a v1.5.0 install may need version constraints to resolve cleanly against newer releases of either. The evaluation side has a similar exposure through Deepeval, RAGAS, BERTScore and the HF evaluate library, each of which can pull model downloads and their own dependency trees. Budget for a lockfile rather than a loose install. On licensing, the code is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant, with the usual requirements around preserving notices and stating changes. The README carries a disclaimer that this is not an official Intel product, so treat support expectations accordingly: issues and pull requests are welcomed per the README, but there is no vendor support contract behind the name. This is not legal advice; check the LICENSE file and your own counsel for anything you ship.
Editorial conclusion
Adopt RAG-FiT if you already run a retriever and want a reproducible record of what the model saw before answering, because the processing stage persists retrieval results and metadata next to the prompt and completion. Skip it if you need a served retrieval API or a hosted evaluation dashboard, since the repository ships four CLI scripts and Hydra config files, not a service. Before committing, verify three things against the docs: which Hydra config group matches your dataset, whether your metric set needs the haystack or deepeval extras, and whether a v1.5.0 install still resolves the TRL and PEFT versions your hardware expects.
Community notes