continuous-eval: Module-Level Scoring for RAG and Agent Pipelines

The failure continuous-eval is built to expose

A RAG system that returns a wrong answer gives you one bit of information. The retriever may have missed the passage, the reranker may have demoted it, or the generator may have ignored it. continuous-eval targets that ambiguity by letting you attach metrics to each module in the chain, so a retriever is scored with retrieval metrics and a generator is scored with answer correctness. The README describes this as modularized evaluation: measure each module in the pipeline with tailored metrics. The intended user is an engineer who owns a multi-stage LLM application and needs to localize a regression, not a product manager who wants a single quality score. The library also covers code generation, agent tool use and classification, so the scope is wider than retrieval alone.

How the Pipeline and Module objects model your system

The mechanism is a graph of Module objects. Each Module takes a name, an input, an output type, and a list of metrics. Inputs can be a dataset field or another module, which is how the retriever to reranker to llm chain is expressed in the README example. Metrics are bound to data through a .use() call that maps metric arguments to dataset columns or to ModuleOutput wrappers. The README shows a helper function page_content that pulls doc["page_content"] out of the retrieved documents, passed as ModuleOutput(page_content), which is how the library handles the fact that your raw pipeline output rarely matches the metric signature. Pipeline([retriever, reranker, llm], dataset=dataset) assembles the graph, and pipeline.graph_repr() renders it in mermaid format for inspection. EvaluationRunner then walks that graph and produces PipelineResults. The data flow is therefore: dataset row enters the first module, each module emits a typed output, metrics read either dataset columns or upstream outputs, and results aggregate per module.

Installing and supplying credentials

The package installs from PyPI with python3 -m pip install continuous-eval. Installing from source uses git clone followed by poetry install --all-extras. LLM-based metrics need at least one LLM API key present in a .env file, and the repository ships a .env.example showing the expected variable names. Deterministic and semantic metrics do not require a key, which matters because it lets you run retrieval scoring in CI without network calls. The README does not enumerate which providers are supported, so check .env.example before assuming your provider is covered.

A single metric call before you commit to a pipeline

The lowest-friction entry point is running one metric on one dictionary. The README example builds a datum with question, retrieved_context, ground_truth_context, answer and ground_truths, then instantiates PrecisionRecallF1 and calls it with keyword arguments. This is worth doing first because it isolates two questions: whether your data has the fields the metric expects, and whether the metric's definition of correctness matches yours. Only after that does wrapping things in SingleModulePipeline and EvaluationRunner pay off.

Tests as thresholds, and where the naming gets fragile

The SingleModulePipeline example adds a tests list containing GreaterOrEqualThan(test_name="Recall", metric_name="context_recall", min_value=0.8). The runner exposes separate evaluate() and test() methods, so scoring and gating are distinct steps: eval_results = runner.evaluate() then test_results = runner.test(eval_results). The weak point is the string metric_name. It must match whatever key the metric emits, and the README example pairs a test named Recall with a metric named context_recall, which is not the class name. There is no compile-time check on that string in the material shown. If you rename or swap a metric, the test can silently stop asserting what you think it asserts. Treat metric_name as a contract you verify by printing eval_results.aggregate() at least once.

Multiprocessing, aggregation and the parts the README leaves open

The single-metric script ends with a comment stating it is important to run the script in a new process to avoid multiprocessing issues, and the example guards main() accordingly. That is a real operational constraint: notebook cells and interactive sessions are the wrong place to invoke EvaluationRunner. The example also times the run with perf_counter and prints elapsed seconds, which hints that LLM-based metrics are slow enough to be worth measuring, though the README publishes no expected duration. What the material does not cover: how results are persisted, whether runs are resumable after a failure partway through a dataset, or how concurrent API calls are throttled. If your dataset is large and your metrics are LLM-based, those gaps are the ones you will hit first.

When a hosted evaluation platform is the better fit

The obvious alternative class is a hosted evaluation service such as LangSmith or Braintrust, where traces are collected automatically and dashboards are provided. The difference in approach is where the work happens. continuous-eval is a library: you assemble the Dataset, Module and Pipeline objects yourself, run them in your own process, and store the results wherever you like. A hosted platform instruments your application and evaluates from the recorded traces, which removes the need to re-run the pipeline to score it but requires your execution to flow through that vendor. continuous-eval keeps evaluation inside your repository and inside your CI, at the cost of you maintaining the dataset plumbing and the ModuleOutput extraction functions. It is the wrong tool if you want a UI, historical trend charts, or evaluation of production traffic that you have not captured in a dataset.

Maintenance, licence and what to check before adopting

The project is Apache-2.0, which permits commercial use and modification, and the repository is not archived. Release cadence visible in the material is uneven: v0.3.14 in January 2025, v0.3.13 in August 2024, v0.3.11 in June 2024, with a later push to the default branch in August 2026. That pattern suggests the library is maintained rather than rapidly evolving, so pin your version and read release notes before upgrading. Apache-2.0 also means you carry the obligation to retain licence and notice files if you redistribute, and the licence text itself is the authority, not this summary. Verify three things first: that every metric you plan to use is reachable without an LLM key if you intend to run it in CI, that your dataset already contains ground_truth_context or ground_truth_answers where the metrics require them, and that the metric_name strings in your tests match the keys in eval_results.aggregate().

Editorial conclusion

Adopt continuous-eval if your pipeline has separable stages and you already have per-stage inputs and outputs to score. Do not adopt it if you only need a single end-to-end quality number, or if you cannot supply ground truth context or answers. Before committing, verify that the metric names in your chosen module match the names you pass to tests, and confirm the LLM API key requirement in .env against your own provider.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. relari-ai/continuous-eval on GitHub
5. Releases