Model or dataset
relari-ai/continuous-eval avatar
relari-ai/continuous-eval

continuous-eval measures a pipeline module by module, and ships telemetry by default

Data-Driven Evaluation for LLM-Powered Applications

519 stars39 forksPythonApache-2.0

At a glance

What is it?
An Apache-2.0 Python package from Relari for data-driven evaluation of LLM applications, with metrics for retrieval, generation, code generation, tool use and classification, plus LLM-as-a-Judge hooks. The dependency matrix is unusually thorough, the Python ceiling is hard, and anonymous usage tracking is on unless you turn it off.
Who is it for?
continuous-eval fits a team that has already built a LLM pipeline and now needs numbers per stage rather than one end-to-end score, since the module abstraction with per-module metrics and pass-fail tests is the part that generalises beyond RAG. It is not a good fit if you have one prompt and one output, because the setup cost is a pipeline definition and a dataset.
Can I use it commercially?
Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 97 days ago.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • relari-ai/continuous-eval519 ★ · +2
Star history
Repository2026-09-112026-11-15
relari-ai/continuous-eval517519

2 Stars in the last 65 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

pip install continuous-eval, poetry install --all-extras from source

The package on PyPI is `continuous-eval` and the import package is `continuous_eval`, so the two names differ only in the separator. The documented install is one command:

bash
python3 -m pip install continuous-eval

Working from the repository means poetry and every extra:

bash
git clone https://github.com/relari-ai/continuous-eval.git && cd continuous-eval
poetry install --all-extras

The interpreter constraint is the first thing to check. It is declared as `>=3.10,<3.13`, so Python 3.13 and later are outside the supported range, which matters more than it looks because two of the optional semantic dependencies narrow it further and require 3.11. Deterministic metrics need nothing else. Running the LLM-based ones requires at least one provider key placed in a `.env` file, with `.env.example` at the repository root as the template. The published version is `0.3.14post2`, a post-release of the 0.3.14 line, which tells you the packaging is still settling.

Six provider blocks and a default LLM of gpt-3.5-turbo-0125

The example environment file is the clearest map of what the metric layer can call. It has six provider blocks: OpenAI, Anthropic, Gemini, Cohere, and Azure OpenAI, which needs three separate variables including an API version pinned to `2023-03-15-preview`, an endpoint URL and a deployment name set to `gpt-35-turbo-16k`. Then there is the switch that decides which model the LLM-based metrics use by default:

code
EVAL_LLM="gpt-3.5-turbo-0125"

That default is worth noticing, because it is an older model than most people would choose for judging today, and it is what a fresh install will use until you change it. The mapping between these variables and the dependency list is direct: `openai` is required while `anthropic`, `google-generativeai`, `cohere` and `azure-ai-inference` arrive through extras, so an Anthropic-only user still installs the OpenAI client. That is a small install cost, not a correctness problem, but it explains why the key file lists providers you may never use.

A metric is a callable that takes one datum dict

The smallest useful entry point is a single metric applied to a single record, and the shape is a plain dictionary with fixed keys. Taking retrieval as the example, the datum carries the question, the retrieved context as a list, the ground truth context, the answer, and the ground truths:

python
from continuous_eval.metrics.retrieval import PrecisionRecallF1

datum = {
    "question": "What is the capital of France?",
    "retrieved_context": [
        "Paris is the capital of France and its largest city.",
        "Lyon is a major city in France.",
    ],
    "ground_truth_context": ["Paris is the capital of France."],
    "answer": "Paris",
    "ground_truths": ["Paris"],
}

metric = PrecisionRecallF1()

print(metric(**datum))

The metric is instantiated with no arguments and then called with the dictionary splatted in, so a metric knows the field names it expects. That contract is what makes metrics composable across modules, and it is also what makes a malformed datum fail at the call rather than at some later scoring stage.

The __main__ guard is what makes the parallelization work

Running a whole dataset goes through `EvaluationRunner`, and a system with several stages is expressed as a `Pipeline` of `Module` objects, each with its own `ModuleOutput` and its own selected metrics, with results collected as `PipelineResults`. The library ships a downloader for example datasets, so `example_data_downloader("retrieval")` returns a ready dataset, and pass-fail conditions are objects such as `GreaterOrEqualThan` imported from `continuous_eval.eval.tests`. Metrics come from the matching module namespace, so generation-side checks like `AnswerCorrectness` live under `continuous_eval.metrics.generation.text` while retrieval checks sit under `continuous_eval.metrics.retrieval`, where `RankedRetrievalMetrics` joins `PrecisionRecallF1`. One instruction in the documentation is easy to skip and expensive to skip:

Wrap your evaluation code in a `main` function behind an `if __name__ == "__main__":` guard, because without it the parallelization does not behave correctly.

Deterministic, Semantic and LLM-based metrics are separate extras

The metric families are described as three kinds that you mix: deterministic checks, semantic similarity, and LLM-based judgement. The dependency structure follows that split rather than pretending everything belongs in the base install. The `semantic` extra pulls in `torch`, `transformers`, `sentence-transformers`, `sentencepiece` and `pandas`, and two of those are additionally constrained to Python 3.11 or newer while the package itself allows 3.10. Provider access is a set of smaller extras: `bedrock` for boto3, `azure` for azure-ai-inference, and one each for anthropic, cohere and google-generativeai. The base dependency list is where the unglamorous work lives, with `thefuzz` and `munkres` for string and assignment matching, `rouge` and `nltk` for overlap scoring, `sqlglot` for SQL parsing, `json-repair` for malformed model output, `tenacity` for retries, `tiktoken` for counting, and `jinja2` for the prompt templates that define each check.

Telemetry ships on by default and the payload is in telemetry.py

The last section of the documentation is about analytics, and it is more concrete than most projects are. Basic anonymous usage statistics are collected to understand user preferences, inform new features and identify areas needing improvement, and the exact list of what is tracked is published in the source at `continuous_eval/utils/telemetry.py`. `posthog` is a required dependency rather than an optional one, so the analytics path is compiled into every install. Opting out means setting the `CONTINUOUS_EVAL_DO_NOT_TRACK` flag to `true`. The point to hold onto is that the switch is opt-out, not opt-in. If your evaluation runs in CI or against data you would rather not have leave the machine, setting that variable is a step you have to remember, and reading the telemetry module first is cheaper than assuming the anonymity claim covers what you expect it to.

Newest tag is v0.3.14 from January 2025, main pushed in August 2026

Version history is where the caution belongs. The visible releases are v0.3.11 on 2024-06-18, v0.3.13 on 2024-08-04 and v0.3.14 on 2025-01-10, so the numbering skips 0.3.12 and the last tag is well over a year older than the most recent push to `main` on 2026-08-10. The repository is not archived, so work continued, but there is no release corresponding to it. Development tooling is modern for its age: pytest, pre-commit, ruff with a line length of 80, and `codeflash`, configured with `continuous_eval` as the module root, `tests` as the tests root and pytest as the framework, which is a code-optimisation tool pointed at the source tree. The repository layout is conventional, with `continuous_eval/`, `tests/`, `docs/`, `examples/`, `poetry.lock` and a `.pre-commit-config.yaml`, and the examples directory holds four runnable files covering a single metric, evaluation on a dataset, modular evaluation and custom LLM criteria. Longer reading sits on Medium, with a two-part guide to RAG pipeline evaluation covering retrieval and generation separately, and a separate `relari-ai/examples` repository for end-to-end applications.

Editorial conclusion

continuous-eval fits a team that has already built a LLM pipeline and now needs numbers per stage rather than one end-to-end score, since the module abstraction with per-module metrics and pass-fail tests is the part that generalises beyond RAG. It is not a good fit if you have one prompt and one output, because the setup cost is a pipeline definition and a dataset. Before you install, check your Python version, since the declared range stops below 3.13 and two of the semantic extras need 3.11 or newer, and decide about the analytics before you do, because posthog is a required dependency and the opt-out is a single environment variable you have to set yourself. Then read the telemetry module so you know what is being sent rather than taking the anonymity claim on trust.

Frequently asked questions

How do I install continuous-eval and run its LLM-based metrics?

Install it with python3 -m pip install continuous-eval, or clone the repository and run poetry install --all-extras. The LLM-based metrics need at least one provider key in a .env file, using .env.example as the template, and the default judge model is set by EVAL_LLM, which ships as gpt-3.5-turbo-0125.

What does continuous-eval add beyond a single end-to-end score?

It is built around modular evaluation, where each stage of a pipeline is a module with its own metrics and its own pass-fail tests. Metrics are organised by task, covering RAG retrieval and generation, code generation, agent tool use and classification, and you can mix deterministic, semantic and LLM-based checks.

Does continuous-eval send telemetry, and how do I stop it?

Yes. Basic anonymous usage statistics are collected by default, posthog is a required dependency, and the exact list of tracked items is published in continuous_eval/utils/telemetry.py. Setting the CONTINUOUS_EVAL_DO_NOT_TRACK environment variable to true disables it.

Which Python versions does continuous-eval support?

The declared range is >=3.10 and below 3.13. Two of the optional semantic dependencies, torch and sentence-transformers, narrow that further to Python 3.11 or newer, so a semantic install on 3.10 is not supported even though the base package allows it.

Official sources

  1. License: Apache-2.0
  2. Project website
  3. README
  4. relari-ai/continuous-eval on GitHub
  5. Releases
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/relari-ai-continuous-eval.svg)](https://hysenlabs.com/projects/relari-ai-continuous-eval)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/relari-ai-continuous-eval.svg)](https://hysenlabs.com/projects/relari-ai-continuous-eval)

Was this useful?