Model or dataset
truera/trulens avatar
truera/trulens

TruLens scores OpenTelemetry spans with LLM judges, on a 2.x wheel still labelled alpha

Project brief: Evaluation and Tracking for LLM Experiments and AI Agents. Tracing is OpenTelemetry-native, so a trace is portable to any OTLP backend, and evaluations run either as traces land or over a dataset after the fact.

3,660 stars398 forksPythonMIT

At a glance

What is it?
TruLens instruments a Python app with a decorator, exports OpenTelemetry spans to any OTLP backend, and grades each span with LLM judges. The PyPI wheel bundles six packages including a dashboard, the release line still carries an alpha classifier, and every metric is a paid provider call.
Who is it for?
TruLens fits one shape of work: a Python app already shipping OTLP spans, where paying a provider per evaluated span costs less than locating the failing step by hand. Check three things first, whether the wheel you pin is the version on main, whether the provider you need has a package, and what one offline batch costs at eight invocation workers.
Can I use it commercially?
Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 2 days ago.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 17, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • truera/trulens3,660 ★ · +177
Star history
Repository2026-08-042026-11-17
truera/trulens3.5k3.7k

177 Stars in the last 105 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

Main reads 2.15.0 while PyPI still serves 2.14.0

`pyproject.toml` names the package version 2.15.0, and the newest published release is trulens-2.14.0, cut on 3 September 2026. Two more minors sit behind that, 2.13.1 from 20 August and 2.12.0 from 6 August, so main sits one release ahead of whatever pip resolves, and the last commit landed on 4 October 2026. What sits next to the version is less encouraging. The classifier block still reads `Development Status :: 3 - Alpha` on a line that already covers agent tracing, and the shipped code comes from the `_trulens` directory rather than `src`, which leaves `src` to the benchmark group, `tools/` and `tests/`. An alpha marker on a 2.x package that drags in a dashboard means the exact wheel you pin matters more than the version range suggests, and `zip_wheels.sh` at the root is there to build the set.

`pip install trulens` pulls in a dashboard and no provider

The base install is a bundle, not a library. The dependency list pulls in trulens-core, trulens-feedback, trulens-dashboard with its `full` extra, and trulens-otel-semconv, plus trulens_eval carrying the inline comment `# Remove after deprecation period`. The OTLP exporter is optional and parked behind the `otlp` extra, so a plain install gives you a dashboard and no way yet to ship spans anywhere. Python is pinned at `^3.10`. Judges are a separate decision: the provider installs sit apart from the base one because a provider package is what turns on feedback evaluation.

bash
pip install trulens trulens-providers-openai   # OpenAI / Azure OpenAI
pip install trulens trulens-providers-orcarouter  # OrcaRouter (OpenAI-compatible gateway)
pip install trulens trulens-providers-litellm  # LiteLLM (Anthropic, Cohere, Mistral, …)
pip install trulens trulens-providers-google   # Google Gemini
pip install trulens trulens-providers-bedrock  # AWS Bedrock
pip install trulens trulens-providers-cortex   # Snowflake Cortex
pip install trulens trulens-providers-huggingface  # HuggingFace
pip install trulens trulens-providers-langchain    # LangChain models

Seven provider packages are on offer. Framework integrations come as two more:

bash
pip install trulens trulens-apps-langchain    # LangChain / LangGraph
pip install trulens trulens-apps-llamaindex  # LlamaIndex

So the ratio the repo asks you to install is seven evaluation backends against two ways to wrap an app.

Span attributes are argument names, not values

Instrumentation is a decorator, and the attribute values are names the recorder looks up rather than text to store. `SpanAttributes.RETRIEVAL.QUERY_TEXT` is set to the string `query`, the parameter on `retrieve`, and `RETRIEVED_CONTEXTS` is set to `return`, the return value. The decorator therefore needs no wrapper around your function, and every recorded call depends on those strings matching the signature.

python
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes

class MyRAG:
    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list: ...

The span type is the other half of the decision. `SpanAttributes.SpanType.RETRIEVAL` marks retrieval; a second one-liner passes `SpanAttributes.SpanType.MCP` to capture Model Context Protocol tool name, arguments, output and latency. Nothing above shows where a recorder or an app object comes from, and the Quick Usage pointer is a Colab notebook that builds a RAG from scratch, so the ends of the API are public while the middle lives in that notebook.

Seven agentic evaluators, each one a separate judge call

Agentic scoring is a fixed table of seven evaluators: LogicalConsistency for reasoning coherence and unsupported assertions, ExecutionEfficiency for redundant steps and wasted retries, PlanAdherence for whether execution followed the stated plan, PlanQuality for the intrinsic strategy rather than the outcome, ToolSelection for the right tool per subtask, ToolCalling for argument validity and output interpretation, ToolQuality for external service reliability. Splitting plan quality from plan adherence is the part that matters when a trace goes wrong, because the two fail for different reasons.

python
# Inline — evaluate as the app runs
with tru_recorder as recording:
    response = my_app.query("What is TruLens?")

# Batch — evaluate a pre-collected dataset using the Run API
from trulens.core.run import RunConfig

run_config = RunConfig(
    run_name="batch_eval_v1",
    dataset_name="eval_questions",
    source_type="TABLE",
    dataset_spec={"input": "QUESTION"},
    invocation_max_workers=8,
    metric_max_workers=4,
)
run = tru_app.add_run(run_config=run_config)
run.start()
run.compute_metrics([relevance, groundedness])

The batch path reads a stored dataset, and the two worker counts are the cost dial: `invocation_max_workers=8` with `metric_max_workers=4` decides how many provider calls sit in flight. `compute_metrics([relevance, groundedness])` names two metrics the README never defines or imports, and it is the only line that shows what scoring costs per invocation.

Metrics are objects, and `provider` is never defined

The Selector API is how a metric says which span fields it reads. A `Metric` takes a name, an `implementation` taken off a provider object, and a `selectors` mapping from slot name to selector function, so `Selector.select_record_input()` supplies the record input and `Selector.select_context()` supplies the retrieved text a context relevance score needs.

python
from trulens.core import Metric, Selector

f_context_relevance = Metric(
    name="Context Relevance",
    implementation=provider.context_relevance,
    selectors={
        "input": Selector.select_record_input(),
        "context": Selector.select_context(),
    },
)

That `provider` is never constructed anywhere in the README, and no connection string or recorder name appears either, so the object graph has a missing root. There is a `docker/` directory and a Makefile at the root, but no documented way to stand up the dashboard the base install already depends on.

Two of the four benchmark rows cite one ranking page

The results table is the strongest thing in the README and the part that rewards close reading. Four rows. 95% agent errors caught with Agent GPA on TRAIL/GAIA, which is 267 of 281 human-annotated errors against 55% for a baseline trace judge, cited to arXiv:2510.08847. Groundedness F1 of 0.81 on LLM-AggreFact, ahead of a fine-tuned Bespoke-MiniCheck-7B on F1, precision and recall over an 11,000-example holdout, cited to a Snowflake engineering blog. Context relevance NDCG@5 of 0.93 and an adversarial win-loss of 4.2:1 over a near-copy with one fact swapped, against 3.3:1 for the next best tool, both cited to the same AIMultiple page dated 23 March 2026.

Two things follow. The last two rows are two columns of one third-party comparison rather than two independent results. And the groundedness number is hosted by Snowflake while the author line in `pyproject.toml` reads `Snowflake Inc. <[email protected]>`. The comparison names WandB Weave, RAGAS, DeepEval and UpTrain as the other four tools, which is why the repo keeps turning up in searches for alternatives.

The provider table ends mid-row, adoption is a link out

The Supported LLM Providers table is the shortest section in the README and it stops partway through a row: OpenAI / Azure OpenAI, LiteLLM (Anthropic, Cohere, Mistral, and more), Google Gemini, AWS Bedrock, and then the Snowflake entry starts with its package name cut off. The install block above it is the fuller inventory, adding OrcaRouter described as an OpenAI-compatible gateway plus HuggingFace, and the LiteLLM comment inside that block ends in an ellipsis of its own. Nothing in the install lines says which LangChain major version `trulens-apps-langchain` targets; the answer lives in `LANGCHAIN_1X_MIGRATION.md` at the root, beside `DEPRECATION.md` and the trulens_eval package still installed as a dependency.

The same pattern runs through the rest of the claims. The adopter list, from Walmart Global Tech and Cisco through J.P. Morgan Chase, Equinix, VMware by Broadcom, Hitachi Digital Services, Thomson Reuters, phData and HID Global, is a paragraph that points at `ADOPTERS.md` rather than a table anyone can check here. Governance lives in `GOVERNANCE.md`, `POLICIES.md`, `CLA.md`, `MAINTAINERS.md` and `CONTRIBUTOR_LADDER.md`.

The contributor recipe needs Poetry 2 and a broken fallback

Contributing runs on Poetry 2, and the reason sits in `pyproject.toml` as a comment: Poetry 1.8 cannot read the committed lock-version 2.1 file, and its lock command re-resolves the whole graph instead of preserving pins, so the build declares `requires-poetry = ">=2.0"`. The Makefile wants `gmake` for a related reason, since `.ONESHELL:` runs every command in a target inside one shell session and the note above it says the default make on macOS is too old.

Three smaller things sit in the same file. `clean-caches` ends its second command with `|| t`, which is not a shell command, so a failing `poetry cache clear` surfaces as a missing binary rather than the intended message. `env-tests` installs a second copy of pytest, ruff and pre-commit by name, unpinned, on top of the versions already pinned in the dev group. And `TOKEN ?= $(shell echo $$TOKEN)` pulls a token out of the environment into the Makefile, while `LAST_TRULENS_EVAL_COMMIT := 4cadb05` pins the commit holding the last pre-namespace trulens_eval package.

Editorial conclusion

TruLens fits one shape of work: a Python app already shipping OTLP spans, where paying a provider per evaluated span costs less than locating the failing step by hand. Check three things first, whether the wheel you pin is the version on main, whether the provider you need has a package, and what one offline batch costs at eight invocation workers. An alpha marker on a 2.x line, a benchmark table where two rows share a single citation, and a deprecation file still pinned as a runtime dependency all argue for pinning exactly and reading the numbers yourself. The tracing layer itself is ordinary OpenTelemetry work and carries no such caveat.

Frequently asked questions

What is TruLens?

A Python library that wraps the functions you care about in an @instrument decorator, records each call as an OpenTelemetry span, and scores spans with LLM judges. The package is MIT licensed, built with Poetry from the _trulens directory, and published as trulens on PyPI.

Is TruLens open source?

Yes. pyproject.toml declares the MIT license and a LICENSE file sits at the repository root, along with CONTRIBUTING.md, GOVERNANCE.md and MAINTAINERS.md. The judges themselves are not free: each metric is a call to a provider package you install separately.

Is TruLens free?

The library is free to install with pip install trulens, but scoring is not free. Every evaluator is a model call against a provider package such as trulens-providers-openai or trulens-providers-bedrock, so the bill scales with how many spans you record and how many metrics you run over them.

How do you use TruLens on a RAG app?

Decorate the retrieval step with @instrument, set span_type to SpanAttributes.SpanType.RETRIEVAL, and name your query parameter and return value in the attributes dict. Execution then happens inside a recorder, and scoring happens either inline or through a Run built from a RunConfig. The Quick Usage pointer leads to a Colab notebook that builds a RAG from scratch.

What are the alternatives to TruLens?

The repository names four in one benchmark row: WandB Weave, RAGAS, DeepEval and UpTrain. It carries no feature-by-feature comparison against any of them, and framework support ships as separate packages (trulens-apps-langchain, trulens-apps-llamaindex), so the switching cost sits mostly in how your traces are exported.

Official sources

  1. Official documentation
  2. Official README
  3. Project repository
  4. Release notes
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/truera-trulens.svg)](https://hysenlabs.com/projects/truera-trulens)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/truera-trulens.svg)](https://hysenlabs.com/projects/truera-trulens)

Was this useful?