leanctx: loss-tolerance routing for prompt compression in Python LLM apps
Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.
At a glance
- What is it?
- leanctx is an MIT-licensed Python wrapper that compresses prompts before they reach OpenAI, Anthropic or Gemini, routing each segment by how much distortion it can survive. The design bet is sound, but the accuracy cost is uneven across task types and the headline savings require a second compressor underneath.
- Who is it for?
- Adopt leanctx if you run RAG or agent workloads where retrieved passages, logs and prior turns dominate the prompt and you already accept some accuracy loss for cost reduction. Do not adopt it if your traffic is short-context single-document QA, since the project's own LongBench v2 sub-buckets show the largest accuracy regressions there.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 24 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem leanctx targets: dynamic prompt content that caching cannot reach
Provider prompt caching wins on stable prefixes. System prompts, tool definitions and retrieved-document pools are reused across requests, so Anthropic, OpenAI and Gemini can bill them at a discount. The README is explicit that caching does not help with dynamic per-query content: chat history, freshly retrieved documents, tool outputs. Those change every call, and they are where the input-token bill accumulates in a long-running agent.
The alternatives each fail in a specific way. Naive truncation drops the middle of a document, which the README says is exactly where many answers live. Hosted compression APIs such as Compresr and Token Company require sending context to a third party. leanctx's positioning is that it runs the model locally and makes no outbound calls except to the provider you already use. The intended users are RAG apps with large retrieved documents, LangChain or LangGraph agents with growing tool-call histories, document-processing pipelines, and coding agents with long tool-call traces.
Loss-tolerance routing: verbatim, LLMLingua-2, or an opt-in self-LLM pass
The mechanism is a classification step that runs before the request leaves your process. Each segment of the prompt is sorted into one of three classes. Zero-tolerance content (code, stack traces, tool_use_id, tool name and input, JSON) is passed through byte for byte. High-tolerance content (documentation, retrieved passages, logs, prior turns) goes through LLMLingua-2 on-device, which the README describes as removing roughly 50 percent. A conditional class covers low-confidence prose and oversized context, and is routed to a self-LLM pass that is opt-in and reported at 41 to 49 percent removal.
After compression the segments are recomposed and checked against invariants. If the check fails or times out, the original prompt goes to the provider instead. That fallback path is the part that makes the routing claim credible: the design accepts that compression is lossy and defines a route to skip it.
The README states the underlying argument directly: compression applied uniformly will eventually damage the one part of a prompt that cannot survive being touched, and will do so unpredictably. v0.3.1 is titled "preserve tool_result code/traceback verbatim", which suggests this class boundary needed tightening after the initial release.
Installing leanctx and the config keys that control routing
The install is a single pip command with extras per provider: pip install 'leanctx[openai,lingua]', or [anthropic] or [gemini] instead. The lingua extra is what pulls in LLMLingua-2. Without it, the README says leanctx falls back to passthrough, so a missing extra degrades silently to no compression rather than raising an error.
Usage replaces the provider import. from leanctx import OpenAI gives the same interface, with a leanctx_config dictionary passed to the constructor. Three keys appear in the README example: mode set to "on", trigger with threshold_tokens set to 2000, and routing with prose set to "lingua". The trigger means short prompts are not compressed at all, which matters because the sidecar adds latency.
Response usage carries two added fields: response.usage.leanctx_tokens_saved and response.usage.leanctx_ratio. The README example prints 1841 and 0.49 respectively. The first Lingua call loads about 1.2 GB of model weights into ~/.cache/huggingface/, and later calls reuse that cache. That download is a real deployment consideration for container images and cold starts.
Two verification commands run without an API key. leanctx bench list reports 7 registered scenarios. leanctx bench run agent-structural --workload agent enforces 5 invariants and exits 0 on pass. The bench CLI arrived in v0.3.0 alongside OTel observability.
What the LongBench v2 numbers actually measure, and what they do not
The headline figure is an extra 18.7 percent token removal on top of ClawRouter's seven structural compression layers, rising to 36.7 percent on prose-heavy traffic, at a cost of 1.8 percentage points of accuracy. The setup matters more than the number. Leg A is ClawRouter alone at 26,397 average tokens per request against 27,865 raw, a 5.3 percent reduction. Leg B adds leanctx as a semantic eighth layer and lands at 21,470 tokens, 23.0 percent below raw and 18.7 percent below Leg A. So the 18.7 percent is a marginal gain on an already-compressed pipeline, not a saving against an uncompressed one.
The accuracy breakdown is where the project is unusually honest. Overall accuracy falls from 45.3 to 43.5 percent across 503 questions. Of those, 275 routed to verbatim and show 0.0 pp change by construction. The 228 that routed to lingua fall 3.9 pp. Because 54.2 percent of Layer-8 input tokens route to verbatim, the compression actually applied to eligible content is 40.8 percent. The README states plainly that compression costs accuracy and does not add it.
The sub-buckets are not uniform and the report publishes all of them. short/lingua is minus 17.6 pp on N=68, and Single-Document QA is minus 11.5 pp on N=78. Long Structured Data Understanding is plus 9.4 pp on N=32. A positive result on one bucket out of many is not evidence that compression improves reasoning; it is more likely variance on a small sample. Sidecar latency is given as 47 ms p50 on GPU, and savings at Sonnet input pricing are about $78 per 1,000 requests. The benchmark was executed and audited by outside contributors, and an audit of an earlier draft found that a headline "+7.4 % on long context" was eval noise. That correction is disclosed in the README.
Where leanctx is the wrong tool
The sub-bucket data points at the failure mode. If your traffic is short-context single-document question answering, the measured regression is 11.5 pp on N=78, and the short/lingua bucket is worse at 17.6 pp on N=68. leanctx's routing sends code, JSON and tool payloads through untouched, but a short prose question with one retrieved passage has little zero-tolerance content to protect and a high proportion of compressible text. The savings will look good and the answers will get worse.
The second constraint is the trigger. With threshold_tokens at 2000 in the README example, anything shorter is not compressed at all, so the tool does nothing for short-prompt workloads. The third is the dependency on a second compressor. The 18.7 percent figure assumes ClawRouter or an equivalent structural layer is already in place. Run leanctx alone against raw prompts and the delta will be larger in absolute terms but measured against a different baseline, so the published number does not transfer.
The fourth is latency and memory. A 1.2 GB model download on first use, plus 47 ms p50 sidecar latency on GPU, is a different profile from a hosted API call. On CPU that figure is not reported in the material, and the README does not give one. Treat GPU latency as the only supported number.
Alternatives: prompt caching, truncation, and hosted compression APIs
The README names three alternatives and describes the difference in approach for each. Provider prompt caching (Anthropic, OpenAI, Gemini) discounts stable prefixes and is complementary rather than competing; the README's advice is to compose the two, not choose. Naive truncation drops the middle of a document, which is where the README says many answers live, and the LongBench v2 results are presented as concrete evidence of that. Hosted compression APIs such as Compresr and Token Company take the opposite architectural position: the context leaves your infrastructure and the compression model is closed. leanctx runs LLMLingua-2 locally and is MIT-licensed, so the trade is your own compute and a model download against a per-call fee and data egress.
ClawRouter is the more interesting comparison because the benchmark layers leanctx on top of it. ClawRouter's seven layers are structural, meaning they operate on the shape of the request rather than on semantic content. leanctx's Layer 8 is a semantic pass. The two are not substitutes: the reported 18.7 percent is what the semantic pass adds after the structural pass has already run. If you are choosing one, the structural layer alone gives 5.3 percent at no measured accuracy cost in this setup, which is a different risk profile from 18.7 percent at 1.8 pp.
Maintenance, licensing and what the release history implies
The licence is MIT, which permits commercial use, modification and redistribution with attribution and without a copyleft obligation. That is a meaningful difference from hosted compression APIs, where the model is closed and the terms are contractual rather than a licence file. This is not legal advice; read the LICENSE file in the repository before relying on it.
The release history is short and recent. v0.3.0 added OTel observability and the leanctx bench CLI. v0.3.1, released the same day, is titled "preserve tool_result code/traceback verbatim", which reads as a correctness fix to the zero-tolerance class rather than a feature. A patch release that tightens what gets passed through verbatim suggests the classifier's boundary was wrong in a way that affected tool output. If you run coding agents, that is the release to check against your own traces.
The upgrade cost is concentrated in two places. The lingua extra pins you to LLMLingua-2 and its model weights, so any change to the upstream compressor is an upgrade you have to validate. The routing config is a small surface (mode, trigger, routing), so config churn is unlikely to be the problem. The real cost is re-running the benchmark after any change to the classifier, because the accuracy delta is bucket-dependent and a routing change can move tokens between the verbatim and lingua classes without changing the headline savings number.
Editorial conclusion
Adopt leanctx if you run RAG or agent workloads where retrieved passages, logs and prior turns dominate the prompt and you already accept some accuracy loss for cost reduction. Do not adopt it if your traffic is short-context single-document QA, since the project's own LongBench v2 sub-buckets show the largest accuracy regressions there. Before committing, run leanctx bench run agent-structural to confirm the five invariants pass on your install, then replicate the layered measurement on your own traffic rather than trusting the 18.7 percent figure, which was produced on top of ClawRouter's seven structural layers and not against a raw baseline.
Community notes