TRACER: routing LLM classification to a classical surrogate with a parity gate

The problem TRACER targets: paying a large model for predictable inputs

The README's framing is blunt. Most LLM classification pipelines call the large model for every input, and most of that traffic is predictable. A logistic regression or a small feed-forward net can reproduce the teacher's label on those inputs. TRACER exists to find that subset automatically instead of by hand.

The audience is narrow and specific. You need a classification task with a fixed label set, a teacher model already producing labels, and enough logged traffic that fitting a surrogate is meaningful. The README's own example is Banking77 with 77 intents and 1,500 traces. That is a small trace set by production standards, which tells you the intended entry point is a pilot, not a migration.

What TRACER is not: it is not a prompt optimizer, not a distillation framework that retrains the teacher, and not a general LLM router that picks between models. It routes between one classical model and one teacher, and the teacher's labels are the ground truth the surrogate is measured against.

The acceptor gate, and why the surrogate is not another LLM

The pipeline in the README is four boxes: embedder, ML surrogate, acceptor gate, then a branch. If the acceptor score is at or above the threshold, the local model answers. Below it, the input is deferred to the LLM.

The surrogate is explicitly not an LLM. The default zoo is logistic regression, SGD, and small feed-forward nets. Tree models (decision tree, random forest, extra-trees, gradient boosting) exist but are opt-in behind tracer fit --trees, and the README calls them heavier. That default matters for cost: inference is described as CPU-bound and sub-millisecond, which is the mechanism behind the savings, not a side effect.

The gate is where the design gets opinionated. A surrogate that answers everything would be cheap and wrong. A surrogate that answers nothing is just the LLM with extra steps. The acceptor is a calibrated threshold on the surrogate's own confidence, and the README ties it to a target: config=tracer.FitConfig(target_teacher_agreement=0.95). You are not tuning for accuracy in the abstract. You are tuning for agreement with the teacher on the traffic the surrogate keeps.

One thing the README does not spell out in the excerpt: how the acceptor is calibrated, or what happens to the threshold when the deferred pool drifts. The concepts guide is referenced for the parity gate, so that detail lives outside the front page.

Getting traces, fitting a router, and serving it

There are two entry points and they are ordered. Watch first, fit second.

The observability path wraps any LLM call and records it as an OpenTelemetry GenAI span locally. No account, no key, nothing leaves the machine:

import tracer watch = tracer.watch("support_classifier", system="my-provider", model="my-model")

@watch def classify(ticket: str) -> str: return call_my_llm(ticket)

Spans append to .tracer/watch/*.jsonl. The README states these map 1:1 to TraceRecord, which is what makes the fit step work off watched traffic rather than a separate logging pipeline. If you want them off-machine, Tracer Cloud observability is free and enabled by setting TRACER_CLOUD_KEY, or passing cloud_key to watch. The README describes the upload as batched and non-throwing.

The fit path takes a JSONL file where each line has input and teacher, plus precomputed embeddings as a numpy array:

result = tracer.fit( "traces.jsonl", embeddings=X, config=tracer.FitConfig(target_teacher_agreement=0.95), )

router = tracer.load_router(".tracer", embedder=embedder) out = router.predict("What is my balance?")

predict returns a dict with label, decision, and accept_score. When decision is handled, the surrogate answered. The fallback kwarg is a callable invoked only on deferral, so the LLM call site stays lazy.

For JavaScript there is no in-process router. The README's pattern is: log traces to JSONL, fit offline with the CLI, run tracer serve as a sidecar, POST the embedding to http://localhost:8000/predict, and call the LLM yourself when decision is deferred. The embedding sent at inference must be from the same model used at fit time. That constraint is stated and it is not optional.

The embedding dependency is the real operational constraint

TRACER does not embed text for you at fit time. You pass embeddings=X, an (n, dim) numpy array. At serve time you pass an embedder to load_router. The README's JavaScript example says it plainly: same model you used at fit time.

This is the failure mode most likely to bite. Swap the embedding model, or upgrade a hosted embedding endpoint that silently changes its output dimension or normalization, and the surrogate is reading vectors from a different space than the one it was trained on. Nothing in the routing policy would catch that. The acceptor gate scores the surrogate's confidence, not whether the input vector is in-distribution with respect to the embedder.

Version 0.2.0 added an OOD safety gate, per the release title. That is the right direction, but the README excerpt does not describe what it compares against or how it is configured. If you are evaluating TRACER, that is the first thing to read in the concepts guide, because an OOD gate that depends on the same embedding model will not save you from an embedder swap.

There is a second constraint the README implies but does not dwell on. The teacher labels in traces.jsonl are treated as correct. If the teacher is wrong on an input, and the surrogate learns to agree with it, the parity metric goes up while the actual task accuracy goes down. TRACER measures agreement with the teacher. It does not measure correctness against human labels, and the README does not claim it does.

What the demo numbers do and do not tell you

tracer demo prints a routing policy block: method l2d, coverage 91.4%, teacher TA 0.920. Then a cost projection for 10k queries/day, showing 10,000 LLM calls/day at $20.00/day dropping to 863 calls/day at $1.73/day, with $6,670 saved/yr.

The coverage and agreement numbers come from a bundled dataset (Banking77, 1,500 traces) and a fixed split. They are a demonstration that the pipeline runs, not a forecast for your traffic. The cost projection is arithmetic on top of the coverage number, and it assumes a specific per-call price that the README does not state. If your LLM call costs a tenth of that, the dollar figure scales down proportionally and the coverage number does not change.

The number worth watching in your own run is teacher TA, not coverage. Coverage is easy to inflate by lowering the acceptor threshold. Teacher TA is the constraint that stops you. FitConfig takes target_teacher_agreement, and the 0.95 in the quickstart is a choice, not a default the README claims. Set it deliberately.

One thing the README does not give: a held-out certification example. Version 0.2.0's release title mentions held-out parity certification, and the concepts guide is referenced for the parity gate, but the front page shows only the demo output. Treat the demo's 0.920 as illustrative.

Where a plain fine-tuned classifier or a semantic cache fits better

The obvious alternative is fine-tuning a small transformer on the same teacher labels and serving it directly, with no gate. The difference is in what you get back. A fine-tuned classifier returns a label and a softmax score. TRACER returns a label, a decision, and an accept_score, plus a deferral path that keeps the LLM reachable for the inputs the surrogate is unsure about. If your traffic has a long tail of inputs the small model handles badly, the gate is the whole point. If your traffic is uniform and the small model is good everywhere, the gate adds a threshold to tune and a branch to maintain for no benefit.

A semantic cache is the other comparison. Caches match on input similarity and return a previously seen answer. They work when inputs repeat. TRACER's surrogate generalizes across inputs that were never seen, which is the case the README is built around: the majority of traffic is predictable, not identical. A cache also has no agreement guarantee against the teacher. It has whatever was stored.

The honest boundary: if your label set changes often, or your task is generative rather than classificatory, TRACER does not apply. The README is explicit that the surrogate is a classifier over a fixed label set. There is no path in the material for open-ended outputs.

Maintenance, refits, and the MIT licence

The self-improving claim rests on a loop: every deferred call produces a new trace, and that trace feeds the next refit. Coverage grows as the deferred pool gets labeled. The README states this as a property of the design. It does not describe a scheduler, a drift trigger, or an automatic refit cadence, so the loop is something you operate, not something that runs itself.

Practically, that means a refit job, a place to accumulate traces, and a way to promote a new router without breaking the embedding contract. The CLI has tracer fit and tracer serve, and the JavaScript guide is referenced for continual learning, but the excerpt does not show the promotion step. Budget for writing that yourself.

On licensing: the repository is MIT. That permits commercial use and modification with the licence and copyright notice retained. Tracer Cloud is a separate hosted service with its own terms, and the README notes the observability tier is free, which is a pricing statement that can change independently of the MIT code. If your compliance review cares about where traces go, the local watch path keeps them on disk and the cloud path sends them to a third party. Those are different data flows and the README treats them as a single env var apart. This is not legal advice; check the terms of the cloud service separately from the repository licence.

Editorial conclusion

Adopt TRACER if you already log input and teacher label pairs for a high-volume classification endpoint and can supply the same embedding model at fit and serve time. Do not adopt it if your labels change weekly, if you cannot pin an embedder, or if you need routing logic inside JavaScript rather than behind an HTTP sidecar. Before fitting anything, run tracer watch for a few days and check that the trace volume and label distribution look like production, then run tracer fit with target_teacher_agreement set and read the held-out certification output rather than the demo coverage number.

Official sources

1. adrida/tracer on GitHub
2. Issues
3. License: MIT
4. README
5. Releases