LanceZPF/agent-as-a-router: ACRouter for Coding Task Model Routing

What ACRouter Actually Decides

Sending every coding task to the strongest available model is simple and expensive. Sending everything to the cheapest model breaks on the hard cases. ACRouter sits in that gap: it takes one coding task, picks a backend model, runs the task, and checks the answer before accepting it. The README describes the project as routing coding tasks to different backend models under a performance-cost tradeoff, and the shipped benchmark is built to measure exactly that tradeoff.

The intended audience is narrow on purpose. This is for people who already have a list of candidate models with known prices and a way to verify whether a generated patch or solution is correct. The repository is the official implementation of a paper, so the primary reader is someone reproducing or extending that result, not someone shopping for a hosted routing service. The two runtime integrations under claude-code-router/ and cc-switch/ show the second audience: engineers who want the routing decision to happen inside an existing Claude Code or Codex workflow rather than in a standalone script.

The Escalation Loop Behind the Routing Decision

The mechanism is visible in the inference API. You construct an ACRouter with a candidate model list, a cheap_chain of models to try first, an escalate_to model for the fallback, and a k parameter. Then run_with_verifier takes the task, a call_model callable, and a verify callable. The router walks the cheap chain, calls your backend through call_model, hands the response to verify, and escalates to the strong model when verification fails. The returned decision exposes chosen_model and final_response.

That design puts the verifier at the center. The router does not judge code quality itself. It delegates that to whatever function you pass in, which means the routing quality is bounded by the quality of your checks. A weak verifier that accepts broken output will keep the router on cheap models and quietly degrade results. The README's api_coding_solver demo makes the same bet, routing one programming problem through an OpenRouter/OpenAI-compatible model list until a verifier passes.

The benchmark side is a data matrix rather than a live loop. CodeRouterBench is a task-by-model release: id_results_long.csv holds 9,999 ID tasks across 8 models, ood176_results_long.csv holds 176 out-of-distribution tasks across the same 8 models, and models.json carries the canonical backend models with USD pricing. Replay reads raw_matrices/phase2_ood/unified/matrix_acrouter_ood176.json for the OOD176 commands. Cost figures in the output come from those prices, so a stale models.json means your Perf/$ column is wrong even when the routing logic is right.

Installing agent-as-a-router and Running a First Replay

The README puts the install inside an open-source-acrouter directory and uses conda with Python 3.11, which matches the requires-python floor in pyproject.toml. The package installs in editable mode as acrouter-repro.

cd open-source-acrouter

conda create -n acrouter python=3.11 -y
conda activate acrouter

python -m pip install --upgrade pip setuptools wheel
python -m pip install -r requirements.txt
python -m pip install -e .

python -m unittest discover -s tests

The requirements file is heavier than the core reproduction needs. Its own comment states that the core ID and OOD112 reproduction uses only the Python standard library, while OOD176 baselines load sklearn checkpoints and run numpy-based replay. It also pins scikit-learn to the 1.8 line because the saved baseline checkpoints were created with sklearn 1.8.x. Installing outside that minor line is the most likely source of pickle replay warnings.

With the environment in place, the OOD176 run is a single script invocation. Output goes to outputs/tmp/ so the checked-in reference outputs stay untouched.

python scripts/run_acrouter_ood176.py --output-dir outputs/tmp/acrouter_ood176
python scripts/run_baselines_ood176.py --output-dir outputs/tmp/baselines_ood176

The README lists the expected headline lines for these runs, including ACRouter-OOD176 with n=176, AvgPerf=73.30, CumReg=15.9, $Total=86.72 and Perf/$=0.85. Treat those as the reference to compare against your own output directory, not as something this article measured. If you would rather not clone the data, the download script can fetch only what OOD176 replay needs.

python scripts/download_hf_assets.py --minimal --dataset-dir .hf/CodeRouterBench

Adding --with-router-model and --model-dir .hf/router_model pulls the optional trained adapter as well. The replay scripts accept --hf-dataset-dir to run straight from that snapshot.

Where the Offline Story Stops

The clean offline path is also the limit of what the repository proves. Replaying OOD176 exercises saved matrices and saved checkpoints. It does not exercise your backend, your verifier, or your task distribution. A router that scores well on 176 out-of-distribution coding tasks can still be the wrong choice for your repository if your verification is weak or your cheap models fail in ways the benchmark never modeled.

The demo routers are explicitly demos. The commercial CLI router is described in the README as router_mvp.py, and its command templates live in a .example.json file you are expected to edit. The api_coding_solver demo requires you to supply OPENROUTER_API_KEY and to edit models.example.json before it does anything useful. Neither is presented as a production gateway. The two integrations that come closer to production shape are the Claude Code Router and cc-switch additions, and those are described as ready-to-adapt rather than finished.

There is also a packaging detail worth noticing. The project metadata declares the package name acrouter-repro, the imports in the README use acrouter_repro, and the quick start operates inside a directory named open-source-acrouter. Anyone wiring this into a larger system should confirm which of those names their install actually produced before writing import statements against it.

How ACRouter Differs From RouteLLM-Style Classifiers

The repository's own requirements file mentions RouteLLM-BERT paths among the legacy utilities, which makes the comparison concrete. A classifier-style router trains a model to predict, from the prompt alone, whether a cheap model will suffice, then sends the request to one backend. The decision is made before any output exists.

ACRouter inverts that ordering. It commits to a cheap model, generates, and then verifies. The routing decision depends on an artifact that did not exist when the request started. That is the whole point of calling it agentic routing, and it is why the inference API takes a verify callable rather than a threshold. The tradeoff is latency and cost on the cheap path: you pay for generation before you know whether you need to escalate. A classifier pays nothing extra when it guesses right. The README's own headline numbers show the tension, with the ID replay reporting Perf/$=2.25 against 0.85 for ACRouter on OOD176, though those two rows cover different task sets and are not a like-for-like comparison.

Maintenance, Licence and What an Upgrade Costs You

The repository is not archived, and the last push was on 2026-06-29. There are no retrieved releases, so there is no tagged version to pin against; the version string in pyproject.toml is 0.1.0. Without releases, upgrades mean tracking the main branch, and the main branch carries the benchmark data alongside the code. A change to the OOD176 matrix or to models.json pricing would move your replay output without any version number changing.

The licence is MIT, declared both in the LICENSE file at the repository root and in the pyproject.toml license field. MIT is permissive, but this repository bundles benchmark data, reference outputs, and a Hugging Face dataset under a separate identifier, Lance1573/CodeRouterBench. Whether the dataset carries the same terms as the code is not stated in the README, so check the dataset card before redistributing it. Nothing here is legal advice.

The practical upgrade cost is the pinned scikit-learn line. requirements.txt holds scikit-learn>=1.8,<1.9 specifically so saved checkpoints replay without warnings. Moving that pin forward means re-validating the baseline replay, and the repository does not document a migration path for that.

Building Your Own Benchmark Into the Pipeline

The config-driven pipeline is the part most likely to outlive the paper. If you already have task and model results, run_pipeline.py takes a config and accepts either a ready matrix or a tasks.jsonl plus model_results.jsonl pair.

python scripts/run_pipeline.py --config configs/eval_pipeline.example.json

Result rows need task_id and model, plus either resolved or score. Token fields are optional, and the README notes they are what allow cost calculation. That is the constraint to plan around: without token counts you get a performance table with no cost column, which defeats the purpose of a cost-aware router. The examples/custom_benchmark/ directory holds the worked example. For inference-time use instead of offline evaluation, examples/inference_demo.py shows a complete mock workflow against the ACRouter class, which is the fastest way to see the escalation loop without spending on API calls.

Editorial conclusion

Adopt this repository if you want to reproduce a published routing result offline or drop a verifier-gated router into an existing backend call, and you are comfortable reading scripts and matrices rather than a polished CLI. Do not adopt it if you need a supported product with a stable API surface, or if your tasks are not coding tasks, because every artifact here is built around code benchmarks and code verifiers. Before building on it, verify three things yourself: that the OOD176 replay on your machine lands near the headline figures, that your candidate models appear in data/coderouterbench/models.json with the pricing you actually pay, and that the installed scikit-learn minor version matches the line the saved checkpoints were pickled with.

Frequently asked questions

Official sources

1. Issues
2. LanceZPF/agent-as-a-router on GitHub
3. License: MIT
4. README