Agent-as-a-Judge: an LLM judge that inspects the workspace, not just the answer

The gap Agent-as-a-Judge is built to close

Most automated evaluation of coding agents reduces a run to a final artifact: did the tests pass, does the file exist, does the output match a reference. That works when the task has one correct endpoint. It fails on open-ended development work, where the deliverable is a repository, the requirements are hierarchical, and a plausible-looking solution can satisfy a surface check while missing the point. The README frames the project around exactly that open-endedness, and the paper's framing (Evaluate Agents with Agents) makes the claim explicit: the judge should itself be an agent.

The audience is narrow but real. If you are training or benchmarking a developer agent and you need a reward signal denser than pass or fail, this is the target use case. The README states the motivation in cost terms, claiming the approach saves 97.72% of time and 97.64% of costs compared to human experts. Treat those figures as the authors' measurement on their own setup, not a property you inherit. Your judge model, your workspace size and your number of requirements will set your own number, and the repository gives you no tool to estimate it before you run.

How the judge gathers evidence before it scores

The mechanism is visible in the script names. run_aaaj.py takes a developer agent, a setting such as black_box, a planning mode, and a benchmark directory. The judge is not handed the diff and asked for a verdict. It is pointed at a workspace and left to inspect it, which is why the repository ships a separate Ask Anything entry point, run_ask.py, that takes a workspace path and a free-form question. That script is the same exploration machinery without the scoring wrapper, and the README points to assets/ask_sample.md as an example of what the exploration produces.

The second half is the benchmark. DevAI contains 55 realistic AI development tasks carrying 365 hierarchical user requirements, and the judge's job is to decide how many of those requirements the developer agent met. Hierarchy matters here: a top-level requirement can be satisfied while its sub-requirements are not, so the output is not a single bit. The README describes the result as continuous, step-by-step feedback usable as reward signals for further agentic training. That is the architectural bet, and it is a bet about granularity rather than about accuracy.

OpenWiki is the same machinery pointed at a different target. The README says it was built by adding two codes to this repository and that it is an open-sourced DeepWiki; the usage is a single command taking a GitHub URL. It is worth noting how thin that claim is in the material. There is no description of which two files change or what the wiki generator does with the evidence the agent collects, so the honest reading is that OpenWiki is a demonstration of reuse, not a documented product.

Getting it running: conda, poetry, and a .env you must edit

The install path is conventional. Clone the repository, create a Python 3.11 conda environment, install poetry, then run poetry install. The README gives no lockfile caveats and no note about platform differences, so the assumption is a standard Linux or macOS shell.

Model access is configured through environment variables. You copy .env.sample to .env and fill in the required APIs and settings, with OPENAI_API_KEY shown as the example value. The README states that LiteLLM handles the LLM calling, which is the detail that matters most for adoption: it means the judge model is not pinned to one vendor in the code, and you can point it at whichever provider your keys cover. Which variables beyond the API key are required is not spelled out in the README, so you will be reading .env.sample itself to find out.

There are three entry points. Ask Anything runs run_ask.py with a workspace path and a question, and the README's example targets a workspace under benchmark/workspaces/OpenHands. The judge run is run_aaaj.py with --developer_agent, --setting, --planning and --benchmark_dir; the documented values are OpenHands, black_box, and efficient (no planning). OpenWiki runs run_wiki.py with a repository URL. All three are invoked with PYTHONPATH=. from the repository root, which tells you the scripts are not packaged for installation as a library. The README also points to scripts/README.md for more comprehensive usage scripts, so the three commands above are a starting subset rather than the full surface.

Where the design strains

The judge is an LLM agent exploring a workspace, which makes it nondeterministic in a way a test runner is not. Two runs over the same workspace with the same developer agent can plausibly produce different scores, and nothing in the README describes a variance check, a repeat protocol, or a confidence interval. For a reward signal feeding training, that noise is not a footnote. It is the thing you have to characterise before you trust a score difference between two checkpoints. The repository gives you no built-in way to do it.

Cost is the second strain, and it cuts against the headline savings. The 97.72% and 97.64% figures compare the judge to human experts. They do not compare the judge to a cheap deterministic check, and they do not tell you what the judge itself costs per evaluation. An agent that explores a workspace, reads files and reasons about 365 requirements burns tokens in a way that a pass/fail test does not, and the README offers no per-task cost estimate. If your loop runs thousands of evaluations, that multiplies.

There is also a boundary problem the README does not address. The judge inspects what is in the workspace. How it handles a developer agent that wrote convincing comments describing work it never did, or that satisfied a requirement in a way the original task author did not intend, is not discussed. An agentic judge that reads code is better positioned against that than a string matcher, but the material makes no claim about adversarial robustness, so do not assume it.

The realistic alternative, and the real difference

The obvious alternative is a deterministic harness: unit tests, type checks, linters and a scoring script that counts passing assertions. The difference in approach is not accuracy, it is what each can express. A test harness gives you a reproducible number and costs almost nothing per run, but it can only check what someone wrote an assertion for. The DevAI requirements are natural-language and hierarchical, and many of them (does the dashboard present the right information, does the pipeline handle the stated edge case) have no crisp assertion. That is the space this project occupies. If your evaluation targets can be written as tests, write the tests; the judge is overhead you do not need.

LLM-as-a-judge is the nearer alternative, and the distinction the README draws is the point of the whole repository. A conventional LLM judge is typically handed a response, a reference answer and a rubric, and it scores the text. Agent-as-a-Judge is handed a workspace and gathers its own evidence before scoring. For code generation that difference is substantive: the judge can look at the files rather than at a summary of the files. The cost is everything that comes with giving a model tools, namely nondeterminism, latency and a larger bill. If your tasks are short and self-contained, a plain LLM judge over the final answer will be cheaper and probably sufficient.

Maintenance, licensing and what the repository does not promise

The license is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are preserved. That is the standard reading, and it is not legal advice; if you are embedding the judge in a product, have counsel confirm how MIT interacts with your dependencies and with the model provider's terms, since the license covers this repository and not the LLM you point it at.

On maintenance, the evidence is limited. There are no releases retrieved, so versioning is by commit and your poetry install resolves against whatever the lockfile pins at the time you clone. The repository is not archived, and the last push is dated 2026-03-28. The README announces acceptance at ICML 2025, which tells you the work was peer reviewed but says nothing about whether the code will be maintained as a library. Plan for reading the scripts rather than importing them.

The upgrade cost is concentrated in one place. Because the judge's behaviour lives in prompts and in the exploration loop, and because the README does not document a prompt versioning scheme, any change you make to the prompt files changes your scores. If you use this for training, record the exact commit hash and the .env configuration alongside every evaluation run, or you will not be able to tell whether a score moved because your agent improved or because the judge changed. The DevAI dataset lives on Hugging Face under DEVAI-benchmark, separate from this repository, and the README points to benchmark/devai/README.md for usage guidelines, so dataset updates and code updates can drift apart.

Editorial conclusion

Adopt it if you are building or training a coding agent and need automated scores against a benchmark like DevAI, and you accept that the repository is a research artifact under MIT rather than a supported product. Do not adopt it if your evaluation needs a stable scoring contract across model upgrades, or if you cannot fund the judge's own token and exploration cost. Before committing, run the Ask Anything script over one of your own workspaces, then edit the prompt files it loads and confirm the score actually moves; if it does not, the judge is not measuring what you think it is.

Official sources

1. Issues
2. License: MIT
3. metauto-ai/agent-as-a-judge on GitHub
4. Project website
5. README