JevHarness: an LLM writes the harness, then the harness runs without it

Authoring happens once, execution happens without the author

The premise of JevHarness is that a large language model is good at broad reasoning but expensive to consult on every single action. The project splits the work: a strong LLM writes a task-specific harness once, and a smaller, faster model named Jev makes the context-sensitive calls inside that harness at runtime.

The harness turns task observations into useful features, builds the Jev questions and criteria, and combines the structured answers into actions. What the authoring model can change during development is listed plainly: the code, the questions, the graph and the memory. Once the harness is fixed, execution uses that code and its Jev calls, and the authoring LLM is not needed on every decision.

The slogan the readme uses captures the division of labour in three clauses: reason deeply during development, freeze the strategy, let Jev make fast, fuzzy decisions. That is the whole architecture in one line, and the rest of the repository is the machinery around it.

The audience is agent builders who hit the same wall: their LLM agent works, but every action pays for a long reasoning trace. JevHarness is aimed at the case where the decision logic can be pulled out into code and the remaining judgement is narrow enough to hand to a cheaper model.

The task adapter is the boundary the harness cannot cross

The design detail that makes the whole approach defensible is a strict division of ownership between the adapter and the harness.

The task adapter owns observations, legal actions, side effects and scoring. The harness owns feature construction, Jev judgments and decision logic. The readme states the reason plainly: that boundary makes it possible to change the harness without letting it rewrite its own reward or read hidden task state.

That is the risk in any LLM-authored component. A model that can see the scoring function will optimise for the score rather than for the task, and a model that can read state it should not see can produce decisions that no real deployment could make. Putting the reward behind an interface the harness cannot reach turns a prompt-level hope into a structural property of the runtime.

The runtime side of the same idea shows up in the execution model. You validate a PipelineSpec and run it through a PipelineRuntime, and reflection is optional. The pipeline is a declared object rather than a pile of prompt strings, which is what makes it possible to freeze it later and to run the same specification again during evaluation.

GEPA reflection works from complete traces, and keeps the rejected ones

The optional improvement path is an integration with GEPA, and the readme describes its mechanics rather than gesturing at it.

The integration selects parents from an instance frontier, compares parent and proposal on the same training batch, and fully evaluates accepted proposals on Eval. It records actual ancestry, including proposals that were rejected. That last point matters more than it sounds: a system that only remembers its successes cannot tell you whether its search was selective or lucky.

The reflection payload is deliberately complete. Each selected episode contributes its full decisions, observations, node inputs and outputs, Jev questions and answers, memory and failures. Lossless deduplication reduces repetition, and an input that exceeds the configured byte cap is archived and rejected without truncation, so a trace is never quietly shortened into something that changes the meaning of the episode.

Jev answers are typed, which is what lets the harness compose them. Three answer types are supported, choice, score and noul, multiple questions can share a single request, and independent graph nodes can run concurrently. A harness can therefore ask one broad question or several narrow ones, and the choice between them is a property of the authored code rather than a fixed convention.

The package is published on PyPI as auto-jev, version 0.1.0, with the command auto-jev pointing at the CLI entry point. Its pinned dependencies include gepa 0.1.4, httpx, python-dotenv, openai, anthropic, fastapi and uvicorn.

The latency table covers twelve Eval games and says what it does not cover

The performance numbers in the readme come with unusually careful caveats, and reading them as a benchmark would be a mistake.

The selected Pokémon harness makes a full decision in 568 ms median, with individual Jev requests at 269 ms median, in the archived Eval run. The initial harness measured 678 ms median for a full decision, and the selected harness has a 95th percentile of 657 ms against 1,495 ms for the initial one, over 238 and 113 decisions respectively. The Jev request line covers 226 requests at a 348 ms 95th percentile.

What the readme is explicit about: these are recorded successful timings from 12 Eval games per harness, with every included Jev call explicitly marked as missing the local response cache. Full decision time includes feature computation, parallel Jev calls and final action selection, while request time includes client and network overhead. Parallel call durations overlap and should not be added together, and battle simulation and authoring or reflection time sit outside the decision timing.

The readme also names what the experiment was not, saying the observations are not a controlled speed comparison and that the experiment did not benchmark an LLM making every runtime decision. Since parallel calls overlap, the median full decision of 568 ms is not the 269 ms request time multiplied by the number of questions.

The quality claim gets the same treatment. The readme reports that after five reflection rounds the selected harness improved the Eval win rate from 25 percent, 3 out of 12, to 75 percent, 9 out of 12, with the search retaining the round-3 candidate. It then says this is an example result on the selection Eval set and not an independent estimate of performance on unseen games.

The example archive runs from Node, and makes no model requests

The repository ships a browsable archive of a real run, and it is the fastest way to understand what the harness actually does.

The archive contains the actual run, the selected harness, a browsable evolution tree, complete decision records and replay highlights synchronised with their Jev answers. Browsing it makes no model requests and starts no games, so it can be inspected without credentials or budget.

Starting it takes a clone and two Node commands:

git clone https://github.com/TianyuCodings/JevHarness.git
cd JevHarness
node website/build.mjs
node website/preview.mjs --port 8768

The viewer then answers on localhost:8768. It needs Node.js and a modern browser, and the battle animations download assets from the official Pokémon Showdown renderer. The archived battle log stays in the browser and is not uploaded to a replay server, which matters for anyone reasoning about what data a demo leaks.

The page also reflects some care about how results are presented. It shows Train and Eval only, distinguishes sampled training coverage, and includes rejected proposals, so the archive is a record of the search rather than a highlight reel of the winner. Deployment and archive verification are documented separately in the website setup notes.

Three plugin commands, and credentials that must not be confused

There are two installation paths in the documentation: a Claude Code plugin, and manual or Codex options covered in the build-your-own-task section.

The Claude Code route is three commands:

/plugin marketplace add https://github.com/TianyuCodings/JevHarness.git
/plugin install jev-harness@jevharness
/reload-plugins

The credential file is where the care shows. The example environment file keeps the AI gateway key, the TypeSafe key, an Azure-compatible OpenAI key, a separate direct OpenAI key and an Anthropic key apart, and the comments say why: never send the gateway key to the direct TypeSafe API, and keep the direct OpenAI credential separate so gateway keys are not sent to another service. A local Claude login does not need an Anthropic variable.

That separation is the kind of detail that usually gets missed in a research repository, and it is the detail that decides whether a reflection run accidentally bills the wrong account.

The repository also carries a .claude-plugin directory, a skills directory, an auto_jev package directory, configs, scripts, docs, examples and tests. The package configuration names the project auto-jev, requires Python 3.11 or newer, and lists an auto-jev console script. The last push was on 2026-10-16, the repository is not archived, and it has no GitHub releases.

Editorial conclusion

Adopt JevHarness if you are building an agent whose decisions are frequent, latency sensitive and repetitive, and if you can afford an authoring pass plus a task adapter that owns observations, legal actions and scoring. Do not adopt it for a one-off task where writing a harness costs more than the reasoning it saves, and do not adopt it as a dependency without checking the licensing, since no licence file appears among the top-level entries and GitHub does not classify one. Verify first that your adapter really isolates reward and hidden state from the harness, that the Jev endpoint answers the choice, score and noul question types you plan to use, and that the environment variables for the AI gateway are set before you run the reflection loop.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. TianyuCodings/JevHarness on GitHub