GVS5H: Ledger-Based Self-Orchestration for Coding Benchmarks
GVS5H: Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard | Fable 5 Level Coding for a Fifth the Price - or on a Single GPU
At a glance
- What is it?
- GVS5H is a research repository that runs multiple fresh instances of one model against LiveCodeBench Hard, coordinating them through a shared filesystem ledger. The README reports large gains for some models and no gain or losses for others, and the code is a benchmark harness rather than a product.
- Who is it for?
- GVS5H is for researchers and engineers who want to reproduce the paper's multi-agent versus single-call comparison on their own backend, and who are comfortable reading the harness code because the README documents only one command line. It is not for anyone looking for a library to import, a serving stack, or a coding assistant, since the repository is a benchmark runner plus a paper and run data.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What GVS5H actually is, and who it is for
The repository is a benchmark harness plus a paper. Its abstract describes "ledger-based zero-shot self-orchestration, a training-free method in which fresh instances of one model decompose problems and coordinate through a shared filesystem holding a plan, notes and current solution." The experiment runs that method and a single-call baseline against the 100 latest hard LiveCodeBench problems across nine open and closed-weight models.
The intended reader is someone who wants to check the claim rather than use a tool. There is no service, no CLI installed on your PATH, no library to import. The README's only usage section is a command that runs a benchmark and prints a pass@1 score. If you want an agent framework to build on, this is not it; if you want to see whether orchestration buys accuracy on your own model and backend, the harness is the point.
The framing in the title, five Qwen3.8 models matching Claude Fable 5, is a result from one configuration on one benchmark slice, not a general property of the code.
How the ledger and the manager work
The mechanism is deliberately unglamorous: a shared filesystem. Fresh instances of the same model write into a workspace that holds "a plan, notes and current solution," and the manager coordinates them. The README does not publish the ledger file format or the message protocol, so anyone wanting to modify the coordination logic has to read escalation/ in the codebase rather than the documentation.
The paper attributes the measured gain to "decomposition and persistent context." That is a narrower claim than "multi-agent is better." The persistent part is the filesystem state, which survives across instances; the decomposition part is the manager splitting the problem. Because each instance is fresh, there is no long conversation to carry context, which is presumably why the ledger exists at all.
The README is explicit that the effect is not uniform: "Gains are not universal: some models are unchanged or worse." That sentence is the most useful one in the repository for anyone deciding whether the approach transfers to their model.
The harness supports two engines, and the difference is the whole experiment. `--engine multiagent` runs the manager; `--engine single` is the one-call baseline.
Installing and running a first benchmark
The README requires uv and an API key for the model under test. There is no pip package and no install step beyond that. The command below is copied from the README, with the model set to the one used in the headline result.
cd codebase/v2-current
export OPENAI_API_KEY=...
LCB_RELEASE=release_v6 \
ESCALATION_CLOUD_MAX_TOKENS=128000 \
ESCALATION_CLOUD_TIMEOUT=7200 \
MULTIAGENT_MODEL=openai:gpt-5.6-terra \
uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \
python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8uv resolves the dependencies listed with --with at run time, so nothing is installed into a project environment. The environment variables set the LiveCodeBench release, the token ceiling, the timeout in seconds, and the model. The README notes that other backends use their own prefixes and keys: `anthropic:<model>`, `dashscope:<model>`, `openrouter:<model>`, each with its own `*_API_KEY`.
The run prints the pass@1 score at the end. Results land in `runs/results.json` and workspaces in `runs/ws/`. The README does not document resuming an interrupted run, so a timeout at ESCALATION_CLOUD_TIMEOUT=7200 means starting over unless the workspace state happens to be reusable.
Where GVS5H breaks down
The largest limitation is stated by the authors themselves: some models are unchanged or worse under orchestration. A method that raises Qwen3.8-27B from 69.2% to 92.4% on this slice and does nothing for another model is not a drop-in upgrade. Anyone assuming the gain is a property of the harness rather than of the model-plus-task combination will be disappointed.
The second limitation is cost accounting. The abstract compares orchestrated GPT-5.6-Terra at 88.0% against Fable 5's 90.4% "at 19% of the cost," and the repository includes a figure titled "What one pass costs." But the harness itself does not enforce a budget; it enforces a token ceiling and a timeout. Five instances plus a manager means several times the calls of the single-call baseline, and the README gives no per-run cost estimate for arbitrary models.
The third is reproducibility. LiveCodeBench problems are pinned through LCB_RELEASE, which is good, but the README does not document how the paper's figures were generated from runs/results.json, and no releases were retrieved, so there is no tagged version to pin the harness to. The paper, figures and run data are under CC BY 4.0, and NOTICE.md is the file that says which license covers which path.
Finally, this is a benchmark runner. It solves LiveCodeBench problems. It does not edit your repository, review your pull requests, or run in CI.
The single-call baseline as the real alternative
The honest comparison is inside the repository: `--engine single`. It uses one call to one model, no manager, no ledger, no workspaces, and it is what the multi-agent numbers are measured against. If your goal is a score rather than a method, running the single engine with a stronger model is the cheaper path, and the README's own cost figure exists precisely to make that trade visible.
The difference in approach matters. A single call gives the model one context window and one shot. The manager approach gives it decomposition plus a filesystem that persists across instances, at the price of more calls, more tokens, and a coordination layer whose internals are undocumented. For problems that fit comfortably in one context, the ledger buys little; the paper's own transcript analysis ties the gain to decomposition and persistent context, which are exactly the things a single call cannot provide.
If you already have a multi-agent framework, porting the ledger idea into it is more work than running this harness, because the protocol is not specified in the README.
Maintenance, licensing and upgrade cost
The last push was on 2026-09-15, so the repository is current, and it is not archived. There are no retrieved releases, which means there is no version to pin and no changelog to read before upgrading. Anyone building on the escalation/ code should expect to track master.
Licensing is split and NOTICE.md is the file to read. Code is MIT. The paper, figures and run data are CC BY 4.0. The README states that the LiveCodeBench fork, the benchmark problem statements and the LaTeX template files keep their own licenses. That means redistributing a modified harness is straightforward under MIT, but republishing the figures or run data is governed by CC BY 4.0, and the bundled benchmark material carries separate terms you should check before shipping it inside another product. This is a description of what the files say, not legal advice.
Upgrade cost is low in the sense that there is no dependency lockfile to migrate, since uv resolves dependencies at run time with --with 'datasets<4' --with numpy --with anthropic. It is high in the sense that an unpinned datasets dependency can change behavior between runs, and the harness has no release tags to fall back to.
Editorial conclusion
GVS5H is for researchers and engineers who want to reproduce the paper's multi-agent versus single-call comparison on their own backend, and who are comfortable reading the harness code because the README documents only one command line. It is not for anyone looking for a library to import, a serving stack, or a coding assistant, since the repository is a benchmark runner plus a paper and run data. Before adopting it, verify three things: that your backend is one of the prefixes the harness accepts (openai:, anthropic:, dashscope:, openrouter:), that your key is exported under the matching *_API_KEY name, and that you are willing to accept the README's own statement that gains are not universal, since some models are unchanged or worse. The paper at paper/iclr2027_conference.pdf is the only place the method itself is described.
Frequently asked questions
What is GVS5H?
It is a research repository containing a benchmark harness and a paper on ledger-based zero-shot self-orchestration, in which fresh instances of one model coordinate through a shared filesystem. The README reports results on the 100 latest hard LiveCodeBench problems across nine models.
How do I install and run GVS5H?
There is no install step. The README requires uv and an API key, then has you cd into codebase/v2-current and run escalation/run_bench.py through uv run --no-project with the model set via MULTIAGENT_MODEL. The pass@1 score prints at the end, and results are written to runs/results.json.
Does GVS5H improve every model?
No. The README states directly that gains are not universal and that some models are unchanged or worse. The reported improvement from 69.2% to 92.4% applies to locally served Qwen3.8-27B on this benchmark slice.
Which model backends does GVS5H support?
The README lists openai:, anthropic:, dashscope: and openrouter: prefixes for MULTIAGENT_MODEL, each with its own *_API_KEY environment variable. The example uses openai:gpt-5.6-terra with OPENAI_API_KEY.
What license is GVS5H under?
The code is MIT, while the paper, figures and run data are CC BY 4.0. The README says the LiveCodeBench fork, benchmark problem statements and LaTeX template files keep their own licenses, and NOTICE.md maps licenses to paths.
Community notes