ScholarLoop: an autonomous AI research loop with deterministic guards

The problem ScholarLoop targets: cheap, honest experiment loops

Most LLM agent frameworks stop at tool calling. ScholarLoop aims at a narrower and harder target: the full research cycle a PhD student runs, on a single-GPU budget. The README describes the sequence as "read papers → find a gap → run real experiments → reflect → write & self-review", and the repository is organised around that claim rather than around a general agent runtime.

The intended user is someone doing machine learning research who wants the boring parts automated: pulling papers, proposing candidate experiments, screening them cheaply, and writing up what survived. The README frames the compute constraint explicitly, calling it "a PhD's workflow on a single-GPU budget". That is a real constraint, not marketing: the shipped engines are digit classification scored by error percentage and diabetes regression scored by RMSE, both of which the README says run "CPU-fast, no download".

The interesting design decision is what the project refuses to delegate to a model. The README states that the LLM does only the open-ended reasoning, while search-space pruning, deduplication, calibration, number-grounding and promotion gates are deterministic, unit-tested code. It also states that the metric is the only optimization target, with no LLM-as-judge in the optimization loop. That is a deliberate rejection of the pattern most agent projects use, and it is the reason to read this repository at all.

Eight agents, one harness, and a frozen metric

The pipeline is a fixed chain of eight roles. A Director reads the ledger plus literature trends and sets topic and budget. A Lit Scout pulls papers from arXiv and OpenAlex, ranks them by citation impact, and produces cited findings. A Reasoner turns constraints, literature and past lessons into the next experiment. A Debate Panel of three personas votes on whether the idea is worth a GPU. A Funnel runs smoke, then verify, then full tiers. A Runner executes a real torch experiment. A Reflector converts the outcome into a lesson in a skill library. An Advisor returns PROCEED, REFINE or PIVOT. Writer and Reviewer turn confirmed findings into a draft and a peer review.

Two mechanisms are worth separating from the rest. The first is the reward-hacking guard. The README describes two-phase frozen scoring, in which train.py cannot fake the metric or see the validation set, combined with an edit allowlist and a VerifiedRegistry that grounds numbers in the text. The project ships a cheater engine as a test fixture, which is an unusually direct way of proving the guard works rather than asserting it.

The second is the outer loop, which the README treats as the product. A governed round proposes N ideas, smoke-screens them in parallel with max_workers, and climbs only survivors through verify (three seeds plus significance) and full (five seeds). A Governor halts on max_cost, max_rounds or dry_patience convergence. A CalibrationLog scores each agent's checkable claims against ground truth and feeds that back into the next prompt, so the loop learns which of its own agents to trust. The SkillLibrary render call takes a query, so retrieved lessons are ranked by relevance to the current idea rather than by recency or weight alone.

Installing ScholarLoop and running the loop in under a second

The README gives a two-step quickstart. The first install pulls pyyaml, jsonschema and pytest, and the example runs the whole loop without a GPU or an API key because tests use a deterministic MockLLM.

pip install -e ".[dev]"
python examples/quickstart.py

The README states this completes in under a second. If you want the real torch engines rather than the mock path, the second install adds torch, numpy and scikit-learn:

pip install -e ".[dev,engines]"

For a real LLM run you also need the llm extra, which installs the Anthropic client. The pyproject file marks this as only needed for real runs, since the test suite does not require it. The README does not give a combined extra syntax for llm and engines together, so add the llm extra on its own after the engines install.

To see the governed loop rather than the minimal one, the README points at examples/governed_campaign.py and describes it as free and deterministic. The repository also ships examples/campaign_demo.py, examples/e2e_live.py and examples/run_to_paper.py, plus two captured sample runs in examples/sample_run/ and examples/sample_run_diabetes/. Those sample directories are the fastest way to see the expected output shape without running anything. Python 3.10 or later is required.

Where the harness is thinner than the README suggests

The reward-hacking defence is scoped to what the harness controls. Frozen scoring prevents an engine from faking the metric, and the edit allowlist constrains what a run may write. Neither stops an agent from picking a weak hypothesis that the frozen metric happily confirms. The README's own framing says the metric is the only optimization target, which is a correctness property and also a ceiling: if the shipped metric does not measure what you care about, the loop will optimise the wrong thing with full determinism.

Adding a domain is described as a YAML profile plus an engine pair, with zero orchestrator changes. That is a clean extension point, but it means your engine sits inside the trust boundary. The cheater engine exists precisely because that boundary is where the interesting failures live, and a custom engine that writes outside the allowlist or reads the validation set is your problem, not the harness's.

The project is also early. The pyproject classifier is Development Status 3 - Alpha, the version is 0.0.1, and the README badge says research preview. There is no published API stability statement and no release history in the repository metadata. The last push was on 2026-06-23, which is roughly three months before the time of writing, so the codebase is not stale but it is also not moving quickly. Treat the orchestrator as something to read, not something to depend on across versions.

Finally, the literature stage depends on arXiv and OpenAlex being reachable and returning what the Scout expects. The README does not document caching, rate-limit handling or offline behaviour for that stage, so a network-restricted environment is a real failure mode for anything past the quickstart.

How ScholarLoop differs from a general agent framework

The obvious comparison is a general-purpose agent framework such as LangGraph or a plain ReAct loop over an LLM. Those give you a graph or a loop and leave the research methodology to you. ScholarLoop bakes in the methodology: a fixed eight-role chain, a three-tier funnel, a governor with a cost ceiling, and a skill library that decays. You get less freedom and more opinion.

The second comparison is an experiment-tracking tool such as MLflow or Weights & Biases. Those record runs you launch yourself and make no decisions. ScholarLoop decides what to run next and records why, which is a different product category. The overlap is the ledger, and the README treats it as durable memory that every stage reads from rather than as a reporting surface.

The third comparison is an LLM-as-judge evaluation harness. ScholarLoop explicitly excludes that pattern from the optimization loop, using deterministic scoring instead and reserving the model for open-ended reasoning. If your work depends on judging open-ended outputs where no frozen metric exists, this framework's central design choice works against you.

Licence, upgrade cost and what version 0.0.1 implies

ScholarLoop is MIT licensed, with the licence file at the repository root and the same identifier declared in pyproject.toml. MIT is permissive: you can use, modify and redistribute it, including commercially, provided the copyright notice and permission notice are retained. That is the standard reading of the text, not legal advice, and if you are embedding it in a product you should have counsel review the actual LICENSE file rather than this summary.

One licence-adjacent point the README does not address: the optional dependencies are not all MIT. The engines extra pulls torch, numpy and scikit-learn, and the llm extra pulls the Anthropic client. Those carry their own licences, and the project does not document a compatibility review. If you ship a container built from the engines extra, your obligations come from that dependency set as much as from ScholarLoop itself.

Upgrade cost is hard to judge because there are no releases. The version is 0.0.1, there is no changelog in the repository listing, and no release notes were retrieved. Pinning to a commit hash is more honest than pinning to a version number here, since 0.0.1 tells you nothing about what changed between two checkouts. The test suite is the practical upgrade gate: the README states there are 108 tests and that they need no API key or GPU, so running pytest after a pull is a cheap way to see whether a change touched the harness.

Editorial conclusion

Adopt ScholarLoop if you want a small, readable research-loop framework you can run end to end without a GPU or an API key, and you are willing to read the orchestrator before trusting its numbers. Do not adopt it if you need a stable API, a published paper pipeline, or an LLM-as-judge evaluation loop, because the project is at version 0.0.1 and declares itself a research preview. Before trusting any campaign, verify three things in the code: that the frozen metric really is frozen across both phases, that the edit allowlist covers every path your engine writes to, and that the MockLLM path and the AnthropicLLM path produce the same schema-valid JSON. The bundled cheater engine is the fastest way to check the first two.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. renee-jia/scholar-loop on GitHub