Scholar Loop: an autonomous research loop that scores real PyTorch runs against a frozen metric
An autonomous AI scientist: a multi-agent loop over literature, experiments, self-critique and write-up, with deterministic guards against reward-hacking and hallucination.
At a glance
- What is it?
- Scholar Loop is an MIT-licensed Python framework in which eight LLM agents read papers, propose experiments and write up results, while deterministic code decides what is kept. Two small experiment domains ship, and the package is at version 0.0.1.
- Who is it for?
- Scholar Loop is a good study object for anyone designing an agent loop that must not game its own score, since everything checkable is deterministic code. It is not a drop-in AI scientist: only two toy-scale domains ship, the package is alpha at 0.0.1, and live runs need an Anthropic key you supply.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- renee-jia/scholar-loop474 ★ · +4
| Repository | 2026-09-21 | 2026-11-15 |
|---|---|---|
| renee-jia/scholar-loop | 470 | 474 |
4 Stars in the last 55 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The metric is frozen in two phases so train.py cannot fake its own score
The central claim of Scholar Loop is that its agents cannot reward-hack. The mechanism has three parts. First, scoring is two-phase and frozen: the experiment script train.py can neither fake the metric nor see the validation set. Second, an edit allowlist limits what an agent is permitted to change. Third, a VerifiedRegistry grounds every number in the final write-up against recorded results.
The project backs this with a bundled engine named cheater, which exists to try the exploit and demonstrate that the guards catch it. The practical value is that the only optimization target is the metric itself. There is no LLM acting as judge inside the optimization loop, so a persuasive but wrong explanation cannot raise a score. If you are building something similar, the lesson is to keep the judge outside the generator's reach rather than asking the generator to behave.
The LLM only reasons, and everything checkable is deterministic code
The design splits work cleanly. The model handles the open-ended parts: choosing a direction, reading papers, proposing ideas, reflecting on failures and drafting text. Search-space pruning, deduplication, calibration, number-grounding and promotion gates are deterministic and unit-tested.
There are eight agents: Director, Lit Scout, Reasoner, Debate, Reflector, Advisor, Writer and Reviewer. They share one harness with typed JSON-schema input and output, validate-then-retry behavior, and a single shared audit trace. Two further components, the ledger and the skill library, are the durable memory every step reads from.
The consequence for testing is large. The project reports 108 tests that need no API key and no GPU, because the whole loop can run against a deterministic MockLLM. That makes the control flow checkable in CI, but it also means those tests exercise scripted answers, not the quality of a live model's research ideas.
A three-tier funnel kills most ideas after one cheap run
Each idea climbs smoke, then verify, then full, and each tier is gated. A cheap smoke screen eliminates the clearly worse ideas, and marginal ones never consume a full run. Before an idea even reaches a GPU, a Debate Panel of three personas votes on whether it is worth the compute.
The first stage is parallel. The method Orchestrator.population_step(k, max_workers) proposes N distinct ideas, smoke-screens all of them concurrently, and promotes only the survivors. The bundled quickstart illustrates a single idea: against a baseline of 4.9% val_top1_err, smoke scored 3.7644%, verify 3.8004% and full 3.7644%, so it climbed three tiers with three kept.
What this does not do is guarantee the cheap screen is a reliable proxy. An idea that looks poor at smoke scale but would have helped at full scale is discarded, and nothing in the loop recovers it.
A governor stops the loop on budget, round cap or convergence
Running unattended requires a stopping rule, and Scholar Loop supplies one in a Governor configured as Governor(max_cost, max_rounds, dry_patience). The loop halts when a dollar budget is spent, when a round cap is reached, or when it has gone a set number of rounds without progress, which the project calls loop-until-dry.
The Director plays a related role earlier in each round. It reads the ledger and literature trends, then sets the next direction, topic and budget. An Advisor then returns PROCEED, REFINE or PIVOT to steer the loop after each outcome.
The point for operators is that cost is bounded by configuration, not by hope. If you allow live API runs, set max_cost deliberately, because the stopping behavior is only as protective as the number you give it.
Calibration scores each agent's predictions, and old lessons decay
Scholar Loop predicts an idea's effect before running it, then compares the prediction with the measured result. This predict-then-verify scoring applies to every agent with a checkable claim, including the Reasoner's expected deltas and the Debate panel's go or no-go votes. The scores land in a CalibrationLog that feeds the next prompt, so the loop learns which of its own agents to trust.
Failures are distilled by the Reflector into lessons held in a skill library. That library is relevance-ranked and time-decaying, and SkillLibrary.render(query=...) surfaces the lessons that bear on the current idea rather than simply the heaviest ones. Literature priors from arXiv and OpenAlex, ranked by citation impact, are fed to the Reasoner at the same time.
The limit is that calibration only measures claims that can be checked against ground truth. Open-ended judgments, such as whether a write-up reads well, stay with the Reviewer agent and are not scored the same way.
Only digit classification and diabetes regression ship as experiment domains
The experiments are real PyTorch runs that are fast on CPU and need no dataset download. Two domains exist today: digit classification, scored by error percentage, and diabetes regression, scored by RMSE. A third domain is described as a YAML profile plus an engine pair, with zero changes to the orchestrator.
This is the honest boundary of the project. These are small, well-known benchmarks, chosen so the loop is cheap to run, and the claim of a PhD's workflow on a single-GPU budget refers to the process rather than the scale of the science. If your question needs a large model, a custom dataset or domain-specific evaluation, you would be writing a new profile and engine yourself, and the loop's guarantees depend on that engine freezing its metric correctly. The repository's profiles and engines folders are where that work would go.
Setup runs offline first, and live API runs are a separate opt-in
The package requires Python 3.10 or newer, is licensed MIT, and is declared at version 0.0.1 with the Development Status Alpha classifier. The core dependencies are pyyaml and jsonschema. Optional extras add pytest, the Anthropic client for live runs, and torch with numpy and scikit-learn for the real engines.
pip install -e ".[dev]" # pyyaml + jsonschema + pytest
python examples/quickstart.py # the whole loop in <1s — no GPU, no API key
pip install -e ".[dev,engines]" # adds torch + scikit-learn for the real experiment engines
python examples/campaign_demo.py # a full campaign on real torch, MockLLM-scripted
pytest -q
# ".[llm]" adds the Anthropic client, only needed for live API runs (e.g. examples/run_to_paper.py)The repository has no GitHub releases, and the last push was on 23 June 2026, so install from a clone at the commit you want. The examples folder also contains governed_campaign.py, e2e_live.py and two sample run directories.
Editorial conclusion
Scholar Loop is a good study object for anyone designing an agent loop that must not game its own score, since everything checkable is deterministic code. It is not a drop-in AI scientist: only two toy-scale domains ship, the package is alpha at 0.0.1, and live runs need an Anthropic key you supply.
Frequently asked questions
What is Scholar Loop used for?
Scholar Loop runs an automated research loop: eight LLM agents read literature, propose experiments, run them on real PyTorch engines, reflect on failures and draft a write-up.
Can Scholar Loop run without an API key or a GPU?
Yes. The 108 tests and the quickstart run the whole loop against a deterministic MockLLM, so no API key or GPU is needed. Live API runs require the optional llm extra.
How does Scholar Loop stop agents from reward-hacking?
It uses two-phase frozen scoring so train.py cannot fake the metric or see the validation set, an edit allowlist, and a VerifiedRegistry that grounds numbers in results.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/renee-jia-scholar-loop)Star history chart for your README
[](https://hysenlabs.com/projects/renee-jia-scholar-loop)Was this useful?