A benchmark that asks an agent to advise a student, then changes the facts

What is being evaluated

This is a benchmark for graduate admissions agents, evaluating agent configurations, which the readme defines as an execution system paired with a model, for admissions advising, application planning and updating advice as student information changes. The current results cover eight agent configurations. The subject matter is the Chinese graduate admissions process, which the project describes in Chinese as a benchmark from the organisation behind the graduate admissions island site, and the tasks are in Chinese. The dataset itself is 70 Chinese language tasks, 23 cases and 83 answer stages, with per-task rubrics and reference programs. The public dataset is available without access approval, while the scoring materials must be withheld from the agents being evaluated. That last point is the part that matters for anyone building on this, because it is the difference between a benchmark that measures capability and one that measures the ability to read the answer sheet.

The leaderboard and what it does not say

The results table shows eight configurations across the 70 public tasks with a mean score and a completion rate. The ranking runs from Codex with GPT Sol 6.1 at 79.08 and Codex with GPT Astra 6 at 78.36, through DSH with DeepSeek V4.1 Flash at 68.16, the Volcengine managed agent with GLM 5.3 Flash at 66.80, Qwen Code with Qwen 3.8 Max at 64.24, Kimi Code with Kimi K3 at 58.96, and Antigravity with Gemini 3.8 Flash at 58.23 and Antigravity with Gemini 3.1 Pro at 55.55. All eight completed 70 tasks and no current tasks timed out. The caveats in the readme are the interesting part. These results combine different real runs with input versions and budgets that are not fully uniform, complete tool traces are not available for every public record, and the leaderboard does not imply uniform execution conditions. That is an unusual amount of honesty for a leaderboard, and it is the sort of caveat that should be quoted whenever the numbers are quoted.

How the tasks are built

The dataset splits into three kinds of task, and the split is what gives the benchmark its shape. There are 21 recommendation tasks, which assess how well the suggested programmes match the student's background and constraints. There are 36 planning and decision tasks, which evaluate preparation plans, risk assessment and actionable advice. And there are 13 two-stage tasks, which release new information within the same session and test whether the agent updates its advice, with both stages receiving one task score. The scoring is a hundred points per task, and the recommendation tasks allocate 70 points to reference-program matching and 30 to recommendation quality. A separate reviewer scores the full answer, and the rubrics stay outside the answering process, which is the design that keeps the reference answers from leaking into the responses.

The two-stage tasks are the interesting experiment

The two-stage design deserves its own look because it is the part that is not a standard question-and-answer benchmark. A two-stage task supplies initial context, labelled S₀, followed by updated information, S₁, in the same session, producing an initial response A₀ and a revised response A₁ for joint scoring. The readme gives a concrete example: the January-to-March scenario and a shift to direct PhD applications, which corresponds to the public task O03-04, and the scoring criteria for that example are labelled E1 to E5. The point of the design is that it tests a property a single-turn benchmark cannot, which is whether an agent revises its earlier advice when the facts change, rather than sticking to its first answer or producing a fresh answer that ignores the earlier one. That is a property that matters for any assistant doing planning work over a long conversation, and it is much closer to real use than a knowledge question is.

Running it, and the deliberate preview mode

The quick start runs from the repository root, and the requirements are unusual. Automated native execution requires macOS Seatbelt and Python 3.10 or newer, with 3.11 or newer needed for the Kimi subscription configuration. The framework uses only the standard library, so no pip install is needed, and you install the official client yourself and prepare your own subscription login or API key. The workflow is three commands for any configuration.

AGENT=gpt-6-1-sol
python3 -m benchmark setup --agent "$AGENT"
python3 -m benchmark cli-check --agent "$AGENT"
python3 -m benchmark cli-batch \
  --selection config/selections/current.json \
  --agent "$AGENT" --out "./runs/current-$AGENT"

The setup command saves local configuration, and after cli-check returns ready_for_trial the readme advises trying one task first. The batch command defaults to a preview of 70 tasks and 83 stages without model calls, and you add --execute to use your account allowance, which is a considerate default because it means you can inspect the selection before spending anything. For multiple models you configure and check each one and then repeat the --agent flag, using a new output directory for each run.

The agent registry separates the harness from the model

The configuration table lists eight agent configurations with their agent IDs, the client each one drives and the template authentication method. Codex with GPT Astra 6 uses the codex client and a ChatGPT subscription, Codex with GPT Sol 6.1 the same, Claude Opus and Claude Sonnet use Claude Code with subscription OAuth, Gemini Flash uses Antigravity with a subscription, DeepSeek Flash uses the DeepSeek Harness with an API, Qwen Max uses Qwen Code with the DashScope API, and Kimi K3 uses Kimi Code with subscription OAuth. The readme makes a point that is easy to miss: these IDs are used with the --agent flag and they are not client model arguments. That separation is the benchmark's central design idea, because what is being measured is the pair, not the model. The same underlying model behind two different execution systems can behave quite differently, and the leaderboard rows make that visible, with Codex ahead on one pair, Qwen Code and Kimi Code and Antigravity all scoring in the high fifties, and the Antigravity rows showing the same client trailing with two different Gemini models.

Scoring, provenance and the repository

Scoring can use a separate Codex subscription session to review exported materials, and optional API judging requires separate configuration, which keeps the judging step independent of the agent under test. The quick start document covers trials, subscription review, API judging and reports, and a separate native CLI guide covers installation, authentication and multiple stages. Provenance is handled carefully, with per-answer scored_draft and teacher_calibration labels left in the captured scoring records, and the difficulty labels described as owner-approved rather than independently double reviewed: 14 easy tasks at 20 percent, 34 medium at 48.57 percent and 22 hard at 31.43 percent. The official flag is described as a separate result-certification flag. The repository is organised with data/ for the tasks, config/ for the agent registry and selections, experiments/ for the records and per-task results, docs/ for the methodology and figures, schemas/, scripts/, tests/, examples/ including a public smoke example, plus a CITATION.cff and a separate LICENSE-DATA for the data.

Where this is the wrong choice

Three limits are worth naming. The first is the domain, which is narrow by design and in one language, so a good score here says something about Chinese graduate admissions advising and about the agent configurations tested, and not much about general capability. The second is the comparability caveat, since the results combine runs with different input versions and budgets and the tool traces are not complete for every public record, which means the leaderboard is a useful signal but not a controlled experiment and should be read with the methodology document rather than the table alone. The third is the operational requirement, because automated native execution needs macOS Seatbelt, so a Linux-only team cannot run the full protocol and a Windows team cannot run it at all, and the scoring step needs a second subscription session of its own. What the benchmark does provide that a general leaderboard does not is a task domain where the rubric has to be written by people who know the domain, and the two-stage tasks give a way to measure revision rather than only answering.

Editorial conclusion

The baoyan-agent-benchmark suits a team that wants to compare coding agents on an advising task with rubrics and a public dataset rather than on a synthetic coding problem, and it suits anyone who wants to see how an agent behaves when the facts change mid conversation. It is the wrong tool if you need a general agent benchmark, since the domain is narrow and the tasks are in Chinese, and it is the wrong choice if you want to run it on Linux, because the automated native execution path requires macOS Seatbelt. Before running it, read the methodology document, because the leaderboard combines runs with different input versions and budgets and does not imply uniform conditions. The last push was on 2026-10-19 and the repository publishes no releases, and the public dataset is on Hugging Face without access approval.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. yeyuxin777/baoyan-agent-benchmark on GitHub