PostTrainBench: A 10-Hour H100 Benchmark for CLI Agents Doing Post-Training

The gap PostTrainBench is trying to measure

Most agent benchmarks ask an agent to edit files, fix a bug, or answer questions. PostTrainBench asks it to conduct AI R&D. The agent receives a base LLM, an evaluation script, an H100 GPU and a ten-hour budget, and its score is the post-trained model's benchmark score. Nothing about the agent's transcript, tool calls or intermediate reasoning enters the score. That is the design choice worth noticing: the benchmark treats the agent as a black box whose only output is a trained model in a folder called final_model, which the prompt instructs it to produce. The audience is narrow. It is for researchers studying automated research and development, and for teams that want a concrete, hardware-bounded task to compare CLI scaffolds against each other. It is not for anyone who wants to post-train a model for production use; ten hours on one H100 is a research constraint, not a training recipe.

One H100, ten hours, and an unrestricted agent

The mechanism is deliberately sparse. The agent prompt states the machine has an Nvidia H100 GPU, that packages like transformers and datasets are installed via uv pip install --system, and that internet access is unrestricted. The agent can query the benchmark through evaluate.py, iterate, and choose its own data sources and training methods. The prompt says: "You have complete freedom in your approach: data sources, training methods, etc." Scoring, then, is not a measure of process quality. It is a measure of whether an agent can find a post-training procedure that generalizes to the held-out evaluation, under a wall-clock cap, without a human in the loop. The repository separates the pieces cleanly: agents in agents/, container definitions and cache downloads in containers/, evaluation tasks in src/eval/, baselines in src/baselines/, and results in results/ with baseline runs prefixed baseline_. Each task folder under src/eval/tasks/ carries a benchmark.txt with the official name, an evaluate.py, and an optional task_context/ directory that can hold information about how evaluation is performed so the agent does not have to guess. That optional directory is the most interesting part of the layout. It is where a task author decides how much of the scoring rule the agent is allowed to see.

Seven tasks, and what they do and do not cover

The benchmark spans reasoning, tool use, knowledge, math, health and code: AIME 2025 for competition math, Arena Hard Writing adapted from ArenaHard v2 for creative writing, BFCL for function calling, GPQA for graduate-level science, GSM8K for grade school math, HealthBench Easy for medical knowledge and reasoning, and HumanEval for code generation. Two of the seven are math tasks and two are code or tool-use tasks, so the suite leans toward verifiable, short-answer evaluation. Arena Hard Writing and HealthBench Easy are the softer ends. There is no long-context task, no multilingual task and no agentic multi-turn task in the list. If your interest is whether an agent can post-train a model to follow instructions across many turns, this suite will not answer that. The score is also the benchmark score of the post-trained model, which means a task where the base model already scores near ceiling has little headroom to show agent skill, and a task where the base model scores near floor may reward any training signal at all. The repository ships src/baselines/ for computing baseline scores, and the README notes baseline runs are stored in results/ with the baseline_ prefix. Reading those baselines before interpreting any agent result is the obvious first step, and it is the step most likely to be skipped.

Getting it running means HTCondor, containers and a .env

The quick start is four commands and an environment file. Install apptainer and fuse-overlayfs, then run bash containers/build_container.sh standard, then bash containers/download_hf_cache/download_hf_cache.sh, then cp example.env .env and fill in values, then bash src/commit_utils/commit.sh. The .env keys are explicit: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENCODE_API_KEY and ZAI_API_KEY for the API-based agents, plus HF_HOME, POST_TRAIN_BENCH_RESULTS_DIR, POST_TRAIN_BENCH_CONTAINERS_DIR, POST_TRAIN_BENCH_CONTAINER_NAME (default standard), POST_TRAIN_BENCH_PROMPT (default prompt) and POST_TRAIN_BENCH_JOB_SCHEDULER (htcondor or htcondor_mpi-is). Shell environment variables take precedence over .env values, which matters when you are debugging a job that picked up a stale key. Jobs are submitted with condor_submit_bid, for example condor_submit_bid 50 -a "agent=codex_non_api" -a "agent_config=gpt-5.3-codex". The subscription-based paths are the fiddliest part. Codex with ChatGPT Pro requires enabling device code login, running codex login --device-auth, and copying ~/.codex/auth.json into agents/codex_non_api/auth.json. Claude Code with a Max subscription requires claude setup-token and writing the token to agents/claude_non_api/oauth_token. Both files are gitignored, and run_task.sh copies them into the job directory only for the agents that need them. The solve.sh scripts then unset the API keys and force the subscription login method, so an agent does not silently fall back to an API key you did not intend to spend.

The HTCondor dependency is the real adoption cost

The README is direct about this: the repository currently targets an internal HPC cluster running HTCondor, and Harbor support for rented cloud GPUs is described as coming soon via PR 8. Until that lands, running PostTrainBench outside an HTCondor environment means working around the submission layer rather than using it. The scheduler variable accepts only htcondor and htcondor_mpi-is, so there is no local or Slurm path in the documented configuration. That is a genuine limitation, not a documentation gap. A second constraint is the container requirement. apptainer and fuse-overlayfs are prerequisites, and the container is built by a project script rather than pulled from a registry, so you are trusting the build to succeed on your host and your kernel. A third is the ten-hour wall clock. It is fixed in the task framing, and it interacts with the H100 requirement: the benchmark is measuring agent behaviour under a specific hardware and time budget, so results from a different GPU or a different time limit are not comparable to the published setup. If your goal is to compare training algorithms rather than agents, this is the wrong tool. The agent's freedom over method means the benchmark cannot isolate which technique produced a gain.

Where it sits next to SWE-bench style harnesses

The closest familiar comparison is a software-engineering agent harness like SWE-bench, where the agent edits a repository and a test suite decides pass or fail. PostTrainBench keeps the pass-or-fail discipline but moves the artifact from a patch to a trained model, and the evaluation from unit tests to benchmark scores. The practical difference is cost and irreversibility. A SWE-bench run is cheap to repeat and easy to inspect; a PostTrainBench run consumes an H100 for ten hours and produces a checkpoint whose quality depends on decisions the agent made along the way. That makes reruns expensive and makes variance across seeds or across agent runs a real concern. It also means the benchmark is measuring something SWE-bench cannot: whether an agent can plan a training run, allocate a fixed compute budget, and stop in time. If you want cheap, high-volume agent evaluation, a code-editing harness is the better fit. If you want to know whether an agent can spend a GPU budget productively, this is one of the few public setups that asks that question directly.

Licence, maintenance and what to check before you commit

The repository is MIT licensed, which permits commercial and private use with the usual requirement to carry the licence and copyright notice; that is a description of the licence text, not legal advice, and you should read the LICENSE file in your own checkout. There are no release tags retrieved, so there is no versioned artifact to pin. The last push recorded is 2026-09-10, and the README advertises Harbor support as an open PR rather than merged code, which means the documented HTCondor path is the only supported one today. Upgrade cost is therefore not a version-bump question; it is a question of tracking the main branch and the PR that changes the scheduler layer. Practically, before you invest an H100 weekend, verify that containers/build_container.sh standard completes on your host, that the HuggingFace cache download finishes, and that evaluate.py in the task folder you intend to score against runs correctly against that cache. Then run a baseline from src/baselines/ and confirm you reproduce a results/ entry with the baseline_ prefix. If the baseline does not reproduce, no agent score on top of it is meaningful.

Editorial conclusion

Adopt PostTrainBench if you already run HTCondor or are willing to wait for the Harbor support described in PR 8, and if your question is specifically whether an agent can move a base model's score on AIME 2025, BFCL, GPQA, GSM8K, HealthBench Easy, HumanEval or Arena Hard Writing within ten H100 hours. Do not adopt it as a general agent evaluation harness: the seven tasks are fixed, the scoring is benchmark-score-of-the-post-trained-model, and the job submission path assumes condor_submit_bid. Before committing, verify three things in your own checkout: that containers/build_container.sh completes on your host, that evaluate.py in the task folder you care about runs against the downloaded HuggingFace cache, and that your chosen agent's auth path (API key versus auth.json or oauth_token) matches what run_task.sh copies into the job directory.

Official sources

1. aisa-group/PostTrainBench on GitHub
2. Issues
3. License: MIT
4. Project website
5. README