RigorPilot Skills: README-First Reproduction with Bounded Execution and Byte-Preserving Annotations

The gap RigorPilot targets: READMEs that describe runs nobody records

A research repository's README is usually the only operating manual. It names a test command, a training script, a config file. What it does not do is record what happened when someone followed it. That record normally lives in a chat log, a shell history, or nowhere. RigorPilot Skills is aimed at that gap. The README describes it as running research repositories "from their README, with bounded execution and auditable evidence," and it adds "section-level results without rewriting the original README." The intended user is an engineer or agent operator who has to decide whether a repository's documented workflow actually executes, and who needs that answer to be inspectable later. The output is not a paper result. It is a trace of a documented procedure, with the commands, the retained files and the observed outcome kept together.

How the annotation mechanism stays reversible

The README states that original commands, prose, badges, images, videos and HTML stay in the source file. RigorPilot splits that file into sections and inserts one evidence-linked card per section. The reversibility claim is specific: removing its insertion blocks restores the retained original README byte for byte. This is the design decision that separates the project from tools that generate a fresh summary document. Because the annotations are additive blocks rather than edits, a diff between the original and the annotated copy should show insertions only. The README also says supporting repository files are kept "so relative links and media retain their original context," which is why the showcase links point into a retained repository checkout rather than to a standalone markdown file. If you plan to rely on this, the check worth running yourself is a byte comparison of the original against the annotated file with the insertion blocks stripped out. The README asserts the property; it does not show the diff.

The status vocabulary and what the colours do not mean

Annotations carry a small status vocabulary: green for selected checks passed, blue for not executed, white for read only, yellow for partial, red for blocked, purple for decision needed. The README is explicit about the two readings people get wrong. Green does not automatically mean paper-result reproduction, and blue is not an execution failure. The four showcase cases illustrate the spread. micrograd is listed as two tests passed in 7.62 s across eight headings and eight annotations. minGPT is listed as a test selected but not executed, with no model download, across eleven headings. PyTorch MNIST is marked partial training with an observed loss of 0.038893. nanoGPT Shakespeare is marked partial with a train loss of 4.1676 and a validation loss of 4.1649. Read together, these are four different kinds of outcome, and only one of them is a passing test suite. The vocabulary is doing real work: it lets a selection boundary be recorded as a result rather than as a failure.

Install commands and the authorization boundary

The installer needs Node.js and npm. The README notes that you should check its Node version requirement if it reports EBADENGINE. Installing every skill is one command: npx skills add lllllllama/rigorpilot-skills --all. Installing only the self-contained reproduction skill is npx skills add lllllllama/rigorpilot-skills --skill ai-research-reproduction. The README states the main skill works alone and that the all-skills option adds companion and leaf entrypoints. The usage pattern is a prompt to a Skills-capable agent with the target repository open. The README's example prompt asks the agent to run the smallest README-documented evaluation, preserve the source, write evidence to repro_outputs/, and produce an annotated copy beside the original README, asking before large downloads or long training. That last clause is the authorization boundary in practice. The README frames it as a default: trusted reproduction is the default and candidate exploration requires explicit authorization. Where the boundary sits is partly a function of the prompt you write, not only of the skill.

The recorded suite: what the numbers cover and what they exclude

The README reports a historical, commit-pinned deterministic run: four of four case protocols passed in 251.0 s, with a peak workspace of 98.67 MiB and zero model API calls. The zero-API figure is scoped by the README itself to that suite only. The same paragraph warns that selection-only and partial cases are not completed evaluations, converged training or reproduced paper scores. That scoping matters more than the headline pass rate. A four-case suite where one case is selection-only and two are partial training runs is not a reproduction benchmark in the sense a paper-replication reader would assume. It is a protocol benchmark: did the documented procedure execute within bounds, and was the outcome recorded faithfully. The repository also lists a local regression figure of 69/69 passed in the badge area. Badge figures are not a substitute for reading the linked case definitions and methodology files, which the README points to under benchmarks/.

Where RigorPilot is the wrong tool

The clearest limitation is stated by the project. If you want a reproduced paper score, RigorPilot does not claim to produce one, and the README says so about its own partial and selection-only cases. A second limit follows from the README-first premise. The tool runs what the README documents. A repository with a correct procedure documented only in a paper appendix, a Makefile or a CI workflow is outside the described input surface. Third, bounded execution is a constraint, not a convenience. The nanoGPT and PyTorch MNIST cases are labelled partial because the run was cut short; a partial training run with a captured loss is evidence that startup worked, not evidence that the model converges. Fourth, the byte-preservation guarantee depends on the retained checkout. If relative links and media are not retained alongside the annotated copy, the annotation context degrades even though the original bytes are intact. Finally, there are no retrieved releases in the material provided, so there is no version pin to install against beyond the repository itself. Treat the main branch as the artifact.

Alternative approach: a reproduction harness you write yourself

The obvious alternative is a hand-rolled harness: a shell script or Makefile target per repository that installs dependencies, runs the documented command under a timeout, captures stdout and stderr, and writes a results file. That approach gives you full control over the timeout, the environment and the output schema, and it needs no Agent Skills runtime. The difference in approach is where the record lives. A hand-rolled harness produces a results artifact that sits beside the README; the README itself remains silent about what was run. RigorPilot's distinguishing move is inserting the evidence into a copy of the README at section level, so a reader sees the claim and its outcome in the same place, and can strip the insertions to recover the original. If your team already has a CI job that runs documented commands and publishes logs, that job covers much of the same ground with less machinery. RigorPilot's advantage is the annotation format and the status vocabulary, not the execution itself.

Licence, maintenance and what to verify before adopting

The repository is MIT licensed, which permits commercial and private use with the usual attribution and warranty disclaimer. That is a statement about the licence text, not legal advice; if the notices matter to your organisation, have someone read LICENSE. On maintenance cost, the material supports only a limited reading. The last push is dated 2026-09-07, the repository is not archived, and no releases were retrieved, so upgrades arrive as commits on main rather than as tagged versions you can pin. The CI workflow at .github/workflows/validate.yml is referenced by the README badge, which suggests validation runs on push, but the README does not describe what that workflow covers. The practical maintenance question is whether the skill keeps pace with the Agent Skills standard it advertises. Before adopting, verify three things: that the annotated README differs from the original only by insertion blocks, that the retained checkout still resolves the relative links and media the annotations point at, and that the case definitions under benchmarks/external_cases.json match the outcomes you care about rather than the ones the suite happens to record.

Editorial conclusion

Adopt RigorPilot Skills if you need a recorded, commit-pinned trace of what a repository's README actually asks you to run, and you want the original README left intact. Do not adopt it expecting paper-score reproduction: the README states that selection-only and partial cases are not completed evaluations or converged training. Before trusting a run, verify that the retained checkout still contains the supporting files your relative links and media depend on, and check the annotated copy against the original to confirm the insertion blocks are the only difference.

Official sources

1. Issues
2. License: MIT
3. lllllllama/RigorPilot-Skills on GitHub
4. README