PaperJury: a Claude Code skill that puts AI review findings on trial before touching your LaTeX
Pre-submission AI review stress-test for research papers. A Claude Code skill: review, verdict, revise, verify.
At a glance
- What is it?
- PaperJury runs a review, verdict, revise, verify loop over a LaTeX paper inside Claude Code. The model reads and drafts; deterministic scripts handle state, voting rules, patch application and stopping conditions. The interesting part is the triage: not every AI reviewer comment gets applied.
- Who is it for?
- Adopt PaperJury if you already write in Claude Code and want a pre-submission pass that separates text-level fixes from questions that need new experiments, with patches you approve before they land. Skip it if you want a tool that generates missing results, or if you cannot supply a LaTeX toolchain and still expect verified compilation.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 33 days ago.
- What is it written in?
- Mainly JavaScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem PaperJury targets: AI review comments that should not all be applied
Ask a language model to review a draft and you get a list. The list mixes three very different things: wording that genuinely obscures a claim, requests for experiments that do not exist yet, and comments that misread the paper outright. A naive loop that applies everything it is told will rewrite sentences it should have left alone and will quietly soften claims the authors can actually defend. The README puts the boundary plainly: PaperJury is a pre-submission self-check tool that cannot replace the author's scientific judgement or peer review, and it cannot be used to fabricate experiments, forge results, add unsupported claims, or hide limitations.
The intended user is a researcher writing LaTeX who already works inside Claude Code, has a draft close enough to review, and wants the mechanical part of the pass handled without surrendering the research decisions. The README's own framing of who benefits lists four situations: a fresh first draft, a final pre-submission check on claim strength and experimental sufficiency, a single paragraph you want tightened without changing the claim, and unattended multi-round revision under explicitly granted auto mode.
Review, verdict, revise, verify: what the loop actually does
PaperJury is packaged as a Claude Code skill, and the pipeline has four named stages. Reviewing produces reviewer-style findings, each carrying evidence, a location, a judgement and a current status. The README is explicit that the tool does not dump a mass of comments into the manuscript body; findings live in a list.
The verdict stage is the design decision that separates this from a plain critique loop. Every finding is routed to exactly one of three outcomes. valid-fixable covers textual problems such as unclear phrasing, an overstated claim, or awkward structure, where no new experiment is needed and the original meaning is preserved; the system drafts a minimal patch and applies it only after passing guardrail checks. author-required covers missing experiments, ablations, data or evidence, and is handed back untouched because the research decision belongs to the author. invalid-drop covers findings where the AI reviewer misread the paper or proposed a change that should not be accepted; those are rejected and recorded.
The division of labour is stated at the top of the README: the model reads, judges and drafts, while state management, voting rules, patch application and stopping conditions are controlled by deterministic code. The arXiv paper (2606.16322) is described as covering which tasks deterministic scripts handle, which judgements go to semantic agents, how disputed findings enter deliberation, and which guardrails apply to edits at different risk levels. The ablation note in the README supports that split indirectly: removing the guardrail chain worsened the unsafe-edit rate, and removing the deliberation stage lowered verdict agreement.
Installing the skill and choosing a mode without memorising commands
Installation is two commands inside Claude Code, quoted from the README:
/plugin marketplace add Spark-To-Paper-Skills/paperjury /plugin install paperjury@Spark-To-Paper-Skills
After that you describe what you want in ordinary language. The README's example is a request to review with emphasis on whether the experiments support the claims; a second example asks for the introduction to be made more compact without changing the claim. There are no subcommands to memorise. PaperJury infers whether you want direct-edit or review from the description, and the README states that auto must be explicitly enabled.
The mode determines when patches reach your file. In direct-edit and review, patches are presented for your confirmation first. In auto, safe edits are applied under a pre-authorised policy while high-risk changes are still returned to the author. That distinction matters more than the install steps: auto is the only mode where a change can land without a per-patch confirmation, which is why the README treats enabling it as an explicit act rather than a default.
One practical caveat is stated in the outputs table. If a LaTeX toolchain is available, PaperJury performs a real compile. If not, it says which checks could not be completed rather than reporting verification it did not perform. Treat the absence of a toolchain as a known degradation, not a silent one.
The guardrail chain and the recorded verdicts are the real product
The output set is three artefacts. A findings list, where each reviewer-style item carries evidence, location, verdict and status. A reviewable patch set, where only safe fixes become minimal patches and high-risk edits are parked pending an author decision. And a verification report, which compiles when the toolchain allows and otherwise states what was skipped.
The guardrails are what make the second artefact credible. The README's ablation line reports that removing the guardrail chain raised the unsafe-edit rate by 0.152, and removing the deliberation stage cut verdict agreement by 0.153. Those numbers come from the project's own evaluation and should be read as the authors' measurements, not independent verification. Still, they point at the correct failure mode: an editing agent that applies everything is cheap to build and expensive to trust.
The verdict record is the other half. Because invalid-drop findings are rejected and logged rather than silently discarded, you can inspect what the reviewer got wrong. The README reports blind-review agreement rates for the three verdict classes: 0.872 for invalid-drop, 0.913 for valid-fixable, and 0.860 for author-required. The lowest agreement sits on the class where the system declines to act, which is worth knowing if you plan to skim the rejected items.
What the evaluation table does and does not tell you
The README presents a comparison across 12 held-out papers, four in Vision, four in NLP and four in ML, against four baselines with a blind expert audit, and links to arXiv 2606.16322. The headline deltas are question quality (panel-relative F1 from 0.519 to 0.656), audit precision (P_verified from 0.663 to 0.847) and unsafe edit rate (ESVR from 0.110 to 0.025, described as 4.4x lower). The per-paper table adds rounds and hours: 3.08 rounds and 2.47 hours per paper for PaperJury, against 3.33 rounds and 2.06 hours for the strongest baseline, the LLM-as-judge loop.
Read the ESVR column with the edit volume, as the README itself instructs. PaperJury applies 13.4 edits per paper and the judge loop 14.3, so the lower unsafe rate is not an artefact of doing almost nothing, but the two figures are close enough that the comparison should be read as a pair rather than as a single win. The README also notes that PaperJury's F1 beat every baseline that produces a findings list on all 12 papers in paired comparison, and that the judge loop hit its round cap on 2 of 12 papers while PaperJury hit it on none. Cost is quoted at 6.76M tokens per paper, against 31.4M for the naive unbounded generator.
These are the authors' numbers from their own harness. Nothing in the supplied material shows an independent replication, and the sample is 12 papers in three subfields. Treat the table as evidence that the design was measured, not as a guarantee about your draft.
Where PaperJury is the wrong tool
The clearest boundary is stated by the project itself: findings that need new experiments, ablations, data or evidence are routed to author-required and handed back unchanged. PaperJury will not invent the missing result, and the README forbids using it to fabricate experiments or add unsupported claims. If your draft's problem is that a claim outruns the evidence and the fix is more work at the bench or the cluster, this tool will tell you so and stop.
A second boundary is toolchain-dependent. The verification report only compiles when LaTeX tooling is present; without it, the README says the tool states which checks could not be completed. That is honest behaviour, but it means the verify stage of review-verdict-revise-verify is partly unavailable in a bare environment.
A third is mode risk. Auto applies safe edits under a pre-authorisation policy. The definition of safe is the project's, enforced by the guardrail chain, and the README's own ablation shows the unsafe-edit rate rises when that chain is removed. If you enable auto on a paper whose claims you have not recently re-read, you are relying on the guardrails to catch claim-strengthening edits on your behalf. The README's warning that the tool cannot replace the author's scientific judgement is not boilerplate; it describes exactly this exposure.
Finally, the interface is Claude Code. There is a Codex version at Spark-To-Paper-Skills/paperjury-codex, aligned at v1.0 per the release news, but if your workflow lives in a different editor or a plain terminal, the skill packaging is the obstacle before any review quality question arises.
Compared with an LLM-as-judge loop, and what maintenance looks like
The natural alternative is the pattern most people already have: prompt a model to critique the paper, apply the suggestions, repeat until the comments stop. The README's baseline table calls this the LLM-as-judge loop, and the structural difference is that it has no verdict layer. Every finding is treated as actionable, which is why the README records an unsafe edit rate of 0.110 for that loop against 0.025 for PaperJury, and why it hit the round cap on 2 of 12 papers. PaperJury inserts a routing decision and a deliberation step between reading and editing, at the cost of roughly 0.4 extra hours per paper in the reported run (2.47 against 2.06).
A second alternative is the naive unbounded generator, which the table puts at 8.37 hours and 31.4M tokens per paper with an ESVR of 0.240. That is the shape of an agent that keeps rewriting until it is satisfied. PaperJury's stopping conditions are deterministic rather than model-decided, which is the mechanism behind the bounded part of the subtitle.
On maintenance, the supplied material supports only a limited reading. Releases run v0.5.0 (2026-06-05), v1.0.0 (2026-06-10) and v1.2.1 (2026-06-15), with the last push to main at 2026-08-14. The v1.0.0 notes describe a soft update reminder that only notifies when a new stable tag appears and does not interrupt current work, so upgrades are opt-in rather than forced. The project is MIT licensed, which permits commercial and academic use and modification; the README also ships a CITATION.bib and asks that the arXiv paper be cited if PaperJury helped your research. That citation request is a norm, not a licence condition, and nothing here is legal advice. A parallel Codex build exists, so feature parity between the two is a coordination cost the maintainers carry rather than something you resolve at install time.
Editorial conclusion
Adopt PaperJury if you already write in Claude Code and want a pre-submission pass that separates text-level fixes from questions that need new experiments, with patches you approve before they land. Skip it if you want a tool that generates missing results, or if you cannot supply a LaTeX toolchain and still expect verified compilation. Before trusting a run, read samples/dogfood/ and the human-checked run report, then confirm which mode you are in, because auto must be explicitly enabled and applies safe edits under a prior-authorisation policy.
Community notes