AgentReview: an LLM-agent simulator for the peer review pipeline
Official Implementation for EMNLP 2024 (Main Track, Oral) "AgentReview: Exploring Academic Peer Review with LLM Agent."
At a glance
- What is it?
- AgentReview turns peer review into a five-phase LLM-agent pipeline with configurable reviewer, author and area chair traits. It is research code for studying how bias and social dynamics move decisions, not a reviewing tool.
- Who is it for?
- Adopt AgentReview only if you are doing computational social science or meta-science and want a controllable stand-in for a review committee; the README's trait axes and five-phase pipeline are the parts worth reusing. Do not adopt it to triage real submissions or to produce review text for actual venues, because the acceptance rate is fixed at 32 percent and the agents are prompt-configured, so the output is a simulation, not an assessment.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 128 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What AgentReview is for, and who should care
The README frames the project as the first LLM-agent simulation of academic peer review, and the abstract gives the motivation directly: prior analyses of peer review rely on exploration and statistics of existing review data, which the authors say do not address the multivariate nature of the process, do not account for latent variables, and are constrained by privacy concerns. AgentReview's answer is to stop analysing recorded reviews and generate synthetic ones instead, so that a single factor such as reviewer commitment can be varied while everything else is held at a baseline. The intended audience is therefore narrow: researchers in computational social science, meta-science and LLM-agent work who want a controllable model of a review committee. The README reports 53,800+ generated peer review documents, 10,460 reviews and rebuttals, 23,535 reviewer to area chair discussions, and 9,414 meta-reviews and final decisions, all drawn from 523 sampled ICLR submissions spanning 2020 to 2023. Those numbers describe a research artifact, not a service. If you need software that helps you review a paper faster, this is the wrong repository.
The five-phase pipeline and its three agent roles
The architecture is a structured five-phase pipeline. Phase I has three reviewers independently evaluate each manuscript. Phase II is author to reviewer discussion, where authors submit rebuttals addressing reviewer concerns. Phase III is reviewer to area chair discussion, in which the AC facilitates and reviewers update their initial ratings. Phase IV has the AC compile a meta-review from all signals, and Phase V is the final accept or reject call. Three role types are instantiated as LLM agents, and each is configured along trait axes set through prompts: reviewers vary by commitment (responsible or irresponsible), intention (benign or malicious) and knowledgeability (knowledgeable or unknowledgeable); authors vary by identity disclosure (anonymous or known); area chairs vary by decision style (authoritarian, conformist or inclusive). The design choice that matters is the one-trait-at-a-time discipline against a fixed baseline. Because every trait is a prompt-level setting rather than a trained parameter, the experiment is reproducible in the sense that the configuration is text, but it also means the causal story rests entirely on how well those prompts isolate the trait they name.
The headline findings and what they actually measure
The README lists five sociological phenomena, each tied to a stated quantitative effect. Social influence, described as conformity to perceived majority opinion, is reported as a 27.2 percent standard deviation reduction in ratings after rebuttals. Altruism fatigue and peer effects are reported as a single under-committed reviewer driving an 18.7 percent drop in commitment across all reviewers. Groupthink and echo chamber effects appear as a 0.17 rating drop among biased reviewers plus a 0.25 spillover onto unbiased reviewers. Authority bias and halo effect are reported as revealing author identity for just 10 percent of papers shifting 27.7 percent of decisions. Anchoring bias is stated qualitatively: the rebuttal phase exerts only a minimal effect on final outcomes. The abstract summarises the overall result as a 37.1 percent variation in paper decisions attributable to reviewer bias. Read these as simulation outputs under a specified configuration. The mixing of units across rows (a standard deviation change, a percentage drop in commitment, raw rating deltas, a percentage of decisions flipped) makes the table hard to compare internally, and the README does not state the baseline values those deltas are measured against. That is a gap worth closing before citing any single figure.
Getting it running: installation, data setup and the demo
The README documents a Python 3.10+ project with a Gradio 5.4 interface, and the repository's primary language is Jupyter Notebook, so expect the workflow to be notebook-driven rather than a packaged CLI. The README's table of contents names the operational sections in order: Installation, Data setup, Quick start, then Customizing your own setting. Customisation is described as prompt-level: the trait axes above are the knobs, and the README says all of them are set via prompts. There is a hosted Gradio demo linked at huggingface.co/spaces/Ahren09/AgentReview, which is the fastest way to see the pipeline without installing anything, and a project site at agentreview.github.io. What the supplied material does not contain is the literal command text: no pip install line, no environment file contents, no dataset download URL and no example invocation appear in the README excerpt provided here. Treat the Installation and Data setup sections in the repository as the authoritative source and read them before planning an experiment, because the data step is the one most likely to depend on external ICLR artifacts that may or may not still be reachable.
Where the simulation design breaks down
The clearest limitation is stated in the pipeline table itself: Phase V uses a fixed acceptance rate of 32 percent. That is a deliberate control for a simulation, and it also means the framework cannot tell you whether a paper would be accepted at a real venue, because the decision threshold is imposed rather than predicted. A second constraint follows from the prompt-based trait axes. Reviewer intention and knowledgeability are induced by instruction text, not by a model trained to behave that way, so any measured effect is an effect of the prompt as much as of the trait, and the README offers no prompt ablation to separate the two. Third, the underlying corpus is ICLR 2020 to 2023 only, 523 sampled submissions split 19 oral, 29 spotlight, 125 poster and 350 reject. The reject class is roughly two thirds of the sample, which is a reasonable reflection of conference reality but a skewed substrate for studying acceptance-side dynamics. Finally, the project has no tagged releases in the supplied metadata. Last push is 2026-05-10, so the code moves, but there is nothing to pin a reproduction to, which is a real problem for a paper artifact whose value is reproducibility.
How this differs from statistical peer review analysis
The obvious alternative is the approach the abstract positions itself against: statistical and exploratory analysis of existing peer review datasets, the kind of work that mines recorded reviews and decisions from venues that release them. The difference is not tooling, it is the direction of inference. A statistical study observes real outcomes and tries to attribute variance to factors after the fact, which is exactly where the authors say confounded latent variables and privacy constraints bite. AgentReview inverts this: it constructs the committee, fixes the acceptance rate, varies one trait, and reads off the outcome. That buys clean identification of a factor and costs external validity, because nothing in the pipeline was ever a real reviewer. A second, closer alternative is simply running an LLM as a single reviewing agent with a fixed prompt. That is cheaper and easier to explain, but it has no mechanism for rebuttals, no area chair, and no way to express reviewer-to-reviewer influence, which is the entire subject of the social influence and groupthink findings here. If your question is about interaction between roles, the single-agent approach cannot answer it at all.
Maintenance, licence and what to check before you build on it
The licence is Apache-2.0, which permits commercial and academic reuse with the usual conditions around notices and attribution; the repository ships a LICENSE file at the root. Nothing in the supplied material suggests a separate data licence, and since the simulation draws on ICLR submissions, the terms attached to any redistributed review artifacts are worth confirming in the repository rather than assuming they follow the code licence. This is not legal advice, and if you plan to redistribute generated reviews, read the licence files and the data setup notes yourself. On maintenance: the last push is 2026-05-10 and there are no retrieved releases, so there is no version to depend on. The practical consequence is that a reproduction you build today is tied to a commit hash you choose yourself, and the Gradio interface pins to 5.4 while the demo runs on Hugging Face, so a mismatch between the space and main is plausible. For a paper artifact, that is the cost you accept: fast-moving research code with a stable licence and an unstable interface.
Editorial conclusion
Adopt AgentReview only if you are doing computational social science or meta-science and want a controllable stand-in for a review committee; the README's trait axes and five-phase pipeline are the parts worth reusing. Do not adopt it to triage real submissions or to produce review text for actual venues, because the acceptance rate is fixed at 32 percent and the agents are prompt-configured, so the output is a simulation, not an assessment. Before building on it, verify two things in the repository: whether the ICLR 2020 to 2023 artifacts referenced under Data setup are actually downloadable, and whether the Gradio 5.4 demo on Hugging Face matches the current main branch, since the release history shows no tagged versions to pin against.
Community notes