AI-paper-reviewer: A Prompt Toolkit for Pre-Submission Self-Audit of ML Papers

What This Project Actually Is

AI-paper-reviewer is not a software package that you install. It is a collection of Markdown prompt files designed to be copied into an LLM chat interface such as Claude, ChatGPT, Gemini, or Codex. The README describes it as a pre-submission self-audit tool for papers targeting top ML conferences like NeurIPS, ICML, ICLR, AAAI, and KDD. The core idea is that many rejection reasons are not single fatal errors but a chain of fixable issues: overclaimed contributions, weak positioning, inconsistent numbers, missing citations, vague writing, and double-blind leaks. The project compiles reviewer guidelines from multiple conference cycles, OpenReview discussions, and the author's own submission and rebuttal experience into a reusable prompt workflow. The primary audience is paper authors who want to catch problems before reviewers do.

The 10-Dimension Framework and 4-Level Red Flag System

The review process is structured around ten dimensions labeled A through J. Dimension A covers logic and argumentation, asking whether premises lead to conclusions and whether counterexamples are considered. Dimension B is empirical rigor, checking cross-table number consistency and whether claims like 'significantly' are backed by p-values. Dimension C targets writing quality, including pronoun ambiguity and passive voice overuse. Dimension D is citation and attribution, checking if every claim has a reference. Dimension E handles mathematical notation consistency, such as the same symbol meaning different things in different places. Dimension F is double-blind anonymity, scanning for identity hints beyond 'we' and leftover metadata. Dimension G covers formatting and IEEE style, including citation title casing and decimal alignment in tables. Dimension H checks language purity, looking for stray Chinese characters, emojis, or bilingual remnants. Dimension I is structure and transitions, verifying topic sentences and that figures or tables are interpreted after they are referenced. Dimension J flags reviewer red-flag words like 'we are the first', 'no matter how complex', and 'obviously'.

How the Prompt Workflow Operates

The workflow is designed for interactive use with an LLM. The README describes a three-minute start: open your AI chat, copy the entire prompts/02_paragraph_audit.md block into the dialog, wait for the AI to reply 'ready', then paste paper content paragraph by paragraph. The AI returns a strict review with directly replaceable rewrite suggestions. The master workflow file, 00_master_workflow.md, is the main entry point for a full 10-dimension audit. The full paper audit, 01_full_paper_audit.md, is for major milestones. The paragraph audit, 02_paragraph_audit.md, is marked as the most frequently used, designed for going through a draft section by section while tracking claims, citations, notation, numbers, and red flags across the whole paper. The README emphasizes that this is not about polishing or packaging the paper but about finding the weakest points from a reviewer's perspective.

Getting Started: Real Commands and File Paths

There are no installation commands because there is no code to run. The repository is a set of Markdown files. To start, you clone the repository or navigate to the GitHub page, then open the file prompts/02_paragraph_audit.md and copy its entire content into an LLM chat. The README explicitly lists the steps: open your AI chat, copy the block, paste paper content, and get the review. For English users, there is a README_EN.md and matching prompt files in prompts_en/. The directory structure includes Part I for core workflow files, Part II for section-specific prompts (abstract, intro, method, experiments, conclusion), Part III for special checks (double-blind, citation format, math notation, anti-AI smell), Part IV for pre- and post-submission (rebuttal drafting, reproducibility checklist), and Part V for a red-flag quick reference. The reproducibility checklist prompt, 13_repro_checklist.md, is described as helping fill out the 21-item NeurIPS reproducibility questionnaire in about one minute.

Documented Examples and Their Value

The repository includes two real case studies. The first, examples/case_01_abstract_before_after.md, shows an abstract before and after revision, with nine key changes. These include replacing 'we are the first' with 'we recast', replacing 'significantly outperforms' with specific numbers, and deleting 'world's largest by volume'. The second case, examples/case_02_method_audit.md, documents the process of fixing a notation inconsistency in an algorithm description where both theta and tau appeared, causing internal contradiction. These examples are useful because they show the concrete output format and the type of suggestions the prompts generate. They also give a realistic sense of what the tool can catch, which is mostly language-level issues and consistency problems, not deep scientific validity. The README does not claim the tool can judge the correctness of your method or experiments, only that it flags places where reviewers are likely to deduct points.

Limitations and Cases Where It Is the Wrong Tool

The most obvious limitation is that AI-paper-reviewer depends entirely on the LLM you paste the prompts into. The quality of the review is bounded by the model's understanding and the prompt's clarity. The README does not provide any guarantee of accuracy, and it does not describe any validation of the prompts against a set of known papers. Another limitation is that the tool cannot verify facts outside the text. It can flag a missing citation or an inconsistent number, but it cannot check whether the cited reference actually exists or whether the experimental setup is fair. The README mentions 'suspected hallucinated citations' as a target, but the prompt can only ask the model to flag suspicious ones, not verify them. The tool is also wrong for authors who need a fully automated pipeline that runs on their local machine without sending text to an external LLM. There is no code to run, no API, and no command-line interface. If you need a self-contained script that parses your LaTeX files and checks consistency, this project is not it.

Alternative Approaches and How They Differ

A direct alternative is to build your own review checklist based on the official reviewer guidelines from the target conference. NeurIPS and ICML publish reviewer guidelines and reproducibility checklists, and you can create a custom prompt that asks an LLM to evaluate your paper against those specific criteria. The difference is that AI-paper-reviewer has already done the work of synthesizing guidelines from multiple venues and organizing them into a 10-dimension framework with severity tiers. A self-built checklist would be more tailored to a single venue but would require more effort to create and maintain. Another alternative is to use a dedicated paper-review tool that runs locally or as a service, such as a script that checks LaTeX for double-blind compliance by scanning for author names in metadata. That approach is more automated but only covers a narrow set of checks, not the full range of logic, writing, and citation issues that the prompt framework addresses. The key distinction is that AI-paper-reviewer is a prompt-based, interactive workflow, while alternatives are either manual checklists or code-based scanners.

Maintenance, Upgrade Cost, and License

The repository is marked as not archived, but the last push is unknown and there are no recent releases retrieved. This means you should check the commit history before relying on it for a current submission cycle. The prompt files are static text, so maintenance cost is low for you as a user: you just copy the files and use them. However, the project itself may not be actively updated, and the guidelines for conferences change over time. You may need to update the prompts yourself if you find that they miss new requirements. The license is MIT, which the README states allows free commercial use, modification, and distribution with the requirement to retain the copyright and license notice. This is a permissive license, so you can adapt the prompts for your own workflow without restriction, as long as you preserve the original notices.

Editorial conclusion

Adopt AI-paper-reviewer if you are an ML researcher preparing a submission to NeurIPS, ICML, ICLR, AAAI, or KDD and you want a structured, repeatable self-audit before reviewers see the work. Do not adopt it if you expect an automated pipeline or a tool that runs locally without an external LLM, because the repository is only prompts and documentation. Before relying on it, verify that the prompt files match the current guidelines of your target venue, and test one prompt on a small section to see if the output severity levels align with your own judgment. The project's value is in its checklist structure and red-flag vocabulary, not in any code execution, so treat it as a starting point for your own review process.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. README
4. Releases