PlanBench and the LLMs-Planning Repository: A PDDL Benchmark for Testing Whether Language Models Can Plan
An extensible benchmark for evaluating large language models on planning
At a glance
- What is it?
- The karthikv792/LLMs-Planning repository packages PlanBench, a planning benchmark built on PDDL domains, plus the code behind four papers on LLM planning ability. It is a research harness for measuring plan validity, not a planner and not a training framework.
- Who is it for?
- Adopt this repository if you are evaluating whether a specific model can produce valid plans in a formal planning language, or if you need the LLM-Modulo back-prompting loop as a reference implementation. Do not adopt it as a production planner or as a general reasoning benchmark; the PDDL domains are narrow and the scores are plan-validity rates, not task success.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 106 days ago.
- What is it written in?
- Mainly PDDL, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Problem PlanBench Was Built to Measure
Most language model evaluations ask whether a model produces a plausible answer. PlanBench asks whether a model produces a valid plan in a formal planning language, where validity is checked rather than judged. The repository README states that the plan-bench subdirectory belongs to the PlanBench paper, described as an extensible benchmark for evaluating large language models on planning and reasoning about change. The audience is narrow: researchers and engineers who need to know whether an LLM can sequence actions under preconditions and effects, not whether it can describe a plan in prose. That distinction matters because a model can write a fluent paragraph about stacking blocks and still emit an action sequence that violates a precondition on step three. The benchmark exists to make that failure visible and countable.
What the Test Sets Actually Contain
The leaderboard in the README names four columns: Blocksworld with natural language prompting over 600 instances, Mystery Blocksworld over 600 instances, Randomized Mystery Blocksworld over 600 instances, and Blocksworld Hard with PDDL prompting over 110 instances. Blocksworld is a classic planning domain where a gripper moves blocks between stacks. Mystery Blocksworld renames the predicates and objects so that surface-level pattern matching from pretraining is less useful, and the randomized variant goes further. The instance counts are part of the design: 600 instances per natural-language column gives a stable denominator, while the 110-instance PDDL column is much smaller and therefore noisier per percentage point. The README notes that for Blocksworld Hard, results live in results/backprompting/, which signals that the hard subset was evaluated with the back-prompting loop rather than a single forward pass. Anyone comparing a PDDL score against a natural-language score is comparing different prompt formats and different instance counts.
How the Repository Is Organized
The README maps code to papers. The plan-bench subdirectory holds the PlanBench code. The llm_planning_analysis subdirectory holds the code for On the Planning Abilities of Large Language Models, a critical investigation. Inside that same subdirectory, back_prompting_parallel.py is described as an implementation of the LLM-Modulo framework, and the README marks that llm_planning_analysis also contains code for a later paper evaluating the reasoning model o1 on planning and scheduling. So the repository is a container for four research artifacts rather than a single library with one entry point. That matters operationally: there is no unified CLI described in the README, and the two subdirectories are separate code paths. If you clone the repository expecting one benchmark runner, you will instead find paper-specific scripts, and the README does not document a shared configuration layer across them.
The Leaderboard Numbers and What They Separate
The README publishes a static test set leaderboard with zero-shot prompting. Deepseek R1 is listed at 99.1% on Blocksworld natural language, 43.3% on Mystery Blocksworld, 25.8% on Randomized Mystery Blocksworld, and 53.6% on Blocksworld Hard PDDL. o1-preview is listed at 97.8%, 52.8%, 37.3%, and 23.65% respectively. GPT-4 is listed at 34.6% on Blocksworld and 0% on Mystery Blocksworld. Claude-3.5 Sonnet, GPT-4o, Claude 3 Opus, and LLaMA-3 70B are all listed at 0% on Mystery Blocksworld. The pattern the table shows is a large drop when predicate names stop carrying meaning. A model near the ceiling on named Blocksworld can fall to zero once the domain is obfuscated, which is the benchmark's central claim about pattern matching versus planning. The README labels the model types as LLM or LRM, and the LRM rows occupy the top of every column where they appear.
The LLM-Modulo Back-Prompting Loop
The one mechanism the README names explicitly is back prompting, implemented in llm_planning_analysis/back_prompting_parallel.py and tied to the LLM-Modulo framework. The file name indicates parallel execution, and its location under results/backprompting/ for the Blocksworld Hard results indicates that the hard subset was scored through this loop. The design idea behind LLM-Modulo is that a language model proposes candidate plans and external critics verify them, with failures fed back as prompts rather than accepted as final answers. The README does not document the critic interface, the number of feedback rounds, or the prompt templates, so the exact loop is only knowable by reading the script. Treat the file as a reference implementation to study, not as a documented API. If you need the loop's parameters, they are in the code, not in the README.
Getting It Running and What the README Does Not Say
The README gives no installation section, no dependency list, no Python version, and no example invocation. What it does give is the layout: plan-bench for the benchmark, llm_planning_analysis for the analysis code, llm_planning_analysis/back_prompting_parallel.py for the LLM-Modulo implementation, and llm_planning_analysis/results/ for detailed result files, with results/backprompting/ holding the Blocksworld Hard output. Running the benchmark therefore means reading those subdirectories, since no entry point is named in the supplied material. The primary language is listed as PDDL, which is the domain description format the benchmark consumes, not the language of the harness. Because the README is silent on setup, this is a case where the honest statement is that installation steps cannot be confirmed from the supplied material, and any command you write will come from the scripts themselves. The leaderboard submission path is documented: open a pull request with a result file.
Where the Benchmark Stops Being the Right Tool
PlanBench scores plan validity in PDDL-style domains, and the README's own columns show how concentrated those domains are: Blocksworld in three variants plus a hard subset. A high score is evidence about planning in that family of domains, not about scheduling a real workflow, handling numeric resources, or operating under uncertainty, none of which the README describes as covered. The Mystery Blocksworld results also cut both ways. A 0% score shows that a model was relying on familiar names rather than reasoning about state change, but it does not tell you how the same model performs on an unfamiliar domain that still has meaningful predicate names. The 110-instance Blocksworld Hard column is small enough that a single instance moves the percentage by roughly 0.9 points, so small gaps between models in that column should not be read as real separation. Finally, this is a static benchmark with a published leaderboard and a pull request submission process, which means it measures models that may have seen related evaluation material during training.
The Alternative: Classical Planners and Validator Tooling
The direct alternative to prompting an LLM on a PDDL problem is running a classical planner on the same problem. A planner such as Fast Downward or a SAT-based planner takes the PDDL domain and problem file as input and returns a plan that is valid by construction, or reports that none exists. The difference in approach is not speed or scale, it is the source of correctness: a classical planner searches the state space and guarantees the plan satisfies the preconditions and effects, while an LLM generates a candidate that must then be checked. That is exactly the gap the LLM-Modulo loop tries to close by adding critics, which makes the repository's own architecture an argument for planner-based verification. If your goal is to get a valid plan, use a planner. If your goal is to measure whether a model can produce one, and how it fails, use PlanBench.
Maintenance, Licence, and What to Verify First
The repository is licensed MIT, which permits reuse and modification with the licence and copyright notice retained. That is a permissive arrangement, and it does not by itself settle the terms of the underlying datasets, the papers, or any model outputs you generate and redistribute, so check those separately if you plan to publish derived results. The last push listed is 2026-06-02, and the two releases are v1.0-beta from June 2023 and v1.0.0-alpha from June 2022, so the tagged releases are old relative to the repository activity and the leaderboard has been updated through pull requests rather than through versioned releases. That means the leaderboard reflects contributions at various times with no release pinning them together. Before you cite a number, open llm_planning_analysis/results/ and identify the file, the prompt format, and the instance count behind it, because the README table alone does not carry that provenance.
Editorial conclusion
Adopt this repository if you are evaluating whether a specific model can produce valid plans in a formal planning language, or if you need the LLM-Modulo back-prompting loop as a reference implementation. Do not adopt it as a production planner or as a general reasoning benchmark; the PDDL domains are narrow and the scores are plan-validity rates, not task success. Before relying on any number, open llm_planning_analysis/results/ and confirm which test set, prompt style, and instance count produced it, because the README's own table mixes natural-language and PDDL prompting columns that are not comparable.
Community notes