BIRD-CRITIC 1.0: A Benchmark for SQL Issue Debugging, Not Text-to-SQL
[NeurIPS 2025 Main] SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
At a glance
- What is it?
- BIRD-CRITIC 1.0 evaluates whether LLMs can diagnose and fix user-reported problems inside real database environments across MySQL, PostgreSQL, SQL Server and Oracle. The top score on its Open split is 34.50, which tells you more about the benchmark's difficulty than about any model's readiness for production.
- Who is it for?
- Adopt BIRD-CRITIC if you are building or evaluating an agent that must repair broken SQL against a live schema, and you need execution-based scoring rather than string matching. Do not adopt it if your goal is text-to-SQL generation from clean natural language; the tasks are bug reports and DBA tickets, not clean prompts.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 176 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem BIRD-CRITIC targets is debugging, not query generation
Most SQL benchmarks hand a model a clean natural-language question and a schema, then check whether the generated query returns the right rows. BIRD-CRITIC 1.0 inverts that. Its stated question is whether LLMs can diagnose and solve user issues within real-world database environments. The input is closer to a support ticket: something is failing, slow, or returning wrong results, and the model has to work out why. The README describes the tasks as realistic user issues across MySQL, PostgreSQL, SQL Server and Oracle, and states that the benchmark expands beyond simple SELECT queries to cover a wider range of SQL operations. That framing matters for who this is for. It is aimed at teams building database agents, DBA copilots, or migration tooling, and at researchers who want an execution-grounded signal instead of a text-similarity one. If your product turns plain questions into SELECT statements, this benchmark measures a different skill than the one you ship.
Four evaluation metrics that map to four kinds of SQL repair
The verification section lists four scoring approaches, and the choice between them is driven by task type rather than by preference. Soft EX applies to SELECT-only tasks. Soft EX plus Parsing handles tasks that carry user-specific requirements or refinements, which suggests the parser exists to check structural constraints that result equivalence alone would miss. For DBA tasks such as CRUD operations, the benchmark uses purpose-built test cases, and the README notes this is also effective when resolving an issue requires multiple sequential SQL queries. For efficiency work and runtime errors, scoring uses the Query Execution Plan, described as evaluating solution SQL at the algorithm level. That last choice is the interesting one. Two queries can return identical rows while one scans a table and the other uses an index, and a plan-based metric is the only way to distinguish them. The cost is fragility: plan shape varies by dialect and version, so a plan-based score is only as stable as the environment it runs in.
How the sandbox and task splits are organised
The README describes an optimized execution-based evaluation environment and a fast eval sandbox built on a PostgreSQL template plus Docker. Each task is stated to have been verified by human experts on three dimensions: reproducing the error in the BIRD environment to prevent data leakage, curating test case functions specific to that task, and creating new relational databases at different scales and in professional domains. The split structure is worth reading carefully because the numbers differ. The overview mentions 600 development tasks and 200 held-out out-of-distribution tests. The Open version, bird-critic-1.0-open, is described as 570 tasks across the four dialects. The PostgreSQL version, bird-critic-1.0-pg, is 530 tasks. The lite version, bird-critic-1.0-flash-exp, is 200 PostgreSQL issues. A SQLite set of 500 issues was announced later. These are not interchangeable, and the README's own performance tables are labelled per split, so a score quoted without its split name is not comparable to another.
Installation and the commands the README gives you
The repository declares Python 3.10 or later and OpenAI 1.40 or later, both visible as badges in the README header. The evaluation path the documentation points to is the PostgreSQL template plus Docker sandbox, and the README states plainly that the evaluation code was updated on 2026-01-08, specifically the remove_distinct function, with the instruction to pull the latest code. That is a direct warning about stale clones: if you forked before that date, your scores are computed by different logic. The dataset itself is not in the repository. It is distributed through Hugging Face under the birdsql organisation, with separate dataset pages for bird-critic-1.0-flash-exp, bird-critic-1.0-postgresql, bird-critic-1.0-sqlite and bird-critic-1.0-open. Full ground truth, including sol_sql and test_cases, is described as available through an email auto-reply system addressed to bird.bench25@gmail.com with the subject tag [bird-critic-1 GT&Test Cases]. That is an unusual distribution channel for a benchmark, and it means an automated pipeline cannot fully bootstrap itself from public artefacts alone.
The scores are low, and that is the point
On the Open split, the README's table puts o3-mini-2025-01-31 at 34.50, deepseek-reasoner at 33.67, o1-preview at 33.33, claude-3-7-sonnet with thinking at 30.67, gemini-2.0-flash-thinking at 30.17 and grok-3-beta at 29.83. The spread between first and sixth is under five points. A benchmark where reasoning-tuned frontier models cluster in the low thirties and cannot separate from each other is measuring something they are all bad at, which is consistent with the stated goal of diagnosing real user issues rather than writing fresh queries. The human numbers point the same way. Per the 2025-07-09 news item, database experts using standard tools but no AI assistants scored below the leaderboard leaders on all three boards, and a second group with the same expertise but access to ChatGPT, Claude or Gemini reached 83.33 on Open, 87.90 on PG and 90.00 on Flash. The gap between unaided and AI-assisted humans is far larger than the gap between the top models, which suggests the bottleneck is tooling and iteration rather than raw model capability.
Where BIRD-CRITIC is the wrong instrument
The benchmark is execution-based and dialect-specific, and both properties impose limits. A task written for Oracle will not score against a MySQL sandbox, and the README's own tables are split by dialect for exactly this reason, so cross-dialect generalisation is not something a single aggregate number here can tell you. The distribution model is the second constraint. Ground truth and test cases arrive by email request rather than a versioned package, which makes reproducibility harder than a pinned pip install and means you cannot diff two ground-truth revisions the way you would diff code. Third, the benchmark targets user issues and DBA tasks, so it says nothing useful about schema design, migration safety, or query generation from clean prompts. If you want to know whether a model writes correct SELECT statements from unambiguous questions, the README points at a different release, LiveSQLBench-Base-Lite, described as 18 end-user databases and 270 tasks with unambiguous queries grounded in external knowledge. Using BIRD-CRITIC for that purpose would measure the wrong thing.
Compared with Spider-style text-to-SQL evaluation
The obvious alternative is the Spider-lineage approach: give the model a question and a schema, generate SQL, and compare result sets against a gold query. The difference in approach is the input distribution. Spider-style evaluation assumes the question is well formed and the schema is correct, and it scores a single query. BIRD-CRITIC assumes something is already broken, supplies a user issue rather than a question, and in the DBA and efficiency cases scores a sequence of statements or a query plan rather than one result set. That makes BIRD-CRITIC harder to set up and harder to compare across papers, because a score depends on which of the four metrics was applied. It also makes it a better proxy for the work a database agent actually does: reading an error, forming a hypothesis, running something, and checking. A team choosing between the two should pick based on whether their product's failure mode is a wrong answer or a broken system.
Maintenance, versioning and licence terms
The repository is MIT licensed and is not archived, with the most recent push dated 2026-03-23. There are no tagged releases, so there is no version number to pin. The README's own news feed is the changelog, and it shows active expansion: a SQLite set of 500 issues in March 2026, the auto-reply ground-truth system the day after, an evaluation-code fix in January 2026, and the paper in June 2025. The practical consequence is that a BIRD-CRITIC score is only meaningful alongside the date you pulled the code and the split you ran. Note the licence mismatch: the repository is MIT, while the README's badge points to CC BY-SA 4.0 for the dataset, and the datasets live on Hugging Face under their own terms. Those are separate grants covering separate artefacts, and the README does not reconcile them. Check both before redistributing anything, and treat the email-based ground-truth delivery as a manual step that belongs in your release checklist rather than a build script.
Editorial conclusion
Adopt BIRD-CRITIC if you are building or evaluating an agent that must repair broken SQL against a live schema, and you need execution-based scoring rather than string matching. Do not adopt it if your goal is text-to-SQL generation from clean natural language; the tasks are bug reports and DBA tickets, not clean prompts. Before running anything, verify that the evaluation sandbox can load the four dialect environments you intend to score, and confirm which split (open, pg, flash-exp, sqlite) your tasks actually come from, because the task counts differ by split.
Community notes