Spider 2.0: the text-to-SQL benchmark that enterprise schemas were built for

Three settings with different shapes and different bills

The benchmark is presented in a table of settings, and reading it tells you most of what you need. Spider 2.0-Snow is a text-to-SQL task with 547 examples against Snowflake, and it costs nothing. Spider 2.0-Lite is also a text-to-SQL task with 547 examples, spread across BigQuery with 214, Snowflake with 198 and SQLite with 135, and it does incur some cost. Spider 2.0-DBT is a different kind of task, a code agent task with 68 examples against DuckDB running DBT, and it is also free. The distinction between Snow and Lite is therefore not size, since both have 547 examples, but dialect coverage and cost. Lite spreads the same volume across three engines so you can run more than one SQL dialect, while Snow concentrates on one hosted warehouse where the schema and the engine are as close to a real enterprise deployment as a public benchmark can manage.

The schemas are the test, not the questions

The project's own explanation of why the benchmark exists, kept in the repository as a commented-out block alongside the overview, is that Spider 2.0 offers a more realistic and challenging test of large language model performance on complex enterprise-level text-to-SQL workflows. The ingredients it names are complex data environments with more than three thousand columns, multiple SQL dialects such as BigQuery and Snowflake, and diverse operations including transformation and analytics. That is the design in one line. A text-to-SQL benchmark over a small normalised schema with a handful of tables measures whether a model can write a join, and a good model does that. A benchmark over a warehouse with thousands of columns measures whether a model can find the right columns at all, which is the skill that actually decides whether a text-to-SQL product works. The same block records the result that even the most advanced models, including GPT-4, solve only 6.0 percent of Spider 2.0 tasks, against 86.6 percent on Spider 1.0 and 57.4 percent on BIRD.

The evaluation suite was rebuilt and the leaderboard moved

The news log documents a benchmark under active correction, and the entry of 2025-10-29 is the one to read first. It announces a major update in three parts. The evaluation suite issue was fixed, so scores are now more accurate and stable, and the affected methods were refreshed on the leaderboard. The Snowflake hosting cost is addressed separately: spider2-snow is free by default, but queries are queued, and the project says that if you are willing to cover the hosting cost it can share the Spider2 Snowflake data directly into your own Snowflake project, with the details in a hosting document in the assets directory. Third, if you hit MF2A connection errors, meaning your credentials cannot access the Snowflake warehouse, there is a pointer to the Snowflake guideline. The instruction in that guideline is not optional either, since the README asks you to check whether your credentials can reach the warehouse at all before blaming the harness.

Snowflake access has been unstable for months

Two entries show that the Snowflake path has been the operational headache. On 2025-11-06 the project apologised for login and credential issues caused by Snowflake's password and MFA policy upgrade, and said both web UI login and Python credential access behaviours had changed, which is why the updated guideline has to be reviewed carefully before continuing. More recently, on 2026-08-12, the project apologised for an ongoing disruption to Spider 2.0-Snow access, saying the Snowflake evaluation account is experiencing an unexpected suspension issue and that the team is working on it with Snowflake. The last push to the repository was on that same date, 2026-08-12. If you are planning an evaluation that depends on the hosted Snowflake account, this is the practical constraint to check first, and the DBT setting is the one that does not depend on it at all.

How much ground truth is released, and what you may use

The release policy is stated in the entry of 2024-12-24, which says that given the many evaluation requirements the decision is to release all examples and gold answers for self-evaluation, while noting that only a small amount of gold SQL is available, and that the leaderboard is still active. To have a method officially validated and scores uploaded to the leaderboard, the submission guidance has to be followed, which is linked from the top of the README alongside the paper and the data update log. The distinction between the two paths is the point. Self-evaluation is fully supported, so you can run the benchmark yourself and compare against the published answers, but the official ranking goes through a submission process. Separately, the entry of 2025-04-20 provides ground-truth tables for spider2-lite and spider2-snow to help with quick benchmarking and analysis, with the condition that anyone using that setting must indicate that they are using oracle tables, since handing over the answer's tables is a materially easier task than letting the model find them.

Agents are the method here, not just prompted models

The methods directory holds the reference agents, and the history of them shows the direction of travel. On 2024-12-26 the project recommended spider-agent-lite and spider-agent-snow for benchmarking your language models, noting the widespread attention to the traditional text-to-SQL setting, and that the final output should be CSV files rather than SQL statements, which is a meaningful change in what a method is asked to produce. On 2025-06-10 it implemented a tool-call-based Spider-Agent for Spider 2.0-Snow that requires no Docker and significantly improves runtime performance, which removes a container dependency from the path a new user has to set up. On 2025-05-22 it created the Spider2-DBT setting with 68 tasks and removed the original Spider2 setting, enabling quick and smooth benchmarking with spider-agent-dbt, and describes it as a comprehensive repository-level text-to-SQL task. So the benchmark measures an agent that explores a repository and a warehouse, not a single completion.

The example set changes under you

The data update log exists because the examples are not frozen. The entry of 2025-01-10 tells readers to refer to the update log to track changes in the evaluation examples, and adds that the leaderboard results will change dynamically accordingly. The entry of 2025-07-13 is a concrete instance: the spider2-snow.jsonl file was updated to resolve ambiguities, and the previous version was renamed to spider2-snow-0713.jsonl for reference. Keeping the old file around rather than overwriting it is the detail that makes the change auditable, and it means you can diff the two to see exactly which examples were touched. The practical advice for anyone running this is to pin the example file you used, because a score computed against the previous jsonl and a score computed against the current one are not comparable, and the leaderboard will not tell you which one it used.

Where Spider 2.0 is the wrong tool

Four limits apply. It measures text-to-SQL and code agents, so a model that is good at the general task of writing queries is not what is being tested, and the benchmark tells you nothing about a schema with fifty tables or a workload with one hundred daily questions. It is not free to run in every configuration, since Lite costs money even where Snow and DBT do not, and the Snow setting depends on a hosted account the project does not control. The gold SQL release is deliberately partial, so you cannot use it to fine-tune on the answers, and the commented-out notice in the repository says as much, that the released Gold SQL for supervised fine-tuning is not recommended because it may affect the fairness of evaluation and hinder proper benchmarking, and that the release is intended to help users design prompts. And the whole thing moves: the evaluation suite was fixed in late 2025 and the example files have been revised since, so any comparison you publish needs the version of the data stated alongside the score.

Editorial conclusion

Use Spider 2.0 to test whether a model can write SQL against a schema shaped like an enterprise warehouse, because the number of columns and the dialect diversity are the whole point and a smaller public benchmark will not reproduce them. Do not read the leaderboard as a settled ranking, since the evaluation suite was fixed as recently as 2025-10-29 and the data update log changes the examples, with the leaderboard changing accordingly. Verify first which setting you can afford, because Snow and DBT are free while Lite incurs some cost, whether you are running a method that needs the Snowflake account at all given the access disruption the project has been reporting since 2026-08-12, and, if you plan to self-evaluate, which examples have gold SQL released, because only a small amount of it is available.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. xlang-ai/Spider2 on GitHub