# TAG-Bench: 80 Queries That Text2SQL Alone Cannot Answer

> TAG-Bench is a benchmark for table-augmented generation, built by modifying BIRD Text2SQL queries so that half of them need world knowledge and half need reasoning the database never stored. It ships the setup scripts, five baseline methods and an analysis script, but it assumes you already have a language model server running.

**TAG-Research/TAG-Bench** — TAG-Bench: A benchmark for table-augmented generation (TAG)

- Repository: https://github.com/TAG-Research/TAG-Bench
- Website: https://arxiv.org/pdf/2408.14717
- Stars: 765 · Forks: 85
- Language: Python
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/tag-research-tag-bench

## Eighty queries built on top of the BIRD databases

Text2SQL benchmarks ask a language model to turn a question into a SQL query and let the database do the work. TAG-Bench starts from a different premise. It describes Table-Augmented Generation, called TAG for short, as a unified and general-purpose paradigm for answering natural language questions over databases, and it positions TAG as a wider set of interactions between the model and the data than Text2SQL or retrieval-augmented generation have explored. The benchmark itself is deliberately small. It is built on the BIRD Text2SQL benchmark, and the authors use and modify a subset of match-based, comparison, ranking and aggregation queries, with 20 queries of each of those four types, 80 in total. BIRD is not re-downloaded by hand in a new format, it is reused, which is good news for anyone who has already worked with it. The scale also means this is a research probe rather than a general leaderboard, and a score on 80 items is a far noisier signal than a score on a large evaluation set.

## Half the queries test knowledge the database never stored

The queries were modified on purpose, and the modification is the interesting part of the project. To raise the difficulty, each query in the TAG benchmark was changed so that answering it requires either world knowledge or semantic reasoning that goes beyond the information explicitly available in the database. The 80 queries split evenly: 40 of them require parametric knowledge from the model, meaning facts it learned during training that appear nowhere in the schema or the rows, and the other 40 require reasoning rather than recall. This is why a plain Text2SQL baseline struggles, and it is the reason the repository says further research is needed rather than presenting a solved problem. The design has a cost worth naming. If your application genuinely has all the facts it needs in the database, a large share of this benchmark is testing something your product does not need, so a TAG-Bench score should be read as evidence about knowledge heavy question answering, not about general database work.

## Four setup scripts turn BIRD tables into dataframes and indexes

Reproduction starts from a fresh conda environment and the pinned dependency list. The setup script downloads the BIRD databases and converts them to Pandas DataFrames, the embeddings script builds indexes over each of those dataframes, and the prompts script attaches a Text2SQL prompt column to the tag_queries.csv file. The environment and install are two separate steps, because the project is also installed in editable mode so the tag package can be imported from the tag directory. The README notes that creating the indexes is recommended on a GPU, which tells you something about the size of the work: this is a preprocessing pass over every table in every downloaded database, not a one-off download of a ready-made artifact. After the scripts finish you should have a dev_folder holding the databases, a pandas_dfs folder for the DataFrames, and an indexes folder. If those three directories do not appear, the pipeline did not finish, and the later evaluation commands will fail on missing paths rather than on model behaviour.

## Running the setup and the first evaluation, step by step

Here is the order the README gives, with the commands exactly as written. The environment is created first, then the project is installed in editable mode so that the tag package resolves locally.

```bash
conda create -n tag python=3.10 -y
conda activate tag
pip install -r requirements.txt
pip install -e .
```

After that, the three setup scripts run from the setup directory. The first downloads the BIRD databases and converts them to Pandas DataFrames.

```bash
cd setup
chmod +x get_dbs.sh
./get_dbs.sh
```

The second creates the indexes over each dataframe, which the README says is best done on a GPU.

```bash
cd setup
chmod +x embed_all_dfs.sh
./embed_all_dfs.sh
```

The third fetches the Text2SQL prompts and adds a Text2SQL prompt column to the tag_queries.csv file.

```bash
cd setup
chmod +x get_text2sql_prompts.sh
./get_text2sql_prompts.sh
```

With the data in place you can run a baseline from the tag directory, for example the Text2SQL method or the reranked RAG variant, and each writes to its own output directory.

```bash
python text2sql.py --output_dir out_text2sql
```

```bash
python rag.py --output_dir out_rerank --rerank
```

Finally the analysis script reads an output directory and reports the accuracy and latency of that run.

```bash
python analyze.py --output_dir out_hand_written
```

## Five baseline methods, one comparison point

The evaluation is deliberately comparative, because the point of the benchmark is that standard methods struggle. The tag directory holds five entry points that cover the space between two extremes: a hand-written TAG pipeline run on LOTUS with an explicit list of pipeline numbers, plain Text2SQL, Text2SQL combined with a language model, plain RAG, and RAG with a reranking stage. Each writes into its own output directory, and a single analysis script then computes the accuracy and latency of a run. That gives you two axes rather than one, and latency is the more revealing of them for a TAG-style system, since the hand-written pipelines chain many operations while a Text2SQL baseline issues one query. The README points at the LOTUS documentation for configuring models, which is the honest signal that this is not a self-contained evaluation harness. You are expected to know what a pipeline is, what the numbers in the list mean, and which model server you want behind them.

## Text2SQL, RAG and reranking differ in where the model touches the rows

The four automatic baselines are worth reading as a ladder of approaches rather than as interchangeable scripts. Text2SQL alone converts the question into a query and lets the database return the answer, so the model never sees a row of data and cannot use anything it happens to know. Text2SQL plus a language model keeps that query as the retrieval step and then lets the model work with the result, which lets it apply knowledge after the fact. RAG retrieves relevant rows from the converted DataFrames and passes them to the model, which is a different path because the retrieval unit is a row rather than a query result. Running RAG again with the rerank flag adds a ranking stage, so the retrieved rows are ordered before the model sees them. The hand-written TAG pipelines are the interesting case, because the README frames them as a large explicit set of pipeline identifiers on LOTUS, which is where the Table-Augmented Generation idea gets its flexibility: the combination of operations is spelled out rather than left to a single translator.

## The language model server is a prerequisite, not an option

Before any of the five commands will produce numbers, you have to point the code at a model. The README says to run the scripts from the tag directory, to edit the lm object in each file so it points at the language model server you want to use, and to consult the LOTUS documentation for configuring models. That single sentence describes most of the real setup cost, since the lm object is edited separately in each file rather than resolved from one shared configuration, and the project does not ship a model, a key, or a default endpoint. The dependency surface is small, with requirements.txt listing a single package, lotus-ai at version 0.1.4 or newer, and the pyproject.toml defining the tag package itself at version 0.1.0 with setuptools as the build backend. The project also configures ruff with a line length of 120 and a target version of py39, while the setup instructions create the conda environment with python=3.10.

## No releases, no leaderboard, an MIT license

TAG-Bench is a research artefact rather than a maintained tool, and the repository reflects that. It has no GitHub releases, and the last push was on 2026-07-25, so there is no versioned changelog to read and no upgrade path between published snapshots. There is no CI badge, no leaderboard file and no hosted results in the top level listing, which contains .gitignore, LICENSE, README.md, assets, pyproject.toml, requirements.txt, a setup directory, the tag directory and tag_queries.csv. Licensing is the simplest part: the repository carries an MIT license, and the README links to the arXiv paper 2408.14717 that describes the paradigm, which is where the method description should be read if you want the argument behind the benchmark rather than the commands. Keep in mind that reproducing the paper results means matching their configuration, and the README does not print expected accuracy numbers for you to compare against.

## Conclusion

Adopt TAG-Bench if you are researching table-augmented generation and want a fixed set of 80 queries that punish a pure Text2SQL approach, since the queries deliberately demand world knowledge and reasoning that the source database does not contain. Skip it if you need a broad text-to-SQL evaluation, a leaderboard, or a pipeline that works out of the box, because it ships no releases and no packaged runner beyond the scripts in the tag directory. Before you build on it, run the five baseline commands with your own language model server configured in the lm object, and check analyze.py output on that run, because the README does not state the accuracy figures you should expect on your own model.

## FAQ

### How do I set up TAG-Bench from scratch?

Create a conda environment with python=3.10, install the dependencies with pip install -r requirements.txt, install the project in editable mode with pip install -e ., then run get_dbs.sh, embed_all_dfs.sh and get_text2sql_prompts.sh from the setup directory. After setup you should have dev_folder, pandas_dfs and indexes folders.

### How many queries are in TAG-Bench and what do they test?

There are 80 queries built from a subset of BIRD, with 20 each of match-based, comparison, ranking and aggregation queries. Half of them, 40 in total, require parametric world knowledge and the other 40 require reasoning beyond what the database states.

### Which baselines does TAG-Bench compare against?

The tag directory holds hand-written TAG pipelines on LOTUS, Text2SQL, Text2SQL plus a language model, RAG, and RAG with reranking via the rerank flag. A separate analyze.py command computes accuracy and latency for each output directory.

### Do I need my own language model to run TAG-Bench?

Yes. The README says to edit the lm object in each file so it points at the language model server you want to use, and to check the LOTUS documentation for configuring models. No model or endpoint ships with the repository.

## Sources

- [Issues](https://github.com/TAG-Research/TAG-Bench/issues)
- [License: MIT](https://github.com/TAG-Research/TAG-Bench/blob/main/LICENSE)
- [Project website](https://arxiv.org/pdf/2408.14717)
- [README](https://github.com/TAG-Research/TAG-Bench/blob/main/README.md)
- [TAG-Research/TAG-Bench on GitHub](https://github.com/TAG-Research/TAG-Bench)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/tag-research-tag-bench