# TravelPlanner: a benchmark for travel plans that hold up to constraints

> TravelPlanner is an ICML 2024 spotlight benchmark that asks a language agent to build a day-by-day trip and then checks it against environment, commonsense and hard constraints, which is the part that separates it from a prompt exercise. It ships two evaluation modes, a postprocessing chain that parses plans into JSON, and a leaderboard, and it takes a firm position on what counts as cheating.

**OSU-NLP-Group/TravelPlanner** — [ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"

- Repository: https://github.com/OSU-NLP-Group/TravelPlanner
- Website: https://osu-nlp-group.github.io/TravelPlanner/
- Stars: 559 · Forks: 89
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/osu-nlp-group-travelplanner

## A benchmark for plans that fail constraints

TravelPlanner is described as a benchmark crafted for evaluating language agents in tool-use and complex planning within multiple constraints. For a given query, the agent is expected to formulate a comprehensive plan including transportation, daily meals, attractions, and accommodation for each day.

The constraint types are what make it a benchmark rather than a toy. From the perspective of real world applications, TravelPlanner includes three types of them: Environment Constraint, Commonsense Constraint, and Hard Constraint. The environment constraints come from the reference database, the commonsense constraints are things a person would consider obvious, and the hard constraints are the ones written into the query itself.

That three-way split is the design decision worth understanding. A plan can be fluent, well ordered and internally consistent, and still be wrong, because it schedules a restaurant that is closed on the day it visits, or a flight that lands after the check-in window, or a hotel that is too far from the requested attraction. Language models are good at the first two and bad at the third, and the whole point of the benchmark is to measure the third.

The paper is the one titled TravelPlanner: A Benchmark for Real-World Planning with Language Agents, arXiv 2402.01622, and the README marks it as an ICML 2024 Spotlight. The code is Python, the licence is MIT, the default branch is main, and the last push to the repository was on 2026-05-24.

## Two modes, and the difference between them

The README describes two ways to run an agent against the benchmark, and the difference is the most important thing to understand before you start.

In the two-stage mode, language agents are tasked with employing various search tools to gather information. Based on the collected information, they are expected to deliver a plan that meets the user's needs in the query and also adheres to commonsense constraints. The agent does the searching itself, using the tools in the agents/ directory, which is why the database of reference information is a prerequisite.

The sole-planning mode is described as an easier mode solely focused on testing the planning ability. The README's wording is that it makes sure no important information is left out, thereby letting agents concentrate on planning itself. In practice the agent is given the information and only has to assemble the plan, so the tool-use half of the problem is removed and the measurement isolates the planning half.

The sole-planning mode also takes a strategy argument, and the supported values are direct, cot, react and reflexion. That set is worth noting: it covers the range of prompting and agent-loop techniques the benchmark was designed to compare, from a single-shot completion through chain of thought, a ReAct loop with tools, and reflexion with self-critique.

The two modes are also the two halves of the cheating question. Since the sole-planning mode hands over the reference information, a model can in principle be tuned on it, and since the test set answers are hidden behind a leaderboard, the maintainers have had to draw a line around what counts as legitimate use.

## Setting up the environment and the database

The setup is a conda environment with the dependencies from requirements.txt.

```bash
conda create -n travelplanner python=3.9
conda activate travelplanner
pip install -r requirements.txt
```

Python 3.9 is pinned in the create command, and the dependency set tells you what the harness is built on: langchain at 0.1.4 with langchain-community, the openai package at 0.27.2, langchain_google_genai, pandas, tiktoken, gradio, datasets, func_timeout, python-dotenv and ollama.

The pinned old versions are the interesting part. openai 0.27.2 is the last release before the client was restructured around the responses and chat completions resources, and langchain 0.1.4 is a generation-old release of the agent framework. Anyone who wants to rebuild this against current library versions will find the tool-calling interfaces have moved, and the harness code in agents/ is written against those older APIs.

The second setup step is the data. You download the database and unzip it to the TravelPlanner directory, at the path your/project/TravelPlanner. The database is a separate download rather than a Git submodule, and the top level of the repository has a database/ directory for the reference information, with an update from 2024/7/14 recording support for reference information in JSON format.

The ollama dependency in the requirements is a local model path, and the list of supported models given in the README is gpt-3.5-turbo-X, gpt-4-1106-preview, gemini, mistral-7B-32K and mixtral.

## Running the agents, and where the plans come out

The two-stage run is configured entirely through environment variables, and the script is invoked from the agents directory.

```bash
export OUTPUT_DIR=path/to/your/output/file
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
export SET_TYPE=validation
cd agents
python tool_agents.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME
```

SET_TYPE is either validation or test, and the generated plan is stored in OUTPUT_DIR/SET_TYPE. The comment in the README notes that if you do not want to test Google models, like gemini, you just input 1 for the Google key, which is a small touch that saves a failed call.

The sole-planning run adds the strategy variable and lives under tools/planner.

```bash
export STRATEGY=direct
cd tools/planner
python sole_planning.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY
```

The output of both is a natural language plan, not a structured one. That is the reason the postprocess directory exists and the reason it is a separate step, and it is also where the pipeline has an unusual dependency on a model: to parse natural language plans into JSON formats the project uses gpt-4, and the maintainers encourage developers to try different parsing prompts to obtain better-formatted plans.

That means the evaluation of an agent's plan is mediated by a second model call. A parsing failure looks like a planning failure unless you are careful, and the encouragement to try different parsing prompts is an admission of that. It is a real cost of the design, since the benchmark has to accept free-form plans from arbitrary agents and normalise them before scoring.

## Postprocessing and offline evaluation

The postprocess chain has three steps, and the README gives them as a sequence.

```bash
cd postprocess
python parsing.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --tmp_dir $TMP_DIR
```

That step converts the natural language plans into JSON, writing intermediate files to TMP_DIR. The second step extracts the individual elements from the parsed plans.

```bash
python element_extraction.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --tmp_dir $TMP_DIR
```

And the third combines the plan files for evaluation, with an example file named example_evaluation.jsonl provided in the postprocess folder.

```bash
python combination.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE  --submission_file_dir $SUBMISSION_DIR
```

The MODE variable is two-stage or sole-planning, so the same postprocessing chain handles both modes and the difference is recorded in the output.

Offline evaluation on the validation set is a single script.

```bash
export SET_TYPE=validation
export EVALUATION_FILE_PATH=your/evaluation/file/path

cd evaluation
python eval.py --set_type $SET_TYPE --evaluation_file_path $EVALUATION_FILE_PATH
```

The README is explicit about why you should not do this for the test set: to avoid data contamination, use the official leaderboard for test set evaluation, and the leaderboard is a Hugging Face space at osunlp/TravelPlannerLeaderboard. The practical consequence is that a test set number is only obtainable by submitting, which is what makes the leaderboard meaningful as a shared reference point, and the A TravelPlannerEnvironment space is also published for running the environment interactively.

There is a format check tool as well, at postprocess/format_check.py, provided for test set submission files, so you can check whether there are format errors in your file before you submit it.

## Released models, and the line against cheating

The README's updates section records that on 2024/10/23 the project released models finetuned on TravelPlanner, with the data in the finetuning_data directory, and that the fine-tuning was done with LLaMA-Factory using its Llama 3 LoRA SFT example script.

Releasing a finetuned model on the training set alongside the benchmark is a deliberate move. It gives new groups a starting point that is not a raw API model, and it gives the benchmark a reference point for what supervised fine-tuning on the training split buys. It also means the comparison between an official finetuned model and someone else's finetuned model is only meaningful if both used the same data, and the finetuning_data directory is published so that they can.

The Warnings section is the other half of the project's stance, and it is unusually direct. The maintainers release the evaluation scripts to encourage new methods and to aid their development, and they encourage the use of evaluation feedback in the training set, such as implementing reinforcement learning techniques, to enhance learning. But they strictly prohibit any form of cheating in the validation and test sets, to uphold the fairness and reliability of the benchmark's evaluation process, and they reserve the right to disqualify results.

The listed violations begin with reverse engineering of the dataset, and the README text available here is cut off in the middle of that item. The distinction the project is drawing is the sensible one: learning from feedback on the training set is fine, and the leaderboard is designed to support that, but the test set is supposed to be a measurement of generalisation rather than a lookup table.

The warning about cheating was updated on 2024/4/28, which suggests the policy was tightened after the first results came in.

## Where TravelPlanner is the wrong tool

Three cases point elsewhere. The first is a team looking for a travel planning product. This is a test harness and a dataset. There is no user interface, no itinerary output formatted for a human, and no booking capability, and the sole-planning mode exists to isolate a capability under measurement rather than to be used. A group that wants a working planner is looking for something else entirely.

The second is a team without the reference database. The two-stage mode depends on the agent searching the provided information, and that information is a separate download that has to be unzipped into the project directory. Without it, the agent has nothing to search, and the sole-planning mode is a fallback that measures something narrower.

The third is a team that wants to evaluate something other than travel planning. The queries are travel queries, the constraints are travel constraints, and the reference data is travel data. The general technique of constrained planning evaluation transfers, but the harness does not, and the domain is a hard one in an instructive way: the constraint types are the ones a person would notice being violated, which is exactly why a fluent model fails it.

The alternative worth weighing is a general agent benchmark built on tool use and instruction following, and the difference in approach is the same class of difference the two modes make. A general benchmark checks whether the agent called the right tool and produced a well-formed answer, while TravelPlanner checks whether the produced plan is satisfiable under constraints that the answer must satisfy simultaneously and that are only discoverable by searching. The second is a much closer approximation of the failure mode that matters in production, and it is also much more expensive to run, since it needs a retrieval corpus and a model-mediated parsing step that a simpler benchmark does not.

## Conclusion

TravelPlanner fits a research group measuring whether a language agent can produce a plan that satisfies many simultaneous constraints rather than a plausible itinerary, and a group that wants a comparable leaderboard number for its own agent. It does not fit a team looking for a trip planner to use, since it is a test harness rather than an application, and it does not fit a team without a database of reference information, since the released database is a prerequisite for the two-stage mode. Before you run it, set the model name and the API key from the environment variables rather than editing scripts, and read the Warnings section, because the maintainers disqualify results that reverse engineer the dataset.

## FAQ

### What is TravelPlanner?

It is a benchmark for evaluating language agents in tool-use and complex planning within multiple constraints, published as an ICML 2024 Spotlight. For a given query the agent must produce a day-by-day plan covering transportation, meals, attractions and accommodation, checked against environment, commonsense and hard constraints.

### What is the difference between the two-stage and sole-planning modes?

In the two-stage mode the agent uses search tools to gather information itself and then plans. The sole-planning mode gives the agent the information so that no important information is left out, letting the agent focus on planning itself, and it supports the direct, cot, react and reflexion strategies.

### How do I set up TravelPlanner?

Create a conda environment with python=3.9, activate it, and run pip install -r requirements.txt. You also need to download the database and unzip it to the TravelPlanner directory, at the path your/project/TravelPlanner, before running the agents in the agents or tools/planner directory.

### Can I evaluate on the TravelPlanner test set locally?

The offline evaluation script is for the validation set, and the README asks you to use the official leaderboard for test set evaluation to avoid data contamination. The leaderboard is published as a Hugging Face space at osunlp/TravelPlannerLeaderboard, and a format check tool is provided at postprocess/format_check.py for submission files.

### What counts as cheating on the TravelPlanner benchmark?

The maintainers encourage using evaluation feedback on the training set, such as reinforcement learning techniques, but strictly prohibit any form of cheating in the validation and test sets, and reserve the right to disqualify results. The listed violations begin with reverse engineering of the dataset, and the warning about cheating was updated on 2024/4/28.

## Sources

- [Issues](https://github.com/OSU-NLP-Group/TravelPlanner/issues)
- [License: MIT](https://github.com/OSU-NLP-Group/TravelPlanner/blob/main/LICENSE)
- [OSU-NLP-Group/TravelPlanner on GitHub](https://github.com/OSU-NLP-Group/TravelPlanner)
- [Project website](https://osu-nlp-group.github.io/TravelPlanner/)
- [README](https://github.com/OSU-NLP-Group/TravelPlanner/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/osu-nlp-group-travelplanner