FSDrive: visual chain of thought for end-to-end driving models

What the paper adds to a driving model

FSDrive is the official implementation of FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving, a NeurIPS 2025 spotlight paper with an arXiv entry at 2505.17685 and a project page hosted at miv-xjtu.github.io.

The claim in the README is that a spatio-temporal chain of thought enables an end-to-end autonomous driving vision language action model to think visually about trajectory planning, unifying visual generation and understanding with minimal data, and advancing autonomous driving toward visual reasoning for the first time.

The mechanism is that the model does not only emit a trajectory as coordinates. It also produces a visual representation of the scene as tokens, and a decoded image of what it thinks it is seeing is compared against the actual camera view. A chain of thought that is expressed as pixels can be inspected by a human, and a mismatch between the generated visual and the real one is a signal that the model is reasoning about the wrong thing.

The authorship spans industry and academia, with the affiliations listed as Amap at Alibaba Group, Xi'an Jiaotong University and the DAMO Academy at Alibaba Group. The repository is the reference code for a paper, and the README is structured as a reproduction guide rather than as a user manual for a tool.

The repository layout and what each directory does

The top level of the repository is a good map of the pipeline. There is an LLaMA-Factory/ directory, a MoVQGAN/ directory, a create_data/ directory, a tools/ directory, a configs/ directory, an assets/ folder and a requirements.txt.

LLaMA-Factory is the fine-tuning framework the work is built on, and it is vendored into the repository rather than installed from upstream. The training entry points are LLaMA-Factory configs, so the model is trained with the llamafactory-cli command rather than a custom loop.

MoVQGAN is the bridge between images and tokens. The README describes extracting the visual tokens of the front view from both pre-trained and fine-tuned data, with separate scripts for the two, and the visual tokens are what the model emits and what is later restored to pixel space for visualisation.

create_data/ holds the scripts that construct the pre-training and fine-tuning datasets in the LLaMA-Factory format, and it also holds the cached_nuscenes_info.pkl file described in the README.

tools/ contains the evaluation and visualisation code, including the matching script, the metric computation under tools/data/evaluation/ and the trajectory and CoT visualisation scripts. The configs/ directory holds the pretrain.yaml and sft.yaml files that the training commands point at.

This structure is worth noting for a reader deciding whether to adopt anything: the data preparation, the model training and the analysis are separate, and each can be run and inspected on its own.

Building the environment with pinned versions

The install is a conda environment with a specific Python and a specific CUDA build of PyTorch, followed by the vendored LLaMA-Factory install.

git clone https://github.com/MIV-XJTU/FSDrive.git && cd FSDrive

conda create -n FSDrive python=3.10 -y && conda activate FSDrive

# CUDA 12.4
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124

cd LLaMA-Factory && pip install -e ".[metrics,deepspeed,liger-kernel,bitsandbytes]" --no-build-isolation

cd .. && pip install -r requirements.txt

The pins are exact rather than minimums. PyTorch 2.5.1, torchvision 0.20.1 and torchaudio 2.5.1 come from the CUDA 12.4 wheel index, and Python is 3.10. The LLaMA-Factory install is an editable install with the metrics, deepspeed, liger-kernel and bitsandbytes extras, and the --no-build-isolation flag matters because the extras need to see the already installed torch rather than resolving their own copy.

The requirements.txt at the root reinforces the pinning, with numpy pinned to 1.26.4, Requests to 2.32.4 and setuptools to 76.0.0, and a floor rather than a pin on transformers at 4.49.0 and vllm at 0.7.3. Other entries show the shape of the stack: nuscenes_devkit for the dataset, pytorch_lightning and peft for training, lmdb and datasets for data handling, vllm for inference, and openai, fastapi, gradio and uvicorn for serving.

The consequence for a reader is that this is a pinned reproduction environment, not a library to install alongside something else.

Preparing nuScenes data and extracting visual tokens

The data preparation has three stages, and the first one is the dataset itself. The README asks for the complete nuScenes dataset to be downloaded and extracted to ./LLaMA-Factory/data/nuscenes, or for a soft link to be created instead.

ln -s /path/to/your/nuscenes LLaMA-Factory/data

The README also mentions pre-cached data hosted on Google Drive, with the cached_nuscenes_info.pkl file located in the ./create_data directory and the metrics folder placed in ./tools/data.

The second stage extracts visual tokens from the front view, separately for the pre-trained and the fine-tuned data, so that the supervised MLLM has what it needs.

python MoVQGAN/pretrain_data.py
python MoVQGAN/sft_data.py

The third stage constructs the datasets themselves in the format LLaMA-Factory expects, one for pre-training and one for fine-tuning.

python create_data/pretrain_data.py
python create_data/sft_data.py --split train # Change to "val" for constructing the validation set

The last step is manual. The README says to follow the LLaMA-Factory tutorial and add the dataset information to the file ./LLaMA-Factory/data/dataset_info.json. That is a small but real friction point, because a file has to be edited by hand before training will find the data.

Two-stage training with llamafactory-cli

Training happens in two stages, both driven from inside the LLaMA-Factory directory with the llamafactory-cli command.

The first stage pre-trains the VLM to activate its visual generation capabilities.

cd LLaMA-Factory
llamafactory-cli train ../configs/pretrain.yaml

The second stage takes the pre-trained parameters and fine-tunes the VLM to think visually about trajectory planning.

llamafactory-cli train ../configs/sft.yaml

The two-stage structure is the practical point here. The visual generation ability has to exist before the model can be taught to plan by generating, so the pre-training run is not optional and a fine-tuning run on its own starts from a base model that has never produced visual tokens.

The configs referenced here are files in the repository's configs/ directory, and the README does not reproduce their contents, so the exact model, learning rate and data mix are only visible by opening the repository. The requirements show the training stack: peft for parameter efficient tuning, pytorch_lightning for the training loop, deepspeed through the LLaMA-Factory extras for distributed training, and liger-kernel as an additional kernel optimisation.

The inference stage runs inside LLaMA-Factory as well, using a vLLM-backed script with a qwen2_vl template, a cutoff length of 32768 and a maximum of 2048 new tokens, at a temperature of 0.1 with top_p 0.1 and top_k 10, writing to results.jsonl.

Evaluation, metrics and visualising the chain of thought

Evaluation is two steps. The first matches the predicted results against the token trajectories so the comparison can be computed, and it runs from the FSDrive directory rather than from LLaMA-Factory.

python tools/match.py \
--pred_trajs_path ./LLaMA-Factory/results.jsonl \
--token_traj_path ./LLaMA-Factory/data/val_cot_motion.json

The second computes the end-to-end trajectory planning metrics, the L2 error and the collision rate.

python tools/evaluation/evaluation.py \
--metric uniad \
--result_file ./LLaMA-Factory/eval_traj.json

The metric argument selects the calculation method, and the README notes that it can be changed to stp3 to use the ST-P3 calculation instead. That is a methodological choice rather than a formatting difference, and the two are not interchangeable, since ST-P3 accounts for the future states of other agents in the scene.

The visualisation scripts are the part that makes the paper's claim inspectable. One draws the predicted trajectory from the results file and the token file.

python tools/visualization/visualize_planning.py \
--pred-trajs-path ./LLaMA-Factory/results.jsonl \
--tokens-path ./LLaMA-Factory/eval_traj.json \
--output-path ./vis_traj

The other restores the visual tokens back into pixel space so the chain of thought can be looked at as an image.

python ./MoVQGAN/vis.py \
--input_json ./LLaMA-Factory/eval_traj.json \
--output_dir ./vis_cot

For a reader evaluating the idea rather than the numbers, that second script is the interesting one, because it is the point at which the visual chain of thought stops being a claim and becomes something a person can look at.

Where FSDrive is the wrong tool

The first limitation is the absence of a stated licence. The licence field on the repository carries no identifier that can be quoted, and the README does not discuss licensing either. The code is readable and the paper is published, but for a research group planning to build on the implementation, the terms need to be confirmed with the authors rather than assumed from the fact that the repository is public.

The second is the scale of the reproduction. nuScenes is a large dataset, the training runs are two full stages of a vision language model, and the requirements include deepspeed and liger-kernel, which are there for multi-GPU training. This is not a codebase someone runs on a laptop to see how it works.

The third is the dependency on vendored and external state. LLaMA-Factory is a fork inside the repository, the dataset registry entry has to be added by hand, and the pre-cached data file is on Google Drive. A reproduction depends on several things being present and matching versions, and the README is a guide to the authors' own setup rather than a specification anyone else can satisfy from scratch.

The fourth is the documentation. The README covers installation, data, training, inference, evaluation and visualisation in a compact form, and points to the LLaMA-Factory tutorial for the dataset format, but the config files, the MoVQGAN details and the data construction are not explained. The Acknowledgement section, which credits the codebases the work is based on, is cut off in the version available here.

The fifth is the project status. The repository has no GitHub releases and the last push was on 2026-05-08, so it is a snapshot of a paper's code rather than a maintained package.

FSDrive compared with UniAD, ST-P3 and a plain VLA baseline

The evaluation options in the README point at the two frameworks this work is compared against. UniAD is the metric name used for the default calculation, and ST-P3 is the alternative that accounts for the future behaviour of other actors rather than treating them as static obstacles. Both are planning frameworks in the end-to-end autonomous driving literature, and the fact that the metric can be switched between them is a sign the authors wanted their numbers comparable with the established baselines.

The difference in approach with UniAD is architectural. UniAD is a planning-centric system in which perception, prediction, occupancy and planning are separate modules optimised jointly, and the trajectory is produced by a dedicated planner head. FSDrive works inside a vision language model that is asked to produce the trajectory as generated tokens, with the visual chain of thought as the intermediate representation. The bet is that a model that can describe the scene in pixels plans better than one that outputs coordinates directly, and that the visualisation of that description is worth the data cost.

The comparison with a plain VLA baseline is the one the paper title argues against. A vision language action model without the visual chain of thought still predicts trajectories, and the README's claim about unifying visual generation and understanding with minimal data is a claim about doing both with one model rather than two.

For a reader choosing where to start, the practical note is that the alternative is not expensive to test. The repository exposes the metric switch, and a baseline that predicts trajectories without the visual tokens can be evaluated with the same matching and metric scripts, which is exactly the comparison a reader who doubts the approach would want to run.

Editorial conclusion

FSDrive suits a research group that already trains vision language action models and wants a concrete implementation of visual reasoning for trajectory planning rather than another dataset pipeline. It is a poor fit for anyone hoping to run an autonomous vehicle, since the repository is a training and evaluation codebase for a published result, and a poor fit for a team without the nuScenes data and several GPU days of pretraining budget. Before using it, check what the licence field on the repository permits, because no licence identifier is recorded and the code is otherwise freely readable, and confirm the CUDA 12.4 and PyTorch 2.5.1 combination in the install steps against the hardware you have, since the pinned versions are not adjustable without reworking the requirements.

Frequently asked questions

Official sources

1. Issues
2. MIV-XJTU/FSDrive on GitHub
3. Project website
4. README