Senna: a vision-language model that plans a driving manoeuvre in words

What the system does

Senna is described as an autonomous driving system that integrates a large vision-language model with an end-to-end model to improve planning safety, robustness and generalization. The paper title puts it more precisely: bridging large vision-language models and end-to-end autonomous driving.

The problem it addresses is a real gap in end-to-end driving research. An end-to-end model maps sensor input to control output directly, which is efficient and learned rather than hand-designed, but it is opaque. When the vehicle does something unexpected there is no explanation, and there is no intermediate representation you can inspect, which makes debugging, safety analysis and error attribution very hard.

Senna's answer is to insert a language step. The vision-language model looks at the multi-view scene and produces a meta-action, a description of the manoeuvre the vehicle intends to make, along with the reasoning behind it. That language output is then what the end-to-end planner consumes.

So the language is not a bolt-on report generated after the decision. It is in the middle of the pipeline, and it is the thing the driving model acts on. The README's two highlight bullets say the system achieves state-of-the-art planning performance and demonstrates strong cross-scenario generalization and transferability.

The word meta-action is the key vocabulary. A meta-action is a coarse, discrete description of what the vehicle is about to do, of the kind that a language model can produce reliably, rather than the continuous trajectory an end-to-end planner ultimately needs. The VLM supplies the coarse decision and the justification, and the end-to-end model refines it into control.

The primary language is Python, the licence is Apache-2.0, the default branch is main, and the homepage is the arXiv paper at 2410.22313. The last push was on 2026-08-12 with the project not archived, and the repository publishes no GitHub releases, so the artefacts are the paper, the code and the weights on Hugging Face.

The authors are from Huazhong University of Science and Technology and Horizon Robotics, which is worth noting because it is a combination of an academic computer vision group and a company that builds production driving systems.

The timeline, and what is actually released

The News section is dated, and reading it tells you how the project has been delivered.

On 2024-10-29 the Senna paper was released, with the note that code and models were coming soon. On 2024-12-08 the code and the weights of Senna-VLM were released, along with the training and evaluation scripts. On 2026-03-16 a follow-up work, Senna-2, was released as a separate arXiv paper. And on 2026-08-11 Senna was accepted by IJCV 2026.

So there is a gap of about six weeks between the paper and the code, which is quick by the standards of research releases, and then a long stretch before the successor work appeared. The acceptance by IJCV, a computer vision journal rather than a conference, is the formal validation of the work, and it comes nearly two years after the arXiv preprint.

The weights are on Hugging Face at the Senna model repository, and the model card is where you would look for the architecture details the README does not repeat. The single row of the weights table gives the configuration: the method Senna, a model size of 7B, a base LLM of vicuna-7b-v1.5, an input view count of 6, 128 tokens per image, and a link to the Hugging Face download.

Six views at 128 tokens each is the input budget. A multi-camera autonomous driving setup usually surrounds the vehicle with cameras looking forward, backward and to the sides, and six views with a small per-image token allowance is a very specific trade: the model sees the whole scene geometrically but each image is heavily compressed in token space, which is the only way a 7B model can hold a surround view at all.

The Vicuna base is worth noting as a historical detail. vicuna-7b-v1.5 was one of the standard open model bases in the LLaVA era, and choosing it means the language half of the system is a fine-tuned derivative of an existing open model rather than something trained from scratch.

The related projects section links VAD and VADv2 and MapTR, also from the same group, which tells you the end-to-end model and the mapping component come from the authors' own prior work.

Setting up, and the data generation step

The installation is four commands, and the environment is pinned to a specific Python version:

git clone [email protected]:hustvl/Senna.git
conda create -n senna python=3.10 -y
conda activate senna
pip install -r requirements.txt

The requirements file is very long, which is a reliable signal of the scale of the dependency set. It pins the training stack across the board: accelerate at 0.29.2, deepspeed at 0.14.1, bitsandbytes at 0.42.0, datasets at 2.14.6, huggingface-hub at 0.22.2, matplotlib at 3.8.4, einops at 0.6.1, aiohttp at 3.9.5 and boto3 at 1.34.92, among many more. The presence of deepspeed and bitsandbytes is the practical detail: this is a distributed training setup with mixed precision and quantisation support, so reproducing the training runs means a multi-GPU machine.

The data preparation step is the one that catches people out, because it needs another model.

A script is provided for generating the question and answer data required for Senna training. The script uses LLaVA-v1.6-34b as the model for generating scene descriptions and planning explanations, and you run it with:

sh data_tools/senna_nusc_converter.sh

So the training data for the driving system is itself generated by a large vision-language model. That is a self-bootstrapping arrangement. A 34-billion-parameter LLaVA model writes the scene descriptions and the explanations of what the driving system should be doing, and those become the supervised target for a 7B model that then drives.

This is the standard way to build a large-scale vision-language driving dataset, and it is also the main reproducibility caveat. The training data is not a static file you download; it is produced by running a large model over the driving dataset, so the exact content depends on the LLaVA checkpoint and on the prompts in the shell script. Anyone regenerating the data will get something close to the original but not identical, which is a real difference from a setup where the labels are fixed.

The script is named senna_nusc_data_converter.py, and the shell wrapper senna_nusc_converter.sh, so the driving dataset in use is nuScenes, which is the standard academic benchmark for this kind of work.

The three training stages, and the full-parameter recommendation

Training is described in two commands covering three stages, and the fine-tuning stage has a parameter-efficient variant.

Stage one is mix pre-training:

sh train_tools/pretrain_senna_llava.sh

Stages two and three, driving fine-tuning and planning fine-tuning, use:

sh train_tools/train_senna_llava.sh

And the LoRA variant of the same two stages is:

sh train_tools/train_senna_llava_lora.sh

The three-stage structure is the interesting part. Stage one pre-trains on a broad mixture, which is where the model learns to describe scenes and reason about them. Stage two fine-tunes on driving data, so the model learns the vocabulary and the conventions of the driving domain. Stage three fine-tunes specifically for planning, which is the task the meta-action output serves.

Splitting the driving and planning fine-tuning is a deliberate separation, and it reflects the two-part structure of the output: a description of the scene, and a decision about what to do. A model can be excellent at describing a roundabout and still be poor at deciding which lane to take, and separating the stages means the second one is not fighting the first.

The README then makes a recommendation and gives the reason, which is unusual and welcome. In their experiments, full-parameter fine-tuning outperforms LoRA fine-tuning, so they recommend it. However, if your machine has limited GPU memory, for example only 24GB, you may consider LoRA fine-tuning as an alternative.

So the default is the expensive option and the cheaper one is the documented fallback, with the reason stated. That is more useful guidance than most research repositories give, and the 24GB figure is a concrete threshold: a single modern accelerator can run the LoRA path, and the full path needs more.

The evaluation and visualisation steps are single commands each. Evaluation uses senna_plan_cmd_eval_multi_img.sh to evaluate the accuracy of Senna meta-action planning, and the multi-image suffix tells you the evaluation is over the six-view input. Visualisation uses senna_plan_visualization.sh, which overlays the predicted meta-actions and the front-view scene descriptions onto the front-view image and saves the results to the specified path.

That visualisation script is the fastest way to see what the system actually does, and the README's qualitative results section links a figure of exactly that output.

The codebase, and why llava/ is in the tree

The top-level listing explains the architecture better than the prose does. There are data_tools/, train_tools/ and eval_tools/, which map onto the three phases of the workflow. And then there are llava/ and llava_next/.

Those two directories are the fork. The acknowledgment section says LLaVA, the codebase we built upon, we sincerely thank the contributors for their great work. So Senna is built on top of LLaVA rather than from scratch, and the llava_next/ directory suggests the newer LLaVA-NeXT generation was also brought in.

Forking a codebase like this is the normal practice in this area of research. LLaVA is a training recipe rather than a model, and the pieces Senna needs, the six-view input handling, the way the vision encoder feeds the language model, the training loop, all have to be modified, and modifying a fork in place is far simpler than upstreaming changes that only one paper needs.

The cost of the fork is that the tree carries a lot of code that is not Senna's own, and that keeping it in step with upstream is somebody's job. It also explains why the requirements file is so long: it covers the training infrastructure of the fork as well as Senna's own additions.

The other top-level entries are assets/ for the images in the README, requirements.txt, and the LICENSE and README themselves. There is no inference server, no deployment configuration and no packaged release, which is consistent with a research code release: the artefacts you consume are the paper, the weights and the scripts.

The project also links Senna-2 as a separate arXiv paper rather than folding it into this repository, so the two lines of work are versioned separately.

For a reader, the practical path through the tree is the order the README uses: data_tools to build the training set, train_tools to run the three stages, eval_tools to measure and visualise the result. The scripts are thin wrappers around the forked code underneath, and the configuration of the actual experiments lives in the shell scripts and in the forked Python.

How to evaluate it, and what to check first

There are five things to consider.

First, what you would actually use it for. Senna is a research system for end-to-end autonomous driving research, and the artefact is a model that produces meta-actions and explanations on the nuScenes benchmark. It is not a driving stack you put in a car, and there is no inference server, no calibration handling and no vehicle interface in the repository. If you are doing research on language-guided planning, it is a strong reference implementation.

Second, the hardware. The training recipe uses deepspeed and bitsandbytes, and the README mentions 24GB of GPU memory as the constrained case where you fall back to LoRA. The data generation step needs LLaVA-v1.6-34b, which is a much larger model than the 7B you are training. So reproducing the pipeline is a multi-stage GPU project rather than a weekend one.

Third, the data generation caveat, which is the thing most often to surprise people. Because the question and answer data is produced by running LLaVA over the driving data, regenerating it does not reproduce the original exactly. If your goal is to match the paper's numbers, the regenerated labels will differ from the authors' labels, and the comparison is no longer exact. If your goal is to train the architecture yourself, that does not matter.

Fourth, the model weight and its licence situation. The weights are published on Hugging Face, and the repository is Apache-2.0, but the base LLM is vicuna-7b-v1.5 and the training data is generated by LLaVA, which has its own terms. For commercial use the terms of the base model and of the data generation model are the parts to check, not the Apache licence on this repository.

Fifth, what came next. Senna-2 exists as a separate paper from 2026, and the same group has VAD, VADv2 and MapTR. So Senna is a step in a line of work rather than a final answer, and if you are choosing between reading this and reading Senna-2, the latter is where the current state of the art in that line is.

What the repository gives you is a complete, honest research release. The code and weights were released within two months of the preprint, the training recipe is three shell commands, the evaluation is one more, and the recommendation about full-parameter versus LoRA fine-tuning is given with a reason and a memory threshold. For a paper in this area, that is a well-organised release.

Editorial conclusion

Adopt Senna if you are researching language-guided end-to-end driving and want a complete reference implementation with the weights, the three-stage training recipe and the evaluation scripts, since the code and the 7B checkpoint were released within two months of the preprint. Do not adopt it as a vehicle stack, because there is no inference server, no calibration handling and no vehicle interface, and do not expect to reproduce the paper's numbers exactly, because the training labels are generated by running LLaVA over the driving data and regenerating them will not match the authors' labels. Before you commit, check the base model's terms rather than the repository's Apache licence, since the weights derive from vicuna-7b-v1.5 and the data from LLaVA, budget for a multi-GPU setup with LLaVA-v1.6-34b needed for the data step, and read Senna-2 as well, since it is the same group's more recent work on the same line.

Frequently asked questions

Official sources

1. hustvl/Senna on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README