Qwen-Drive-1.0 keeps one vision-language model and attaches three heads
An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
At a glance
- What is it?
- The Qwen team and Huazhong University built Qwen-Drive-1.0 by keeping the pretrained Qwen3.5-4B vision-language model intact and hanging a BEV perception head and a planning expert off its representations, leaving the language decoder untouched. The perception head is described in the paper as a probe rather than a module, and the results tables show the reinforcement-learned planner beating the imitation-trained one on two of three planning benchmarks and losing on the third.
- Who is it for?
- Read this as an architecture argument rather than as a driving system, because what is published is an initial step towards a driving foundation model plus its inference code, not an end to end vehicle stack. The design bet is that a general multimodal backbone already encodes usable 3D structure and that you should probe it rather than replace it, which is exactly what the BEV head is described as doing.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 74 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- QwenLM/Qwen-Drive-1.0557 ★ · +95
| Repository | 2026-09-21 | 2026-11-15 |
|---|---|---|
| QwenLM/Qwen-Drive-1.0 | 462 | 557 |
95 Stars in the last 55 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The decoder is left alone on purpose
The architecture is a statement about where to intervene. Qwen-Drive-1.0 retains the architecture of the pretrained Qwen3.5 vision-language model and integrates 3D perception, visual question answering and motion planning inside a unified framework, with the natively multimodal 4-billion-parameter model serving as the shared backbone. Two external modules attach to it. The original language decoder is explicitly unchanged and still handles both general and driving question answering. That last point is the one to hold onto when reading the results, because it means every general-purpose benchmark in the tables is measuring the same language capability the base model already had, and any movement there is a side effect of training rather than a redesign. The paper frames the project as an initial step towards a vision-language foundation model for autonomous driving, from the Qwen team together with Huazhong University of Science and Technology, with a technical report, a blog post, and weights on both Hugging Face and ModelScope. Behind the architecture is a staged training strategy that integrates perception, language and planning objectives rather than optimising them separately, and it is kept honest by a unified data pipeline with three specific jobs. It maps heterogeneous perception annotations from different sources into one shared label space, so a detection label means the same thing whichever dataset it came from. It re-annotates driving question answering responses to ensure format and factual consistency, which is a step most projects skip and then wonder why their fine-tuned answers drift in shape from their base model's. And it standardises trajectories from several public driving datasets into a single waypoint representation, which is what makes multi-dataset planning training possible at all, since two datasets rarely agree on what a trajectory looks like. Each of those three is a normalisation task rather than a modelling one, and together they are most of the engineering.
The perception head is called a probe, and that word is load-bearing
The BEV perception head is described in a way that is unusual for a component list. It jointly performs 3D object detection, semantic occupancy prediction and BEV map segmentation, three tasks that normally need dedicated supervision heads. But the stated purpose is not that it predicts these things well; it serves as a probe of the 3D information accessible from the shared representations, and it provides an explicit, inspectable interface to 3D scene structure. Read carefully, that is two claims. The first is that the general vision-language backbone's internal states already encode enough about 3D structure to train a head that reads it out. The second is that the head's real deliverable is a human-inspectable picture of the scene rather than a set of downstream detections, which is what makes it a research instrument rather than a product component. Perception results are not in the README tables at all; the detection, occupancy and map segmentation numbers are in the technical report, with the full planning tables in a separate evaluation document.
Reward optimisation wins two planning benchmarks and loses the third
The planning table has two columns and the comparison is more interesting than a uniform improvement. The imitation-trained expert is called the SFT column and the same expert after reward optimisation on the benchmark objectives is the RL column, and both share one vision-language model, so the difference is training rather than capacity. On the NAVSIM navtest benchmark scored by PDMS, the SFT model reaches 88.2 and reward optimisation lifts it to 90.7, with best-of-six figures of 89.3 and 91.4 alongside. On the Waymo Open Dataset end to end test scored by RFS the gap is much smaller, 7.78 to 7.91. On the NVIDIA PhysicalAI open-loop benchmark scored by average displacement error over three seconds, the order reverses: the imitation-trained model is better at 0.34 metres and reward optimisation makes it worse at 0.38. That regression is the most informative line in the table, since optimising against benchmark objectives rather than against imitation data is exactly the kind of change that can trade one benchmark's notion of good driving for another's.
One benchmark was rescored with a different judge, and both numbers shipped
The driving question-answering table is where the caveats live. A note states that one benchmark, LingoQA, is scored with a different model acting as judge instead of the official judge, because the official judge was found to score leniently and inconsistently across scenarios. Under the official protocol the same model obtains a higher LingoScore, and that figure is published next to the substituted one rather than left out. This is the right way to handle a protocol disagreement and it also means the headline column is not the official score, so anyone comparing against a leaderboard that used the official judge is comparing different measurements. The rest of the driving table is a clean sweep for this model on the listed sets, including a spatial reasoning benchmark and a Chinese urban driving decision set described as in-house, which is the one column where the underlying data is not public and therefore not independently checkable. A dash in the table marks a response that was invalid or could not be parsed, which is a scoring failure mode rather than a wrong answer.
General benchmarks still favour the base model in several columns
The general question-answering table is the honest counterweight to the driving tables, and it should be read column by column rather than as a row. On the driving-tuned model against its own base, the model wins on the second general benchmark listed and on a real-world question set, but the base model is ahead on the first, on the college-level reasoning set, on both variants of the harder version of it, on a chart interpretation benchmark, on optical character recognition, on a simple visual question set and on a counting benchmark. Put plainly, specialising for driving cost ground on general capability, which is the trade the staged training strategy is built to manage rather than eliminate. The stated approach is a staged one integrating perception, language and planning objectives, combining driving-specific supervision with general-purpose vision-language data so the model retains broad visual understanding and instruction following. How much is retained is what those columns measure, and the answer is mostly but not entirely. Comparisons are reproduced under one protocol with near-deterministic decoding, which is the right caveat for a table of this width.
Spatial grounding is a mixed column rather than a clean win
The spatial understanding and grounding table is worth isolating because it is the one place where the base model holds two of the five columns outright. Against the 4-billion-parameter base, the driving-tuned model is ahead on an embodied spatial benchmark, on a referring-expression question set, and on out-of-distribution detection, but behind on referring spatial grounding and on a 3D grounding benchmark. That pattern is informative rather than disappointing. The two benchmarks the base model wins are both reference-to-region style tasks, where the model has to point at a specific place in an image, and driving training appears to shift that capability slightly. The two it wins are classification and cross-domain robustness style tasks. So the specialisation is not making the model better at everything spatial; it is moving it along one axis and slightly back along another. For anyone building on this, the practical implication is that a system needing both driving QA and precise referring grounding may want to evaluate the base model as a candidate rather than assuming the driving model subsumes it.
One directory, three subfolders, and sizes listed
Distribution is unusually explicit, which helps when you are working out what you actually need to download. The weights come from Hugging Face or ModelScope, and everything ships in one directory with the vision-language model at its root, shared by every task, and each task head in a subfolder beside it. The sizes are given: 9.1 GB for the model itself, 2.1 GB for each of the two planning experts, and 0.5 GB for the perception head. That root directory is itself useful on its own, because the model alone serves the question-answering mode, so a user who never plans anything downloads 9.1 GB rather than 13.7. A head is attached at load time by naming the subfolder:
model = QwenDriveForPlanning.from_pretrained(
"Qwen-Drive-1.0-4B", planner="Qwen-Drive-1.0-4B/planner-rl", dtype=torch.bfloat16
)And there is a constraint attached to that choice which the naming does not reveal. The reward-optimised planner was optimised only on reasoning-conditioned rollouts, so it has to be run in the reasoning planning mode; the imitation-trained planner covers both direct and reasoning planning. Using the wrong one is not a quality difference, it is a mismatch of training and inference.
The verified environment is pinned and needs a special install flag
The requirements file is not a convenience list, it is a record of what the released weights were verified against, and it says so. The environment is Python 3.12 with a specific CUDA release, and the instruction is to install without build isolation so that the attention package and the causal convolution package build against the already-installed torch rather than a fresh copy. That flag is easy to miss and its absence produces a build that succeeds against the wrong torch. Every version is pinned exactly, from torch and torchvision through the transformers version, the safetensors version, the array library, the imaging library, the attention and convolution packages, the linear attention package, an OpenCV build, plotting and progress. The project metadata expresses the same dependencies as floors and ranges rather than pins, including an upper bound on the transformers major and minor version, plus optional extras for an archive format and for object storage access. So there are two different dependency contracts in the repository, and the pinned one is the one the weights were checked on. The rest of the tree is small and matches a research release rather than an installed product: the licence, the readme, an assets directory for the figures, a data directory, documentation, the two dependency files, a scripts directory and the source tree the package is discovered from. There is no test suite at the top level and no continuous integration configuration, which is consistent with a codebase whose verification story is a frozen environment plus a paper table rather than a regression suite. The optional extras are equally telling about intended use, one for reading an archive format and one for object storage client access, which together suggest the intended consumers are people loading recorded data and pushing results somewhere rather than people running a service.
Editorial conclusion
Read this as an architecture argument rather than as a driving system, because what is published is an initial step towards a driving foundation model plus its inference code, not an end to end vehicle stack. The design bet is that a general multimodal backbone already encodes usable 3D structure and that you should probe it rather than replace it, which is exactly what the BEV head is described as doing. Two things to weigh before you rely on the numbers. The judgement protocol, because one driving benchmark was rescored with a different judge and the alternative figure under the official protocol is published alongside. And the trade-off on general benchmarks, since several of those columns still favour the unmodified base model, so the driving gains are not free. Also read the pinned environment rather than the dependency floors, since the verified configuration names exact versions, a specific CUDA release, and an install flag needed to build two of the packages against the installed torch.
Frequently asked questions
Is there any open-source autonomous driving software available?
There is open source inference code for a driving vision-language model. Qwen-Drive-1.0 is Apache-2.0 licensed, with weights published on Hugging Face and ModelScope and the inference code, packaging and pinned environment in the repository. It is described as an initial step towards a driving foundation model rather than as a complete vehicle stack.
What is Qwen-Drive-1.0?
A vision-language model for autonomous driving that keeps the architecture of the pretrained Qwen3.5-4B model and attaches two external modules: a BEV perception head doing 3D detection, semantic occupancy and BEV map segmentation, and a planning expert that generates future ego trajectories. The original language decoder is unchanged and still handles general and driving question answering.
How do I load the planning head in Qwen-Drive-1.0?
Name the head subfolder when loading the shared model, passing the planner argument to the from_pretrained call along with a dtype. The reward-optimised planner must be run in reasoning planning mode because it was optimised only on reasoning-conditioned rollouts; the imitation-trained planner supports both direct and reasoning planning.
What is the difference between planner-sft and planner-rl?
Both are the same planning expert architecture sharing one vision-language model. The imitation-trained one is the supervised column and the other is the same expert after reward optimisation on benchmark objectives. It scores higher on two planning benchmarks and slightly worse on an open-loop displacement error benchmark.
What environment were the Qwen-Drive-1.0 weights verified with?
Python 3.12 with a specific CUDA release, with every dependency pinned exactly, including torch, transformers, an attention package and a causal convolution package. The install must run without build isolation so those two packages compile against the installed torch rather than a separate copy.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/qwenlm-qwen-drive-1-0)Star history chart for your README
[](https://hysenlabs.com/projects/qwenlm-qwen-drive-1-0)Was this useful?