Qwen-Drive-1.0: a vision-language foundation model for autonomous driving

What Qwen-Drive-1.0 actually is, and who it is for

Most driving stacks are assembled from separate specialists: one detector for 3D boxes, one occupancy network, one planner. Qwen-Drive-1.0 takes the opposite route. It keeps the architecture of the pretrained Qwen3.5 vision-language model and bolts on two external modules, so that 3D perception, visual question answering and motion planning all run through one shared representation. The README describes this as "a unified framework" and frames the release as an initial step rather than a finished product.

The intended reader is a research engineer or graduate student who wants to probe how far a general vision-language backbone can be pushed into driving tasks without giving up general visual understanding. The performance tables make that trade-off explicit: Qwen-Drive-1.0-SFT scores 85.5 on MMBench and 72.7 on MMMU, against 87.1 and 73.4 for the untouched Qwen3.5-4B base. Driving-specific numbers move much more, with LingoQA going from 70.4 to 77.8. So the pitch is not "better at everything". It is "much better at driving, slightly worse at general benchmarks".

If you are building a production vehicle stack, this is not the artifact you are looking for. There is no training code in the repository layout, no data preparation scripts beyond a data/ directory, and no deployment tooling. What you get is inference code plus released weights.

How the shared VLM, BEV head and Planning Expert fit together

The architecture has three parts and one constraint. The constraint is that the VLM is shared: the natively multimodal Qwen3.5-4B is the single backbone, and every task routes through it.

The BEV Perception Head performs 3D object detection, semantic occupancy prediction and BEV map segmentation jointly. The README is unusually candid about its role, calling it "a probe of the 3D information accessible from the shared representations" that "provides an explicit, inspectable interface to 3D scene structure". That framing matters. The head is not presented as the strongest possible detector; it is presented as a way to read out what the language model already encodes. If you want a state-of-the-art occupancy network, this is the wrong layer to look at.

The Planning Expert conditions on the shared VLM representations to generate future ego trajectories. Two variants ship: planner-sft, trained by imitation, and planner-rl, the same expert after reward optimization on benchmark objectives. The performance table shows why both are kept. On NAVSIM v1.1 navtest, SFT reaches 88.2 PDMS while RL reaches 90.7. On the NVIDIA PhysicalAI open-loop benchmark the ordering flips: SFT gets 0.34 m minADE at 3 s, RL gets 0.38 m. Neither variant dominates, which is a real finding rather than a packaging convenience.

The LLM Decoder is left unchanged and handles both General VQA and Driving VQA. Training used a staged strategy with a unified data pipeline: heterogeneous perception annotations mapped into a shared label space, driving VQA responses re-annotated for format and factual consistency, and trajectories from multiple public driving datasets standardized into a unified waypoint representation. The repository ships the inference side of that pipeline, not the pipeline itself.

Installing Qwen-Drive-1.0 and running the VLM alone

The package is named qwen_drive and requires Python 3.10 or newer, though requirements.txt pins the environment the released weights were verified against: Python 3.12 and CUDA 12.8. The dependency list is not light. It includes torch, transformers, flash-attn, causal-conv1d and flash-linear-attention, all of which need a working CUDA toolchain.

Start by installing the pinned requirements. The comment at the top of requirements.txt is explicit that flash-attn and causal-conv1d should build against the already-installed torch:

pip install -r requirements.txt --no-build-isolation

If that succeeds you have the exact stack the weights were verified with. Skipping --no-build-isolation is the most common way this step fails, because the build backend will try to pull its own torch.

For a lighter install, pyproject.toml allows a range rather than pins, with transformers constrained to a narrow window:

pip install -e .

That resolves transformers>=5.14.0,<5.15.0 and torch>=2.8.0. Two optional extras exist: archive pulls pyarrow, and oss pulls oss2. Neither is needed for basic inference.

Weights come from Hugging Face or ModelScope, and everything lands in one directory. The VLM sits at the root and each task head sits in a subfolder beside it:

Qwen-Drive-1.0-4B/          9.1 GB  the VLM, which on its own serves the VQA mode
├── planner-sft/            2.1 GB  Planning Expert, imitation-trained
├── planner-rl/             2.1 GB  Planning Expert after reward optimization
└── perception/             0.5 GB  BEV perception head

The root VLM alone is enough for VQA. You only need to download a head subfolder if you intend to run perception or planning. Budget roughly 9.1 GB for the VLM, plus 2.1 GB for one planner variant, plus 0.5 GB if you also want the perception head. The README does not document a CLI entry point, so expect to drive the model from Python through transformers and accelerate rather than from a shell command.

The limitation that matters most: inference only, and a shared backbone

The repository layout is LICENSE, README.md, assets/, data/, docs/, pyproject.toml, requirements.txt, scripts/ and src/. The pyproject.toml description says it plainly: "Inference code for Qwen-Drive-1.0". There is no training loop, no fine-tuning recipe, no reward-model code for the RL planner, and no instructions for reproducing the staged training strategy described in the README. If your goal is to adapt the model to your own fleet data, you are on your own.

The shared backbone is the second constraint, and it is architectural rather than a packaging choice. Because one VLM feeds the perception head, the planning expert and the language decoder, you cannot upgrade the perception head without retraining or at least re-validating the other two paths. Teams used to swapping a detector behind a frozen feature extractor will find that coupling uncomfortable.

There is also a hardware floor. flash-attn and causal-conv1d are hard dependencies, and requirements.txt targets CUDA 12.8. The README does not document a CPU path or a CPU fallback. Anyone hoping to run this on a laptop, or on an inference server without a recent NVIDIA GPU, should treat that as out of scope until the repository says otherwise.

Finally, the benchmark story comes with a caveat the README states itself. LingoQA is scored with Qwen-Plus as the judge rather than the official LingoJudge, which the authors say "scored leniently and inconsistently across scenarios". Under the official protocol, Qwen-Drive-1.0-SFT obtains 79.4 instead of 77.8. That is still a strong number, but it means the driving VQA row is not directly comparable to published LingoQA results from other groups.

How it differs from Cosmos-Reason and MiMo-Embodied

The README's own comparison tables make the alternatives concrete, and the differences are architectural rather than incremental.

Cosmos-Reason2-8B is the closest in spirit: a reasoning model applied to physical and driving scenes. On driving VQA it scores 59.6 on LingoQA and 12.62 Ego3D RMSE, against 77.8 and 7.78 for Qwen-Drive-1.0-SFT. On general benchmarks the two are much closer, with Cosmos-Reason2-8B at 82.8 MMBench and 59.1 MMMU against 85.5 and 72.7. The difference in approach is that Cosmos-Reason2 does not ship a BEV perception head or a dedicated planning expert in this comparison; it is a reasoning model evaluated on driving data.

MiMo-Embodied-7B is the more interesting counterexample. It beats Qwen-Drive-1.0-SFT on LingoQA, 72.0 against 77.8 in favor of Qwen-Drive, but it beats it on Ego3D RMSE with 9.85 against 7.78, also in favor of Qwen-Drive, and it leads on WaymoQA safety at 66.5 against 70.7, again in favor of Qwen-Drive. Where MiMo-Embodied pulls ahead is general reasoning: 22.4 on MMStar and 27.4 on MMMU-Pro std are far below Qwen-Drive-1.0-SFT's 75.9 and 62.7, and the README marks several of its general-VQA cells as invalid or unparsable. So the honest reading is that MiMo-Embodied is a driving specialist with degraded general ability, while Qwen-Drive-1.0 tries to hold both.

Alpamayo-1.5-10B sits at the other extreme. It scores 7.5 on MMBench and 3.2 on OCRBench, which suggests a model that has largely abandoned general vision-language competence. Qwen-Drive-1.0's design bet is that this trade is avoidable, and the general VQA table is the evidence it offers.

Licence, maintenance and what an upgrade costs you

The repository is licensed Apache-2.0, with LICENSE at the top level and pyproject.toml referencing it via license = { file = "LICENSE" }. Apache-2.0 is a permissive licence with an explicit patent grant, which is generally friendlier for commercial deployment than copyleft alternatives, but the weights themselves may carry separate terms on Hugging Face or ModelScope that this repository does not restate. Check the model card before you ship anything. Nothing here is legal advice.

The repository is not archived, and the last push was on 2026-09-03. There are no retrieved releases, so there is no versioned artifact to pin against and no changelog to read. Upgrades therefore mean tracking the main branch, which is a real cost: requirements.txt pins an exact environment including transformers==5.14.1 and torch==2.8.0, while pyproject.toml allows a range. Those two files can drift apart, and a fresh pip install -e . may resolve to a combination the released weights were not verified against.

The practical upgrade burden is the CUDA stack. flash-attn==2.8.3, causal-conv1d==1.6.2.post1 and flash-linear-attention==0.5.1 all compile against torch, so any torch bump forces a rebuild of all three. If you pin to requirements.txt and freeze the environment, upgrades are a deliberate quarterly exercise rather than a background task. If you follow the range in pyproject.toml, expect to debug build failures.

Editorial conclusion

Adopt Qwen-Drive-1.0 if you need a single checkpoint that answers driving questions, emits 3D perception output and produces waypoints, and you are willing to run it on CUDA 12.8 with Python 3.12. Skip it if you need a training or fine-tuning pipeline, a CPU-only deployment, or a component you can swap out independently, because the repository ships inference code only and the VLM is shared by every head. Before committing, verify the released weights actually load under transformers 5.14.1, check that flash-attn 2.8.3 builds against your torch 2.8.0 install with --no-build-isolation, and confirm your GPU has room for the 9.1 GB VLM plus whichever head you need.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. QwenLM/Qwen-Drive-1.0 on GitHub
4. README