PRISM-VL: vision-language reasoning on RAW-derived measurement inputs instead of ISP-rendered RGB
PRISM-VL studies measurement-grounded VLM learning with RAW-derived Meas.-XYZ inputs, camera-conditioned grounding, and exposure-bracketed supervision transfer.
At a glance
- What is it?
- PRISM-VL, styled PRSIMVL in the repository, is a research release that swaps the visual input of Qwen3-VL from post-ISP RGB to RAW-derived Meas.-XYZ plus camera metadata, and ships the benchmark, training corpus, evaluation scripts and LoRA checkpoints needed to reproduce the comparison.
- Who is it for?
- Take PRISM-VL if your question is whether something in the sensor signal survives ISP rendering, because the release gives you the matched benchmark, the training corpus, the LoRA adapters for 2B, 4B and 8B Qwen3-VL and a dry-run evaluation path, so the comparison is reproducible rather than claimed.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 115 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The question the release is built around
The repository opens with an allegory and then states the question plainly: when the RGB image has already lost sensor evidence, can a vision-language model reason better from measurement-domain observations? RGB is described as a display-oriented product of an image signal processor, useful, compact and familiar, but one that may clip, denoise, tone map or quantize away the evidence a downstream model needs. PRISM-VL changes the visual interface rather than the model: the input becomes RAW-derived Meas.-XYZ, a linear three-channel view derived from RAW measurements, plus camera metadata such as ISO, exposure time and aperture. Everything else keeps the familiar Qwen3-VL training and inference workflow, which is what makes the release usable for people who already run that stack. The base models are Qwen3-VL at 2B, 4B and 8B sizes with released LoRA adapters.
What ships in the release
The release is organised so each artifact has its own documented entry point. The benchmark is MeasL-Bench-V1, 2,183 held-out matched examples spread over 14 measurement-sensitive capability slices. The training corpus is MeasL-150K-V1, 152,517 instruction-tuning examples with 48,000 release images. Evaluation, dataset cards, schemas and Hugging Face loading snippets live under eval/, eval_data/ and training_data/, and RELEASE_MANIFEST.md records what is included, what was pruned and which large artifacts are expected to arrive separately. The benchmark and corpus download from Hugging Face at kepeng/MeasL-Bench-V1 and kepeng/MeasL-150K-V1, and the LoRA weights sit at kepeng/PRSIMVL-LoRA-V1, to be restored under exps/ before running adapter inference or a full evaluation.
Getting it running
The README keeps the start to three commands: clone the repository, enter the directory and run the editable install script.
git clone <repo-url> PRSIMVL
cd PRSIMVL
bash install_editable.shThe evaluation wrapper supports a dry-run that checks the setup without launching model inference, which is a sensible first move given the artifact sizes.
MODEL_SIZE=2b CUDA_VISIBLE_DEVICES=0 bash eval/run_infer_and_eval.sh --dry-runA real run evaluates one adapter on the default Meas.-XYZ benchmark split.
MODEL_SIZE=2b CUDA_VISIBLE_DEVICES=0 bash eval/run_infer_and_eval.shThe large artifacts are expected at release-local paths that the README spells out.
eval_data/ # MeasL-Bench-V1 JSONL + image/
training_data/ # MeasL-150K-V1 JSONL + image/
exps/ # released LoRA adaptersFor a single question rather than a benchmark run, the inference entry point starts a swift deploy service and sends a local image to it with ask_service.py, then prints the answer.
Why measurement grounding matters, with examples
The README makes the argument with two low-illumination text cases rather than abstractions. In the first, a shop name photographed in poor light, the RGB input leads the model to answer Hua Tian Hua, which is wrong, while the Meas.-XYZ observation leads PRISM-VL to a correct answer. In the second, text on a yellow sign, the RGB path answers diamond, also wrong, while the measurement path answers BLACK. Zoomed evidence crops for both cases are included so a reader can see what the RGB rendering destroyed. The mechanism is that ISP rendering, tuned for human viewing, compresses and remaps pixel values in ways that can erase the difference between text and background in hard lighting, while the linear measurement representation keeps the recoverable structure. Whether that advantage generalises beyond such slices is exactly what the 14 capability slices in the benchmark are designed to test.
The numbers on MeasL-Bench
The headline comparison is against an RGB-trained Qwen3-VL-8B on the held-out MeasL-Bench protocol, where PRSIMVL-8B improves by 0.1074 BLEU, 0.1071 ROUGE-L and 4.46 LLM-Judge points, with the judge reported as accuracy percentage. The results table notes that BLEU and ROUGE-L are lexical metrics, which matters here because measurement-grounded answers are often about reading text correctly, and exact-string agreement moves those metrics sharply. The full table in the README covers the model family across sizes, and the evaluation pipeline is packaged so the same wrapper can evaluate either Meas.-XYZ or matched RGB input, which is the comparison that gives the numbers meaning. No claim is made beyond the benchmark: the README does not assert general VLM superiority, and the framing stays on measurement-sensitive capabilities.
Where this comes from
The release builds on an earlier project the authors link from the README: an end-to-end RAW synergy paper on OpenReview, which introduced Raw-VLM with a learnable ISP frontend and RAW tokenization for vision-language reasoning, studied through synthetic RAW data on captioning, VQA and hallucination behavior. PRISM-VL moves from synthetic RAW to measurement-grounded real inputs, adds camera metadata, and ships the benchmark, corpus and adapters publicly. The project page and an arXiv entry are linked, along with a Chinese README. The training and inference plumbing visibly inherits from a mature framework: the bundled setup.py carries an Alibaba copyright header and reads its version from swift/version.py, meaning the release is built on top of the ms-swift training ecosystem rather than a from-scratch harness, which is good for compatibility and worth knowing when you read the install script.
Limits to plan around
The heavy parts are not in the git repository. Benchmark images, training images and LoRA checkpoints arrive from Hugging Face, so a clone without those downloads can do nothing except dry-run checks, and RELEASE_MANIFEST.md exists precisely to explain that split. Inference needs a CUDA GPU through the swift tooling, and the README examples all pin a single device. There are no published releases in the repository and the evaluation entry points assume the release snapshot layout, so treating the repository as a versioned research artifact rather than a maintained product is the right mental model. The gains documented are on measurement-sensitive slices, so for well-lit ordinary photographs there is no documented benefit over plain RGB Qwen3-VL, and switching input representation has a cost: your data pipeline must produce Meas.-XYZ plus metadata, which ordinary image datasets do not carry.
Compared with fine-tuning on RGB
The obvious alternative is to keep RGB and spend the same effort on better fine-tuning, data curation or prompting of Qwen3-VL. That route needs no RAW capture, works with every existing dataset, and is where most of the field's progress is happening. It cannot recover information the ISP has already destroyed, which is the specific failure the two README examples demonstrate. PRISM-VL is the controlled experiment for the other direction: matched examples, the same base models, the only difference being the input representation. Practically, the measurement route makes sense when you control the capture side, such as camera pipelines or research data collection, and your failures are concentrated in hard lighting or fine text. For general assistant workloads on ordinary images, the RGB path remains the sensible default, and this release is best read as evidence about when to pay the RAW tax rather than as a replacement.
Editorial conclusion
Take PRISM-VL if your question is whether something in the sensor signal survives ISP rendering, because the release gives you the matched benchmark, the training corpus, the LoRA adapters for 2B, 4B and 8B Qwen3-VL and a dry-run evaluation path, so the comparison is reproducible rather than claimed. Skip it if you want a drop-in better VLM for ordinary photos, since the gains are concentrated on measurement-sensitive slices such as low-illumination text, and the pipeline assumes GPU inference through the swift tooling. The honest framing comes from the authors themselves: this is a research release around one under-tested question, not a production model, and the evidence tables in the README, including the two wrong RGB answers recovered from Meas.-XYZ, are the right way to judge it.
Frequently asked questions
What base models does PRISM-VL use?
The model family is Qwen3-VL at 2B, 4B and 8B sizes, with released PRSIMVL LoRA adapters hosted on Hugging Face at kepeng/PRSIMVL-LoRA-V1, restored under exps/ before inference.
What is MeasL-Bench-V1 and how big is it?
It is the released benchmark of 2,183 held-out matched examples spread over 14 measurement-sensitive capability slices, used to compare Meas.-XYZ input against matched RGB input under the packaged evaluation wrapper.
Does PRISM-VL improve ordinary photo understanding?
The documented gains are on measurement-sensitive slices such as low-illumination text, where the README shows RGB answers being wrong and Meas.-XYZ answers correct. For well-lit ordinary photographs the README documents no benefit over plain RGB Qwen3-VL.
Community notes