VQASynth: generating spatial VQA datasets from ordinary image collections

The gap VQASynth targets: spatial language is missing from pretraining data

The README opens with a specific claim: spatial reasoning matters for embodied AI such as robotics, and samples that teach it are rare in AI pretraining datasets. That is the problem statement, and it is narrower than general visual question answering. A model trained on caption data learns to name objects. It does not learn that a pallet sits roughly sixty centimetres from a worker, or that a chair is to the left of a bookshelf rather than behind it. VQASynth exists to manufacture that second kind of supervision from images you already have. The intended user is an engineer who owns an image collection on the Hugging Face Hub and wants to instruction-tune a vision language model on spatial queries without hand-labelling distances. The README frames the project as an open-source reproduction of SpatialVLM, which describes a 3D scene reconstruction pipeline plus prompt templates. So the deliverable is not a model. It is a dataset generator, and the models listed in the repository (SpaceLLaVA, SpaceQwen2.5-VL-3B-Instruct, SpaceThinker-Qwen2.5VL-3B, SpaceOm) exist as downstream evidence that the generated data can be used for low-rank adapter tuning.

From image to question: the four-stage data flow

The pipeline is exposed as importable classes, and the README's in-process example shows the order plainly. Localizer runs first with a captioner_type argument (the example uses "florence") and returns masks, captions and a third value the example discards. This is the detection and segmentation stage, and the README notes that SAM2 replaced SAM in the localization refinement step. SpatialSceneConstructor then takes the image plus those masks and lifts them to 3D. The README states that VGGT emits per-object point clouds, depth and intrinsics in one pass, and that this replaced DepthPro to improve metric depth estimation speed and accuracy. The constructor returns point cloud file paths plus a canonicalized structure, writing scenes to a directory you pass in. PromptGenerator consumes the captions, the point cloud paths and the canonicalized output, and returns QA pairs. The README's sample output is a question about how close a man in a red hat is walking from a wooden pallet, answered with a figure in centimetres. That chain is the whole design: semantics come from captions, metric scale comes from the reconstruction, and language comes from templates. The canonicalization step is what lets the generated answers reference consistent surfaces such as floors rather than arbitrary object-relative frames.

What the localizer and reconstructor actually contribute

Two design choices in the README are worth separating from marketing language. First, captioning is described as object-grounded, produced by point prompting with Molmo, and the README links to an issue rather than a paper section. Grounded captions are what allow a question to name a specific object (the man in the red hat) rather than a region. Second, the swap from DepthPro to VGGT is presented as improving both speed and accuracy of metric depth estimation. That matters because the answers are numeric. A pipeline that outputs "approximately 60.13 centimetres" is only as good as the metric scale it recovered, and the README offers no error bounds, no calibration procedure and no accuracy table. The two-decimal precision in the sample output is a formatting choice, not a claim about measurement error. Anyone reading that number as a tolerance should treat it as a template artefact. The chain also assumes static geometry: nothing in the described stages models object motion, so a question about where someone is walking is answered from a single frame.

Getting it running: Docker Compose, a config file and 24GB of VRAM

The documented path is container-first. Prerequisites are Python 3.10 or later, Docker, Docker Compose V2, and the NVIDIA Container Toolkit, with at least 24GB of VRAM on an A10 or larger and 16GB of system RAM. The run sequence is three commands: huggingface-cli login to authenticate for pushing to the Hub, cd into the repository, then bash run.sh. Dataset selection happens by editing config/config.yaml before the run, so switching source datasets is a config change rather than a code change. The README also mentions a Colab notebook that requires an A100 runtime. Output lands in a cache directory, and pushing is done through the Dataloader class: Dataloader(cache_dir).push_to_hub(final_dataset, target_repo_name). The in-process route skips Docker entirely if you want to process a single image, importing Localizer, SpatialSceneConstructor and PromptGenerator and calling .run() on each. That split is useful: the class-level API is testable on one image, while run.sh is the batch path. Note that the README does not enumerate the keys inside config.yaml, so you will be reading the file to learn what is configurable.

Hardware floor, static scenes and the missing evaluation story

The clearest constraint is the 24GB VRAM requirement. That is not a soft suggestion; it follows from running segmentation, captioning and a 3D reconstruction model in the same pipeline. If your only GPU is a 12GB card, the documented Docker path is closed to you, and the README does not describe a reduced-precision or staged alternative. The second limitation is temporal. Every stage operates on a single image, and the described outputs are per-object point clouds and depth from one view. Questions about trajectories, contact or intent cannot be grounded by this pipeline. Third, the README lists the datasets and models produced with VQASynth but gives no evaluation protocol, no comparison against human-annotated spatial data, and no statement of how often the prompt templates produce malformed or ambiguous questions. The CoT examples shown in the model card are model outputs, not pipeline validation. If your acceptance criterion is a measured distance error, you will have to build that measurement yourself, because the repository material does not provide one.

How this differs from caption-based dataset builders

The obvious alternative for anyone with images and a need for training data is a captioning or instruction-generation pipeline, of which there are many, including the captioners VQASynth itself calls. The difference in approach is the source of the answer. A caption-based generator produces questions and answers that are grounded in language and appearance: what is in the image, what colour, what is happening. VQASynth inserts a reconstruction step between the image and the text so that the answer to a distance or orientation question is derived from a 3D scene rather than from a model's visual guess. That is a heavier pipeline with a higher hardware cost, and it buys you metric answers that a captioner cannot produce at all. The trade-off is coverage. A captioner will happily describe a crowded street, an abstract diagram, or a photo where depth is ambiguous. VQASynth's reconstruction stage is where those inputs are most likely to fail, and the README does not describe fallback behaviour when a scene cannot be lifted to 3D. Choose based on whether your target capability is spatial or descriptive, not on which pipeline is smaller.

Maintenance, licensing and what to check before you commit

The repository is Apache-2.0 and not archived, with the last push recorded in August 2026. The README states there are no retrieved releases, so expect to track the main branch rather than pin a version. That has a direct cost: the pipeline depends on external models (Molmo for point-prompted captions, SAM2 for mask refinement, VGGT for reconstruction), and a change in any of those upstream projects can alter your generated dataset without a version bump in VQASynth itself. Regenerating data is therefore not a stable operation unless you pin those dependencies yourself. Apache-2.0 permits commercial use and modification, and it includes a patent grant, but it also means the project ships without warranty; the datasets and models you produce inherit the licences of whatever source images and checkpoints you feed in, which is a separate question the repository does not answer. Before adopting, confirm three things: that config/config.yaml targets your dataset and that you understand its keys, that VGGT recovers sensible scale on your image domain rather than on the warehouse and sports examples in the README, and that your GPU meets the 24GB floor. If all three hold, the class-level API in the README is small enough to test on one image before you commit to a full run.

Editorial conclusion

Adopt VQASynth if you already hold an image dataset on the Hugging Face Hub and have an A10-class GPU with 24GB of VRAM, because the pipeline is built to convert that collection rather than to be a general annotation tool. Do not adopt it if you need CPU-only processing, sub-second latency, or reasoning about motion and interaction, since the reconstruction step assumes static scenes and the documented setup requires Docker Compose V2 plus the NVIDIA Container Toolkit. Before committing, verify that config/config.yaml points at your dataset and that VGGT produces usable per-object point clouds on your image domain, because the distance answers inherit whatever scale the reconstruction recovers.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. remyxai/VQASynth on GitHub