ComfyUI VLM Nodes: One Interface for Vision-Language Models, Plus Typed Detection Sockets

The gap between ComfyUI graphs and vision-language models

ComfyUI's native node set covers diffusion sampling, image manipulation and a growing set of utility operations. It does not cover vision-language inference. Anyone who wants to caption a batch of images, run OCR over a scanned page, ask a question about a frame, or ground a phrase to a bounding box has had to leave the graph, run a Python script, and import the results back as text or masks. ComfyUI VLM Nodes exists to close that loop. The README describes the package as production-oriented vision-language, structured prompting, audio and utility nodes, and the model list spans Qwen3-VL, SmolVLM2, InternVL, Gemma 3, Florence-2, Moondream and MiniCPM-V, with GGUF execution through llama.cpp and hosted API nodes alongside local ones. The audience is therefore narrow and specific: people who build ComfyUI workflows and want the model output to land on a socket they can wire into the next node. If you do not run ComfyUI, almost nothing here applies to you.

What the Modern VLM node actually does

The central abstraction is a single node, Modern VLM, with a twelve-choice picker. The README lists Qwen 3.5 0.8B and 4B, Qwen 3 VL 2B, 4B and 8B Instruct, SmolVLM2 500M and 2.2B Video, Liquid LFM2.5-VL 450M, InternVL 3.5 1B, Granite Vision 4.1 4B, Gemma 3 4B IT, and one custom Hugging Face image-to-text repository. Everything else, including the 27B and 35B-A3B Qwen tiers, Qwen 2.5 VL, Gemma 3 12B and 27B, and the older Molmo, Kosmos-2 and MiniCPM-V loaders, sits in a separate node labeled [Legacy] Modern VLM Compatibility. That split is the most interesting design decision in the project. The server still accepts every known catalog value, so previously saved workflows keep running even when their model has moved to Legacy, but the picker a new user sees is curated. The default is Qwen 3 VL 2B, which the README justifies on load time rather than accuracy. Sixteen sub-4B choices are marked internally as a small-and-fast tier. The mechanism behind this is a catalog that maps a stable string to a model repository and an inference path, with native Transformers implementations for curated entries and remote repository code enabled only for the explicit custom-model option.

Typed sockets replace model-specific lists

The detection and segmentation side of the package does not pass Python lists between nodes. It defines four socket types with versioned JSON schemas: VLM_DETECTIONS for per-frame boxes, labels, scores, optional polygons and in-process masks; VLM_TRACKS for durable object IDs with ordered observations over time; VLM_POINTS for pixel-coordinate points including detection centers; and VLM_EVENTS. Each carries a schema identifier such as comfyui-vlm/detections at version 1. This matters because it makes the graph inspectable and lets a tracking node consume the output of a detection node without either knowing the other's model. The Moondream 3 Preview segment node is a good illustration of how much work sits behind one socket. According to the README, it produces native SVG segmentation, preserves the SVG and also converts it into antialiased MASK, black-and-white previews, foreground cutouts, overlays, polygons, canonical VLM_DETECTIONS and core bounding boxes. Detection and pointing submit frames concurrently so the Photon runtime can batch them, and every run reports measured worker FPS, end-to-end FPS and real-time factor. Those are the project's own reported metrics, not figures I have reproduced.

Streaming text without breaking the STRING output

Modern VLM streams decoded text through ComfyUI's native progress_text WebSocket channel by default. A connected ViewText node updates while tokens arrive, shows the final response after execution, and restores the last result when ComfyUI rehydrates workflow output history. The README is explicit that streaming is best-effort and never changes the final STRING output or causes inference to fail, and that stream_output can be disabled for API-only or headless runs. That honesty is worth noting. Live token streaming in a graph editor is a UI convenience, and the project treats it as one rather than as part of the contract. The same section of the README documents the text toolkit that grew around this: Text (formerly SimpleText), View Text (Streaming), JSON to Text, Text Join, Text Template, Text Clean, Text Replace, JSON Extract, Text Split / Batch, and Text Inspector. The JSON utilities never evaluate code, follow references, access files or make network requests, and template fields are direct names rather than Python attribute or index expressions. The README also flags that approx_tokens is a rough UTF-8 budget estimate and that the target model tokenizer should be used when exact billing or context accounting matters.

Installing it and getting a model to load

The README states that version 3.4 supports ComfyUI's selected NVIDIA CUDA, AMD ROCm, Apple Metal, Intel XPU and CPU device without replacing its PyTorch build, and that it removes startup installers and global accelerator cache flushes. That is the main operational promise: the nodes should not fight your existing ComfyUI environment. The package is distributed through the ComfyUI registry, with the homepage given as registry.comfy.org/nodes/comfyui_vlm_nodes, so installation follows the usual custom-node path rather than a bespoke script. The repository itself is Python with Apache-2.0 licensing. Once loaded, the workflow is: add a Modern VLM node, pick one of the twelve curated models, connect an image or video batch, and read the STRING output or wire it into View Text (Streaming). The README gives the example of disabling stream_output for headless runs, which is the one config key it names explicitly. For detection and segmentation you add the relevant node, for example Florence-2 for captioning, OCR, detection, region captioning and referring expression segmentation, and consume VLM_DETECTIONS or VLM_TRACKS downstream. What the README does not give is a dependency list or a pinned Transformers version, and that is a real gap for anyone trying to reproduce an environment.

Where the design shows its seams

Two limitations are visible in the material. First, the Legacy split means the model you want may not be in the picker. The README says previously saved ModernVLM workflows remain valid even when their selected model moved to Legacy, which implies the move happened and will happen again. Anyone building on a specific checkpoint should expect to check whether it is still curated after an update. Second, the Moondream2 entry documents a concrete failure mode in the wider ecosystem: the README states the official checkpoint is loaded through its native safetensors state dict to avoid what it calls the silent empty-output regression in Transformers 5. That is a dependency-level hazard, not a bug in this package, but it tells you the project is tracking upstream breakage and that upgrading Transformers independently of this package is risky. The streaming path is a third soft spot. Because streaming is described as best-effort and never affects the final output, you should not build any logic that depends on incremental tokens arriving. And the custom Hugging Face repository option, while useful, is the point where the README says remote repository code is enabled, which is a different trust posture from the curated entries.

Compared with running the models yourself

The obvious alternative is to call the same models directly through Transformers or through a dedicated inference server, and the difference is not accuracy but where the graph lives. A direct Transformers script gives you full control over the tokenizer, the quantization, the batching strategy and the exact revision of every dependency. It also gives you no visual graph, no socket types and no history rehydration. ComfyUI VLM Nodes trades that control for wiring: the model choice is a picker, the output is a STRING or a typed socket, and the same graph can chain a Florence-2 detection into a SAM2 segmentation into a mask. If your work is exploratory, or you need to prove a pipeline visually before committing to it, the node package wins on iteration speed. If your work is a batch job over a million images, or you need a stable HTTP API for other services, a script or a server is the better fit and this package will get in the way. The README's own framing supports that reading: it targets ComfyUI users, and the API nodes it includes are for calling hosted models from inside a graph, not for exposing your graph as an API.

Maintenance cost and licence

The maintenance surface here is larger than a typical custom-node pack because the package tracks upstream model releases, Transformers interfaces and llama.cpp. The README's own history shows the cost: model tiers get promoted and demoted, node IDs are kept stable for saved workflows, and at least one entry exists specifically to work around an upstream regression. Budget for re-testing your graphs after each upgrade rather than assuming compatibility. On licensing, the package itself is Apache-2.0, which is permissive for commercial use of the code. That says nothing about the models it downloads. The README states the catalog intentionally uses official model repositories and maintained Transformers interfaces rather than unverified community quantizations, and that Florence-2 uses Transformers-native converted checkpoints instead of Microsoft's legacy repository code. Each of those upstream checkpoints carries its own licence, and the README does not summarize them. Check the model card for whichever checkpoint you deploy before shipping anything commercial. This is not legal advice; it is a pointer to the file you need to read.

Editorial conclusion

Adopt ComfyUI VLM Nodes if you already run ComfyUI and need captioning, OCR, detection or segmentation inside the same graph as your image pipeline, and you are willing to pin the version and re-check saved workflows after upgrades. Do not adopt it if you need a standalone inference server, a stable public API for other applications, or guaranteed streaming in headless runs. Before committing, verify three things in your own checkout: that your ComfyUI PyTorch build matches the accelerator the README lists, that the model you need sits in the curated 12-choice picker rather than the Legacy node, and that your saved workflows still resolve their node IDs after the update.

Official sources

1. gokayfem/ComfyUI_VLM_nodes on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README