ShowUI: a 2B vision-language-action model for GUI agents

The problem ShowUI targets: grounding a click on a screen

A GUI agent has to do something a text model cannot: look at a screenshot and output a coordinate. The README frames ShowUI as an end-to-end vision-language-action model for GUI agents and computer use, and the paper title given in the citation is "ShowUI: One Vision-Language-Action Model for GUI Visual Agent". The intended user is someone building or studying that loop, not someone looking for a desktop automation tool with a settings dialog. The repository is Python, licensed Apache-2.0, and the primary artifact is a 2B checkpoint published as showlab/ShowUI-2B, with a second mirror on ModelScope. Because the checkpoint is 2B parameters, the local path is within reach of a single consumer GPU, which is the stated reason the project describes itself as lightweight. The README also links a companion repository, computer_use_ootb, for actually driving a PC, which tells you the boundary the authors draw: ShowUI is the perception-and-action model, OOTB is the harness around it.

UI-guided token selection and why screenshot resolution is the real cost

The mechanism the README singles out is UI-guided token selection, implemented in the showui module and demonstrated in test.ipynb. The repository ships two example images side by side: examples/chrome.png, annotated as 1296 screenshot patches, and examples/demo.png, annotated as 167 UI components after a UI graph is applied. That pair is the whole argument in one picture. A full screenshot at high resolution becomes a very long visual token sequence, and attention cost grows with it. If you instead detect UI components and keep only those regions, the sequence shrinks by roughly the ratio shown, about eight to one in that example. The README notes the notebook supports Qwen2VL models, and a later update adds Qwen2.5-VL fine-tuning and inference. The trade-off is not stated in the README and is worth naming: the reduction depends on a component detector, so a missed component is a missed click target, and canvas-style or game UIs without a clean component structure are the wrong input for this path. The README gives no accuracy comparison between the full-patch and UI-graph settings, so treat the token counts as a cost figure, not a quality figure.

Getting it running: notebooks, api.py and the vllm path

The README keeps installation instructions out of the front page and points to three documents: QUICK_START.md for local model usage, GRADIO.md for the local Gradio install, and TRAIN.md for training setup. That is the first thing to read, because the README itself does not list dependencies or a pip command. Inference has two routes. The first is inference_vllm.ipynb, and the README notes you can adjust the gpu_num parameter to use multiple GPUs for faster inference. The second is the API path: run python3 api.py with a screenshot and a query. The README states that this path is built on the Hugging Face Gradio client, so no GPU is needed locally, which makes it the cheapest way to evaluate output quality before you provision hardware. Training is documented as supporting grounding and navigation on Mind2Web, AITW and Miniwob, with self-customized models including ShowUI, Qwen2VL and Qwen2.5VL, DeepSpeed, BF16, QLoRA, SDPA or FlashAttention2, Liger-Kernel, mixed datasets, interleaved data streaming, random image resize with crop and pad, Wandb monitoring, and multi-GPU or multi-node runs. That is a long feature list for a research repository, and the practical risk is version drift across that stack; the README does not pin versions, so budget time for dependency work before you budget time for training.

Where ShowUI is the wrong tool

Two limits are visible in the README itself. First, dragging. The update log for 2025.12.31 announces ShowUI-pi for GUI dragging as a separate repository, which means drag interactions are not handled by the main ShowUI line. If your agent must move a slider or drag a file, this repository is not the one to start from. Second, human demonstration workflows: the same date carries a separate release for ShowUI-Aloha, again outside this repository. The project has grown into a family, and the main repo covers grounding and navigation, not every interaction type. A third limit is quieter. The README describes iterative refinement, added on 2024.12.1, as improving grounding accuracy and points to the HF Spaces demo to try it. Nothing in the README says how many refinement rounds are used, what the stopping rule is, or what the latency cost per round is. If you plan to depend on that accuracy, you will have to measure the round count yourself. Finally, the training support list is a statement of what the codebase handles, not a report of results on each benchmark; the README publishes no numbers, so any accuracy expectation has to come from the paper, which is outside this repository.

How it differs from SeeClick and from prompting a general VLM

The README acknowledges SeeClick for providing codes and datasets, which makes it the natural comparison. The difference is scope. SeeClick is a GUI grounding model: screenshot in, element location out. ShowUI is described as a vision-language-action model with navigation training on Mind2Web, AITW and Miniwob, so the same checkpoint is meant to cover both where an element is and what to do next. That matters if you want one model in the loop rather than a grounding model plus a separate planner. The second alternative is not a repository at all: sending a screenshot to a large hosted vision-language model and parsing coordinates from the text. That approach avoids training and avoids GPU provisioning, and the api.py path in ShowUI shows the authors are comfortable with a hosted round trip. The difference is control and cost. A hosted model gives you no token-selection control, so you pay for every patch of a 1296-patch screenshot, and you cannot fine-tune it on your own annotation format. ShowUI lets you run the weights locally, quantize to int8 as the 2024.12.9 update notes, and train on mixed datasets. If your UI is one internal tool and your volume is low, the hosted route is less work. If you need a fixed per-call cost or your data cannot leave your network, the local 2B checkpoint is the reason to pick this repository.

Maintenance, licensing and what the release history implies

The license is Apache-2.0, which permits commercial use and modification, but the repository also ships model weights and datasets under separate Hugging Face pages, and a licence on the code does not automatically settle the terms attached to a checkpoint or a dataset. Check each artifact you download. On maintenance, the last push recorded for the repository is 2026-04-24, and the update log shows a steady cadence through 2025 and into 2026, including ShowUI-pi accepted at CVPR 2026. The pattern to note is that new capabilities increasingly land in sibling repositories rather than here, so watching this repository alone will not tell you what the project can do. There are no retrieved releases, so there is no tagged version to pin against; you are tracking the main branch. For a research dependency that is normal, and it means upgrade cost is not a version bump but a re-read of the update log plus a re-check of QUICK_START.md and TRAIN.md against whatever base model you are using, since Qwen2VL and Qwen2.5VL are both listed as supported and are not interchangeable.

Who should pick this up, and the first thing to verify

The fit is narrow and clear. You are working on GUI grounding or navigation, you want a small checkpoint you can fine-tune, and you are comfortable reading notebooks instead of a CLI reference. The training stack covers Mind2Web, AITW and Miniwob, which are the standard navigation benchmarks, and the int8 and vllm options give you a path from experiment to something faster. If you only want to drive your own desktop, start from the computer_use_ootb link rather than this repository, because the README treats that as the harness. The first verification step is concrete: open QUICK_START.md and confirm the local inference path runs against the ShowUI-2B checkpoint on your hardware, or run python3 api.py first if you have no GPU and want to see output before installing anything. Only after that is it worth reading TRAIN.md, because the training dependencies are the expensive part and the README does not pin them.

Editorial conclusion

Adopt ShowUI if you are doing GUI grounding or navigation research and want a 2B checkpoint with training code for Mind2Web, AITW and Miniwob, or if you want to run a computer-use agent locally through the OOTB integration. Do not adopt it if you need a supported product with a stability guarantee, or if your task is GUI dragging or long human-demonstration workflows, which the README points to separate repositories for. Before committing, check QUICK_START.md against your installed transformers version, confirm the ShowUI-2B checkpoint still loads on Hugging Face, and decide whether you are running the local weights or the Gradio client path, because only the second one avoids GPU provisioning.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. showlab/ShowUI on GitHub