# OmniParser: turning a screenshot into clickable element boxes for a vision agent

> Microsoft's screen parsing tool converts UI screenshots into detected regions plus icon captions, so a vision model can name and ground the elements it wants to click. It is a component, not an agent.

**microsoft/OmniParser** — A simple screen parsing tool towards pure vision based GUI agent

- Repository: https://github.com/microsoft/OmniParser
- Stars: 25,513 · Forks: 2,238
- Language: Jupyter Notebook
- License: CC-BY-4.0
- Published: 2026-10-08 · Updated: 2026-10-08 · Language: en
- Canonical page: https://hysenlabs.com/projects/microsoft-omniparser

## Why a vision agent needs a screen parser at all

The stated problem is grounding. A multimodal model can look at a screenshot and describe what is on screen, but deciding where to click requires coordinates, and a caption like 'Close button' does not say where the close button is. OmniParser sits in that gap. It parses a screenshot into structured elements, which the README says improves the ability of GPT-4V to generate actions grounded in the right region of the interface.

The architecture follows from that. Rather than one end-to-end model, the pipeline is built from separable stages with separate weights: an object detection model that finds interactive regions, and a captioning model that describes what each icon is. A third piece, added more recently, is a detector for interactive regions specifically, so the model can distinguish something clickable from something decorative.

The repository tree is consistent with a research codebase rather than a library. `demo.ipynb` holds the examples, `gradio_demo.py` is a demo entry point, `omnitool/` holds the agent-side tooling, `util/` holds helpers, `eval/` holds the benchmark harness, and `docs/` holds the evaluation write-up. There is no `setup.py` and no package entry in the tree, so there is nothing to `pip install` from this repository. You clone it and run scripts inside it.

## Setting up the environment with conda and requirements.txt

The install is documented as four lines, and the Python version is pinned rather than left to chance:

```python
cd OmniParser
conda create -n "omni" python==3.12
conda activate omni
pip install -r requirements.txt
```

`requirements.txt` is worth reading before you commit to this, because it is longer than a parsing demo needs and it tells you what else the repository expects to run. It pins `numpy` at 1.26.4, `einops` at 0.8.0, `ultralytics` at 8.3.70, `supervision` at 0.18.0 and `openai` at 1.3.5, and it includes `easyocr` alongside `paddleocr` and `paddlepaddle`, so both a classical and a deep OCR path are available. It also carries tooling that has nothing to do with inference: `ruff` 0.6.7, `pre-commit` 3.8.0, `pytest` 8.3.3 and `pytest-asyncio`.

That last group matters if you plan to contribute. The project runs a formatter and a test suite, and the release notes for v1.5.0 show ordinary maintenance pull requests: a spelling fix in `requirements.txt`, a dependency update, a typo in an `einops` pin, and a change that wrapped something in `torch.inference_mode`.

Once the environment exists, the demo is a single command:

```python
python gradio_demo.py
```

The README's only example material is `demo.ipynb`, and it is where the V1.5 behaviour is shown, including the prediction of whether an element is interactable at all.

## Fetching weights from a pull request instead of a release

This is the part of the install that will cost you an afternoon if you skim it. The newest detector, `icon_detect_v3`, is not on the main model repository yet. The README is explicit about where it lives and why:

```bash
huggingface-cli download microsoft/OmniParser-v2.0 icon_detect_v3/model.pt \
  --revision refs/pr/37 --local-dir weights
```

So the current detector is fetched from a pull request ref on the Hugging Face hub, and the README says the code prefers this local weight. The second download pulls the caption model and renames the directory, which is the step that trips people up:

```bash
for f in icon_caption/{config.json,generation_config.json,model.safetensors}; do huggingface-cli download microsoft/OmniParser-v2.0 "$f" --local-dir weights; done
mv weights/icon_caption weights/icon_caption_florence
```

That directory rename to `icon_caption_florence` is not cosmetic. It reflects which caption model is in use, and the older layout in the README's deprecated section still refers to `icon_caption_blip2` alongside the Florence one, so both caption backends were once shipped side by side. The README says that once the pull request is merged, the same weight downloads automatically on first use, which makes the manual step a temporary arrangement rather than a permanent one.

The install section also explains why the local download exists at all: the code prefers the local weight. If you run the demo before downloading, you are relying on the automatic path and on network access at inference time.

## OmniTool turns the parser into a Windows 11 control loop

The `omnitool/` directory holds the piece that most people will actually want, and it arrived in February 2025. OmniTool controls a Windows 11 virtual machine using OmniParser plus a vision model of your choice. The README lists the models it supports out of the box: OpenAI 4o, o1 and o3-mini, DeepSeek R1, Qwen 2.5VL, and Anthropic Computer Use.

That list is the interesting design decision. Instead of tying the screen parser to one model, the parser produces structured elements and the agent's planner consumes them, so the model can be swapped. If you already have a hosted vision model in your stack, the argument for adopting this parser is stronger than if you do not.

The requirements file backs that up. Alongside `pyautogui` 0.9.54, `uiautomation`, `screeninfo` and `streamlit` 1.38.0, it lists `anthropic[bedrock,vertex]>=0.37.1`, `dashscope`, `groq`, `boto3` and `google-auth`, which is a broad set of provider clients rather than a single vendor integration.

A later entry in the news list adds local trajectory logging so OmniParser plus OmniTool can be used to build training data for an agent in your own domain. The documentation for that is marked work in progress, which is the honest signal to weigh: the data pipeline is announced and the docs are not finished. The July 2026 addition, a YOLOv9-E interactive region detector, is available as an inference-only weight through the same pull request path as the v3 detector.

## Licensing splits three ways and the badge oversimplifies it

There is a real conflict here and it is worth stating without picking a winner for you. GitHub reports the repository license as CC-BY-4.0, the README badge shows MIT, and the README's own model weights section splits the components three ways: `icon_detect_v3` is based on the MIT-licensed YOLOv9 implementation, the earlier Ultralytics-based icon detectors keep their original AGPL licence, and the caption models are under MIT.

The practical consequence is that the licence of the code and the licence of the weights are separate questions, and the AGPL clause is the one that can reach into your distribution. If you ship a product that bundles an older icon detector, AGPL obligations travel with it. If you use the v3 detector and the Florence caption model, the weights are under permissive terms. A data licence of CC-BY-4.0 on the repository is a different category again and concerns the documentation and assets rather than the model behaviour.

The same split shows up in the version history. v2.0.0 was published on 2025-02-13 with a larger and cleaner icon caption and grounding dataset and a claimed 60 percent latency improvement over V1. v2.0.1 followed on 2025-09-12 with security updates, dependency version fixes and documentation improvements, and nothing else. v1.5.0 was also published on 2025-02-13, and its notes are where the project says it adopted semantic versioning. The last push on the repository was on 2026-07-20.

## What the 39.5 percent claim covers and what it does not

The benchmark number attached to V2 is 39.5 percent, in a January 2025 news entry, on ScreenSpot Pro, with a pointer to `docs/Evaluation.md` in the repository. The v2.0.0 release notes give the figure as 39.6 average accuracy on the same benchmark, so the two official sources differ by a tenth of a point and neither explains the difference.

More useful context: ScreenSpot Pro is a grounding benchmark for professional desktop software, which is a harder distribution than the mobile-oriented benchmarks this line of work started on. The project also reports best performance on Windows Agent Arena as of September 2024, which is an end-to-end agent result rather than a parsing result, so it is not directly comparable to the grounding number.

What none of these figures tell you is the cost side. ScreenSpot Pro measures whether the model points at the right region. It does not tell you how long a full parse takes on your resolution, whether the caption model hallucinates on unfamiliar iconography, or whether a detection is stable enough to act on. The repository's own answer to the last question is the interactable prediction added in V1.5 and the interactive region detector added in July 2026, both of which exist because the earlier versions could not separate clickable from decorative.

If you are evaluating this against alternatives, the comparison is usually not with another parser but with a platform-native accessibility tree. Operating systems already expose a structured UI tree, and `uiautomation` in the requirements file is how OmniTool reaches into Windows directly. Screen parsing is the fallback for interfaces that do not expose one, which is the honest framing for where this belongs.

## Conclusion

OmniParser earns its place when you are building an agent that has to act on a screen nobody hand-wrote a selector for, which is the case for desktop automation, Windows VM control and accessibility work on arbitrary interfaces. The documented path is clear: create the conda environment, install `requirements.txt`, pull the detector weights, then launch `gradio_demo.py` and look at the overlay. Check three things before building on it: whether the 39.5 percent ScreenSpot Pro figure quoted for V2 is still the version you get from `microsoft/OmniParser-v2.0`, whether the AGPL terms on the older icon detectors matter for your distribution, and whether a raw detection plus caption pair is enough grounding for your model, or whether you need the interactive region detector added in July 2026. It is a parsing component, not an agent loop, so you still owe it a planner and an executor.

## FAQ

### How do I use OmniParser?

Clone the repository, create a conda environment on Python 3.12, and install `requirements.txt`. Then download the `icon_detect_v3` weight and the Florence caption weights into a `weights` directory, and launch `python gradio_demo.py` to see the parsed overlay. `demo.ipynb` holds the worked examples.

### What license applies to the OmniParser model weights?

It depends on the component. `icon_detect_v3` builds on the MIT-licensed YOLOv9 implementation and the caption models are MIT, but the earlier Ultralytics-based icon detectors stay under AGPL. The repository itself carries a different, CC-BY-4.0 license field, so check the component you actually ship.

### Is OmniParser a full computer use agent?

No, it is the parsing component. The `omnitool/` directory provides the agent-side tooling that drives a Windows 11 VM using OmniParser plus a vision model such as OpenAI 4o, DeepSeek R1, Qwen 2.5VL or Anthropic Computer Use. You still need a planner and an executor.

### Can OmniParser be run without downloading the weights manually?

Eventually, yes. The README says that once the Hugging Face pull request holding `icon_detect_v3` is merged, the same weight downloads automatically on first use. Until then, the local download from the pull request ref is the documented route, and the code prefers the local weight.

### What score does OmniParser report on ScreenSpot Pro?

Two official sources give slightly different numbers. The January 2025 news entry reports 39.5 percent with OmniParser V2, and the v2.0.0 release notes give 39.6 average accuracy on the same benchmark. Neither figure covers latency, caption accuracy or detection stability.

## Sources

- [Issues](https://github.com/microsoft/OmniParser/issues)
- [License: CC-BY-4.0](https://github.com/microsoft/OmniParser/blob/master/LICENSE)
- [microsoft/OmniParser on GitHub](https://github.com/microsoft/OmniParser)
- [README](https://github.com/microsoft/OmniParser/blob/master/README.md)
- [Releases](https://github.com/microsoft/OmniParser/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/microsoft-omniparser