CogAgent: an open source vision language model that drives a graphical interface

What a GUI agent actually is

CogAgent is an open-sourced end-to-end VLM based GUI agent, and the name in the repository description is the whole explanation of the category.

A GUI agent is a model that works on a screen rather than on a document. It takes a screenshot and an instruction, and it produces the actions that move the task forward, a click at a coordinate, a keystroke, a scroll, a drag. There is no API to call and no structured output to parse, there is a picture of a window and the next action in it.

That framing explains the model choices and the benchmark names. The release claims state of the art across multiple platforms and categories in GUI agent tasks and in GUI grounding benchmarks, and those are the two halves of the problem, understanding what is on the screen and deciding where to act on it.

The model is bilingual, supporting Chinese and English interaction through both screen captures and natural language, which is worth noting because most of the grounding benchmarks are English and the Chinese step-wise in-house benchmark is one of the four it leads.

The repository carries the three directories you would expect for a model release of this kind, the inference code, the fine-tuning code and a demo application, plus the action space documentation that describes the output format the model produces.

The timeline, from a first release to a highlight

The readme opens with a short history, and it is the history of the project rather than of a marketing launch.

In December 2023 the team open-sourced the first GUI agent, CogAgent, with the former repository available under a different organisation, and published the corresponding paper.

In June 2024 CogAgent was accepted by CVPR 2024 and recognised as a conference highlight, which the readme gives as the top 3 percent.

And in December 2024 they open-sourced the latest version of the model, CogAgent-9B-20241220, which is the current release and the subject of everything else in the document.

That last release is described as having significant improvements over the previous version of CogAgent, in four named areas: GUI perception, reasoning accuracy, action space completeness, and task universality and generalization. The action space point is the one with the most practical consequence, since a fuller action space means the model can express more of what a user would do, and the model input and output sections of the readme spell out the format.

So the project is about a year old at this point, with a published paper, a peer reviewed highlight and an open weight release behind it.

The base model and where the weights live

CogAgent-9B-20241220 is based on GLM-4V-9B, a bilingual open-source vision language base model, and the readme describes what was done on top of it: data collection and optimization, multi-stage training, and strategy improvements.

The result is described as achieving significant advancements in GUI perception, inference prediction accuracy, action space completeness and generalizability across tasks, and the model supports bilingual interaction with both screenshots and language input. The readme also notes that this version has already been applied in ZhipuAI's own desktop product, which is the most useful validation in the document, because a model that only works in a research harness is a different thing from one that drives a shipped application.

The weights are distributed widely, which matters for a model that is supposed to be self-hosted. There are download links on HuggingFace, ModelScope, WiseModel and Modelers, the last of which is the Ascend distribution. There is a technical blog with the report, a practical guide in Chinese, and online demo spaces on all three of the HuggingFace, ModelScope and Modelers platforms, so the model can be tried without a local GPU at all.

The practical guide being Chinese only is worth flagging for an international team, since the rest of the documentation is in English and Chinese with the Chinese readme as a separate file.

The benchmark claims, and who they were compared against

The capability section is specific about the comparisons, and that is what makes the claim checkable.

The model was compared against three groups. There are API-based commercial models, which the readme names as GPT-4o-20240806 and Claude-3.5-Sonnet. There are commercial API models combined with GUI grounding models, which are GPT-4o with UGround and GPT-4o with OS-ATLAS. And there are open-source GUI agent models, which are Qwen2-VL, ShowUI and SeeClick.

Against that field, the readme claims the model leads in four places: GUI localization on the Screenspot benchmark, single-step operations on OmniAct, a Chinese step-wise in-house benchmark called CogAgentBench-basic-cn, and multi-step operations on OSWorld.

And it states the one place it does not lead, with only a slight disadvantage on OSWorld compared to Claude-3.5-Sonnet, which specialises in computer use, and GPT-4o combined with external GUI grounding models. A claim that names the benchmark where it loses is worth more than one that does not.

There are demonstration videos too, of the model sending Christmas greetings to contacts and of it opening an issue tracker and writing an issue. Those are the two tasks people ask a GUI agent to do first, and the fact that both are shown end to end rather than as a click sequence is a good sign about the reliability of the action space.

What it costs to run

The inference and fine-tuning cost section is where the adoption decision is really made, and the number is large.

The readme states that the model requires at least 29 gigabytes of video memory for inference at BF16 precision, and then describes the lower precision path with INT4. There is no consumer card that holds 29 gigabytes comfortably, so the practical reading is that this is a multi-gigabyte server requirement or a workstation class machine.

The requirements file shows the software side, and it is a conventional modern machine learning stack.

transformers>=4.47.0
torch>=2.5.1
numpy==1.26.4
gradio>=5.23.2

The full list covers the transformers and torch core, the vision and audio pieces, the hub client, the tokeniser, a template engine, a validation library, a vision library, a tokeniser, a pinned numpy, the accelerate and sentence transformers libraries, a gradio interface, an OpenAI client, an einops library, imaging, a server-sent events library for streaming, a bits and bytes library for quantisation and the spaces library. A separate group of dependencies is commented out for the fine-tuning path, with the natural language toolkit, a Chinese segmenter, a YAML library, the datasets library, a parameter-efficient tuning library and a Chinese Rouge metric.

So the inference path is a normal transformers and gradio setup, and the fine-tuning path adds the usual training stack.

The repository, the licences and the deployment reality

The repository layout is short, and it is arranged around the three jobs.

There is the inference directory, the fine-tuning directory and an app directory, which is the demo application, alongside the assets used by the demos, the action space documents in English and Chinese, the requirements file, a Chinese readme and the licence files.

There are two licence files, which is worth noticing. There is the general licence for the code and a separate model licence, and that separation is normal for a model release, because the weights carry their own terms about what the model may be used for and how those terms differ from the code around it. Anyone deploying this commercially needs to read the model licence, not just the code licence.

The deployment reality is the last thing to weigh. The model is large, the inference cost is stated in video memory rather than in parameters, and the readme says the release is intended to help researchers and developers advance the research and applications of GUI agents based on vision language models.

So the honest summary is that this is a research release with a real deployment behind it, a peer reviewed highlight, weights you can download and fine-tune, and a cost profile that puts it out of reach of a laptop. The alternative is the hosted route the readme compares against, where you pay per call and run no model yourself, and the difference between them is ownership of the weights and the ability to adapt them to your own interface.

Editorial conclusion

Adopt CogAgent if you need a GUI agent whose weights you control, because the model is published for download and the repository carries the inference and fine-tuning code, and that is a different position from calling a hosted agent API. Do not adopt it without hardware, because the README puts the inference requirement at 29 gigabytes of video memory at BF16 precision before any fine-tuning. Before you start, read the action space documents in the repository, because the model output format is what your integration code has to parse, and the cost section in the readme is the number that decides the project.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. zai-org/CogAgent on GitHub