KnowAct-GUIClaw: A Self-Evolving GUI Assistant for Android, iOS and Desktop

What KnowAct-GUIClaw automates, and for whom

The project is a personal assistant that operates graphical interfaces rather than APIs. The README lists desktop, Android, iOS and HarmonyOS as targets, and the CLI exposes a separate backend for each: adb for Android, ios through WebDriverAgent, hdc for HarmonyOS, and local for foreground desktop automation. That breadth is the point. Most GUI agents pick one platform and build around its accessibility tree. GUIClaw abstracts the platform behind a backend flag and keeps the agent loop identical.

The intended user is someone who already has a multimodal model endpoint and wants an agent that controls a screen. The repository ships two entry points for two different habits. If you want a host with files, shell, web, MCP and memory tools alongside screen control, you install the repository environment and use nanobot, where GUIClaw registers as a task named gui_task. If you want a single executable you can call from a terminal, a script, or another host such as Hermes or OpenClaw, you install the local project as a uv tool and call guiclaw. The README is explicit that the standalone path does not import nanobot at runtime, so the two paths are genuinely separable rather than one wrapping the other.

The claim that separates this project from a plain automation script is persistence. The README describes a Know-Route-Act-Reflect loop, with induced GUI memory written to ~/.guiclaw/memory/gui_memory_bank.jsonl and validated shortcuts plus extracted skills written to ~/.guiclaw/skill/skills.py. A one-shot script forgets; this one accumulates.

The Know-Route-Act-Reflect loop and where state lives

The architecture diagram in the repository names four phases. Know covers memory and skills that inform the task. Route selects which skills or shortcuts apply. Act executes steps against the device. Reflect feeds results back into memory and skill extraction. The README does not spell out the internals of each phase, so treat the diagram as the project's own summary rather than a specification.

What is documented is the storage layer, and it is unusually concrete. Runtime data sits outside the host workspace, under ~/.guiclaw/. Screenshots, compact trajectories and task results go to ~/.guiclaw/gui_runs/. Statically inferred Android shortcuts go to ~/.guiclaw/shortcut_cache/, and runtime validation records to ~/.guiclaw/shortcut_cache_validation/. The shared skill file is ~/.guiclaw/skill/skills.py, described as validated shortcuts followed by extracted skills. GUI memory lands in ~/.guiclaw/memory/gui_memory_bank.jsonl, and an always-injected policy lives in ~/.guiclaw/memory/policy.md.

The skills.py detail deserves attention. Skills are stored as Python, not as a data format the agent parses at runtime. That means the skill layer is inspectable and editable by a human, and it also means the file is executable code the tool loads. Reviewing it before a run is not optional hygiene; it is reading the instructions the agent will follow.

The policy file works differently from a sandbox. When no POLICY entry exists, the first GUI task initializes a conservative permission policy that denies, cancels or defers requests unless the task explicitly authorizes them. Users can edit the structured entry in policy.md. The README then states plainly that these rules are model guidance, not guaranteed enforcement, and points to OS, backend, host approval or sandbox controls for mandatory restrictions. That sentence is the most important one in the document, and it should shape how you deploy this.

Installing GUIClaw as a standalone CLI

The standalone path needs Python 3.11 or later and uv. Clone the repository and install the command from the repository root. The README gives a plain install for Android, HarmonyOS or dry-run use, and a desktop extra for local desktop automation, which you would install instead when the target is your own machine.

git clone https://github.com/HITsz-TMG/KnowAct.git
cd KnowAct

# Android, HarmonyOS, or dry-run
uv tool install ./GUIClaw

# Use this instead for local desktop automation
# uv tool install './GUIClaw[desktop]'

Next create the configuration file at ~/.guiclaw/config.yaml. The README's example sets the provider base URL and model name, and four runtime values. max_steps caps the loop at 15 in the example, image_scale_ratio of 0.5 halves screenshots before they reach the model, and agent_profile selects the default profile. stagnation_limit is present but set to 0 in the example; the README does not explain what that value means, so leave it as shown until you read the CLI reference.

provider:
  base_url: "https://api.example.com/v1"
  model: "your-vision-model"

max_steps: 15
stagnation_limit: 0
image_scale_ratio: 0.5
agent_profile: default

Export the API key and run a smoke test. The first command uses --dry-run, which the README describes as testing the model and the agent loop without changing a real device. The second targets an Android device over adb and asks it to open Settings and enable Wi-Fi. If the dry run returns a description of the current screen, your provider, model name and key are wired correctly.

export OPENAI_API_KEY="your-api-key"

guiclaw --dry-run "Describe the current screen and finish"
guiclaw --backend adb "Open Settings and enable Wi-Fi"

For scripts and other agents, add --json to get machine-readable output. The README shows this form with a task that opens Contacts and searches for a name. The CLI reference at GUIClaw/docs/guiclaw-cli.md is where the README points for installation extras, every flag, complete defaults, backend setup, shortcut validation and subprocess integration.

guiclaw --backend adb --json --task "Open Contacts and search for John"

The nanobot host path and the gui_task registration

If you want one runtime that holds screen control next to file, shell, web and MCP tools, the host path is the other option. You install the repository environment with uv sync and pass the web, desktop and cjk extras, then run the onboarding wizard. The distribution is named nanobot-ai and ships both packages.

git clone https://github.com/HITsz-TMG/KnowAct.git
cd KnowAct/GUIClaw
uv sync --extra web --extra desktop --extra cjk
uv run nanobot onboard --wizard

The wizard handles the host, but the GUI side needs a gui block added to ~/.nanobot/config.json, and the provider must also be configured for nanobot. The README's example nests a custom provider with an apiKey and apiBase, then a gui block naming the backend, provider, model and agent profile, with maxSteps at 15 and three skill-related switches. enableSkillExecution and enablePromptSkillSelection both appear as true in the example, while promptShortcutOnly and promptSkillAppFilter are false. Note that the defaults table says prompt skill selection and skill extraction are disabled by default, so the example is showing an enabled configuration rather than the out-of-the-box state.

{
  "providers": {
    "custom": {
      "apiKey": "your-api-key",
      "apiBase": "https://api.example.com/v1"
    },
    "gui": {
      "backend": "adb",
      "provider": "custom",
      "model": "your-vision-model",
      "agentProfile": "default",
      "maxSteps": 15
    }
  }
}

Start the WebUI with uv run nanobot webui. The README states that the host registers GUIClaw as gui_task and routes screen work to it while keeping the other tools in the host runtime. The adapter contract at GUIClaw/ADAPTERS.md is the document to read if you intend to wire GUIClaw into a host of your own rather than nanobot.

Shortcut inference and what validation actually gates

The shortcut commands are the part of the CLI that most distinguishes GUIClaw from a generic screen agent, and they are also the part the README documents least. The workflow has three stages. First, infer Android shortcut candidates from an app manifest or a manifest directory. Second, validate those candidates on a real ADB device. Third, promote the eligible ones into the shared skill file.

guiclaw shortcuts SOURCE
guiclaw shortcuts CACHE.json --validate
guiclaw shortcuts CACHE.json --validate --promote

The separation matters because it puts a device check between static inference and the skill file. A shortcut guessed from a manifest does not reach skills.py until it has been validated on hardware and then explicitly promoted. That is a sensible gate, and it is the reason the cache directories are split into shortcut_cache for inference and shortcut_cache_validation for runtime records.

What the README does not say is how many candidates survive validation, what a failed validation looks like in the output, or whether promotion is reversible. The CLI reference is the place to look. Until you have read it, treat --promote as a one-way action against a file the agent will later load.

Where GUIClaw is the wrong tool

The permission model is the first limitation, and the README states it without hedging. Policy entries in policy.md are guidance to the model, not enforcement. If your threat model requires that the agent cannot perform an action, a written rule in a markdown file does not satisfy it. The README itself directs you to OS, backend, host approval or sandbox controls for mandatory restrictions. Anyone who needs a hard boundary around what the agent can touch should build that boundary outside GUIClaw, or choose a tool that runs inside one.

The second limitation is cost and latency by construction. Every step in the loop sends a screenshot to a multimodal model. The image_scale_ratio default of 0.5 exists precisely because full-resolution screenshots are expensive, and the default maximum of 15 steps bounds a single task. Long-horizon work on a device means many model calls, and the README's benchmark claim about long-horizon performance does not change the per-step economics.

The third is platform coverage in practice. Four backends are listed, but iOS depends on WebDriverAgent and HarmonyOS on hdc, each with its own setup that the README does not walk through here. A team that only needs to script one Android app may find that a direct instrumentation framework is less machinery than a vision loop, because the loop's advantage, generalizing across interfaces, is worth nothing when the interface never changes.

Finally, the README as published ends mid-sentence in an Android section. Anything about iOS setup, HarmonyOS setup, memory extraction behavior or failure recovery is not in the portion reviewed here, which is a reason to read the full repository documents before committing.

How this compares to scripted UI automation

The obvious alternative is a scripted automation framework such as Appium or a platform-native tool like Android's UIAutomator. The difference in approach is fundamental rather than incremental. A scripted framework addresses elements through selectors, resource IDs or accessibility labels that a developer writes and maintains. It is deterministic, fast, and free of model calls, and it fails loudly when a selector stops matching.

GUIClaw addresses the screen through a vision model. It does not need selectors, which is why it can move between Android, iOS, HarmonyOS and desktop without rewriting the task. The cost is that every action is a model inference, results are probabilistic, and the failure mode is a plausible wrong click rather than a clean exception. The memory and skill layers are the project's answer to that instability: instead of fixing behavior in code, it records what worked and reuses it.

That trade-off suggests the split. Use scripted automation for a fixed flow on a fixed app where reliability and cost per run matter. Use GUIClaw for tasks that span apps or platforms, where writing selectors for every screen would cost more than the model calls, and where an occasional retry is acceptable. The two are not exclusive; a scripted job can call guiclaw --json for the parts that resist selectors.

Licence, maintenance and the cost of upgrading

The repository is MIT licensed, which permits commercial use and modification with the licence and copyright notice retained. That is permissive, and it also means no warranty and no support obligation from the authors. If you embed GUIClaw in a product, the MIT notice needs to travel with the distribution; this is a description of the licence text, not legal advice, and your counsel should review how it interacts with your own dependencies.

Maintenance signals are mixed. The repository is not archived, and the last push was on 2026-08-26, roughly three weeks before this writing. The most recent release listed is GUIClaw-Result from 2026-07-12, which the README describes as real-device experimental logs and trajectories rather than a software version. There is no conventional versioned release cadence visible in the repository, so pinning a version means pinning a commit rather than a tag.

Upgrade cost concentrates in two places. The first is the configuration surface: the standalone CLI reads ~/.guiclaw/config.yaml while the nanobot adapter reads a gui block in ~/.nanobot/config.json, and the two have different field names for overlapping concepts, such as max_steps against maxSteps. A change to one does not automatically reach the other. The second is accumulated state. Memory in gui_memory_bank.jsonl and skills in skills.py are built up over runs, and the README does not document a migration path or a schema version for either. Back both files up before upgrading, and expect to re-validate shortcuts if the shortcut cache format changes, because the README does not promise compatibility.

Editorial conclusion

Adopt GUIClaw if you already run a multimodal model endpoint and want device automation that accumulates memory and validated shortcuts across sessions, and if you can keep the agent on a test device or emulator. Do not adopt it if you need enforced permission boundaries, since the README states the policy rules are model guidance rather than guaranteed enforcement, or if you want a managed product with a support contract. Verify first that your provider is reachable from ~/.guiclaw/config.yaml, that --dry-run completes against your chosen backend, and that the ~/.guiclaw/skill/skills.py and ~/.guiclaw/memory/ files land where you can review them before you point the agent at a device holding real accounts.

Frequently asked questions

Official sources

1. HITsz-TMG/KnowAct on GitHub
2. License: MIT
3. Project website
4. README
5. Releases