ok-script: computer vision automation for Windows apps and Android devices

Automation that reads pixels instead of calling window APIs

ok-script is a Python automation framework built on computer vision. The project describes it as a computer vision based Python automation framework aimed at Windows applications, Windows games, Android emulators and ADB devices. Its topic list carries automation, computervision, game and python.

The mechanism is the differentiator. Rather than driving an application through its accessibility tree or its window handles, the framework takes a screenshot, looks for what it expects to find, and issues synthetic input. That works on anything that renders pixels, which is why the same task code covers a Windows desktop program and an Android emulator, and why the README can claim that one set of task code supports the Windows client, the Android emulator and ADB devices.

The capabilities listed cover the full loop of such a system: UI, screenshots and input, device control, OCR, template matching, a visual debug overlay, automated testing, packaging and incremental updates. Vision approaches are pluggable, with OCR, OpenCV template matching and YOLO all named as options.

The audience is therefore teams writing bots or test harnesses against graphical interfaces, where no API exists. The project ships a directory of real deployments built on it, several of them game clients, which tells you both what it is used for and what kind of maintenance the ecosystem expects from it.

The project is on PyPI as `ok-script`, published from this repository with a `setup.py` that reads the current published version and computes the next one, and releases are frequent: v2.0.8, v2.0.9 and v2.0.10 all landed in October 2026.

One core, three front ends: run_task, gui and web

The most consequential architectural change in the project is that the automation core is decoupled from Qt. The README states that the core is now separated from Qt, so the framework can run with no interface at all, with the original desktop interface through the `qt` extra, or with a FastAPI browser interface through the `web` extra. The corresponding commands are `ok run_task`, `ok gui` and `ok web`.

That matters for how the framework is deployed. A headless install has no GUI toolkit on the machine at all, which is what you want on a build agent or a server. The web mode exposes the same tooling through a browser, and the desktop mode keeps the original Qt experience.

The package manifest confirms the split. The base dependencies are Windows oriented: `pywin32>=306,!=312`, `pyappify>=1.0.9`, `typing-extensions`, `requests`, `psutil`, `pydirectinput==1.0.4`, `pycaw==20240210`, `mouse==0.7.1` and `pynput>=1.8.1`. The screenshot helper is `ok-d3dshot`, pinned in the `default` group along with `numpy`, `Pillow` and `OpenCC`. The `qt` group brings `PySide6-Essentials`, `PySideSix-Frameless-Window`, `darkdetect` and `PySide6-Fluent-Widgets`, the `web` group brings `fastapi`, `uvicorn[standard]` and `pywebview`, the `adb` group brings `adbutils`, and the `ocr` group brings `onnxocr-ppocrv5`.

So the input, screenshot and window control layers come from the standard Windows automation stack, while the vision layer is optional and chosen by you.

Install with pip and choose your own OpenCV variant

The framework needs Python 3.11 or newer, and the README recommends Python 3.12. The base install is a single pip command:

python -m pip install ok-script

But the README steers you away from a plain install and towards extras, because the base install pulls only what every project needs. Each capability is a named group:

python -m pip install "ok-script[default]"
python -m pip install "ok-script[qt]"
python -m pip install "ok-script[web]"
python -m pip install "ok-script[adb]"
python -m pip install "ok-script[ocr]"

The `default` group is the headless mode, and the Web, Qt, ADB and OCR capabilities are added on demand.

One instruction is unusually strict. The framework does not choose OpenCV for you: you must pick the variant that matches your application and install exactly one of `opencv-python`, `opencv-contrib-python`, `opencv-python-headless` or `opencv-contrib-python-headless`. That is a sensible rule, since mixing the packaged variants is a well known source of conflicts, but it means an install that omits the choice will fail later rather than at install time.

If you are working inside this repository rather than consuming the package, the development install is different. The README says to install both the `default` and `dev` dependency groups from `pyproject.toml`, and to add `--group web`, `--group qt`, `--group adb` or `--group ocr` as needed:

python -m pip install --editable . --group default --group dev

The project also recommends starting new automation work from the `ok-script-app` template repository rather than writing business scripts inside the framework repository itself.

COCO formatted template assets that scale with screen resolution

Template management is where most of the engineering sits in a computer vision automation framework, and ok-script makes a specific choice: it manages template assets in COCO format, the annotation standard used for object detection datasets, and scales them automatically according to screen resolution.

The reasoning is practical. A template cropped from a 1920 by 1080 screen will not match on a 1440 by 900 laptop, so the framework scales stored assets to the resolution it is running at. Using COCO annotations as the storage format means template metadata rides along in a format that vision tooling already understands.

Recognition itself is layered. The README names OCR, OpenCV template matching and YOLO as the supported visual approaches, and the OCR path is notable for what it does not require. The acknowledgements credit `OnnxOCR`, described as providing OCR with Intel NPU support, inferring through OpenVINO or ONNX Runtime and using PaddleOCR v5 models. That combination means text can be read on a local machine without a GPU, and the `onnxocr-ppocrv5` package in the `ocr` extra is the whole dependency for it.

For debugging, the framework includes a visual overlay for template recognition, and the README points to a separate document, `docs/interface.md`, for the complete description of the interface and development tools. Four screenshots in the README show task configuration, screenshot and interaction, the scripting and API view, and template management.

The task level API is documented separately under `docs/api_doc/README.md`, which covers `Box`, `BaseTask` and the common task APIs, while `docs/intro_to_automation/README.md` explains the screenshot, recognition, decision and input loop that the whole framework is built around.

The web mode is a React and Fluent UI front end over the Python core

The browser interface is not a wrapper around a remote desktop. The repository contains a separate `web_src/` directory with its own `package.json`, and that manifest describes a modern TypeScript and React application.

The package is named `ok-script-web-ui` and is marked private, so it is not published separately. Its dependencies are `@fluentui/react-components` and `@fluentui/react-icons` alongside React and React DOM, and the build runs through Vite with the React plugin and TypeScript 5. The scripts show the shape of the workflow: `dev` starts the Vite server against `web_src/vite.config.ts`, `build` compiles the two TypeScript projects and then bundles, and `typecheck` runs the same compilation with no emit. There is a `predev` and `prebuild` step that both run `npm run i18n`, which executes a Python script through `scripts/run_python.mjs`, so the internationalisation strings are generated from Python sources before the JavaScript side builds.

That last detail says something about the project as a whole. The framework is Python, but the web interface pulls its translation strings out of Python and feeds them to a TypeScript build, which is a deliberate coupling rather than a duplicated set of locale files.

On the Python side, the `web` extra serves this through FastAPI with Uvicorn, and `pywebview` is included as well so the same build can run in a native window on a desktop. The `web` extra in the manifest is what makes both `ok web` and the pywebview path available.

A setup.py that computes the next version from PyPI

The build configuration has one unusual piece that is worth reading before you fork the project. `setup.py` does not hardcode a version. It first checks an environment variable:

import os
import setuptools

os.environ["PYTHONIOENCODING"] = "utf-8"

VERSION_NUM = os.environ.get('OK_SCRIPT_BUILD_VERSION')
if VERSION_NUM:
    print(f'building explicit version {VERSION_NUM}')
else:
    from get_pypi_latest_version import GetPyPiLatestVersion

    obtainer = GetPyPiLatestVersion()
    latest_version = obtainer("ok-script")
    VERSION_NUM = obtainer.version_add_one(latest_version, add_patch=True)
    print(f'latest_version is {latest_version} new version is {VERSION_NUM}')

setuptools.setup(version=VERSION_NUM)

With `OK_SCRIPT_BUILD_VERSION` set, the build uses that version. Without it, the script queries PyPI for the latest published `ok-script` release, adds one to the patch number, and uses that. The helper lives in the repository root as `get_pypi_latest_version.py`.

The effect is that a local build of an unmodified checkout produces a version one patch above what is on PyPI, which is a workable scheme for a single maintainer publishing frequently but a sharp edge for anyone who wants reproducible builds or wants to release from a fork. A local build also needs network access to PyPI at build time, unless the environment variable is set.

Otherwise the packaging is conventional setuptools with a `pyproject.toml` declaring `setuptools>=77` as the build requirement, a `setup.cfg`, a `MANIFEST.in`, a `pytest.ini` for tests and `mkdocs.yml` for the documentation site.

The Commons Clause limits selling the framework itself

The licence is not a standard one, and GitHub cannot classify it. The README names it as the Apache License 2.0 plus the Commons Clause and additional terms, with the text in `LICENSE.txt`.

The permitted uses are broad. Closed source projects are allowed, and commercial projects are allowed. Forking and improving the project is allowed provided the fork stays open source and free.

The restrictions are narrower but decisive for some readers. You may not sell the project directly, nor rename it, modify it slightly and then sell or promote it as a competing product. Use for hacking, unauthorised access, malware, fraud or other illegal activity is prohibited. And there is an attribution obligation that applies to every project using ok-script: at least one of the project website, the GitHub page or the software itself must mention and link to ok-script.com or the ok-script GitHub repository. Using the Qt interface also means complying with the PyQt-Fluent-Widgets licence.

In other words, the Commons Clause plus the extra terms means you can build a commercial product on top of the framework, provided you attribute it, but you cannot turn the framework itself into a product. That is a statement about the terms, not advice on how to apply them, and the difference between those two cases is exactly where a legal reading of `LICENSE.txt` matters.

The acknowledgement list also names the upstream projects whose own licences apply independently: PyAppify for updates and packaging, OnnxOCR, PySide6, PyQt-Fluent-Widgets, OpenCV, NumPy, Pillow, FastAPI, Uvicorn, pywebview and adbutils.

Against writing pywin32 and pydirectinput scripts yourself

The honest alternative is not another framework but the direct approach, because the dependency list shows exactly what the direct approach costs.

Writing it yourself means combining `pywin32` for window handles and process control, `pydirectinput` for synthetic keyboard and mouse input at the driver level, `pynput` or `mouse` for input events, `pycaw` for the Windows UI Automation API, and `psutil` for process management. The screenshot layer is the piece no library gives you for free, and the recognition layer is where the real work sits: you need to locate a target on screen, decide what it means, and act, repeatedly, without the screen changing underneath you in a way that breaks the match.

That is the difference in approach. A hand-written script is a sequence of steps against a specific window and a specific resolution. A computer vision framework is a task loop with template assets, resolution scaling, a debug overlay and a plugin point for the recognition model, so the same task code survives a change of resolution and can be pointed at a different application.

The cost of that generality is visible in the project itself. A framework with template management, internationalisation, packaging, incremental updates and three front ends carries far more surface area than a hundred lines of pyautogui. It also carries a licence that forbids competing products, which a script of your own does not.

The other alternative is a hosted device farm or a commercial RPA suite, which trade the framework weight for per-seat licensing and a closed model. If you need scheduling, recording and a dashboard rather than a library you control, that is a different product, and the choice comes down to whether you want the vision loop to be yours.

Editorial conclusion

Adopt ok-script when you need to automate a Windows desktop application or an Android device through screen recognition, and when the attribution requirement in LICENSE.txt is acceptable. Do not adopt it if you need a permissive licence or a product that sells a derivative of the framework itself, since the Commons Clause forbids that, and check first that the OpenCV variant you install matches your environment, because the framework deliberately does not choose one for you.

Frequently asked questions

Official sources

1. Issues
2. ok-oldking/ok-script on GitHub
3. Project website
4. README
5. Releases