MarkBench test harnesses: scripted benchmarks for games and desktop applications

What the repository holds and how it fits together

This is the home of the test harnesses used in LTT Labs MarkBench, and the README is short enough to state the whole design.

A harness is a pair of files. There is a TOML descriptor, and a Python entry point specified by the TOML's `py_script` field. The harnesses can also be run as standalone scripts, so a harness is not useless without the orchestrator.

The division of labour is explicit. The TOML contains the orchestration metadata, and the Python entry point handles the whole lifecycle: launch and setup, UI automation, benchmark timing, artifact capture, report writing, cleanup, and the exit status.

That split is the interesting design decision. Everything declarative sits in the TOML, so the same harness can be pointed at a different executable or a different benchmark scene without touching Python. The Python side is where the imperative work lives, and it is a script per application rather than a framework that a script plugs into.

The repository layout follows the same logic. There are four directories of harnesses, games/, non_games/, stress/ and utility/, plus harness_utils/ for the shared code, a deprecated/ directory for retired harnesses, a pyproject.toml, a CHANGELOG.md, a cspell.json and a licenses.md.

The licence field on the repository says GPL-3.0, the primary language is Python, and the last push was on 2026-10-19, so the harnesses are being worked on daily.

The four harness categories

The harnesses are grouped into four categories, and the categories are directories in the repository root: games/, non_games/, stress/ and utility/.

The category field in the TOML mirrors that structure, with the example using category = "game". So the category is not decorative, it is how the orchestrator and the directory layout agree on what a harness is.

The distinction between games and non_games is the obvious one and the most useful, because a game benchmark and a desktop application benchmark have different shapes. A game typically needs a scene selected from a menu, a launch sequence, a scene to be played for a duration, and a score at the end. A desktop application might need a configuration applied, a workload run and a measurement taken without a score.

The stress category is the third, and it is for workloads whose purpose is to push the machine rather than to measure the application, which is a different objective from the first two and would pollute a results set if it were mixed in.

The utility category is the fourth, and it holds harnesses that measure something about the machine or the environment rather than about a program under test. It is the natural home for a harness that reports baseline numbers other harnesses can be compared against.

The directory names match the category values, and the deprecated/ directory keeps retired harnesses out of the active set while preserving them for reference.

The TOML descriptor and its option types

The README gives a generic TOML that includes all current fields and option types, and it is short enough to read as a specification.

friendly_name = "Example Game"
py_script = "example_game.py"
executable = "ExampleGame.exe"
presentmon_enabled = true
presentmon_ignore_exit = true
category = "game"
ocr = true

The top-level fields are the display name, the Python entry point, the executable to launch, the PresentMon toggle, the category, and the OCR toggle. The `presentmon_ignore_exit` field is optional and defaults to FALSE, which is worth understanding: it controls whether a non-zero exit from the capture tool is treated as a failure of the harness, and the default treats it as a failure.

The repeatable part is `[[args]]`, which is optional and can appear any number of times. Each entry has a name, a type, a tooltip, and a default, and the type determines what the orchestrator presents.

[[args]]
name = "benchmark"
type = "select"
default = "battle"

[[args]]
name = "duration_seconds"
type = "input"
default = "120"

The select type uses a list of values, so the user picks one of them, with battle and campaign as the example choices. The input type accepts free-form text, which is how a duration in seconds is supplied. The tooltip is optional help text, and the default is optional.

The result is that the parameters of a benchmark are data. A harness author defines the scene list in the TOML, and the orchestrator renders a control for it, without the Python code knowing what a scene is.

The output contract: report.json and the artifacts directory

Every run writes to a run/ directory under the harness, and the layout is fixed enough to be a contract.

run/harness.log is the execution log. run/report.json is the machine-readable result. run/artifacts/ holds the screenshots, the copied files and an artifacts.yaml file.

The report.json has a set of common fields, and the list is short enough to be worth knowing by heart. The resolution is recorded as WIDTHxHEIGHT. The start_time and end_time are in epoch milliseconds, which makes the duration a subtraction rather than a parsing exercise. The version is the game or benchmark version. And there is a score with a unit, both when applicable.

The phrase when applicable is doing real work. A utility harness measuring frame time has no score, and a stress harness has no version, and a schema that required them would force every harness to invent placeholder values. Making the optional fields genuinely optional means a consumer of report.json has to check what it got, which is a small cost that buys honest data.

The epoch millisecond choice is worth noting too. Since the timing and the result live in the same file, a consumer can compute an elapsed time without parsing the log, and the log stays free to be a human-readable narrative of the run.

The Windows-first dependency set

The pyproject.toml is where the platform reality of the project shows, and it is worth reading in detail.

The package requires Python 3.11 exactly, pinned with a version specifier rather than a floor, so the harness code can rely on a specific interpreter.

Several dependencies are conditional on Windows. dxcam, which is a DirectX capture library, is required only on win32. pygetwindow, for window manipulation, is also Windows only. pywin32, the Windows API extension, is likewise conditional.

The cross-platform ones are the interesting contrast. mss is a cross-platform screen capture library, and opencv-python-headless is the headless build of OpenCV, which is the right choice for a machine that has no display server. psutil gives process and system information, requests handles HTTP, pyyaml parses YAML, vdf reads Valve's VDF format, and vgamepad is for virtual gamepad input.

The headless OpenCV build is a deliberate signal. A benchmark harness wants to run without a visible window, and a GUI-linked OpenCV would pull in display libraries it does not need.

The optional dependencies section describes the platforms. The windows extra is empty, which suggests the Windows dependencies are already the defaults. The linux extra lists its system dependencies, namely ydotool, ydotoold and spectacle, and it carries a note that the Linux virtual gamepad requires pre-provisioned /dev/uinput access, which is the kind of statement that tells you a Linux gamepad benchmark is possible but not turnkey.

Where these harnesses are the wrong tool

The first limitation is that this is not a general test framework. The contract is specific: a TOML with a py_script field, a folder name that matches the TOML file name, a run/ directory with a fixed layout, and a report.json with a known set of fields. A harness written for another runner will not work here, and a general purpose automation tool would be the better choice for testing an application whose performance is not the question.

The second is the platform assumption. The dependency list is Windows-first, with three of the capture and windowing libraries conditional on win32, and the Linux path needs a specific set of system tools and pre-provisioned device access. A team that runs its benchmarks on Linux is taking on a materially harder setup.

The third is that these harnesses measure applications, not systems. The result is a score for one game on one machine, and comparing two machines means controlling for driver versions, background load and the specific build. The report schema is deliberately small, and it does not carry the environmental metadata that would make a cross-machine comparison rigorous.

The fourth is the strict interpreter pin. Python 3.11 exactly is a constraint on the environment, and it will need revisiting whenever the project's Python moves on.

The fifth is the deprecated/ directory. Retired harnesses are kept in the repository, which is good for reference and bad for someone counting the harnesses in the tree and assuming they all work.

How these harnesses compare to a general UI test framework

The comparison with a general UI automation framework is the one that defines the purpose, and the difference in approach is what the harness is measuring.

A general UI test framework drives an application to check that it behaves correctly, that a button appears, that a flow completes. Its unit of success is a pass or fail, and its output is a test report. A benchmark harness drives the same application for a different reason, to hold it in a known state for a known duration while a timing tool records what the machine does, and its output is a number plus the evidence behind it.

The evidence is the artifacts. Screenshots and copied files under run/artifacts/ mean that when a score looks wrong you can look at what was on screen at the time, which is a debugging step a pass or fail framework does not usually provide.

The third comparison is with a game benchmark tool built for one title. Those tools often have deeper instrumentation of the game itself, and they are usually closed. The value of a harness set like this is the opposite: the same structure, the same report schema and the same orchestration for every application, so two unrelated programs produce comparable files.

The fourth is with a synthetic benchmark. A harness that launches the real application and plays it is slower and noisier than a synthetic loop, and it is the only one of the two that measures what a user would experience.

Licence, tooling and how to add a harness

The licence field on the repository says GPL-3.0, and the repository has a licenses.md file that records the licensing of the material, which matters here because a benchmark harness may ship with game assets or data files that carry their own terms. There is a CODE_OF_CONDUCT.md, a CONTRIBUTING.md and a CHANGELOG.md.

The tooling is light but deliberate. The project is configured for ruff with a py311 target, and the lint configuration ignores E402, BLE001, TRY002 and RUF015. Each of those suppressions encodes a decision. E402 allows module level imports that are not at the top of the file, which is what a harness that has to set up capture before importing its own modules needs. BLE001 allows blind except, which a cleanup path that must run whatever happens needs. RUF015 allows a list to be indexed by a value that might be out of range.

There is also a cspell.json at the root, which is how a project with a large vocabulary of vendor names, executable names and window titles keeps its spell checker usable.

Five steps to add a harness, and the last one is the one people miss. Create the folder in the right category directory. Write the TOML descriptor with a friendly_name, the py_script, the executable and the category. Write the Python entry point that handles launch, automation, timing, capture, reporting, cleanup and the exit status. Reuse the shared helpers in harness_utils/ rather than reimplementing capture. And make sure the name of the harness folder and the name of the harness TOML file match, because the README states that they must match to work with Markbench, and a mismatch is a silent failure rather than an error message.

Editorial conclusion

These harnesses suit a team that wants repeatable, comparable measurements of how a game or a desktop application performs on a given machine, with the run configuration expressed as data rather than code. They are the wrong choice for a project that needs a general test framework, since the contract is specific to MarkBench, and the wrong choice if you need to run them on Linux, since the dependency set is Windows-first and the Linux path needs pre-provisioned input device access. Before writing a new harness, read the generic TOML example in the README, and note the one naming rule that silently breaks everything: the harness folder name and the harness TOML file name must match.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. LTTLabsOSS/markbench-tests on GitHub
4. README