OpenHTF: a test framework where a test is a plain Python program

A test is a Python file, and the framework runs it for you

OpenHTF is a Python library that provides a set of convenient abstractions designed to remove as much boilerplate as possible from hardware test setup and execution, so that test engineers can focus primarily on test logic. The README says it aspires to do so in a lightweight and minimalistic fashion, and that it is general enough to be useful in a variety of hardware testing scenarios, from the lab bench to the manufacturing floor.

The central design decision is that a test is a regular Python program. In the OpenHTF paradigm, tests are expressed as .py files that import and instantiate the Test class from the openhtf module, and what distinguishes an OpenHTF test from any other Python program is that it imports the package, instantiates the Test class and calls its Execute() function.

From there, the framework takes over the parts that are the same in every hardware test. It manages the setup, execution and teardown of the test, keeps track of anything gathered, and provides a pass or fail result. The README is explicit about the benefit of that choice, that it provides the flexibility to do anything in a test that can normally be done in Python. A test engineer is not restricted to a DSL, and can call an instrument, write a file, or branch on a measurement inside a phase.

The disclaimer at the top of the README is short and worth quoting exactly, because it sets the expectation: this is not an official Google product. The repository is the open source framework without the Google apparatus around it, and the AUTHORS file at the top of the tree is credited to The OpenHTF Authors rather than to a Google team.

Phases, measurements and plugs, and how a result is decided

Three concepts do most of the work in an OpenHTF test, and the README documents each of them as nomenclature.

A phase is a logical block of the test. Phases are no more than normal Python callables, usually functions, combined with the needed metadata, and writing a test is a matter of writing phase functions and specifying the order in which they should be executed.

A measurement is the data the framework gathers about the device under test, and the important part is that a measurement is declared along with a specification describing what constitutes a passing value. The rule for the verdict is stated directly: if OpenHTF finishes the test run and one or more measurements were out of spec, the result of the whole test run is considered a fail. That is a different model from a framework where a phase returns a boolean, and it means the pass or fail logic lives in the specification rather than scattered through the test code.

A plug is the piece of code that lets OpenHTF talk to hardware. The README describes it as the essence of an OpenHTF test, since the point of the test is to interact with a device under test to exercise it in various ways and observe the result. Sometimes that interaction is direct with the DUT, and sometimes it is with a piece of test equipment to which the DUT is attached. OpenHTF ships with a growing collection of useful plugs, and supports the creation of custom plugs as well.

The third concept, the attachment, covers the case where the data is more complex or free-form than a measurement. An attachment can link arbitrary binary data to a test record, with an optional MIME type. That is how a captured image, a log dump or a firmware blob ends up attached to a test record without being forced through a measurement specification.

Test run, test recipe and test script, and the station breadcrumb

The README spends a paragraph disambiguating the word test, and the distinctions it draws are the ones a team running a production line will care about.

A test run is a single start-to-finish execution of a specific test. A test recipe is a test definition that may be executed multiple times, each time as a distinct test run. A test script is a .py file that contains a test recipe.

The station concept sits on top of those. A station captures the notion that a given test ran at some point and may run again, and it loosely reflects the idea of physical test stations that process multiple DUTs over time. The mechanism is a breadcrumb: OpenHTF writes one to the filesystem, in a directory that can be set using the --rundir flag, each time a test runs, and all tests that have the same name are considered to be of the same station. This is what lets the web frontend display a consolidated list of known tests as a list of stations.

That design decision is worth pausing on, because it is unusual. The identity of a station is the name of the test, not a configuration file or a database row, and the grouping is derived from breadcrumbs on disk rather than declared. The consequence is that the filesystem becomes the index of what has been tested, and the same test name run on two different benches collapses into one station, which is convenient for a consolidated view and ambiguous if you wanted to distinguish the benches.

The DUT term is used throughout for the individual piece of hardware being evaluated, exercised, or tested.

Installing the package, and the protocol buffer wrinkle in the examples

The installation guidance recommends using virtualenv to create an isolated Python environment, so as to protect system-wide Python packages the OS depends upon, and the instructions assume you have already created and activated one.

The recommended route is pip, which installs the most recent production release.

pip install openhtf

Installing from source is the alternative, and the README gives the reason you would want it: if you want a new feature that has not made it to the production release yet. The source is downloaded via git or other means and installed into the environment using the standard build command, with the details in CONTRIBUTING.md.

There is a wrinkle for the examples, and it is the one practical gotcha in the setup. The README notes that some of the examples require protocol buffer code to be generated via a python setup.py build command, which requires the protocol buffer compiler library to be installed, with additional instructions in CONTRIBUTING.md for setting up the development environment.

That is not a small thing, because the protocol buffers are structural. The pyproject configuration lists four protobufs under the setuptools-protobuf section, for the assembly event, the guzzle, the manufacturing event and the test runs. The build backend is setuptools with setuptools-protobuf, so a plain pip install of a source checkout has to compile those schemas, and the compiler is a system-level dependency rather than a Python package.

The project metadata pins requires-python at 3.11 or later, and the version is 1.6.3. The direct dependencies are worth a look because they describe the runtime: attrs, colorama, contextlib2, inflection, google-auth, mutablerecords, protobuf, PyYAML, pyOpenSSL, requests, sockjs_tornado, tornado pinned at exactly 6.5.9, and typing_extensions. The google-auth and pyOpenSSL entries are the reason there is a credentials angle to the framework, and the sockjs_tornado and tornado pair is the web frontend's transport.

The web frontend, the examples directory, and the plugins you must write

The examples directory is where the README sends a newcomer, and the file list is a good map of the framework's feature set. There is hello_world.py, measurements.py, checkpoints.py, phase_branches.py, phase_groups.py, repeat.py, stop_on_first_failure.py, ignore_early_canceled_tests.py, with_plugs.py, example_plugs.py, all_the_things.py, and frontend_example.py, plus a resistor_tutorial directory and an example_config.yaml.

Reading that list tells you what the framework considers its core concerns. phase_branches and phase_groups are about control flow inside a test. stop_on_first_failure is about verdict semantics. repeat and checkpoints are about running the same recipe many times and resuming partway. frontend_example is about the web view, and with_plugs and example_plugs are about the hardware interface.

The web frontend is a Tornado application, and the sockjs_tornado dependency is the websocket layer it uses. The package data configuration in pyproject.toml points at output/web_gui/dist with its css, js and img subdirectories, so the frontend is a compiled bundle shipped inside the package.

The plug situation is where a user does real work. The optional dependency groups in the project metadata show how the extra hardware support is packaged: usb_plugs pulls in libusb1 and M2Crypto, update_units pulls in xlrd, serial_collection_plug pulls in pyserial, and examples pulls in pandas. So USB, serial, spreadsheet and notebook support are separate extras rather than part of the core install, and the pyOpenSSL and google-auth dependencies in the main list sit alongside them.

That structure is the honest cost of the framework. It gives you the execution model, the data model, the verdict logic and the frontend, and leaves the part specific to your lab to you.

Maintenance, the roll-up release cadence, and what the licence allows

The project is alive and the release naming tells you how it is managed. The recent releases are v1.6.3, described as a roll-up release, on 2026-10-08, and 1.6.0, also a roll-up release, on 2025-03-05, with a test publish workflow tag in between. The last push was on 2026-10-20, and the pyproject version matches the newest tag at 1.6.3.

A roll-up release is the term for a release that gathers a set of changes into one version rather than shipping a patch for each. For a framework like this that is the right pattern, because the surface a hardware test touches is wide: a change to a plug, to the measurement handling, or to the frontend can all land together. The cost is that upgrading is a larger event, since you take everything in the window rather than choosing a small change, and the gap between 1.6.0 in March 2025 and 1.6.3 in October 2026 suggests the windows are long.

The repository carries the artefacts you would expect from a project with this much history: a CHANGELOG file, a CONTRIBUTING.md, a tox.ini for test environments, a .coveragerc with a Coveralls badge, a pylintrc alongside a pylint_plugins directory, a MANIFEST.in, and a continuous integration workflow badge on the master branch.

The licence is Apache-2.0, and the project metadata points at the LICENSE file. The disclaimer that it is not an official Google product does not change the licence, and the copyright in the file governs what a user can do with the code. For an engineer evaluating it, the practical questions are the ones the framework itself raises: how the test records are stored, what the web frontend exposes, and what the protobuf schemas commit you to.

Editorial conclusion

OpenHTF fits a team that already has instrumented hardware and needs reproducible, logged test runs with a web view of results, and it does not fit a team that wants a graphical test editor or a test that a non-Python engineer can write. Before adopting it, look at the examples directory rather than the framework documentation, because the README points there for the fastest way in, and check the plugin structure for the two or three instruments you actually need, since each plug is code you write against a specific protocol and a plug for hardware OpenHTF does not ship is work you are taking on yourself.

Frequently asked questions

Official sources

1. google/openhtf on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases