labgrid: an abstraction layer for embedded board testing

What the project is

labgrid is a Python library for embedded board control, with a focus on testing, development and general automation. And it includes a remote control layer for boards that are connected to other hosts.

That last clause is the one that changes what the project is. A library that controls a board over a serial port is a convenience. A library that lets a test on your laptop drive a board sitting in a rack two rooms away, through a machine that has the board's power switch and its serial lines attached, is infrastructure.

And the features list is the inventory of a hardware lab. There is a client, an exporter and a coordinator architecture, so that boards can be made available from different computers on a network. There is a plugin for the standard Python test framework, so that a test can bring a board into a known state. There is a command line interface and a library interface for development and automation. There is interaction with a bootloader and with a Linux shell over a serial console or over a network shell. There is power and reset management through drivers for power switches. There is uploading of binaries and bootstrapping of devices over a universal serial bus. There is control of digital outputs, of memory cards and of the multiplexers that let several boards share one serial line. There is integration of audio, video and measurement instruments for remote development and testing. And there is integration with the two usual ways of running a system without the hardware: containers and the emulator.

Read that list as a set of nouns rather than as a set of features, because that is what it is. The library's job is to make a power switch, a serial console, a bootloader, a multiplexer and a measurement instrument look like the same kind of thing, so that a test can be written once and run against whatever the lab happens to have.

And the readme is careful about the boundary of the project. It is not a testing framework; it is intended to be combined with the standard Python one and its plugins. That is a sensible division, because the test framework is a solved problem and the hardware control is not.

The background is short and worth having. The work started at a company that builds embedded systems, in late 2016, and is currently in active use and development there. The licences are the LGPL variant, and the version numbers are calendar based, with the release history showing a major version in the middle of each year, which is a convention the project has evidently settled into.

The client, the exporter and the coordinator

The architecture is three roles, and understanding them is the key to the whole library.

The exporter is the machine that has the hardware attached. It knows which power switch controls which board, which serial line goes to which board, and how to talk to the instruments. It exposes that capability.

The client is the program that wants something done to a board. It asks for a board to be made available, waits for it, and then works with it.

The coordinator is the thing that matches the two. It is a registry: clients say which board they want and how they want it configured, and the coordinator finds an exporter that can provide it and hands over the connection details.

That three-part model is what makes the remote control layer work, and it is a model you have seen before in other guises. It is the same shape as a job queue, or as a device farm, or as the way a build farm allocates a machine.

The reason it matters is that it makes the hardware fungible. A test does not know which physical board it is talking to, and it does not know which rack it is in. It asks for a board with a certain name and a certain configuration, and the coordinator arranges it. That is what allows a hundred tests to run in parallel against a rack of ten boards, with the allocation of boards to tests being the coordinator's problem rather than the test's.

It also means the physical arrangement of the lab is a configuration rather than a fact about the test. If a board moves to a different exporter, or a power switch is swapped, the change is in the inventory, and the tests do not change. For a lab that is being re-cabled, which is the normal state of a lab, that is the difference between a test suite that survives the change and one that has to be edited.

And the interaction with the hardware is layered on top. The exporter provides a serial console or a network shell, the exporter can drive a bootloader, and the exporter can reset and power the board. A test that wants to check that a device boots correctly says: reserve a board, reset it, watch the console for a message, and fail if it does not appear.

The plugin, and the shape of a test

The library is not a test framework, and the readme says so, but the way it meets the test framework is the part worth understanding.

A test does not describe a board. It describes a strategy, and the strategy is a named piece of configuration that says what to do with the board before the test runs and what state to leave it in afterwards. There is a strategy for a board that is already running a Linux system and is reached over a serial console, one for a board reached over a network shell, one for a board that has to be flashed over a serial line first, one for a board with a memory card, and so on.

The test then says: give me a board with this strategy, and let the framework handle the acquisition, the reset, the flashing and the cleanup.

That is the abstraction doing its work. The test is written in terms of what the board should be, and the strategy says what has to happen physically for the board to be that way. A test that needs a freshly flashed board and a test that needs a running Linux system share the same acquisition code and differ only in the strategy.

The command line interface is the same idea outside the test framework. The readme lists a client and an exporter and a coordinator, and the same inventory appears in the example directory, where there is a directory per scenario: a bare-metal board, a relay board, a container, a library, a serial protocol converter, a network test, a network manager setup, a power switch, a measurement instrument over a standard protocol, virtual machines with networking, a shell over a serial line, a signal analyser, a shell over the network, a strategy, a general purpose input and output pin, a universal serial bus, a power switch over that bus, and a multiplexer over that bus.

That directory listing is the most informative thing in the repository, because it is a list of the hardware problems the library has actually been pointed at. Each of those is a driver somebody had to write, and each is a piece of hardware that would otherwise be a special case in a test suite.

The readme is also honest about the state of the project. Several planned features are not yet implemented, and the interfaces may change as more use cases appear. And it asks people to get in touch before starting larger changes, so that the design trade-offs can be discussed early and redundant work avoided. That is a request for design discussion rather than for a patch, and it is the behaviour you want from a library whose whole value is the shape of its interfaces.

Installing it, and the two states

The installation section offers a released version and a development state, and the difference is the usual one between installing a package and working on it.

For the released version, the instructions are a virtual environment on a recent Python, the environment activated, the package manager upgraded, and the package installed.

For the development state, the repository is cloned, and then the same virtual environment steps, and then the package is installed from the checkout.

And then the tests, which take a configuration argument naming the environment to run against.

That argument is the interesting part of the test command. A test suite for embedded hardware cannot run anywhere; it needs to know which lab it is talking to. So the environment configuration is a file that says what boards exist, where they are, and how they are wired, and it is passed to the test run. The same file is what the coordinator and the exporter read.

So the laboratory is described in a file, and the same description is used by the test run, by the coordinator and by the exporters. That is the whole design in one sentence: the inventory is data, and everything else is code that reads it.

And there is the documentation extra, which is installed as an optional group rather than as part of the main package, because it pulls in the documentation generator and the dependencies needed to build the manual pages.

Making the documentation a separate extra is the kind of small decision that matters in a project that is installed in a lot of environments, because the documentation toolchain is large and nobody running tests needs it.

The repository, and the packaging around it

The top level of the repository is a Python project's layout with a few entries that tell you about how it is distributed and how it is checked.

There is the package directory, the tests, and the documentation. There is an examples directory with the scenario folders, and a separate directory of helper scripts. There is a directory of container files, which is the packaging for running the exporters inside containers, and a directory for the manual pages, which means the command line interface has its own reference documentation.

And then there are the packaging directories, which are the interesting ones for a project that ends up on machines it does not control. There is a directory for the Debian packaging and a directory of contributed material. For a library whose users are hardware laboratories, a distribution package matters more than it would for a web library, because the machines running the exporters are often locked-down images that somebody else maintains, and a package that fits the distribution's conventions is the difference between the library being installable and not.

The quality tooling is thorough. There is a coverage configuration and a coverage badge, a linter configuration, a static type checker configuration, a file that marks certain paths as excluded from the version control attributes, and a continuous integration configuration for two different services, one of them the one that has since been folded into the other.

There is a file listing the files that are archived with the source distribution, which is the standard way to say that some of the files in the repository should not be shipped. There is a code owners file, which means the project has contributors with write access and a convention for who reviews what. And there is a changelog in the same markup dialect as the documentation, because the project's documentation is written in that dialect and keeps the release notes in it too.

The readme itself is in that dialect as well, which is worth noting because it is the oldest file in the repository and the one a newcomer reads first, and it is still in the format the project uses for everything else. The readme is also the shortest part of the documentation: it points at a tutorial playlist on a video channel for people who would rather watch, at the documentation site for people who would rather read, and at a page of design decisions for people who want to know why the interfaces look the way they do.

Editorial conclusion

labgrid fits a team that tests embedded hardware in CI, and a developer who wants to script a board's power, console and flashing steps instead of doing them by hand. It does not fit somebody with one board on their desk, where a serial terminal and a power switch are enough. Verify first that the drivers exist for the hardware in your lab, since the library's value is the driver layer rather than the framework, that you can reach the boards from the machine running the tests, since the whole model assumes a network of controllers, and that you are prepared to write a small driver if your board's quirks are not covered, because the extension point is the point.

Frequently asked questions

Official sources

1. Issues
2. labgrid-project/labgrid on GitHub
3. Project website
4. README
5. Releases