orchard: orchestrating virtual machines across a cluster of Apple Silicon machines

What the system is

Orchard is an orchestration system for Tart. Create a cluster of bare-metal Apple Silicon machines and manage dozens of virtual machines with ease.

That is the whole pitch, and it is worth being clear about what it is not. It is not a virtual machine manager, because Tart is that. It is not a scheduler in the general sense, because the readme does not describe one. It is the layer that sits between a person who wants to start a virtual machine and a room full of Macs, and it presents that room as one cluster.

The names in the readme give the shape of it: a controller and a worker. The controller is the thing you talk to, and the workers are the things that actually run the virtual machines, one per host machine. The local development mode starts a controller and a single worker on your own machine, which is the fastest way to see the system working and the standard way to develop against it.

Once you have a cluster, you interact with it in two ways. There is a command line tool, and there is a built-in REST API server, and the readme is explicit that both are available. The programming interface matters here, because the point of an orchestration system is that something else drives it: a test harness that needs a machine, a continuous integration job that needs a clean environment, a script that spins up a fleet and tears it down. The dependency list even includes a generated client library, which is the clearest sign that the API is meant to be called by other programs rather than only typed at.

The project is written in Go, the licence file in the repository is one GitHub does not classify as a standard licence, and the release history shows a rapid cadence with a version published in the most recent week of the year and two more in the preceding months.

One detail in the repository metadata is worth a paragraph, because it explains a small inconsistency you will notice. The module path in the dependency manifest points at an organisation called Cirrus Labs, and the links inside the readme point at that same organisation's copy of the repository, while the repository itself lives under OpenAI and the container image label names the OpenAI location. So the code has been moved, and the module path and the issue links have not been updated to match. That is the kind of thing that happens when a project changes hands, and it is worth knowing about before you file an issue in the wrong place.

The permission problem, and the two ways around it

The readme opens with a warning box rather than a description, and the warning is about the operating system rather than about the software.

macOS 15 introduced a new local network permission, and on each host machine that runs an Orchard Worker it requires accepting a graphical dialog. On a machine you sit in front of, that is a click. On twenty machines in a rack, it is a provisioning problem, and on a machine with no one logged in, it is a blocker.

The readme documents two workarounds, and the choice between them is instructive.

The first is to invoke the worker run command as root with an additional argument naming your regular, non-privileged user on the host machine. The worker then starts a small helper process in the background and drops privileges to the user you named.

The helper process is the interesting part, and it is privileged and needed to establish network connections on behalf of the worker without triggering the dialog. And the readme says plainly that this approach is more secure than simply running the worker as root, because only a small part of the worker runs privileged and the only functionality that part has is establishing new connections.

That is a well-constructed answer to a permissions problem. The naive fix is to run everything as root, which removes the dialog but gives the whole worker the privileges of the superuser. The design here splits the worker in two: a narrow helper whose entire job is to make the connections the system will not permit an unprivileged process to make, and the rest of the worker, which drops to an ordinary user as soon as the helper is up. The privilege is confined to the one operation that needs it.

The second workaround is to set the local network privacy preferences so that all of the private address space that could potentially be used for virtual machines is excluded. Those are the three familiar ranges from the private address registry, and they are excluded for both the wired and the wireless case.

sudo defaults write com.apple.network.local-network AllowedEthernetLocalNetworkAddresses -array "10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/16"
sudo defaults write com.apple.network.local-network AllowedWiFiLocalNetworkAddresses -array "10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/16"

And then a reboot. The word is in the readme as an ellipsis before the instruction, which is a reminder that this is a heavier operation than the first workaround: you are changing a system-wide privacy setting and restarting every host.

The two options trade different things. The first keeps the permission system intact and confines the exception to one small process. The second leaves the system setting alone but asks the operating system to ignore a whole class of addresses, which is a broader statement about the machine and needs a restart to take effect.

Getting a cluster in front of you in two commands

The fastest way to get started is the local development mode, and it is two commands.

brew install openai/tools/orchard
orchard dev

The install comes from a package manager tap, and the second command starts an Orchard Controller and a single Orchard Worker on the local machine.

That is the whole quick start, and it is worth noticing what it implies about the system's shape. A development mode that brings up a controller and a worker together, in one command, means the two roles are separable programs that happen to be able to run side by side on one machine. The production topology is the same two roles, spread across machines.

And a local worker on the developer's own machine is exactly what a virtualization system needs for the loop of writing and testing an operating system image. A developer changes something, wants a clean machine, and does not want to wait for a machine in the rack. Having the same controller talk to a local worker and to remote ones means the code path is the same either way.

The readme then points at the official documentation for more, and at the issue tracker for the remaining questions. The division is sensible: the quick start here is deliberately short, and the documentation site carries the content that would make this readme long.

One small detail in the readme is the image at the top, which points at a picture hosted in the old repository location. That is the same organisational move described earlier, and it is a useful signal about the state of the project: the documentation site and the code have moved, and the readme has some pointers that have not caught up.

What the dependencies say about the design

The dependency manifest is the best description of the system that the readme itself gives, because every entry is a decision somebody made.

The web layer is a popular Go web framework with a logger bolted on through a middleware package, and there is an explicit replacement rule in the manifest pinning the framework to an earlier version to work around a reported issue in the newer one. That is a small thing and a good sign: the workaround is documented in the manifest rather than hidden in a commit message, and the comment points at the upstream issue so that it can be removed when the fix lands.

The command line is built with the usual command library, and the terminal interface uses a prompt library and a pretty terminal library together with a table writer, which is the combination you would expect for an interactive tool with a table of virtual machines in it.

The metrics and the tracing are OpenTelemetry, with an instrumented middleware for the web layer, a metrics SDK, and two exporters: one over gRPC and one over HTTP. The presence of both exporters is a deployment decision, since a cluster running on a private network may have a collector reachable over one protocol and not the other, and the environment variable in the container image selects the release mode of the web framework, which is the other half of the same kind of decision.

Storage is an embedded key value store, and there is a library for file locking, which suggests the controller owns state that has to be consistent across processes.

Then there are the pieces that say what this system actually talks to. A guest agent from the Tart project is a dependency, which is how the controller reaches inside a running virtual machine rather than only orchestrating its lifecycle. A websocket library and a gRPC connection helper cover the two ways a controller talks to a worker, with websockets for the long-lived connection and gRPC for the calls. A library for generating passwords appears next to them, which is how the credentials for a virtual machine are created.

Two details are worth singling out because they are the kind of thing you only notice by reading the list. There are two versions of the retry library, one for each major release, which usually means a migration is in progress rather than a mistake. And there are two functional programming helper libraries, one for generic collection operations and one for monadic composition, which tells you the codebase uses a functional style in places where it is clearer than the alternative.

The manifest also names a Go version and pins the toolchain, and the language directive is a recent one, so this is a project built with current tooling rather than an old one that has been left alone.

The container image

The container file is short, and every line of it is a decision.

The base is the distroless base image from a public registry, with a build argument for the target platform. Distroless means the image contains the binary and essentially nothing else: no shell, no package manager, no distribution files. For a service that exposes a port and talks to workers, that is a good trade, and it is the same reasoning a security team would apply to any image that runs with network access.

The label records the source repository, which is one of the two places in the repository where the new organisational location appears.

Then two environment variables. One sets the web framework to its release mode. The other sets the home directory for the application to a path inside the data volume, which is where the embedded database and the state live.

The port is exposed, the binary is copied into the image from the build stage, and the entry point is that binary.

ENV GIN_MODE=release
ENV ORCHARD_HOME=/data
EXPOSE 6120

And the default command is the controller in its run mode, which is the right default for an image that is mostly the controller: a worker needs to know which controller to join, whereas a controller needs to know where to listen.

The build argument for the target platform is the detail that tells you the image is built for more than one architecture, and for a system whose whole point is Apple Silicon hardware that is a meaningful choice rather than a formality.

The manifest also names a release tool configuration, which is how the binaries for each platform are produced and signed, and a lint configuration for the Go code, and a development document, and a separate directory for the protocol definitions with the configuration files for the code generator that produces them.

How the code is laid out

The repository layout is the standard split for a Go service with a public protocol, and each directory has an obvious job.

There is a directory for the command line programs, one for the internal implementation, and one for the packages that are meant to be imported. There is a directory for the interface definition, one for the remote procedure call definitions, and one for the tests, plus a separate configuration file for the interface compiler and a workspace file that tells the compiler to generate across several modules.

The split between the internal and the public packages is the one that matters for anyone building on the project. The public packages are the API surface, and the internal ones are the implementation that can change without a version bump. For an orchestration system whose clients are generated from the interface definition, that separation is what lets the protocol evolve while the library stays stable.

The tests directory as a separate top-level entry rather than being spread through the source is a convention some Go projects adopt and others do not, and here it is consistent with the rest of the layout: the repository is organised by role rather than by feature.

And the release cadence is the last thing worth noting, because it says what kind of project this is. Three releases in the three months up to the most recent push, each one a version that ends in zero, and the readme warns about a macOS release that changes the behaviour of the workers. That combination suggests a system that has to move quickly because the platform under it moves, and that the version number is the only thing telling a cluster operator which combination of controller and workers is known to work together.

Editorial conclusion

Orchard fits a team that has a rack of Apple Silicon machines and wants to treat them as one pool of virtual machines, and a developer who wants to try the idea on a single laptop first. It does not fit somebody who needs the virtual machines to survive the host in a way this design does not provide, or a team without the hardware. Verify first which of the two workarounds for the local network permission you will use, because one of them needs a privileged helper and the other needs a reboot on every host, that the host machines can actually reach each other over the private address ranges you exclude, and that the client library version you depend on matches the server version, since the protocol between them is versioned.

Frequently asked questions

Official sources

1. Issues
2. openai/orchard on GitHub
3. README
4. Releases