go-workflows: durable workflows embedded in a Go program

What it is, and the family it belongs to

go-workflows provides embedded durable workflows for Go, in the style of a small number of systems that share a common ancestor. The readme says the project borrows heavily from one of them, and that since it is a fork it also borrows from another, and it mentions a third.

The lineage is worth spelling out, because it explains the shape of the whole thing. The oldest of the three was built at a ride-sharing company as an internal orchestration service for long-running business processes. That code was donated, became a separate project, and was then forked into the engine that is now the dominant one in the market. A later project took a different approach, building on a task framework rather than on the earlier code, and go-workflows is the Go branch of that second line.

So the project is a fork of a fork, and the readme says so plainly, with links. That candour is worth noticing, because a project that hides its ancestry makes it harder to reason about what will and will not work. Here you can read the upstream sources and see which behaviours are inherited and which are Go-specific.

What the Go branch adds is the thing the upstream lacks: a language with goroutines, with channels, with a scheduler, and with a type system that makes a certain class of bug impossible. A workflow engine is a state machine with a scheduler inside it, and a language with a scheduler built in is an interesting host for one.

The documentation is published as a site, and the readme points at two blog posts from 2022, one on the first part and one on the second, which is the conventional naming for a series of articles on building the thing. Those two posts are probably the best available explanation of why the design is the way it is.

The release history is a curiosity worth mentioning: the tags are all nightly builds, with a version, a build number and a timestamp in the name, and there is one every month or so. So the project is releasing continuously and not cutting stable versions, which is a choice that suits a library whose main consumer is its author and a few other people.

Workflows, activities, and the one rule

The contract is stated in two short paragraphs, and it is the clearest thing in the readme.

Workflows are written in Go code. The only exception is that they must not use any of Go's non-deterministic features, and the examples given are a select statement and iteration over a map. Inputs and outputs for workflows and activities have to be serialisable.

And then the second half.

Activities can have side effects and do not have to be deterministic. They will be executed only once, and the result is persisted.

That is the entire design, and it is worth understanding why it works. A durable workflow is a function that has to be able to run again after a crash, from the beginning, and produce the same result. If the function reads a clock, picks a random number, or iterates a map, running it again produces a different history, and the engine's record of what happened no longer matches what the code would do. So the engine replays the workflow's history to rebuild its state, and for the replay to be correct the workflow function has to be a pure function of its inputs and its history.

Activities are where that constraint is lifted, because an activity's side effect must happen exactly once, and the engine guarantees it by recording the activity's completion and its result in the event log before moving on. So the rule for a developer is simple and practical: put anything impure inside an activity.

The example shows it. The workflow calls an activity with two integers and gets the sum back, and it calls a second activity with no arguments and gets a constant. The workflow itself does nothing but call them and check the results, and the calling convention is explicit about the result type, which the sample declares as a type parameter, and about the options, which the sample passes as a default.

And the activity side is the mirror image: a function that takes ordinary Go context and arguments and returns a value and an error, with no restriction at all.

That contrast is the thing to take away. If you have ever written a workflow in a system where the definition had to be expressed in a data format rather than in code, the difference here is that your workflow is a Go function you can read, with a compiler checking it, and the only thing you have to remember is which side of the line each function is on.

The worker and the backend, and who does what

Two more pieces complete the model, and the readme's division of labour between them is the same one the upstream engines use.

The worker is responsible for executing workflows and activities, and both have to be registered with it. The example creates a worker with the backend and an optional second argument, registers the workflow function and the two activity functions, and then starts it. The start call returns an error, and the example treats that as fatal, which is the right thing to do: a worker that fails to start is a worker that will not execute anything, and the failure has to be visible.

The backend is responsible for persisting the workflow events, and creating one is a single call.

b := sqlite.NewSqliteBackend("simple.sqlite")

The readme lists what exists today: an in-memory implementation for testing, one using SQLite, one using MySQL or PostgreSQL, and one using Redis.

The word events is the important one, and it explains the whole architecture. The state of a running workflow is not stored as a snapshot of the workflow's variables. It is stored as the list of things that have happened: an activity was scheduled, an activity completed with this result, a timer was set, a signal arrived. The engine reconstructs the workflow's state by replaying that log.

That design has a consequence for the choice of backend, and it is the reason the list of implementations matters. The log is append-only and read in order, so a backend has to support ordered appends and ordered reads. An in-memory implementation satisfies that trivially, which is why it exists and why it is for testing. A relational store gives you durability with transactions. A key-value store gives you durability with a simpler write path. A cache is somewhere in between and is probably not what you want for a production workflow store, but it is in the list because it is useful for development and for a small deployment.

So the durability guarantee you get is exactly the durability guarantee of the store behind it, and the choice of backend is the choice of how much you trust the machine the workflow runs on.

The client, and running the two halves together

The last piece is the client, and the readme's final example is the one that shows how the whole thing fits together.

A workflow instance is created with an identifier and a set of options, and the identifier is a random string generated by the identifier library. The workflow function and its input are passed alongside. And the example then does something worth noticing: it starts the worker in a goroutine in the same process, and then creates the client from the same backend.

So the readme's demonstration is of the embedded arrangement, where the client, the worker and the backend are all in one program. The readme says explicitly that you can do this or run them separately, and there is a sample directory for the split arrangement, so both are supported.

That distinction matters more in Go than it does in the systems this project descends from. The reason to run the worker separately is usually operational: the worker is a long-running process that holds the workflow state and executes the activities, and you want to scale it independently, or run it somewhere with different permissions, or restart it without restarting the API. The reason to embed it is that you have one process and no reason to have two.

The example also shows the shutdown handling, and it is a bit rougher than the rest: a channel is made with a buffer of one, a signal handler is registered for the interrupt signal, and the program then waits on the channel. The worker is never asked to stop, and the goroutine is never joined. For an example that is fine, and for a program that is going to run in production it is the sort of thing that leaves an activity half-finished on shutdown, which is exactly the case the engine's exactly-once guarantee is supposed to cover but which still deserves an explicit stop.

The instance identifier is worth one more thought, because it is how a workflow is found again. It is generated rather than chosen, so the caller does not have to invent an identifier, and the same identifier is what a client would use to signal a running workflow or to ask about its state, both of which have samples in the repository.

The samples, which are the real documentation

The samples directory has about twenty-five entries, and the names are a complete list of the features you would want from a workflow engine, which makes it the most useful part of the repository to read.

There is the simple one, and a variant that splits the worker across two processes. There are samples for cancellation, for retries, for timers, for signals in both directions, for a subworkflow and for signalling a subworkflow. There is a sample for continuing as new, which is how a workflow with unbounded history is restarted with a fresh one, and a sample for queues, which is how work is routed to a particular worker. There is one for an orchestrator pattern, and one for a converter, and one for errors, and one for complex parameters, and one for activity registration and one for workflow registration, which are about how the engine finds the functions you want it to run. There is one for context propagation, one for concurrency, one for concurrency with an error group, one for tracing, and one for scale.

And there is a web sample, which tells you the project has a browser-facing story, and a directory for a runner, which is the harness the samples are executed by.

What the list reveals is the shape of the project. It is not a thin wrapper over somebody else's engine. It has its own idea of how a workflow is defined, what the activity interface looks like, how the worker registers things, and how the engine talks to the storage. The upstream code informed the design, but the Go implementation has its own vocabulary, and the samples are where that vocabulary is written down.

The two samples that are worth pointing a newcomer at are the simple one and the one for continuing as new. The first shows the three-part contract in its shortest form. The second shows the concept that a Go developer is least likely to have met, because a language with goroutines does not obviously want a workflow that replays itself. The idea of a workflow whose history grows without bound, and the need to retire that history by starting a new run with the old one's final state, is a consequence of the replay design, and it is the thing that makes long-running workflows practical.

The repository and the dependencies

The top level of the repository shows a project with a clear internal structure, and a couple of the entries are worth explaining.

There are the obvious packages: the workflow definition, the activity definition, the worker, the client, the backend, the core types, and a registry. The registry is worth a note, because it is the piece that answers the question of how the engine finds the function you registered, and it is separated from the core for the same reason the compiler and the runtime are separated in a language.

Then there are the ones that are about building the thing rather than using it: an analyser, a tester, a second tester for activities, a directory of diagnostics, a benchmark directory, an internal directory, and a web directory. Having a benchmark directory and a diagnostics directory in a library repository is a sign of a project whose author cared about how it performs and what it does when it goes wrong.

The top level also has a make file, a compose file, a development document, a linter configuration, and a file instructing coding assistants. The compose file is the local environment, and for a project whose backends include a relational database server, a key-value store and a cache, having a compose file that starts them is how a contributor gets a working environment in one command.

The dependency list is where the design shows. There is a database migration library, which tells you the backends are expected to have a schema that evolves. There is a pure-Go relational driver and another one for the other server, so that the engine does not need a compiler toolchain to be used. There is a cache library and a key-value store client, matching the backend list. There is the identifier library for instance names. There is the telemetry standard with three exporters, one over the binary protocol and one over HTTP and one to standard output, which is a sensible set for a library that will be used both in a service and in a development run. There is an error handling library, which suggests the engine's errors are first class values rather than formatted strings, which is what you want when an error crosses a process boundary and has to be serialised. And there is a goroutine leak detector in the test dependencies, which for a project that schedules goroutines is not a nicety but a requirement.

The language version is recent, and the module path carries the author's name, so this is a personal project with a public following rather than a company project with a maintainer rota.

Editorial conclusion

go-workflows fits a Go service that has a multi-step process which must survive a restart, and a team that wants the guarantees of a durable engine without running a separate server. It does not fit somebody who needs the full operational surface of a hosted engine. Verify first which backend you will use, because the in-memory one is for tests and the durability guarantee is only as good as the store behind it, that your workflow functions avoid the non-deterministic constructs the readme lists, since a workflow that iterates a map will replay differently, and that the version you pin is a stable one rather than one of the nightly builds the release history is made of.

Frequently asked questions

Official sources

1. cschleiden/go-workflows on GitHub
2. License: MIT
3. Project website
4. README
5. Releases