failpoint: injecting errors into Go code without paying for it in production

The marker is real Go code, not a comment

The design principle that sets this library apart from the usual approaches to failure injection is that a failpoint is defined in valid Golang code, not in comments or anything else. The consequence is that the compiler sees it, the tooling sees it, and a reader of the code sees an ordinary function call. A failpoint itself is described as a code snippet that is only executed when the corresponding failpoint is active, and the closure will never be executed if the failpoint is disabled. The closure can reach variables from an enclosing scope, which is why the surrounding code remains legal after transformation. There is a second principle about cost, and it is stated as three separate claims: the failpoint will not take effect on regular logic, it will not cause regular code performance regression, and the failpoint code will not appear in the final binary. The mechanism that achieves all three is that the marker function is inlined at compile time and emits nothing to the binary, and the injected block is compiled only when the failpoint is active. The marker functions are described as empty functions whose only jobs are to hint the rewriter to rewrite with an equality statement and to receive parameters as the rewrite rule.

Two routes to enable a failpoint, and they are not equivalent

There are two documented ways to use the library, and the difference between them determines how the code reaches your build. The first is failpoint-ctl, which transforms your source. You build the tool from source with a clone and make:

git clone https://github.com/pingcap/failpoint.git
cd failpoint
make
ls bin/failpoint-ctl

Then you inject markers into your program, transform the code with the enable command, build with go build, and enable the failpoints through the GO_FAILPOINTS environment variable. The second is failpoint-toolexec, which intercepts the build instead of rewriting files. You use a separate build cache to avoid mixing caches with and without the tool executor, and pass the toolexec path to go build. The toolexec route has the advantage of leaving your working tree alone, and the ctl route has the advantage of being explicit about what changed on disk. The reason the separate cache matters is stated plainly: without it you can end up with a build cache containing objects produced with and without the rewriting, and the result is a mystery. The environment variable syntax is the same in both cases, so a test written against one route works against the other, which is a good property for a library whose whole purpose is to be used from tests.

The GO_FAILPOINTS syntax is borrowed from FreeBSD sysctls

The activation mechanism is deliberately modelled on something older. Failpoints can be enabled by exporting environment variables with a pattern that the documentation says is quite similar to the FreeBSD failpoint sysctl variables, and the reference is given as a manual page. The pattern is:

[<percent>%][<count>*]<type>[(args...)][-><more terms>]

The type argument specifies which action to take, and the list of actions is broader than simply returning an error. Off takes no action and does not trigger the failpoint code. Return triggers the failpoint with the specified argument. Sleep sleeps for the specified number of milliseconds. Panic panics. Break executes gdb and breaks into the debugger. Print prints the failpoint path for the inject variable. Pause pauses until the failpoint is disabled. The percent and count prefixes mean you can fire a failpoint a percentage of the time or a given number of times, and the arrow syntax chains more terms, which is what makes probabilistic fault injection possible from a single environment variable. The sleep and pause actions are the ones that turn this from an error injector into a general fault injector, since a slow dependency and a hung dependency are two of the hardest failure modes to reproduce on demand.

Nine marker functions, and one of them cannot be switched on by env var

The marker function list is the API surface, and it is small enough to read in one go. There is Inject, which takes a name and a block, InjectContext, which adds a context so that parallel tests can be supported, and InjectCall, which takes variadic arguments instead of a closure. Then there are the control flow markers, Break, Goto, Continue, Fallthrough, Return and Label, each taking the label or results it needs. The two exceptions to the usual rules are worth knowing before you design a test around them. InjectCall cannot be enabled by environment variables, and the documentation repeats that note in the environment variable section as well as in the quick start, where it says that GO_FAILPOINTS does not work with the InjectCall type of marker. That means a test that depends on InjectCall has to activate it some other way, and it is a trap for anyone who assumes every marker behaves the same. The other exception is that the failpoint name is scoped to a package, so the environment variable value carries a package prefix, as in the example where a marker named testPanic in package main is enabled with the value main/testPanic.

What the rewrite actually produces

Understanding the generated code is useful because it explains both the zero cost claim and the debugging experience. When you call Inject with a name and a closure, the closure is expanded as the body of an if statement, and the converted code looks like this:

if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
    return "unit-test", val
}

The generated code calls an evaluation function with the package-qualified name, and if that evaluation succeeds the block runs. When the failpoint is not active the evaluation returns an error and the branch is skipped, which is why the whole thing compiles away in a normal build where the marker is inlined and emits nothing. The value that reaches the closure is whatever was passed to the enable call, for example a return of 5, and the documentation notes that it can be ignored, which is why there are two forms of the marker, one taking a failpoint.Value and one taking nothing at all. The design goals also require that line numbers stay the same with the injecting codes, so that debugging still points at the right line, and that the generated code be easy to read. Both of those are about the developer experience of a fault that fires in the middle of a stack trace.

Compile-time checking and parallelism with context

Two of the design goals are about safety rather than convenience. The first is that the failpoint routine is writable and readable and should be checked by a compiler, and the mechanism is stated as introducing a compiler check for failpoints which cannot compile in the regular mode if the failpoint code is invalid. In other words, if the body of your injected closure does not type check, the build fails, which is a stronger guarantee than a comment-based approach can offer. The second is that generated code by the failpoint definition is easy to read, and the last two goals are that the parser should be simple and maintainable over time, because parsers have a tendency to get really confusing really fast, so readability and debug-ability is a high priority, and that it should be testable, in the sense that the parser should produce provably valid parse trees, achieved by defining and continuously testing a set of invariants about the tree. The parallel testing support is a small API addition with a large effect. InjectContext takes a context, which means a failpoint can be enabled for one goroutine and not another, and that is what makes it possible to write a test that fails one request and not its neighbours.

Building the tools and running the library's own tests

Building the two binaries is a matter of running make in the repository, and the Makefile shows exactly what is compiled and how. The build target produces both bin/failpoint-ctl and bin/failpoint-toolexec from their respective main.go files, with linker flags that stamp the release version from a git describe, the build timestamp, the git hash, the branch and the Go version into the binary, so a tool you ship can be traced back to a commit. The test target is more interesting than it looks. The test command sets GO111MODULE=on, runs the go test with a parallel package count of 4, enables CGO, and also sets its own GO_FAILPOINTS environment variable with two entries, one returning 10 and one returning true. The library is therefore testing itself with failpoints active, which is the intended usage pattern: the same mechanism that consumers use to test their own code is used to test the rewriting tool. The dependencies are correspondingly small, with the pingcap errors package, a diff library used by the rewriting logic, testify for assertions, goleak for detecting goroutine leaks, and golang.org/x/mod for module handling, with the module declaring Go 1.18.

Editorial conclusion

failpoint is a good fit for a distributed Go system where you need to exercise the failure paths that are otherwise untestable, such as a peer going away mid-request or a write failing after it has been logged, and for a team that wants those failures reproducible from a test rather than simulated by hand. It is the wrong tool for a codebase that does not want a source rewriting step in its build, since both routes in the documentation require either transforming the source or using a tool executor, and it is the wrong tool for anyone who needs the fault to be triggered from inside a running production binary, because the design principle is that failpoint code will not appear in the final binary. Before adopting it, decide which of the two routes you want, since the failpoint-ctl route transforms files in place and the toolexec route intercepts the build, and note that the InjectCall marker cannot be enabled by environment variable, so any test relying on it needs another mechanism. The licence is Apache-2.0, the repository publishes no GitHub releases, and the last push was on 2026-08-11.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. pingcap/failpoint on GitHub
4. README