goloader: running Go code that was compiled after your program started

It works like a linker, not like a script interpreter

Goloader can load and run Go code at runtime, and the project is careful about what that does and does not mean. It works like a linker: it relocates the address of symbols in an object file, generates runnable code, and then reuses the runtime function and the type pointer of the loader. To make that work with Go's runtime and garbage collector, goloader provides some information to the runtime and the GC, which is what allows it to work correctly with them. The project says plainly that goloader is not a scripting engine. It reads the output of the Go compiler and makes it runnable, all features of Go are supported, and the code runs just as fast and lightly as native Go code. That is the key difference from a scripting layer, where every operation crosses a boundary and the type system stops applying.

Reusing the runtime, and unloading the result

The comparison the project makes with plugin systems is about the runtime, and it is the reason to use this rather than a conventional plugin architecture. Goloader reuses the Go runtime, which makes it much smaller, and the code loaded by goloader is unloadable. Unloadable is the interesting half of that claim, because a plugin system that can never let go of a loaded module leaks its types, its goroutines and its memory for the life of the process. There is also a profiling consequence that is easy to miss. Goloader supports the pprof tool, so you can see code loaded by goloader in pprof, which means a hot function inside dynamically loaded code shows up in a profile alongside the statically compiled code rather than being invisible.

Building it, and the step the compiler will not let you skip

You need go at least 1.8, and there is one preparation step that has to happen before you build and test.

  cp -r $GOROOT/src/cmd/internal $GOROOT/src/cmd/objfile

The reason is given plainly. Goloader relies on the internal package, which is forbidden by the Go compiler, so the internal package is copied into the object file package to make it reachable. This is the point where the project stops being an ordinary Go library, because it depends on parts of the toolchain that are not importable and on the layout of the Go source tree. The module declares go 1.11 and the package has no dependencies of its own, but the build process is not a plain go build. The example loader is built with a command that names the example package path.

  go build github.com/pkujhd/goloader/examples/loader

The repository layout reflects the internals: an obj/ and objabi/ pair for reading Go object files, a link/ directory for the relocation work, libdl/ for the dynamic loading, mmap/ for memory mapping, a stackobject/ directory, and constants/.

Compiling an object file and running it

The example flow compiles a Go source file to an object file with the go tool and then runs it through the loader. With a go module, or on go 1.20 and newer, the import configuration has to be built first so the compiler can find the packages the file depends on.

  export GO111MODULE=on
  go list -export -deps -f '{{if .Export}}packagefile {{.ImportPath}}={{.Export}}{{end}}' $GOPATH/src/github.com/pkujhd/goloader/examples/schedule/schedule.go > importcfg
  go tool compile -importcfg importcfg $GOPATH/src/github.com/pkujhd/goloader/examples/schedule/schedule.go
  ./loader -o schedule.o -run main.main -times 10

The loader is told which object file to load, which symbol to run and how many times to run it. The older go path route skips the import configuration step, and the project also shows a variant that compiles a package archive first and then loads it, which is how you get a reusable archive rather than a one-off object file. The example directory covers the cases worth knowing about, with base, basecontext, const, gctest, http, inline, inter, loader and schedule examples, and the gctest example in particular is the one that exercises the interaction with the garbage collector.

goloaderbuilder, and compiling with dependencies

There is a builder that handles the package dependency management, and the project credits a contributor with proposing the practical ideas to support cgo and to automate that dependency management. Compiling only a package archive is two commands.

./builder -f $GOPATH/src/github.com/pkujhd/goloader/examples/inter/inter.go -p inter -b
./loader -o ./target/github.com/pkujhd/goloader/examples/inter/inter.a -run inter.main -times 10

Compiling with the dependency packages included is the same idea with the runner wired in.

./builder -e ../runner/runner -f $GOPATH/src/github.com/pkujhd/goloader/examples/inter/inter.go -p inter
../runner/runner -f target/inter.goloader -r inter.main

The two forms differ in what the builder is given and what runs the result. The first produces a package archive that the loader then loads by symbol name, and the second produces a .goloader artifact that a runner takes and executes against a named entry point. The goloader extension on the artifact is how the project distinguishes a prepared file from an ordinary Go archive.

The warnings, and the platform matrix

Two warnings in the project overview are worth treating as hard requirements. Do not use the -s and -w compile arguments, because that strips the symbol table the loader needs. And do not use go run or go test, because the -s and -w compile arguments are the default in those commands. The second one is a trap, because go run is the obvious way to try something out and it will fail in a way that looks like a loader bug. The tested platform matrix is given per architecture. Go 1.8 to 1.27 is tested on x64 and x86 across darwin, linux and windows. Go 1.10 to 1.27 is tested on arm with linux and android. Go 1.8 to 1.27 is tested on arm64 with linux and android, and Go 1.16 to 1.27 is tested on arm64 with darwin. The release tags track the same range, with the newest at 0.0.25 covering Go 1.8.x to 1.27.x, and the project is a fork of dearplain's goloader taken over because the original author was no longer maintaining it.

Editorial conclusion

goloader fits a Go developer who needs to compile and run code after the host process has started, for a plugin system or a scheduled job, and who wants the loaded code to be unloadable rather than a permanent part of the binary. It does not fit a Go project that compiles with the -s and -w flags, because that strips the symbol table the loader depends on, and the project warns against it explicitly, along with go run and go test which enable those flags by default. It also does not fit someone on a Go version outside the tested range. Before you build it, copy the internal package into the object file package as the project requires, since the loader depends on internals the Go compiler forbids importing, and check the platform matrix, because support covers x86, arm and arm64 across Linux, macOS, Windows and Android with different minimum versions per platform.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. pkujhd/goloader on GitHub
4. README
5. Releases