yzma: running llama.cpp from Go without CGo

What yzma solves for Go programs that need local inference

Most Go code that talks to a language model today talks to a server over HTTP. yzma takes the other route: the model runs inside your process. The README describes it as a way to "write Go applications that directly integrate llama.cpp for fully local inference using hardware acceleration." The audience is Go developers who want inference embedded in a CLI, a daemon, a desktop tool or a browser page, and who do not want to run a separate llama.cpp server and manage its lifecycle.

The binding is built on purego and ffi rather than CGo. That matters for build ergonomics: a CGo build needs a C toolchain, slows cross-compilation and complicates CI images. yzma instead loads the llama.cpp shared libraries at runtime, which is why the README tells you to set YZMA_LIB to a directory containing those libraries. The cost of that choice is that the libraries are a runtime dependency you have to place on the target machine.

The project also ships a command line tool of the same name. It downloads the prebuilt llama.cpp libraries for your platform, and it downloads GGUF model files. That is the piece that keeps the setup from turning into a manual hunt for the right binary.

How the binding works: dlopen, a token loop, and a sampler chain

The hello example in the README is the clearest statement of the architecture. The program calls llama.Load(libPath), where libPath comes from os.Getenv("YZMA_LIB"). That call is the runtime link step: purego resolves the symbols in the shared library. Then llama.Init() initialises the backend, llama.ModelLoadFromFile reads a GGUF file, and llama.InitFromModel creates a context from it.

From there the flow is the familiar llama.cpp loop. llama.ModelGetVocab returns the vocabulary, llama.Tokenize turns the prompt into tokens, and llama.BatchGetOne wraps them into a batch. A sampler chain is created with llama.SamplerChainInit and given a greedy sampler through llama.SamplerChainAdd. The loop calls llama.Decode, samples a token, checks llama.VocabIsEOG for end of generation, converts the token back to text with llama.TokenToPiece, and feeds it back as the next one-token batch.

Two details are worth noting. First, the example ignores the errors returned by ModelLoadFromFile and InitFromModel, which is fine for a demo and wrong for production; a missing GGUF file will not produce a helpful message. Second, the loop is bounded by responseLength, a fixed token budget, not by a stop condition the model chooses. If you want chat behaviour, the examples/chat directory is the place to look, and examples/vlm shows the multimodal path with a separate mmproj file passed by flag.

Installing yzma and running the hello example

The README gives a three-command install. The first installs the CLI, the second downloads the llama.cpp libraries into a directory you choose, and the third exports the environment variable the runtime reads.

go install github.com/hybridgroup/yzma@latest
yzma install --lib /path/to/lib
export YZMA_LIB=/path/to/lib

After the install, yzma verify checks the installation. The README states that each downloaded file is checked against the SHA-256 digest published with the release, so verify is a real integrity check rather than a version print.

yzma verify

Next, fetch a model. The README uses SmolLM2-135M-GGUF for the hello example and shows the model get subcommand with a -u URL pointing at the Hugging Face resolve endpoint. The Makefile in the repository uses the same subcommand with additional flags, including -y, -o and --show-progress=false, so those flags exist in the CLI.

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

With YZMA_LIB set and the model present, the example runs with go run. The README shows the expected shape of the output: a short reply to the prompt "Are you ready to go?" If the library path is wrong, the failure surfaces at llama.Load, before any model work begins.

go run ./examples/hello/

The repository layout reinforces this path. There is an examples/installer directory next to examples/hello, and the README notes that an application can download the libraries itself, with auto-detection for CUDA and ROCm, instead of relying on the CLI.

Where yzma is the wrong tool

The runtime library dependency is the first real constraint. Because yzma loads llama.cpp shared libraries rather than linking them in, a yzma program is not a self-contained binary. Shipping one means shipping the .so, .dylib or .dll files for each target platform, or writing download logic into the application itself. Teams that treat a single static Go binary as a hard requirement will find this disqualifying before they evaluate anything else.

The second constraint is version coupling. The README says yzma "works with the newest llama.cpp releases so you can use the latest features, performance improvements, and bugfixes." That is a benefit when you want new model support, and a liability when you want a frozen API. The releases in the repository move quickly: v1.26.0, v1.26.1 and v1.27.0 all landed in September 2026, and the last push was on 2026-09-14. A Go program pinned to an older yzma may not match the llama.cpp library version it loads, and the README does not document a compatibility matrix between yzma versions and llama.cpp versions.

The third case is scale. yzma gives you the raw loop: tokenize, decode, sample, detokenize. It does not give you a scheduler, request batching across concurrent callers, or an OpenAI-compatible HTTP surface. If several services need to share one GPU, a llama.cpp server process is a better fit than embedding the runtime in each service. yzma is for the case where the process that needs inference is the process that owns the model.

How yzma differs from calling llama.cpp through CGo

The obvious alternative is a CGo binding to llama.cpp. The difference is where the complexity lands. With CGo, the compiler links against llama.cpp at build time. You get a binary that carries the code, and the build machine needs the C toolchain, the headers and the right link flags. Cross-compiling to another architecture generally means a cross toolchain as well.

With yzma, the build is plain Go. The library is resolved at runtime by purego, and the ffi package handles the calling convention. The same Go source can run against a CPU-only library on one machine and a CUDA build on another, as long as YZMA_LIB points at the right directory. That is a genuine operational difference, not a stylistic one: it moves the platform decision from build time to deploy time.

The price is that symbol resolution failures become runtime failures. A missing library, a wrong directory or an ABI mismatch shows up when the program starts, not when it compiles. The yzma install and yzma verify commands exist to reduce that risk, and the SHA-256 checking described in the README means a corrupted download is caught rather than loaded.

A second alternative is to skip Go bindings entirely and run llama.cpp as a server, calling it over HTTP from Go. That keeps the Go program free of native dependencies and lets many callers share one model in memory. It also adds a process to supervise, a port to configure and a serialization hop per request. The choice between the two is really a question of whether the model belongs to one process or to a fleet.

Licence and upgrade cost

The repository reports the licence as NOASSERTION, which means GitHub could not map the LICENSE file to a known identifier. That is not the same as having no licence, and it is not the same as any particular open source licence. If you plan to redistribute yzma or the libraries it downloads, read the LICENSE file in the repository and the llama.cpp licence directly rather than assuming terms. This is a factual note about what the metadata shows, not legal advice.

The upgrade cost is tied to the version coupling described earlier. yzma tracks llama.cpp, so a yzma upgrade can change which llama.cpp features are available and, potentially, which Go functions exist. The repository has a version.go and a version_test.go at the top level, which suggests the project tracks its own version in code, but the README does not describe a migration policy or a deprecation window. Plan to re-run your own examples after each bump rather than assuming the API is frozen.

On the positive side, the upgrade path is short. Because the libraries are downloaded rather than compiled in, moving to a newer llama.cpp build is a matter of re-running the install into the same directory. There is no rebuild of a C dependency and no CGo cache to invalidate. The work is in testing that your Go code still behaves, not in the toolchain.

Editorial conclusion

Adopt yzma if you are shipping a Go program that must embed inference and you can accept shipping llama.cpp shared libraries alongside it, and if your target platforms are among those the install docs cover: Linux, macOS, Windows, Raspberry Pi, Jetson Orin, Arduino UNO Q or the browser through TinyGo and WebAssembly. Do not adopt it if you want a single static binary with no external .so or .dylib files, or if you need a stable C API contract, because yzma tracks llama.cpp releases and the Go surface can move with them. Before committing, run yzma install --lib into a directory you control, export YZMA_LIB to that path, run yzma verify, and then run go run ./examples/hello/ to confirm the library loads and a model decodes on your actual hardware.

Frequently asked questions

Official sources

1. hybridgroup/yzma on GitHub
2. Issues
3. Project website
4. README
5. Releases