# Gufo: a vertically optimized local inference engine for AMD Strix Halo

> Gufo is a C++ inference engine built for one piece of hardware, the AMD Strix Halo APU with its Radeon 8060S graphics and up to 128 GiB of unified memory. The README publishes per-model throughput numbers and the caveats needed to read them, and the project ships a Podman quickstart for running text, speech, image and video models on that single target.

**gufo-org/gufo** — The Open Source Strix Halo AI engine. Qwen 3.8  Flash Next Q4_K_XL: 1,801.01 pp, 73.17 tg single user, 194.37 tok/s 8 users; Qwen 3.8 27B Q4_K_XL: 656.33pp, 70.56tg tok/s single user with DFlash2

- Repository: https://github.com/gufo-org/gufo
- Website: https://gufo.blog
- Stars: 874 · Forks: 135
- Language: C++
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/gufo-org-gufo

## One APU, one engine, no portability goal

Gufo describes itself as a vertical local inference engine specifically built and optimized for AMD Strix Halo hardware: Ryzen AI MAX+ 395 systems with Radeon 8060S graphics, the gfx1151 target, and up to 128 GiB of unified memory. The word vertical is the design statement. Instead of a general-purpose inference backend that supports whatever accelerator happens to be present, this one commits to a single chip, its unified memory model, and the models that fit inside 128 GiB.

That commitment is the whole value proposition and the whole limitation. The unified memory pool on these machines is large enough that a quantized 27B model, or a set of smaller models, can be resident at once, which removes the PCIe transfer bottleneck that normally separates CPU and GPU tiers. Gufo's job is to exploit that, and the project's stated ambition is to be the one-stop shop for Strix Halo local AI users with text, audio, image and video support included rather than bolted on.

The README is also explicit about what the project will not do. Smaller memory configurations should still work and keep the speed benefits for models that can fit, but the optimization target is the 128 GiB part. And the maintainers commit to supporting only the best available models for their size that can run on this hardware, on the reasoning that fewer models means less code to maintain and more focused optimization and testing. That is a sensible trade for a single-hardware project, and it is also why the catalogue looks short next to a general-purpose runtime.

## What is actually in the model catalogue

The model documentation lives under docs/models, and the README table lists what each entry supports today.

For text generation there are three. Qwen3.8 27B is offered in Q4 and Q8 with image input, autoregressive decoding and DFlash2 speculative decoding. Qwen3.8 Flash-Next is listed with Q4, images, autoregressive decoding and MTP. DeepSeek V4 Flash is listed with autoregressive decoding and DSpark. Each row links to Hugging Face weights, usually Unsloth or z-lab GGUF builds, and to separate benchmark and quality documents.

The audio and media entries broaden the scope. Qwen3-ASR 1.7B is a speech recognition model with BF16 weights, and Qwen3-TTS 1.7B covers speech synthesis and voice cloning with three weight variants: CustomVoice, VoiceDesign and Base. Qwen-Image-2.1 covers image generation and editing with a complete pipeline, and MiniMax H3 handles text to video and audio. The last two are marked in progress in the benchmark column, which is an honest signal that they are not finished.

The pattern across the table is that speculative decoding is a first-class mode rather than an afterthought. Three different speculative schemes appear, DFlash2, MTP and DSpark, each paired with a second weight file that the engine loads alongside the main model.

## How to read the published throughput figures

The README publishes numbers, and it also publishes the conditions under which those numbers were reached. That second part matters more than the first.

The measurements distinguish prompt processing, abbreviated pp, from token generation, abbreviated tg, and the README states that text pp is autoregressive while tg uses the named speculative mode. It warns that peaks include repetitive output, and that aggregate tg sums the individual request decode rates across concurrent requests rather than measuring a single shared stream. For the 27B model, the single-user peak comes from a short-prompt workload identified as C1, and the audio figures exclude model loading.

On those terms the README reports Qwen3.8 27B at Q4 reaching 648.89 tok/s for prompt processing, up to 71.55 tok/s for token generation with a single user, and 128.42 aggregated tok/s across 8 concurrent requests with DFlash2. Qwen3.8 Flash-Next reaches 1,801.01 tok/s pp with autoregressive decoding, up to 116.93 tok/s tg for a single user, and 245.81 aggregated tok/s on 8 concurrent requests with MTP. DeepSeek V4 Flash is listed at 484.62 tok/s pp, up to 26.62 tok/s tg and 54.74 aggregated tok/s with DSpark.

The audio numbers are on a different scale, which is expected because they measure audio rather than text: Qwen3-ASR 1.7B reaches 15.27 times realtime, and Qwen3-TTS 1.7B up to 2.54 times realtime with 201 milliseconds to first audio on CustomVoice. Read these as best case ceilings on a specific machine, not as a service level.

## Fetching weights with pinned revisions

The quickstart begins with model weights, downloaded through the Hugging Face CLI with an explicit revision and a local directory. Pinning the revision is the practice the README demonstrates, and it matters here because a floating tag would make a deployment non-reproducible:

```bash
hf download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-UD-Q8_K_XL.gguf \
  --revision 4ca720788d1e01f1bff70c033e0d0028fd02e502 \
  --repo-type model \
  --local-dir models/Qwen3.8-27B-GGUF
hf download z-lab/Qwen3.8-27B-DFlash2-GGUF \
  Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  --revision 2d9571f8ce46e151f61c6499c99dee6079e1d610 \
  --repo-type model \
  --local-dir models/Qwen3.8-27B-DFlash2-GGUF
```

The two downloads are the main model and the speculative companion that DFlash2 needs, and each lands in its own directory under models. The same pattern repeats for Flash-Next with its MTP weights. Every model guide under docs/models lists the required files and the complete benchmark settings, which is the place to look when a run does not reproduce the published numbers.

## Serving Gufo in a container on the GPU device nodes

The second half of the quickstart runs the engine in a container pulled from the project's own registry, with the accelerator device nodes passed through:

```bash
podman pull ghcr.io/gufo-org/toolboxes/gufo-runtime:latest
podman run --rm \
  --userns=keep-id:uid=1000,gid=1000 \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add keep-groups \
  --ulimit memlock=-1 \
  -p 8080:8080 \
  -v ./models:/models:ro \
  ghcr.io/gufo-org/toolboxes/gufo-runtime:latest \
  gufo serve --host 0.0.0.0 --port 8080 llm \
  --model /models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q8_K_XL.gguf \
  --speculative dflash2 \
  --dflash-model /models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
```

Several details are load-bearing. The kernel fusion and compute device nodes, /dev/kfd and /dev/dri, are passed explicitly. The memlock limit is lifted with an unlimited value, which is what a process that pins large amounts of unified memory needs. The models directory is mounted read-only, so the container cannot rewrite the weights. And the serve subcommand takes the task type, here llm, alongside the model path and the speculative settings.

The README then documents the host prerequisites, which is where most first attempts fail. Rootless Podman needs crun for the keep-groups option, because the container's named video and render groups do not carry the host's supplementary groups. The host user must have read and write access to /dev/kfd and to the render nodes under /dev/dri, normally through the render and video groups, and has to log out and back in after changing group membership.

## The engineering choices the project commits to

The README has a philosophy section, and a few of its points describe how the code is likely to be shaped.

The strongest one is about kernels. The maintainers say they do not reuse kernels across different models, to limit the blast radius of a code change. That is a deliberate cost: duplicated, model-specific kernel work instead of a shared, general implementation, and it only makes sense if you intend to keep optimizing a small, fixed set of models rather than chase a long tail.

Quality is treated as a separate axis from speed. Each model has a quality report that records independent numerical checks, execution consistency and unresolved gaps, and the rule is to preserve quality when optimizing. Concurrency, cancellation and conversation caching are named as first-class workloads rather than afterthoughts, which is what you would expect from a serving engine rather than a benchmark script.

The build and repository structure match that style. The project uses CMake with CMakePresets.json and a cmake directory, keeps formatting and linting in .clang-format and .clang-tidy, has tests and tools directories, and pins development environments with a Nix flake and flake.lock. Releases are automated with release-please, tracked in .release-please-manifest.json and a version.txt file, and the recent tags, v0.17.0, v0.17.1 and v0.18.0, landed on three consecutive days in October 2026.

## Where Gufo is the wrong tool

The hardware constraint is the obvious one. If your machine is not a Strix Halo system with a Radeon 8060S on gfx1151, this engine is not for you, and there is no generic fallback path in the project. A team standardising on discrete GPUs or on NVIDIA hardware is looking at the wrong repository.

The catalogue is also incomplete in two places. Qwen-Image-2.1 and MiniMax H3 are marked in progress in the README's own benchmark column, so image and video generation are the areas where the project is least ready, even though the philosophy section claims text, audio, image and video coverage as the goal.

The platform story has gaps. The README points out two variants that have not been merged: a Windows port maintained in another repository, and Gufo RDMA for dual Strix Halo systems, which is how a two-machine setup would talk to each other. Anyone on Windows, or anyone planning to scale beyond a single APU, is depending on a fork that is not part of this codebase.

Licensing is a further constraint on distribution. Gufo itself is MIT, but the models it serves carry their own terms, and the repository takes this seriously enough to ship a NOTICE file, a THIRD_PARTY_NOTICES.md and a licenses directory. Running the weights internally is a different proposition from redistributing a service built on top of them, and the MIT grant on the engine does not settle that.

## Maintenance, releases and what a fork inherits

The repository is not archived and the last push was on 2026-10-22, three days after the v0.18.0 release and four days after v0.17.1. That cadence, three releases across three days in the middle of October 2026, suggests a project in heavy development rather than a stable one, and the changelog and GitHub Releases pages are the places the README points users to for user-facing changes.

The licensing setup is unusually complete for a project of this size. Alongside the MIT LICENSE there is a NOTICE file, a THIRD_PARTY_NOTICES.md and a licenses directory, which is what a project looks like when it has vendored or adapted code and wants the obligations recorded per component. A fork inherits that bookkeeping as a maintenance task rather than a one-off, since every new dependency or adapted kernel needs its entry kept current.

The documentation model is also worth noting for contributors. AGENTS.md, CLAUDE.md and an.agents directory sit at the root alongside a.devops directory and an.envrc, so the project is set up for both human and automated contributors, and the model documentation is versioned in the repository alongside the code rather than kept on an external site.

## Conclusion

Adopt Gufo if you own a Ryzen AI MAX+ 395 system and you want to run large quantized models locally with a serving stack that was tuned for that exact APU, and accept that the project is deliberately not portable. Do not adopt it for general GPU fleets or for platforms other than gfx1151, and do not treat the published peak figures as throughput you will see on real prompts. Verify first that your unit exposes the required device nodes to the container, that the model you want is listed with a completed benchmark rather than marked in progress, and that the third-party licence files cover the weights you plan to redistribute, since Gufo's own MIT licence does not extend to the models.

## FAQ

### What hardware does Gufo require?

It targets AMD Strix Halo systems, specifically Ryzen AI MAX+ 395 with Radeon 8060S graphics on the gfx1151 target, with up to 128 GiB of unified memory. Smaller memory configurations should still work for models that fit, but the optimization target is the 128 GiB part.

### How do I serve a model with Gufo?

The quickstart runs the published container image with podman, passing the /dev/kfd and /dev/dri device nodes, lifting the memlock limit and mounting the models directory read-only, then starting gufo serve with the task type, the model path and any speculative model settings.

### What licence applies to Gufo and the models it runs?

Gufo itself is released under the MIT licence, with NOTICE, THIRD_PARTY_NOTICES.md and a licenses directory recording third-party terms. The model weights carry their own licences, which the MIT grant on the engine does not extend to.

### Is there a Windows or multi-machine version of Gufo?

The README points to two unmerged variants: a Windows port in the pixmaate/gufo repository and Gufo RDMA for dual Strix Halo systems in the neuhaus/gufo repository. Neither is part of the main codebase.

## Sources

- [gufo-org/gufo on GitHub](https://github.com/gufo-org/gufo)
- [License: MIT](https://github.com/gufo-org/gufo/blob/main/LICENSE)
- [Project website](https://gufo.blog)
- [README](https://github.com/gufo-org/gufo/blob/main/README.md)
- [Releases](https://github.com/gufo-org/gufo/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/gufo-org-gufo