HUATUO: kernel-level observability for Linux fleets, and the cases it does not cover
eBPF-based Linux kernel observability 🚀🚀
At a glance
- What is it?
- HUATUO is a Go and eBPF project from Didi, incubated under the CCF, that captures kernel runtime context and profiles Linux subsystems. It is aimed at operators who need kernel evidence during incidents rather than application-level traces.
- Who is it for?
- Adopt HUATUO if you run Linux 4.18 or later at fleet scale and your hard incidents live below the application layer: page faults, scheduling delay, block I/O, load spikes. Skip it if your team cannot grant a privileged container host PID, host cgroup and host network access, or if your debugging stops at HTTP latency, where an application tracing agent is a better fit.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Go, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap HUATUO is built to fill
Application tracing starts at the process boundary. When a request stalls because the scheduler deprioritised a runnable thread, because a page fault pulled in a cold mapping, or because block I/O queued behind a noisy neighbour, the application span shows latency and nothing about the cause. HUATUO targets exactly that layer. The README describes it as delivering kernel-level observability for cloud-native computing, AI computing and bare-metal infrastructure, using kprobe, tracepoint, ftrace and eBPF to instrument kernel subsystems rather than application code. The intended reader is an operator or SRE on a Linux fleet who has to explain a system-level stall, not a developer adding spans to a service. The project's own framing is worth noting: it says it is deployed at scale in Didi's production environment, which tells you the design assumptions come from a large internal fleet rather than a single cluster.
How the capture mechanism actually works
The architecture has two distinct halves, and the README separates them clearly. The first is continuous: BPF programs attached to kernel subsystems (MM, CPU scheduling, networking, block I/O) emit metrics, and the project claims performance overhead below 1 percent. The second is event-driven. HUATUO instruments kernel slow paths and triggers capture when something crosses a threshold: the README names page faults, scheduling delays, CPU idle drops, CPU sys spikes, I/O surges and Loadavg spikes. The result is a snapshot retained for later analysis rather than a live stream you must be watching at the moment of the incident. That distinction matters operationally. A latency spike at 03:00 is useful only if the agent kept the context, and the AutoTracing feature exists precisely because nobody is tailing a dashboard at 03:00. On the data side, the agent exposes a metrics endpoint and the README lists integration with Prometheus, Grafana, Pyroscope and Elasticsearch, plus automatic association of Kubernetes container labels and annotations. The Go client in client/ (node files prefixed node_) is a control plane, not a data plane: the README's example calls UpdateConfig with a config key Runtime.CPULimitCores set to 1.5, which is how you tune the agent's own resource use.
Running it: two paths and one warning
The README gives a single-container path and a full-stack path. The container command is docker run --privileged --pid=host --cgroupns=host --network=host -v /sys:/sys -v /run:/run huatuo/huatuo-bamai:latest, and metrics are then read with curl -s localhost:19704/metrics. The privileged flags and host namespace sharing are not incidental. Kernel tracing needs visibility into host processes and cgroups, so this is not an agent you can run with a restricted security context. The full stack is docker compose --project-directory ./build/docker up, which brings up Elasticsearch, Prometheus, Grafana and huatuo together, with the dashboard at http://localhost:3000. Note the README's own warning: do not deploy images with the latest tag to production, because that tag is a development and testing image. Use a formal release image or binary. That is a project-stated constraint, and it means a production rollout should pin a version such as v2.3.0 rather than track latest.
Kernel and distribution coverage is the real gate
The README includes a tested matrix, and it is the first thing to check against your fleet. Support starts at kernel 4.18. The listed combinations run from 4.18.x on CentOS 8.x, through 5.4.x on OpenCloudOS V8 and Ubuntu 20.04, 5.10.x on OpenEuler 22.03 and Anolis OS 8.10, 5.15.x on Ubuntu 22.04, 6.6.x on OpenEuler 24.03, Anolis OS 23.3 and OpenCloudOS V9, 6.8.x on Ubuntu 24.04, and 6.14.x on Fedora 42. The matrix also lists 2.3.0 against kernel 7.0.x on Ubuntu 26.04. Read that table as a statement about what the maintainers primarily test, not as a guarantee of what will load. eBPF attachment points vary across kernel versions and vendor backports, and the README does not enumerate per-distro caveats. If your hosts run a vendor kernel that is not on the list, the honest position is that compatibility is unverified from the supplied material. This is the single largest adoption risk, and it is discoverable in an afternoon on one representative host.
Where HUATUO is the wrong tool
Two limits stand out. First, the deployment model. Host PID, host cgroup, host network and privileged mode mean the agent sits close to the kernel boundary, and on managed Kubernetes services, hardened node images or environments with strict pod security admission, that combination may be blocked outright. If you cannot get those permissions, the project is not a fit regardless of its feature list. Second, scope. HUATUO observes the operating system and the hardware beneath it: the README lists CPUs, memory, PCIe interconnects, network adapters, storage, and AI accelerators such as GPUs and NPUs. It does not replace distributed tracing inside your services, and it will not tell you which function in your Go handler is slow. A team whose incidents are mostly application logic will get little from kernel snapshots. The overhead claim of below 1 percent is also worth treating as a claim rather than a measurement: the README states it, but the conditions under which it holds (which subsystems are enabled, what sampling rates, which kernel) are not specified in the material available here.
How it differs from a general metrics agent
The obvious comparison is node_exporter, the standard Prometheus node agent. Both expose a metrics endpoint scraped by Prometheus, and both can be visualised in Grafana, so on a dashboard they can look interchangeable. The difference is where the numbers come from. node_exporter reads counters the kernel already exposes through /proc and /sys: it reports what the kernel has chosen to publish. HUATUO attaches BPF programs to kernel functions and tracepoints, so it can produce data the kernel does not export as a counter and can retain context around a specific slow path. That is why the AutoTracing snapshot exists in HUATUO and has no equivalent in node_exporter. The trade-off runs the other way too. node_exporter is a single unprivileged binary with no kernel version matrix to consult and no host namespace requirements. If your questions can be answered from existing kernel counters, node_exporter is the lower-risk choice, and HUATUO adds deployment complexity you will pay for on every node.
Maintenance, releases and licence
The release cadence visible in the repository is roughly two to three releases per year: v2.1.0 in November 2025, v2.2.0 in March 2026, v2.3.0 in August 2026, with the last push to main in September 2026. That is a moderate pace, and it implies upgrade work when the kernel matrix moves: the 2.3.0 row adds kernel 7.0.x on Ubuntu 26.04, so staying current means periodically re-validating against new distributions. The upgrade cost is not just the binary. Because the agent depends on kernel attachment points, a host kernel upgrade is a potential breaking change, and the tested matrix is the reference for whether a given combination has been exercised. On licensing, the project is Apache-2.0, which permits commercial use and modification; the README also notes the project is incubated under the CCF and listed in the CNCF Landscape. That is a description of governance and licence, not legal advice, and if you redistribute the agent inside a product you should read the LICENSE file and your own counsel's guidance rather than this summary.
Editorial conclusion
Adopt HUATUO if you run Linux 4.18 or later at fleet scale and your hard incidents live below the application layer: page faults, scheduling delay, block I/O, load spikes. Skip it if your team cannot grant a privileged container host PID, host cgroup and host network access, or if your debugging stops at HTTP latency, where an application tracing agent is a better fit. Before rolling out, verify two things against your own hosts: that your kernel appears in the tested matrix, and that a v2.3.0 release image (not the latest tag) runs cleanly under your container runtime with /sys and /run mounted.
Community notes