# Kindling: an eBPF monitoring agent that profiles a trace from kernel to code

> Kindling is an eBPF based cloud native monitoring agent that pairs network flow monitoring with trace profiling, correlating on-CPU and off-CPU events, the logs each thread printed and the network and file syscalls that ran during a trace. It needs a kernel newer than 4.14, and trace profiling needs 4.17 or newer, with a Sysdig derived kernel module as a fallback for older systems.

**harmonycloud/kindling** — eBPF-based Cloud Native Monitoring Tool

- Repository: https://github.com/harmonycloud/kindling
- Website: http://kindling.harmonycloud.cn
- Stars: 1,063 · Forks: 167
- Language: Go
- License: Apache-2.0
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/harmonycloud-kindling

## The idea: triage from the network, then pinpoint with a trace

Kindling is an eBPF based cloud native monitoring tool which aims to help users understand the app behavior from kernel to code stack. The stated hope is that with trace profiling the user can understand the application's behavior easily and find the root cause in seconds.

The problem it is aimed at is described in the section Why do we build Kindling, and it is a real operational gap. When you talk about observability you already have plenty of tools, like Skywalking for tracing, ELK for logging, and Prometheus for metrics. So why another one?

The answer given is about the sequence of investigation. For applications on Kubernetes, we don't know the network flows between the services until we can instrument the apps, and we can't tell which part to blame when a product issue arises. Is Kubernetes configured correctly? Are there bugs in the virtual network like Calico or Flannel? Does the application code cause it?

The company behind the project is based in Hangzhou and used to provide a Kubernetes distribution, so they were the people fielding those questions from their own customers without a good answer. The proposal is that network view triage first, identifying the issue roughly, and then digging into the code. The framing in the documentation is that if the network view says it is a network problem, the code works fine and you should dig into the Calico configuration, and if the infrastructure is fine then the app code should be blamed and you go to the logs or the APM dashboard.

The APM point is worth dwelling on because it is the honest limit of the tool. APM is good for applications in Java which can be instrumented automatically, while for Go programs the code has to be rewritten for instrumenting. And even with an APM solution you still can't tell whether an issue is caused by network problems, and many issues can't be pinpointed to a root cause easily.

So Kindling is not trying to replace APM or tracing. It is trying to supply the layer above both, the one that answers whether the network or the code is at fault, and then to give you a single trace that shows the code executing.

## What trace profiling actually records

The premise of the feature is stated directly. With traces, metrics, and logs, many issues still can't be understood easily. Trace profiling tries to integrate the OnCPU and OffCPU events within the traces and collect the logs output during the trace execution timeframe.

OnCPU events are like a flame graph, but the code has been collected at the thread level instead of the process level, and since the trace was executed by one thread, you can understand how the trace was executing on the CPU.

OffCPU events are the opposite. As for trace analysis, most traces spend their lifetime waiting for locks, database queries, remote process calls, file reading or file writing, and all of these cause the thread to be in a waiting status, which is what is considered an OffCPU event. That is the honest observation the feature is built on: a latency trace that only shows you CPU time is showing you the part of the trace where nothing interesting was happening.

The output is a replay. How all threads were executed is recorded and can be replayed, and the specific correlations are listed. The exact thread which executed the trace span is highlighted. The logs printed by each thread are collected and correlated to the relative thread with its timestamp. The code execution flame graph is correlated to the time series where the CPU is busy. The network-related metrics are correlated to the time series where the network syscalls are executing. And the file-related metrics are correlated to the time series where the file syscalls are executing.

So the value is the joining of five views that normally live in separate tools: which thread ran, what it printed, where the CPU time went, where the network syscalls happened, and where the file syscalls happened, all against one trace timeline. The documented example is understanding how ElasticSearch executes a bulk insert operation, which is a good illustration because a bulk insert is exactly the case where the interesting time is in the network and the disk rather than in the CPU.

The UI reference in the documentation is network exploration in Chrome, and the usage is described as the same: users can tell which part to dig into to find the root cause of the problem in the production environment. It is a navigation metaphor rather than a technical one, but it is a useful one for a tool whose job is deciding where to look next.

## The eBPF and kernel module split, and the version floor

The kernel requirement is the first practical gate, and the documentation is explicit about the numbers.

The Kindling eBPF module depends on a kernel version newer than 4.14. For trace profiling, the kernel version has to be newer than 4.17, with the hope that more work will bring trace profiling to older kernel versions. This is described as an eBPF constraint, meaning the eBPF module cannot work for older kernel versions.

For users who want to try the functionality on an old kernel, the project uses the kernel module from the Sysdig open source project, with enhancement and verification. The stated idea is to use a kernel module for tracking the kernel tracepoints, and the thanks to Sysdig is because that project provides a tracepoint instrument framework for older kernel versions.

The current state of the fallback is specific and worth knowing before you plan on it. The kernel module works as expected as the eBPF module during the project's test, except for trace profiling, and it is recommended to use the eBPF module in the production environment as it is safer than the kernel module. Issues with the kernel module are welcome.

So the fallback gives you network monitoring and the flow dashboards on an old kernel, but not the trace profiling, which is the headline feature. That is a meaningful split, because trace profiling is the reason to run Kindling rather than a packet capture.

The reason for eBPF at all is a performance argument, and it is specific. The libpcap way of analyzing flows in a Kubernetes environment is too expensive for the CPU and the network, while the eBPF way of data capture costs much less than libpcap. eBPF is described as the most popular technology to track the Linux kernel where the virtual network, built by veth-pair and iptables, works, so it is a proper technique for tracking how the kernel responds to application requests.

The veth-pair detail is the key to why this is a Kubernetes tool rather than a general network tool. In a container network, the interesting traffic does not traverse a physical interface, it is created and destroyed in software between veth pairs and shaped by iptables rules. A capture point at the wire level misses it, and libpcap at the host level is expensive because of the volume. eBPF hooks in the kernel see the containers' traffic where it actually happens.

## The agent, the exporters and the DaemonSet deployment

The architecture is described from a high level view. The agent runs as a DaemonSet in Kubernetes, and it collects all syscalls and some other tracepoints. Different exporters are used for different distributions.

The example given is a Prometheus exporter, which exports data that can be stored in Prometheus and displayed in a Grafana plugin. But for the trace profiling module the UI is different, and that is a standalone module. So the project has two delivery paths, and the split follows the data model rather than the deployment.

The network monitoring side is the conventional path. Flow data is metrics shaped data, so it goes to Prometheus and Grafana, and the built-in dashboards are named in the description: TCP retransmit, DNS, throughput and TPS. Those are the four questions you ask of a network: is there loss, is name resolution working, how much is moving and how many transactions per second.

The trace profiling side cannot work that way, because a trace with per-thread flame graphs, correlated logs and syscall timelines is not a metric. It needs its own storage and its own UI, and the documentation links to a standalone module for it.

The repository layout confirms the split. The collector/ directory is the Go agent, the probe/ directory holds the eBPF and kernel module probe, the grafana-plugins/ directory holds the dashboard plugins, the deploy/ directory holds the Kubernetes deployment, and there is a camera-front/ directory and a contribute/ directory. The CHANGELOG.md, CONTRIBUTING.md and CODE_OF_CONDUCT.md are in the root along with a Makefile and a docs/ directory.

The Makefile shows how the two halves are built and how they are coupled. The collector target depends on libkindling, and the collector is built with go build and linker flags that stamp the git commit into the binary through a version.CodeVersion variable, so a running agent reports the commit it was built from.

The libkindling target depends on patch-libs, and it builds the probe with cmake using -DBUILD_DRIVER=OFF, a probe version of 0.1.1dev and an agent libs source directory pointing at probe/libs/agent-libs. The built libkindling.so is then copied into collector/docker/libso/ and also installed into /usr/lib64/. The fact that a shared object is copied into the collector's docker directory and into the system library path tells you the collector loads the probe dynamically rather than linking it, which is what allows the same collector image to ship with either the eBPF probe or the kernel module probe.

The build also uses two container images, kindlingproject/kernel-builder:latest for the probe and kindlingproject/agent-builder:latest for the collector, and the builder-images target pulls whichever is missing.

## Version history, licence and what to expect from the project

The project is an Apache-2.0 licensed effort from a Chinese company, with the homepage at kindling.harmonycloud.cn and documentation under a docs/overview-and-concepts/overview path. The topic tags are ebpf, kubernetes, monitoring, observability and profiling.

The release history is the thing to weigh. The three most recent tags are v0.9.1 on 2024-02-27, v0.9.0 on 2024-02-02 and v0.8.1 on 2023-09-01. So the last tagged release is more than two years before the last push, which was on 2026-07-21, and the repository is not archived.

That gap has two possible readings and you should decide which applies to your evaluation. Either the project is still being developed on the branch without cutting releases, or the release process has slowed while maintenance continued. The repository has a CHANGELOG.md, and the fact that the branch is receiving pushes is real. A team can pin the v0.9.1 release or select and test a specific newer commit, bearing in mind that unreleased code may not have the same release validation.

The project is also structured as a set of directories that look like separate pieces of a bigger system, camera-front/, collector/, probe/, grafana-plugins/, deploy/ and contribute/, with a .gitmodules file indicating submodules are involved. That shape suggests the agent is one component in a wider stack, and the surrounding parts such as the Grafana plugins are as much part of the adoption as the agent itself.

The practical checklist for evaluating Kindling is short. Confirm the kernel version on every node, because the 4.14 and 4.17 floors are per node and a heterogeneous cluster will have nodes below both. Decide whether the trace profiling feature is essential, because that is the part the kernel module fallback cannot deliver. Read the deploy/ directory to see the DaemonSet configuration and what privileges the agent needs, since an eBPF agent that collects all syscalls is not a low privilege workload. And check whether the flow data you need fits the Prometheus and Grafana path, since that is the part that is conventional and well understood.

## Conclusion

Kindling fits a Kubernetes platform team that cannot instrument every application, that wants to triage a production issue from the network side first and then drill into a single trace with the code, logs and syscalls in one view. It is the wrong tool if you run Java services with a working APM and do not need the kernel level view, and it is the wrong tool on kernels older than 4.17 if trace profiling is the point, since the Sysdig derived kernel module does not support it. Before adopting, check the kernel version on every node you plan to cover, read the deploy/ directory for the DaemonSet configuration, and be aware that the most recent tagged release is v0.9.1 from 2024-02-27 while the last push was on 2026-07-21.

## FAQ

### What does Kindling's trace profiling add over a normal distributed trace?

It integrates OnCPU and OffCPU events into the trace and collects the logs printed during the trace timeframe. OnCPU events are collected at the thread level, off-CPU events capture time spent waiting on locks, database queries, remote calls and file operations, and the logs, CPU flame graph, network syscalls and file syscalls are all correlated to one timeline.

### What kernel version does Kindling require?

The eBPF module requires a kernel newer than 4.14, and trace profiling specifically requires newer than 4.17. For older kernels the project provides a kernel module derived from the Sysdig open source project, which works as expected except for trace profiling, and the eBPF module is the recommended choice in production.

### How is Kindling deployed in Kubernetes?

The agent runs as a DaemonSet and collects all syscalls and some other tracepoints. Network flow data is exported through a Prometheus exporter and displayed in Grafana plugins with built-in dashboards for TCP retransmit, DNS, throughput and TPS, while the trace profiling module is a standalone UI.

### Why does Kindling use eBPF rather than libpcap?

The stated reason is cost. Analyzing flows in a Kubernetes environment with libpcap is too expensive for the CPU and the network, while the eBPF way of data capture costs much less. eBPF also tracks the Linux kernel where the container virtual network, built from veth pairs and iptables, actually operates.

## Sources

- [harmonycloud/kindling on GitHub](https://github.com/harmonycloud/kindling)
- [License: Apache-2.0](https://github.com/harmonycloud/kindling/blob/main/LICENSE)
- [Project website](http://kindling.harmonycloud.cn)
- [README](https://github.com/harmonycloud/kindling/blob/main/README.md)
- [Releases](https://github.com/harmonycloud/kindling/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/harmonycloud-kindling