# pwru: tracing a single packet through the Linux kernel with eBPF

> pwru attaches eBPF programs to kernel functions and prints every skb it sees, so you can find the exact function where a packet dies. It is a debugging tool for kernel and network engineers, not a general observability platform.

**cilium/pwru** — Packet, where are you? -- eBPF-based Linux kernel networking debugger

- Repository: https://github.com/cilium/pwru
- Stars: 3,833 · Forks: 233
- Language: C
- License: Apache-2.0
- Published: 2026-09-23 · Updated: 2026-09-23 · Language: en
- Canonical page: https://hysenlabs.com/projects/cilium-pwru

## The problem pwru solves: a packet that vanishes between two interfaces

A packet leaves one interface and never arrives on the other. tcpdump on the sending side shows the frame leaving; tcpdump on the receiving side shows nothing. The drop happened inside the kernel, in a function that has no counter and no log line. pwru exists for that gap. It attaches eBPF programs to kernel networking functions and reports each packet as it passes through them, so the last function a packet was seen in becomes the place to look.

The README frames it as "fine-grained introspection of kernel state to facilitate debugging network connectivity issues", and the demo it ships is exactly the case above: a curl request whose packets stop after an iptables rule is installed. The audience is narrow and technical: kernel networking developers, CNI and service-mesh maintainers, and anyone who has already exhausted conntrack, nftables counters and interface statistics. It is a diagnostic instrument you reach for when the normal tools have gone quiet.

## How pwru attaches to kernel functions and follows one skb

pwru is written in C for the eBPF side and Go for the userspace side. The Go module vendor directory and go.mod show the eBPF work is done through github.com/cilium/ebpf, with pcap filter expressions compiled to cBPF through github.com/cloudflare/cbpfc and the BPF C compiled at build time via bpf2go. The Makefile builds a statically linked libpcap into the binary, which is why the released executables have no runtime libpcap dependency.

At runtime pwru probes kernel functions and filters events by a pcap-filter expression. The kernel side emits an event per matching skb per probed function; the userspace side formats it. Two behaviours matter for correctness. First, filtering is not limited to the initial match: --filter-track-skb keeps tracing a packet even after it stops matching the filters, which the help text attributes to NAT and tunnel decapsulation, and --filter-track-skb-by-stackid continues after the skb is kfreed, for traffic through a bridge. Without those flags a packet can leave your trace at exactly the moment it becomes interesting.

Second, the probe backend is chosen automatically unless you set it. --backend accepts kprobe or kprobe-multi; kprobe-multi needs kernel 5.18 or newer and the FUNCTION_TRACER and FPROBE config options, and it is the reason --filter-func can be given a RE2 regular expression. The README is explicit that --filter-func=foo matches only foo(), and that a wildcard needs --filter-func=".*foo.*". A plain substring will silently match nothing.

## Installing pwru and running a first trace

The README points at the release page for statically linked x86_64 and arm64 executables, so the fastest path is to download the binary rather than build it. Building from source requires clang, llvm, libpcap headers and a Go toolchain; the Makefile target pwru generates the BPF objects and then builds the Go binary with CGO enabled. If you prefer containers, the project publishes images at hub.docker.com/r/cilium/pwru.

Before any of that, check the kernel. pwru needs 5.3 or newer, 5.9 for --output-skb, and 5.18 for the kprobe-multi backend. The README also lists required kernel configuration options and suggests verifying them one at a time:

```bash
zgrep CONFIG_DEBUG_INFO_BTF /proc/config.gz
zgrep CONFIG_KPROBES /proc/config.gz
zgrep CONFIG_BPF_SYSCALL /proc/config.gz
```

Each command should print a line ending in =y. CONFIG_DEBUG_INFO_BTF is required by both backends and has been available since 5.3. debugfs is optional but must be mounted at /sys/kernel/debug if you use it:

```bash
mount -t debugfs none /sys/kernel/debug
```

A first trace, taken from the README's own example, follows packets to and from 1.1.1.1 and prints the L4 tuple:

```bash
./pwru --output-tuple 'host 1.1.1.1'
```

You should see one line per kernel function the packet passes through, each with the function name and the tuple. If the output stops before the packet reaches its destination, the last function printed is where to look next. The same invocation works from the published image, which the README shows mounting debugfs and running with host PID namespace:

```bash
docker run --privileged --rm -t --pid=host -v /sys/kernel/debug/:/sys/kernel/debug/ cilium/pwru pwru --output-tuple 'host 1.1.1.1'
```

## Where pwru stops being the right tool

pwru needs privileges and a cooperating kernel. The Docker example uses --privileged and mounts /sys/kernel/debug; the Kubernetes example sets privileged: true. On a managed cluster where privileged pods are blocked by policy, or on a kernel built without the BTF and kprobe options listed in the README, the tool cannot attach at all. The project keeps a KNOWN_ISSUES.md at the repository root, which is the file to read before filing anything: it is where behaviour that depends on kernel version is recorded.

Cost is the second limit. Every probed function produces events, so a broad pcap filter combined with --all-kmods or --filter-track-bpf-helpers multiplies the volume quickly. The flags --output-limit-lines, --output-file and --output-file-max-size exist precisely because a trace can outgrow a terminal, and the file output rotates at 100MB by default with optional gzip compression. There is no sampling mode documented in the help output; you reduce cost by narrowing the filter, not by asking pwru to look less often.

Finally, pwru answers "which function saw this packet last", not "why is throughput low" or "which service is slow". It is a packet path debugger. For latency attribution across a cluster you want a profiler or a distributed tracing system, not a kprobe on the receive path.

## pwru compared with tracing the same path by hand

The obvious alternative is ftrace through /sys/kernel/debug/tracing, or bpftrace scripts that attach to the same kernel functions. Both are already installed on many systems and need no new binary. The difference is in what you have to write. With ftrace you pick functions yourself, enable a tracer, and correlate timestamps across a function set you guessed in advance. bpftrace gives you a language, so you can express the filter and the output you want, but you write and maintain that script.

pwru packages the guesswork. You supply a pcap-filter expression, the tool decides which functions to probe, follows the skb through NAT and tunnel decapsulation when asked, and prints a formatted line. The trade-off is the opposite of bpftrace's: pwru is opinionated about what an event looks like and which functions are interesting, and it will not answer a question outside packet path tracing. If your question is about a specific kernel function's arguments rather than a packet's journey, a bpftrace one-liner is still the shorter route. Note also that pwru's kprobe-multi backend and ftrace share the same underlying FUNCTION_TRACER and FPROBE kernel options, so the two approaches are not independent of kernel configuration.

## Maintenance, releases and the Apache-2.0 licence

The repository is not archived and the last push was on 2026-09-21. Releases are infrequent and versioned: v1.0.10 in August 2025, v1.0.11 in January 2026, v1.0.12 in July 2026. Between releases the main branch moves, so pinning to a tagged binary is the cheaper operational choice than building from a commit.

Upgrade cost is dominated by the kernel, not by pwru. A newer pwru may default to a backend your kernel does not support, and the README ties each backend and output mode to a minimum kernel version. Read the release notes before upgrading on a fleet with mixed kernels, and keep the zgrep checks in your provisioning runbook so a host that lacks CONFIG_DEBUG_INFO_BTF fails early rather than at trace time. The dependency list in go.mod, including github.com/cilium/ebpf and the vendored libpcap, moves with each release; renovate.json5 at the repository root suggests dependency updates are automated, which means the main branch can change under you.

Licensing is Apache-2.0, a permissive licence that permits commercial use and modification provided you keep the licence and notices. That is the whole of what the repository states; whether your distribution or your legal team accepts Apache-2.0 terms in your context is a question for them, not something this article can settle.

## Conclusion

Adopt pwru when a packet disappears somewhere between the NIC and the socket and tcpdump on both ends shows nothing useful. Do not adopt it as a permanent monitoring agent: it attaches kprobes across kernel networking functions, needs privileged access and debugfs, and its output is a per-packet trace, not a metric. Before rolling it out, confirm the kernel version and the required CONFIG_ options with zgrep against /proc/config.gz, and check KNOWN_ISSUES.md for behaviour that matches your kernel.

## FAQ

### What kernel version does pwru require?

The README states pwru needs kernel 5.3 or newer. The --output-skb option needs 5.9 or newer, and the kprobe-multi backend needs 5.18 or newer.

### How do I install pwru?

The README points to the release page for statically linked x86_64 and arm64 executables, and notes that Docker images are published at hub.docker.com/r/cilium/pwru. Building from source uses the Makefile's pwru target, which requires clang, llvm, libpcap headers and a Go toolchain.

### Does pwru need privileged access to run?

The README's Docker example runs the container with --privileged and mounts /sys/kernel/debug, and the Kubernetes example sets privileged: true on the container. debugfs is described as optional but must be mounted at /sys/kernel/debug when used.

### Why does --filter-func not match my function name?

The README states that --filter-func does an exact match on function names, so --filter-func=foo matches only foo(). For a wildcarded match it suggests --filter-func=".*foo.*" instead, which requires the kprobe-multi backend.

## Sources

- [cilium/pwru on GitHub](https://github.com/cilium/pwru)
- [Issues](https://github.com/cilium/pwru/issues)
- [License: Apache-2.0](https://github.com/cilium/pwru/blob/main/LICENSE)
- [README](https://github.com/cilium/pwru/blob/main/README.md)
- [Releases](https://github.com/cilium/pwru/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/cilium-pwru