# kubectl trace: running bpftrace across a Kubernetes cluster from the command line

> A kubectl plugin from the iovisor organisation that schedules bpftrace programs as cluster jobs, so you can attach kernel and userspace probes to nodes and containers without leaving your laptop.

**iovisor/kubectl-trace** — Schedule bpftrace programs on your kubernetes cluster using the kubectl

- Repository: https://github.com/iovisor/kubectl-trace
- Stars: 2,188 · Forks: 176
- Language: Go
- License: MIT
- Published: 2026-10-06 · Updated: 2026-10-06 · Language: en
- Canonical page: https://hysenlabs.com/projects/iovisor-kubectl-trace

## A kubectl plugin that wraps bpftrace in a job

`kubectl trace` is described in its own README as a kubectl plugin that schedules the execution of bpftrace programs inside a Kubernetes cluster. The distribution channel matches that description: it installs through Krew, the package manager for kubectl plugins, and a successful install drops a `kubectl-trace` binary somewhere on your PATH so the subcommand appears.

The reason this wrapping exists is privilege, not convenience. bpftrace needs to load BPF programs, read kernel tracepoints, and attach uprobes, which means elevated privileges and a kernel interface that is not exposed to an ordinary container. On a single host you solve that with sudo. On a cluster you solve it with a pod spec, an image built for the purpose, and the scheduling machinery you already have. The plugin takes care of building that pod spec.

The Go module is `github.com/iovisor/kubectl-trace`, the language is Go, the license is MIT, and the default branch is `master`. The dependency list in `go.mod` gives a fair picture of the design: it uses `k8s.io/kubectl`, `k8s.io/client-go` and `k8s.io/cli-runtime`, all at v0.27.2, plus `spf13/cobra` and `spf13/afero`. The presence of both cobra and afero tells you this is a client tool written against the kubectl plugin conventions, with an injectable filesystem layer for testing. `sigs.k8s.io/kind` in the same file says the integration tests run against a throwaway kind cluster.

## Installing through Krew, or building from source

The shortest documented path uses Krew, and the README presents it as a one line install followed by nothing else to do.

```bash
kubectl krew install trace
```

Pre-built archives exist for Linux, macOS and Windows, and the README spells out the Linux and macOS shape of the process, which is download the tarball, extract it, move the binary into `/usr/local/bin`. Because a kubectl plugin is invoked by name, the filename matters as much as the location: `kubectl-trace` on your PATH is what makes `kubectl trace` resolve.

Building from source goes through Go modules, and the documented command installs the command package at its latest tag:

```
GO111MODULE=on go get github.com/iovisor/kubectl-trace/cmd/kubectl-trace@latest
```

The README adds a specific caveat here that is worth repeating. It recommends building only tagged revisions if you want stability, and warns that building `master` or the `latest` tag can give you a build that has had less QA than a tagged release. For a tool that creates privileged jobs on your nodes, that is the correct default posture.

There is a version command too, and the README shows how to bake the ref you built into the binary with an ldflag so that `kubectl-trace version` can report it back. The Makefile does the same thing for release builds, injecting `buildTime`, `gitCommit`, and the runner and init container image names and tags through `github.com/iovisor/kubectl-trace/pkg/version` and `pkg/cmd`.

## Tracing a node with a program from the command line

The usage section opens with a warning that is easy to skim past. The README says you do not need to set anything up on the cluster before using the plugin, and immediately asks that you not use it on a production system just because it is not yet 100 percent ready. Both halves of that sentence matter: there is no agent to deploy, and there is also no promise about production safety.

The simplest invocation probes a tracepoint on a node, with the bpftrace program passed inline as a string:

```bash
kubectl trace run ip-180-12-0-152.ec2.internal -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"
```

That is a syscall counter keyed by probe name, which is a sensible smoke test: if you get a map back with per syscall counts, the plumbing works.

Longer programs come from a file instead of the command line, using `-f`:

```bash
kubectl trace run ip-180-12-0-152.ec2.internal -f read.bt
```

The README's screenshot for this case shows the output of the `read.bt` example, so the file based route is the documented way to run anything substantial. Note that a node is the default target. You can also name it explicitly with the `node/node-name` syntax, which matters when you are writing scripts and do not want the implicit default to carry a mistake.

## What the pod target actually provides

This is the part of the README that most deserves a careful read, because the naming invites a wrong model of what happens.

The README is direct about it: running against a pod is just a helper to resolve the context of a container. The program itself does not run inside your pod. The plugin resolves the pod, hands your bpftrace program a variable called `$container_pid` holding the pid of the container's root process in the root pid namespace, and runs the trace from the node side. The README says it in as many words, noting that the trace is never contained in the pod and that the pod target only passes knowledge of the container's context.

The worked example attaches a uretprobe to a Go binary at `/caturday` in a pod, hooking a function called `main.counterValue`:

```bash
kubectl trace run -e 'uretprobe:/proc/$container_pid/exe:"main.counterValue" { printf("%d\n", retval) }' pod/caturday-566d99889-8glv9 -a -n caturday
```

Resolving the binary through `/proc/$container_pid/exe` is the neat part, because it means you do not need to know the binary's path inside the image filesystem. The guidance on choosing a target follows from this. Kernel level things, meaning kprobes, kretprobes, tracepoints, software and hardware events, and profile events, belong on a node. Userspace work with uprobes, uretprobes or USDT probes belongs on a pod, because that is how you get the process id.

The README also notes that you could achieve the same thing on a node by learning the pid yourself over ssh. The pod target is convenience, not a different execution model.

## Images, versions and the build pipeline behind the plugin

The Makefile shows how much of this project's structure lives in the build. Two images are named and tagged separately: an init container image and a runner image, both on quay.io under the iovisor organisation. The file derives four tags from Git state, a branch tag, a commit tag, a git describe tag and a latest tag, for both images, which is the standard pattern for a project whose release artifacts and runtime images must stay in step.

The bpftrace version the images are built around is set in one variable:

```
BPFTRACEVERSION ?= "v0.19.1"
```

That single line is where you should look when a probe behaves unexpectedly, because bpftrace syntax and builtin availability move between releases and the plugin controls the version inside its images. The release history tells the same story from the other side. v0.1.2, published on 2021-07-13, is mostly packaging fixes, krew selector corrections for 32 bit Windows, a zip extension for Windows releases, and one feature commit that bumps bpftrace to 0.13.0. v0.1.1-rc.0 from 2021-05-27 is the substantial one, adding the ability to apply patches to the job spec, moving integration tests to testify, using the kubectl factory for Kubernetes access, and setting up automated releases through Krew. v0.1.0-rc.1 from 2019-09-19 added job deadlines and a configurable deadline grace period, along with a pre-stop hook to print maps before exit.

Two of those are worth remembering when reading the plugin's output. Job deadlines exist so that a trace job terminates on its own, and the pre-stop hook exists so that the maps a bpftrace program accumulated get printed instead of lost when the job is torn down.

## Service accounts, job spec patches, and the incomplete edge of the docs

Scheduling privileged work inside someone else's cluster is a permissions question before it is a technical one, and the README starts to address it. By default `kubectl trace` uses the `default` service account in the target namespace, which is also `default`, to schedule the pods that carry your bpftrace program.

Then the README breaks off mid sentence. The custom service account section is titled `Using a custom service account`, and the text stops at "If you need to pass a s". Whatever followed, whether it was an example manifest or a flag name, is not available to read. The same happens to the sections listed in the table of contents after it, including the section on running under Pod Security Policies, on using a patch to customise the trace job, and on finding more bpftrace programs. The table of contents lists them; the body does not carry them. The v0.1.1-rc.0 release notes confirm that patch support and pod security policy material existed, so the documentation was there at some point and the README no longer carries it in this state.

That gap matters for anyone evaluating the plugin for a locked down cluster, because Pod Security Admission and restricted policies are exactly the environments where this plugin needs a documented answer. The repository tree does carry the usual supporting material, including a docs directory, so the deeper architecture notes are likely reachable there rather than in the README.

## What the repository layout says about how it is tested

The dependency on `sigs.k8s.io/kind` is the strongest signal in the repository. Kind spins up a Kubernetes cluster inside a container, which is the only practical way to run an integration test suite for something that creates pods, waits for them to attach, and reads their output. The v0.1.1-rc.0 notes show that test work was real rather than aspirational: the integration tests moved to testify, they were updated to use the newer kind cluster creation API, and errors encountered during `Attach()` started being printed instead of swallowed. One commit reads as a symptom of flaky infrastructure, which is the honest description of "Attacher continues retrying if pod not found".

A separate detail points the same way. The `go.mod` requires `github.com/fake-gcs-server`, a local stand in for Google Cloud Storage. That is not part of tracing at all, and its presence suggests the test fixtures fetch or store artefacts through a storage abstraction, which is then faked so tests do not need cloud credentials.

Taken together, the test story is coherent: a kind cluster for the Kubernetes behaviour, a fake object store for artefact handling, testify for assertions, and afero for filesystem isolation. For a project whose releases stopped in 2021, that is the part of the codebase most likely to have decayed against newer Kubernetes versions, since the pinned client libraries at v0.27.2 are now well behind the current client.

## Conclusion

kubectl trace exists because the bpftrace you want to run is almost never on the machine where you are sitting. The plugin takes a program string or a `.bt` file, turns it into a job, and lets the cluster schedule the container that needs privileged kernel access. That is the whole idea, and it is a good one. What you give up is control: the program runs somewhere you did not choose, on an image pinned by the plugin, inside a namespace you did not create. Before running anything like this against a real workload, read the plugin's own warning at the top of the usage section and remember that the newest tagged release in the repository is v0.1.2, published on 2021-07-13. The last recorded push to the default branch was on 2026-04-16, so the code moves even though the release line has not advanced in years. Whether the pinned bpftrace in the Makefile, `v0.19.1`, matches what your kernel and your probe scripts expect is the first thing to check on your own cluster.

## FAQ

### How do I install kubectl trace?

Through Krew, the package manager for kubectl plugins, with a single command: `kubectl krew install trace`. Pre-built archives for Linux, macOS and Windows are also published on the releases page, and source builds go through Go modules at a tagged revision, which the README recommends over building the default branch.

### What is the difference between running a trace against a node and against a pod?

Kernel level probes such as kprobes, tracepoints, software and hardware events run against nodes, which are the default target. Userspace uprobes, uretprobes and USDT probes run against a pod, because the pod target only resolves the container and exposes its root process id to your program as `$container_pid`. The trace itself still executes outside the pod.

### Does my bpftrace program run inside the pod I target?

No. The README states that running against a pod is a facilitator for finding the process id, and that it does not mean the program is contained in that pod. The probe is loaded from the node side, with only the container's root pid handed to the program.

### What is the latest release of kubectl trace?

v0.1.2, published on 2021-07-13. It is mostly packaging and Krew selector fixes plus a bpftrace bump to 0.13.0. The last recorded push to the default branch was on 2026-04-16, so commits continue to land even though the tagged release line has not moved since 2021.

### Is kubectl trace safe to run in production?

The README asks that you not use it on a production system on the grounds that it is not yet 100 percent ready. It also confirms that no setup is required on the cluster beforehand, since the plugin schedules the job itself using the `default` service account in the target namespace unless you configure otherwise.

### Why is Kubernetes so hard to debug with standard kubectl commands?

kubectl gives you the control plane view of a workload: its spec, its status, its logs. Kernel level behaviour inside a node, and userspace probes on a specific binary inside a container, sit below what those commands reach. kubectl trace exists to close that gap, which is why a search for kubectl troubleshooting often lands on tracing rather than on describe or logs.

## Sources

- [iovisor/kubectl-trace on GitHub](https://github.com/iovisor/kubectl-trace)
- [Issues](https://github.com/iovisor/kubectl-trace/issues)
- [License: MIT](https://github.com/iovisor/kubectl-trace/blob/master/LICENSE)
- [README](https://github.com/iovisor/kubectl-trace/blob/master/README.md)
- [Releases](https://github.com/iovisor/kubectl-trace/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/iovisor-kubectl-trace