Goldpinger: Node-to-Node Connectivity Testing Inside Kubernetes

The problem Goldpinger targets: silent node-to-node failures

Kubernetes networking fails in ways that application logs rarely show. A pod starts, passes its readiness probe, serves traffic from its own node, and still cannot reach a pod on another node because of a CNI misconfiguration, an MTU mismatch, a stale iptables rule or a dropped packet on one link. The README describes the origin directly: Goldpinger was built at Bloomberg "to troubleshoot, visualise and alert on our networking layer while adopting Kubernetes". That is the scope. It is a connectivity prober for the cluster fabric, not an APM tool.

The audience is platform and infrastructure engineers who own the cluster network and need evidence, not guesses. The README calls it "the go-to tool to see connectivity and slowness issues", and the binary is described as small at roughly 16MB, which matters when you schedule one copy per node.

How Goldpinger discovers peers and probes them

Goldpinger does not ship a static peer list. According to the README, it "works by asking Kubernetes for pods with particular labels (app=goldpinger)". Each instance queries the Kubernetes API for pods carrying that label, then calls the other instances over HTTP. Because it runs as a DaemonSet, one pod lands on each node, so the set of peers is effectively the set of nodes.

Two environment variables shape what those calls look like. HOSTNAME is injected from spec.nodeName, and the README notes that this "will make for easier to understand graphs/metrics", so the graph labels are node names rather than random pod names. POD_IP comes from status.podIP and, in the README's words, "is used to select a randomized subset of nodes to ping". That randomization matters at scale: a full mesh of every node against every other node grows quadratically, and sampling keeps the call volume bounded. The rendezvous hashing dependency in go.mod (github.com/stuartnelson3/go-rendezvous) is consistent with a deterministic peer-selection scheme, though the README does not spell out the selection algorithm.

The results are exposed as Prometheus metrics on port 8080, and the DaemonSet example carries the annotations prometheus.io/scrape: 'true' and prometheus.io/port: '8080' so a standard Prometheus Kubernetes service discovery picks it up. A built-in UI renders the connectivity graph, and the repository also carries a swagger.yml, from which the Makefile can regenerate a server and client with the swagger target.

Installing Goldpinger with Helm and reading the first graph

The fastest path is the Helm chart published from the project's own repository. The README gives these three commands to add the repo, refresh the index and install the chart:

helm repo add goldpinger https://bloomberg.github.io/goldpinger
helm repo update
helm install goldpinger goldpinger/goldpinger

After the install, a DaemonSet named goldpinger should schedule one pod per node. The manual example exposes the Service as type NodePort with nodePort 30080, so on a cluster where that port is reachable you can open the UI in a browser against any node address. The graph shows nodes as points and the calls between them as edges; the README's screenshot shows this view. The same port serves the Prometheus metrics and a /healthz endpoint used by both the readiness and liveness probes.

If you prefer plain manifests, the README provides a full example with a ServiceAccount, a DaemonSet and a NodePort Service on 30080. The critical detail is RBAC: Goldpinger must be allowed to list pods, and the README warns that "you will also need to add an RBAC rule to allow Goldpinger to list other pods", suggesting a view-all ClusterRoleBinding only for experimentation. For authentication it accepts either a kubeconfig via --kubeconfig-path or the in-cluster service account. The container image referenced in the example is docker.io/bloomberg/goldpinger:v3.0.0; the Makefile in the repository sets version to v3.11.3.

You can also run the binary outside the cluster. The README's quick start shows:

go get github.com/bloomberg/goldpinger/cmd/goldpinger
goldpinger --help

Building locally requires Go 1.15 or newer and Docker, per the README, and make bin/goldpinger produces ./bin/goldpinger.

Where Goldpinger stops being the right tool

Goldpinger measures its own calls between its own instances. It does not instrument your application's traffic, so a service that cannot reach its database will not appear as a broken edge unless the failure happens to correlate with a node-level path problem. If you need per-request traces or service-level latency, this is the wrong instrument.

The probing model has a second consequence. Because POD_IP is used to select a randomized subset of nodes, the graph is a sample, not a complete mesh. A single bad link can be missed on one refresh and caught on the next. That is a reasonable trade at scale, but it means the tool is better at spotting persistent degradation than at proving a one-off packet drop.

There is also an operational cost that the README does not quantify. Every Goldpinger pod holds a watch against the Kubernetes API server to enumerate peers, and the DaemonSet runs on every node including control-plane nodes, since the example tolerates node-role.kubernetes.io/master with effect NoSchedule. On a large cluster that is a lot of API traffic from a debugging tool. The example resource requests are modest (1m CPU, 40Mi memory requested, 80Mi limit), so the pods themselves are cheap; the API server load is the part to watch.

Finally, the README does not document rollback, upgrade ordering or chart values. The manual path is a DaemonSet with updateStrategy RollingUpdate, so a bad image will roll node by node, but nothing in the README describes how to pin or revert a chart release.

How Goldpinger differs from a general-purpose mesh or service monitor

A service mesh such as Istio or Linkerd also reports connectivity between workloads, but it does so by intercepting application traffic through sidecars or per-node proxies. Its telemetry describes real requests: which service called which, with what status and latency. Goldpinger takes the opposite approach. It injects its own synthetic calls between its own pods and reports on the network path itself. That means Goldpinger can tell you the fabric is broken before any application notices, while a mesh can only tell you about traffic that actually flowed.

The trade is coverage. A mesh sees every service; Goldpinger sees only node-to-node reachability as experienced by its own DaemonSet pods. If your question is "is my network healthy", Goldpinger answers it with far less deployment surface than a mesh. If your question is "which service is slow", Goldpinger has nothing to say. The README also points at a UDP probe mode for packet loss, hop count and RTT, which extends the picture beyond HTTP reachability, and at an Alert Manager integration for turning the metrics into alerts.

Maintenance, licensing and upgrade cost

The repository is not archived, and the last push was on 2026-09-14, with v3.11.3 tagged the same day. That is recent enough to treat the project as maintained, and the go.mod pins current Kubernetes client libraries (k8s.io/client-go v0.35.0) and Go 1.25 in the Dockerfile, so it tracks upstream rather than drifting.

Upgrade cost is low by design. The deliverable is a single static binary in a distroless image built from gcr.io/distroless/static:nonroot, with the static assets and config copied alongside it. There is no database, no CRD and no migration step. Upgrading means changing an image tag or bumping a chart release. The Makefile exposes a build-release target, which uses docker buildx to push multi-architecture images for linux/amd64 and linux/arm64.

Goldpinger is licensed under Apache-2.0, and the LICENSE file sits at the repository root. Apache-2.0 permits commercial use and modification and includes a patent grant; it also requires that you preserve the licence and notice files when redistributing. That is the general shape of the licence, not advice for your situation. The repository also carries a DCO.md, so contributions are accepted under a Developer Certificate of Origin rather than a CLA.

Editorial conclusion

Adopt Goldpinger if you run Kubernetes and need to know whether pod-to-pod traffic between nodes is actually working, and you already have Prometheus to scrape port 8080. Skip it if you need application-level tracing or per-service latency breakdowns; it only measures its own calls between its own instances. Before rolling it out, verify that the ClusterRoleBinding grants pod list permission, that the ServiceAccount name in the DaemonSet matches the one you created, and that the readinessProbe on /healthz passes on your nodes.

Frequently asked questions

Official sources

1. bloomberg/goldpinger on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases