Kuberhealthy: Running Synthetic Checks as Kubernetes Pods

What Kuberhealthy Solves and Who It Is For

Kuberhealthy addresses a specific gap in Kubernetes monitoring: verifying that your cluster and applications actually work, not just that they are up. Standard metrics tell you if a pod is running, but they do not tell you if a user can log in, if a database connection succeeds, or if a deployment can roll out. Kuberhealthy runs synthetic checks as short-lived pods on a schedule. Each check pod executes your validation logic and reports success or failure back to the controller. This is for platform engineers who want to codify their monitoring as Kubernetes manifests, ship them with their applications, and expose the results to Prometheus. The README describes it as an operator for synthetic monitoring and continuous validation, and it emphasizes that checks are Kubernetes resources, so they live in the same Git repository as your app. The target user is someone who already runs Kubernetes and Prometheus and wants a way to test real user workflows, not just scrape metrics.

The Mechanism: HealthCheck CRDs and Check Pods

The core mechanism is the HealthCheck custom resource definition. A HealthCheck specifies a podSpec, a runInterval, and a timeout. The controller watches these CRDs, schedules a checker pod at the specified interval, and waits for the pod to report back. The checker pod runs your logic, then sends a POST request to the controller at /check. The README shows this data flow in a Mermaid diagram: the controller watches CRDs, schedules pods, and the pod reports back. The controller aggregates results and exposes them through a service on port 80, which feeds the Prometheus /metrics endpoint, a JSON API at /json, and a web UI. The podSpec is a standard Kubernetes pod spec, so you can set resource requests and limits, environment variables, and any image you want. The controller enforces the timeout: if the pod does not report within the specified timeout, the check is considered failed. This design means your check logic is completely decoupled from the controller; it just needs to speak the reporting protocol.

Getting It Running: Helm, Kustomize, and ArgoCD

Installation is straightforward. The README recommends Helm, but also offers Kustomize and ArgoCD manifests. With Helm, you run helm install kuberhealthy deploy/helm/kuberhealthy -n kuberhealthy --create-namespace. With Kustomize, you run kubectl apply -k github.com/kuberhealthy/kuberhealthy/deploy/kustomize/base?ref=main. For ArgoCD, you apply deploy/argocd/kuberhealthy.yaml. After installation, you port-forward the service: kubectl -n kuberhealthy port-forward svc/kuberhealthy 8080:80, then open http://localhost:8080 to see the status UI. You then apply a HealthCheck manifest. The README gives a concrete example: a deployment check that creates a test deployment, rolls it out, and tears it down. The manifest uses apiVersion kuberhealthy.github.io/v2, kind HealthCheck, and spec fields runInterval, timeout, and podSpec. The example sets runInterval: 10m and timeout: 5m, and the podSpec runs the image docker.io/kuberhealthy/deployment-check:v0.1.1 with environment variables like CHECK_DEPLOYMENT_REPLICAS and CHECK_DEPLOYMENT_ROLLING_UPDATE. You can also write your own checks in Go, Python, Rust, bash, or any language that fits in a container, using the provided check clients.

Consuming Results: Prometheus Metrics and JSON API

The output is designed for Prometheus. The /metrics endpoint exposes metrics like kuberhealthy_check{check="api-smoke-test",namespace="kuberhealthy",status="1"} 1, which is a gauge that is 1 when the check is OK and 0 when it fails. There is also kuberhealthy_check_duration_seconds for run duration and kuberhealthy_check_success_total as a counter. This is a clean contract for alerting: you can alert on status="0" or on a sudden increase in duration. The JSON API at /json returns a structured object with an ok flag and per-check details, including lastRun and runDuration. The web UI is a convenience, but the real value is the Prometheus integration. One thing to note: the README does not mention any built-in alerting. Kuberhealthy reports status, but you must configure your own Prometheus alerts. That is a limitation if you expect the operator to page you.

Writing Your Own Checks: The Check Client Contract

Writing a check is straightforward if you use one of the language clients. The README shows a Go example that hits an internal API endpoint and calls checkclient.ReportFailure or checkclient.ReportSuccess. The client handles the reporting URL, the run UUID, and deadline enforcement automatically. The injected environment variables are KH_REPORTING_URL and KH_RUN_UUID, which the client uses to post results back. This means your check code does not need to know where the controller is or how to format the report. The README lists clients for Go, Python, TypeScript, JavaScript, Rust, Ruby, Java, and Bash. This multi-language support is a strength, but it also means you have to trust the client library for your language. The Go client is part of the main repo, but the others are separate repositories, and the README does not state their maintenance status. If you write a check in a language with a less mature client, you may need to implement the reporting protocol yourself.

Limitations and Failure Modes

The most obvious limitation is that every check runs as a separate pod. That means each check consumes cluster resources, and you must set resource requests and limits carefully. The example deployment check requests 25m CPU and 15Mi memory, but sets a limit of 1 CPU, which is a wide range. If you run many checks with high limits, you could exhaust node resources. Another failure mode: the check pod must be able to reach the controller's service. If the controller is down or the network is partitioned, the check pod cannot report, and the check will time out. The README does not describe what happens to in-flight checks during a controller restart. Also, the timeout is a hard limit; if your check logic takes longer than the timeout, it will be marked failed even if it eventually succeeds. This is by design, but it means you must set timeouts generously for slow workflows. Finally, the README does not mention any built-in retry or backoff for failed checks; a failed check just stays failed until the next run.

Alternatives and How They Differ

The most direct alternative is to write your own cron-based checker using Kubernetes CronJobs. You could schedule a CronJob that runs your validation script and exposes metrics via a custom exporter. The difference is that Kuberhealthy manages the scheduling, the reporting protocol, and the status aggregation for you. With CronJobs, you have to build the reporting and the Prometheus exporter yourself. Another alternative is a tool like kube-monkey or Litmus, but those are more about chaos engineering than synthetic monitoring. For pure synthetic monitoring, you could also use a hosted service like Pingdom, but that runs from outside your cluster and cannot test internal services. Kuberhealthy's approach is to run inside the cluster, so it can test internal APIs and services that are not exposed externally. That is a key difference: it verifies the internal health of your cluster, not just the public endpoints.

Maintenance and License Considerations

Kuberhealthy is licensed under Apache-2.0, which is permissive and allows commercial use without copyleft obligations. The project is actively maintained, with releases v3.0.14, v3.0.13, and v3.0.12 all from August 2026, and the last push on 2026-08-26. The repository is not archived. This suggests active development, but you should check the changelog between versions to understand the upgrade path. The project uses semantic versioning, so minor version bumps should be backwards compatible, but major version changes, like from v2 to v3, may require changes to your HealthCheck manifests. The README does not document an upgrade procedure, so you should test upgrades in a staging cluster. The maintenance cost is mainly in keeping your check images updated and ensuring the controller version matches the CRD version. Since checks are just pods, you are responsible for updating the images you reference in your HealthCheck manifests.

Editorial conclusion

Adopt Kuberhealthy if you want synthetic checks defined as Kubernetes manifests, with Prometheus metrics and a simple status UI, and if you can accept the operational overhead of running check pods. Do not adopt it if you need checks that run inside the controller process or if you require built-in alerting; you will need to pair it with Prometheus alerts. Before adopting, verify that your cluster can tolerate the resource requests and limits you set on check pods, and confirm that the check image you plan to use is maintained and supports the injected environment variables like KH_REPORTING_URL and KH_RUN_UUID.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes