kwatch: one crash message per cause, not one per pod

Twenty pod alerts collapse into one cause line

A failing release usually produces a crowd of separate Kubernetes events. Pods enter CrashLoopBackOff, a Service loses its endpoints, an Ingress starts returning 503s, and health checks fail across workloads that have nothing wrong with them. Each event is accurate on its own, and together they tell an on-call engineer nothing about where to look. kwatch is built for that specific gap. The project describes itself as the on-call teammate for a cluster, and its job is to take a set of symptoms and report the single thing that caused them, together with the command that fixes it. It targets operators who handle deployment-triggered incidents rather than teams looking for a metrics dashboard. The comparison kwatch draws is with alert pipelines that emit one message per object: in its example, a rollout of api:2.3 into the shop namespace produced per-pod CrashLoopBackOff notices, a Service with no endpoints and a storefront Ingress returning 503s, all of which share one cause.

Watch, detect, explain, tell: the pipeline inside the pod

The project documents four stages. Watch keeps a live map of the cluster: every object, how the objects connect, and a timeline of what changed and who changed it. Detect identifies unhealthy things, listed as crash loops, unready nodes, Services with no endpoints, full volumes and failing webhooks. Explain is the interesting part. From each failure kwatch walks to what the failed object depends on, discards suspects that are neither broken nor recently changed, and then selects the causes that account for the whole set, preferring stronger evidence over simply covering more failures. Tell produces one message per cause, updates it only when something new is true, and closes it once when the condition is fixed. The Go dependency list supports this design. The module github.com/abahmed/kwatch declares go 1.27.2 and pulls in k8s.io/client-go, k8s.io/api and k8s.io/apimachinery at v0.37.1 for cluster access, plus discordgo and slack-go for chat delivery, mail.v2 for email, bbolt for local storage, and the Prometheus client for metrics.

The change timeline that remembers what kwatch missed

A diagnosis is only useful if it can point at the change that came before the failure, so kwatch records rollouts, configuration edits and image updates on a timeline with the identity of whoever made them. That is how a message can say the failure started after the 14:02 release of api:2.3 by alice, with the previous image api:2.2, and that only pods of the new revision fail. The same mechanism is what makes the root cause claim in the project's own example plausible rather than a guess about image tags. The project also states that it remembers changes made while kwatch itself was restarting, which matters because a controller restart during a bad release is exactly when a naive watcher would lose the one piece of context it needs. A separate recovery message closes the loop, for example a line saying the api in shop was healthy again after being down for eight minutes. The timeline is built from in-cluster events, and the project states that cluster data never leaves the cluster except as the alerts you configure.

Deploying kwatch: the docs point, the repo builds

The README links out for setup rather than embedding a copy-paste manifest, pointing to the getting started guide at kwatch.dev and to a one command entry point in its own table of contents. The repository itself carries a deploy/ directory, a Dockerfile, a Makefile and a cmd/ directory, so the pieces needed to build and place the binary are present in the tree. The Makefile defines a local gate called verify and a ci target that runs the same suite with the CI tooling checks and module hygiene added. Both run the tests once with the race detector and coverage and then again for the wall-clock latency and memory budgets:

make verify
make ci

The Dockerfile builds from a golang:1.27.2-alpine base with CGO disabled and cross-compiles from the native runner architecture, using TARGETOS and TARGETARCH supplied by buildx. Version metadata is injected at link time through ldflags pointing at github.com/abahmed/kwatch/internal/version, so a local build reports dev unless git describe finds a tag. The README gives no helm values and no namespace manifest, so the exact install command has to be taken from the getting started guide rather than reconstructed.

Quiet by default, with a hard ceiling on the storm

The design choice most likely to surprise you is what does not get sent. Successful rollouts, scaling operations, node drains, finished Jobs and ordinary pod restarts never notify anybody. Updates are sent only when something new is true about a cause, and the project states that a storm of 1,000 failing pods from a single cause produces at most three messages. That ceiling is a design decision rather than a configuration knob visible in the tree. Messages are formatted as short lines with resource names in bold, text taken from pod output in code, and the suggested command in a code block. Rich providers such as Slack, Discord, Telegram and Jira render that structure, while SMS, push and pager providers receive the same lines as plain text. A notification that names the cause and ends with a command is meant to be acted on directly, and the project notes that new team members can act on it without knowing the cluster. The example below is taken from the project's node memory incident, and the command in it is the actual one to run for that case:

kubectl describe node n1

The other recurring case, a rollout that broke the workload, points at the rollout state the same way:

kubectl rollout status deployment/api -n shop

What kwatch deliberately does not tell you

kwatch reports what is happening, not what could happen. Advice that would be useful but speculative, such as suggesting a single replica or a missing probe, is never listed in a message. That is a real constraint for anyone expecting a configuration auditor. The cause list is also specific: a bad release, a full node, an expired certificate, or a database outside the cluster. Dependencies reaching outside the cluster are named as suspects but are not inspected, so an unreachable external database produces a diagnosis that points outward and stops there. The read-only design sets another limit. kwatch can read your resources, including Gateway API objects, snapshots and your own CRDs, but it cannot change anything, so a message is always followed by a human running a command. The project also does not document rollback of its own deployment, nor how to migrate a configuration between kwatch versions, so both need checking before a change.

How this differs from a Prometheus alert rule

The alternative most teams already run is a Prometheus alert rule on crash loop counters or an on-call tool that sends one notification per Kubernetes event. The difference is in what the unit of notification is. A rule fires on a metric crossing a threshold, so a bad release still produces a flood of alerts, and the engineer must correlate them by hand. kwatch makes the correlation its main work, walking dependencies upstream and choosing the causes that explain the whole set, then emitting one message per cause with a command attached. The project also tracks a change timeline, which a threshold rule has no equivalent for, so the message can name a person and a time. The trade-off is that kwatch is an event-driven notifier rather than a queryable time series, so it does not answer what the error rate was over the last week, and the README does not describe any retention or history view beyond the bbolt storage dependency in go.mod.

Editorial conclusion

Adopt kwatch if your on-call noise comes from many pods failing for one reason and you want a message that names the cause and the command to run. Skip it if you need latency percentiles, resource saturation trends, or long-term SLO reporting, since it is built around crash and unhealth events rather than time series. Before deploying, check the licence file yourself, because GitHub classifies it as Other while the project badge shows ELv2, and confirm that the notifications configured in deploy/ point only at channels your team is allowed to receive cluster detail in.

Frequently asked questions

Official sources

1. abahmed/kwatch on GitHub
2. Issues
3. Project website
4. README
5. Releases