RelayPulse: probing LLM relay services with real tokens instead of HTTP pings

The failure mode RelayPulse was built to catch

Uptime Kuma and similar tools answer one question: did the endpoint return a successful HTTP response. In an LLM relay, that answer is frequently useless. The README states the problem directly: an HTTP 200 that carries an empty body or an error code is a common pattern in this space, and it calls this a fake-alive condition. A relay can accept your request, route it, and hand back a well-formed envelope with nothing usable inside.

The project's response is to stop treating the socket as the unit of measurement. RelayPulse sends scheduled API requests that consume real tokens and then validates the response content. The README's phrasing is that only when the LLM actually produces output does it count as available. That single design decision drives everything else in the system, including its cost profile and its storage requirements.

The audience is narrow and identifiable. The README lists self-hosted or purchased LLM relay services, multi-vendor quality comparison, and monitoring of external API dependencies. If you pay a relay provider and your application breaks while their status page stays green, you are the intended user.

Probe scheduling, token budget, and what validation actually means

A probe is a real completion request. The README gives the cost shape: max_tokens is set to 1, so each probe is roughly 20 input tokens plus 1 output token. At the default cadence of once per minute, that works out to about 30,000 tokens per day per service. This is the number to check against your own provider pricing before you scale the monitor list, because the cost scales linearly with the number of monitored services and inversely with the interval.

Monitoring entries are declared under a monitors key in config.yaml. Each entry carries provider, service, category, sponsor, sponsor_level, base_url, template, and api_key. The template field points at a file under a templates directory, and the README notes that model and request_model are preset by the template rather than written in the monitor block. That indirection is the part worth understanding before you write your first config: the probe payload comes from the template, not from the monitor entry, so a wrong template produces a probe that tests something other than what you intended.

The README also mentions parent-child inheritance for multi-model monitoring, documented in docs/user/config.md. The supplied material does not describe the inheritance rules in detail, so treat that document as the source rather than guessing at the merge behaviour.

Probes write results into a sliding 24-hour window, and the dashboard renders availability as a heatmap over 24h, 7d, and 30d. A slow_latency key sets the threshold above which a request is classified as slow.

Storage choice and the hot/cold board model

Two backends are supported. SQLite is the default and the README positions it for single-machine deployments and development, with zero configuration. PostgreSQL is positioned for Kubernetes and multi-replica deployments, where high availability and horizontal scaling matter. The docker compose file exposes both paths: docker compose up -d monitor for the SQLite variant, and docker compose up -d postgres monitor-pg for the PostgreSQL one. If you are running more than one replica, SQLite is not a candidate, and the README is explicit about that split rather than presenting SQLite as universally sufficient.

The board concept deserves attention because it appears in two different shapes. In the per-monitor results returned by /api/status, board is one of hot, secondary, or cold. In the StatusQuery response, board is described as a folded activity result at the channel level with only two values. A channel is hot if any active monitor under it is configured as hot or secondary; it is cold only when every non-disabled monitor beneath it is cold. The README flags this difference itself, which is a good sign that the distinction has bitten someone. If you are building alerting on top of the query API, read the channel-level field, not the per-monitor one.

Getting it running: Docker path and the local build trap

The documented Docker flow is four commands. Download docker-compose.yaml and config.yaml.example with curl from the raw GitHub URLs, create a config directory and copy the example into config/config.yaml, edit in your API key, then run docker compose up -d and open http://localhost:8080. The README also notes that the API key can come from an environment variable instead, using the pattern MONITOR_88CODE_CC_API_KEY, where the middle segments derive from the provider and service names.

The local development path has a step that will cost you time if you skip it. You must run ./scripts/setup-dev.sh before the first build and after any frontend change. The script builds the frontend and copies frontend/dist into internal/api/frontend/dist, which is the path the Go embed directive expects. The README states that symlinks are not supported there, and that skipping the script produces the error pattern frontend/dist: no matching files found. After that, go mod tidy and make dev, or go run cmd/server/main.go directly.

Configuration reloads without a restart, handled by fsnotify. That matters operationally: adding a monitor or rotating a key does not require a container restart, so a bad config edit shows up as a reload failure rather than a crash loop.

The HTTP surface is small and readable. /health and /api/version are unauthenticated. /api/status accepts period and board query parameters. /api/status/query handles single lookups by provider, service, and channel, or up to 20 compact queries in the q=provider/service/channel form. /api/status/batch takes a POST body with up to 50 queries. The event endpoints, /api/events and /api/events/latest, require a Bearer token configured through events.api_token or the EVENTS_API_TOKEN environment variable.

Where the sliding window and the token spend become your problem

The API uses a sliding window, and the README spells out the consequence rather than hiding it. A request for period=24h returns data from the current moment back 24 hours, so the time bucket boundaries shift with every request. Provider rankings always reflect the most recent 24 hours. The README's own recommendation is to sample at a fixed cadence, such as on the hour, if you need stable data points for integration. That is an honest disclosure of a design constraint, and it is also a real constraint: you cannot treat two /api/status responses as comparable snapshots unless you control when you ask.

The cost model is the second constraint. Real tokens mean real money, and the README's own figure of roughly 30,000 tokens per day per service is per service, not per installation. A deployment watching twenty relays at one-minute intervals is spending twenty times that. The interval key is the lever, and there is no discussed mechanism for per-monitor intervals in the supplied material, so the cadence appears to be global.

The third limitation is scope. RelayPulse measures availability and latency from wherever it runs. The README's disclaimer notes that results can be affected by network variation, geography, and cache delay, and that the output is a technical probe result rather than a judgement about a provider's compliance or commercial standing. If your users are in a region your monitor is not, the monitor is measuring a path your users do not take. That is a limitation of the approach, not a bug, and it means a single RelayPulse instance is not a substitute for probing from multiple vantage points.

How it differs from a generic uptime monitor

The obvious comparison is Uptime Kuma, which the README names directly. The difference is not in the dashboard or the alerting. It is in what constitutes a successful check. Uptime Kuma asks whether the endpoint responded; RelayPulse asks whether the model produced output, and it pays tokens to find out. That changes the failure taxonomy you can build: a relay that returns 200 with an empty body is invisible to a connectivity monitor and visible here.

The trade is cost and complexity. A connectivity monitor costs nothing per check and needs no API key. RelayPulse needs credentials for every monitored service, spends tokens continuously, and requires you to maintain a config file with templates. For a service where an HTTP 200 genuinely does mean the service is working, that overhead buys you nothing.

A second comparison point is building the probe yourself. A cron job with curl against the chat completions endpoint and a JSON field check is not hard to write, and it is the honest baseline. What RelayPulse adds over that is the storage layer, the heatmap aggregation across three time windows, the hot/cold board folding, the config reload, and the query and batch APIs. If you only need one endpoint checked and you already have alerting, the cron job is the smaller system. If you need history, per-channel status queries, and a dashboard that non-engineers can read, the aggregation is the part you would otherwise rebuild.

Licence, maintenance cadence, and upgrade cost

The project is MIT licensed, which permits commercial use, modification, and redistribution provided the copyright notice and licence text are retained. The README's disclaimer adds a data-nature caveat: displayed status comes from automated technical probing, may contain errors from network variation or cache delay, and is not a statement about any provider's compliance, legality, qualifications, or commercial reputation. If you plan to publish RelayPulse output, that framing is worth carrying over. Nothing here is legal advice, and the disclaimer text is the author's position rather than a reviewed legal instrument.

On cadence: the repository is not archived, and the release history shows v2.87.0, v2.88.0, and v2.89.0 all dated 2026-09-10, with the last push on the same day. Three releases in one day is a fast-moving main branch, and it suggests the version numbers should not be read as stability guarantees. For an operator, that means pinning a specific image tag rather than tracking latest, and reading release notes before moving. The README points at GitHub Releases as the changelog, with no separate CHANGELOG file mentioned.

Upgrade cost is largely configuration-shaped. Because monitors reference templates by name and the template supplies the model and request_model, a change to a template file affects every monitor that references it, and the config reload via fsnotify means that change takes effect without a restart. That is convenient and also means a template edit propagates immediately. The documentation set is split across QUICKSTART.md, docs/user/config.md, docs/user/methodology.md, docs/user/docker.md, docs/user/deploy-postgres.md, and notifier/README.md, so an upgrade that touches probe semantics is best checked against methodology.md rather than the README alone.

Editorial conclusion

Adopt RelayPulse if you run or resell LLM relay endpoints and need to distinguish a live model from a live HTTP socket, and if you are comfortable running a Go service plus SQLite or PostgreSQL yourself. Do not adopt it if you need a hosted, zero-ops monitor, or if you cannot budget the token spend for continuous probing. Before deploying, verify three things against your own setup: that the token cost per day per service matches what you expect at your chosen interval, that your config.yaml monitor entries resolve through the templates directory without errors, and that the sliding-window semantics of /api/status fit whatever you plan to integrate with it.

Official sources

1. License: MIT
2. prehisle/relay-pulse on GitHub
3. Project website
4. README
5. Releases