promxy makes many Prometheus shards look like one endpoint

The problem is a Prometheus that has no clustering

Promxy is a Prometheus proxy that makes many shards of Prometheus appear as a single API endpoint to the user, and the documentation says this significantly simplifies operations and use of Prometheus at scale when you have more than one Prometheus host.

The motivation section in the repository, which is written out separately in MOTIVATION.md, states the problem plainly. Prometheus itself provides no real HA or clustering support, so the best practice is to run multiple hosts with the same configuration. And Prometheus has no real built-in query federation, which means you end up with N sources in Grafana, which is confusing to Grafana users and has no support for aggregation across the sources.

So the cost of running redundant Prometheus is paid at the query layer, and every consumer of the data, a dashboard, an alert, a person with a browser tab, has to deal with N sources. That is the specific pain Promxy removes.

What it does about it is two things. It merges the data from the duplicate hosts, so if there is a gap in one, Promxy fills it with the other. And it provides a single datasource for all PromQL queries, which means Grafana can have one source and you can have globally aggregated PromQL queries.

And the constraint that makes it deployable is stated in the same breath: Promxy delivers this unified access endpoint without requiring any sidecars, custom builds, or other changes to your Prometheus infrastructure. Another entry in the FAQ is even more direct. What changes are required to my Prometheus infrastructure for Promxy? None. Promxy is simply an aggregating proxy that sends requests to Prometheus.

That is the property to hold on to. In a monitoring estate, the component that is hardest to change is the thing that is actually scraping your targets, and Promxy deliberately does not ask you to touch it.

A ServerGroup is a set of identical hosts, and it is the unit of merging

The central concept in the documentation is the ServerGroup, and the FAQ defines it precisely.

A ServerGroup is a set of Prometheus hosts configured the same. The documentation notes that this is a common best practice for Prometheus infrastructure, since Prometheus itself does not support any HA or clustering, and it is what allows Promxy to merge data from multiple hosts in the group, with all of them until one becomes a priority.

So the merging happens inside a group, and groups are how you describe redundancy. If you have two Prometheus servers scraping the same targets with the same config, they are one ServerGroup, and a query against that group sees the union of their data with the holes filled from whichever host has them. The promotion of one host to a priority is how you handle a planned maintenance window, and the link in the documentation points at the tracking issue for that behaviour.

The reason to model it this way rather than simply listing hosts is that a merge across hosts that are not configured identically produces nonsense. Two servers scraping different targets, or the same targets with different relabel rules, do not have a union, they have a conflict. The ServerGroup boundary is where the assumption is stated, and stating it is what makes the rest of the system safe.

The second thing a ServerGroup gives you is the failure semantics, which are covered in more detail further on. And there is one more property worth noting. Promxy can be a downstream of Promxy, so it aggregates other Prometheus API endpoints together and you can definitely layer it. Similarly you can mix Prometheus API endpoints, for example you could have Prometheus, Promxy and VictoriaMetrics all as downstreams of a Promxy host, since they all have Prometheus compatible APIs.

That last point is the escape hatch. Promxy does not require its downstreams to be Prometheus, only that they speak the compatible API.

Building it and pointing it at your servers

Release binaries are available on the releases page, so the quick path is to download one. If you want to hack on Promxy or run your own build, the documentation gives the sequence, and it starts inside the command directory because the build tags and the vendored dependencies matter:

git clone [email protected]:jacksontj/promxy.git
cd promxy/cmd/promxy && go build -mod=vendor -tags netgo,builtinassets

The -mod=vendor flag builds against the vendor directory that is committed to the repository, and the netgo and builtinassets build tags are what the project's own build uses, so the binary you produce matches the released one.

An example configuration file is available in the repository, at cmd/promxy/config.yaml, and it is where the server groups and the downstreams are declared. Once you have modified it, running the proxy is one command:

./promxy --config=config.yaml

The Makefile shows the full set of targets a maintainer uses. There is fmt and imports for formatting, static-check running staticcheck over the tree, and test running the suite with the race detector, the vendor mode and the same netgo and builtinassets tags. The release target calls a build.bash script for each of the two commands, and the vendor target runs go mod tidy followed by go mod vendor.

The container image follows the same pattern. The Dockerfile has a builder stage on golang:1.25-alpine3.21 that compiles two binaries with CGO_ENABLED=0, Promxy itself and a remote_write_exporter, and a runtime stage on alpine:3.21.3 that copies them in, exposes port 8082, runs as the nobody user and sets the entrypoint to /bin/promxy. The image is published on Quay under jacksontj/promxy.

Scatter-gather to every group, on every query

How Promxy knows which server to route a query to has a simple answer in the documentation: Promxy currently does a complete scatter-gather to all configured server groups.

That is the mechanism, and it has a known cost. Every query fans out to every downstream in every group, and the response is assembled from all of them. There are plans to reduce scatter-gather queries, tracked in the repository, but the documentation is candid that in practice the current scatter-gather always implementation has not been a bottleneck.

The performance target is stated too, and it is a useful one to hold in mind when you read a benchmark from someone else. Promxy's goal is to be the same performance as the slowest Prometheus server it has to talk to. If you have a query that is significantly slower through Promxy than on Prometheus direct, the documentation asks you to open an issue.

There is one version specific caveat. If you are running Prometheus older than 2.2 you may notice slow performance when running queries that access large amounts of data, and the reason given is inefficient JSON marshaling in Prometheus itself. The documented workaround is to configure Promxy to use the remote_read API instead.

That is worth understanding as a design consequence rather than a bug. Promxy sits in the request path of every dashboard and every alert evaluation, so its latency is a floor under the whole monitoring stack, and the only way to reduce it is to reduce the work it asks the backends to do.

Recording rules work, but they need somewhere to write

The FAQ section on alerting and recording rules contains one of the most useful facts in the documentation, and it is the kind of fact that changes a deployment plan.

Promxy is simply an aggregating proxy in front of your Prometheus infrastructure, and as such you can use Promxy to create alerting and recording rules which will execute across your entire Prometheus infrastructure. The example given is knowing whether the global error rate was under 10 percent, which would be impossible on the individual Prometheus hosts, without federation or re-scraping, but is trivial in Promxy.

So a single Promxy becomes the place where cross-fleet rules are defined and evaluated. That is a real capability, and it is not something a plain Prometheus fleet gives you.

The catch is in the note below it. Recording rules in regular Prometheus write to their local TSDB, and Promxy has no local TSDB. So if you want to use recording rules, or see the metrics from alerting rules, a remote_write endpoint must be defined in the Promxy configuration, and that is where it will send those metrics.

In other words, the evaluation moves to Promxy but the storage does not, and you have to point Promxy at a destination yourself. If you are evaluating Promxy for a fleet that already has recording rules producing the metrics your dashboards read, this is the question to ask first, because the answer determines whether you need a remote write receiver in the design at all.

It returns an error rather than an inaccurate value

The behaviour when things break is worth a paragraph of its own, because it is the design decision that distinguishes Promxy from a naive merge.

The default behaviour in the event of a ServerGroup being down is to return an error. The reasoning is given explicitly. If all nodes in a ServerGroup are down the resulting data can be inaccurate, with missing data and so on, and so the project would rather by default return an error than an inaccurate value, since alerting and so on might rely on it and they do not want to hide a problem.

That is the right default for a monitoring system. A silently shortened series is worse than a failed query, because a failed query is noticed and a shortened series is graphed.

If you would rather have some or all ServerGroups be optional, meaning the errors are ignored and the response is served anyway, you can do that with the ignore_error option on the ServerGroup in the configuration file. That is the escape hatch for the case where a group covers a less critical estate and you would rather have partial data than an error page.

So the knob is per group, not global, which is the right granularity. The default protects you and the option lets you make an informed, local decision about which parts of your estate can tolerate a hole.

A forked Prometheus, a vendored tree, and a 0.0.x version line

Promxy is currently using a fork based on Prometheus 2.24, and the documentation says this version is not supremely important, but it is relevant for PromQL features such as subqueries and for service discovery config options.

The fork is visible in the build files. The go.mod pins github.com/prometheus/prometheus at a pseudo version from July 2021, and the Makefile has a dedicated target, update-prom-fork, that rewrites the module requirement to point at jacksontj/prometheus at the fork tag v0.2.37.5-fork and then re-vendors. So the fork is a deliberate, mechanical part of the upgrade path rather than something hidden in a branch.

The dependencies tell you the rest of the shape. The Prometheus client libraries for the exposition format, the common package for shared model types, the exporter toolkit for the HTTP surface, go-kit and logrus for logging, jessevdk/go-flags for the command line, httprouter for routing, yaml.v2 for the configuration, and the Azure autorest libraries, which are there for authenticating against managed Prometheus installations.

The compatibility question is answered separately in the FAQ. Promxy uses the /v1 API of Prometheus under the hood, so it simply requires that API to be present, and it has been used with Prometheus as early as 1.7 and as recent as 2.13.

The release numbering is the thing to note. The versions are v0.0.95, v0.0.96 and v0.0.97, published on 2026-06-26, 2026-07-06 and 2026-09-04, and the repository is not archived with the last push on 2026-10-12. A 0.0.x line means the project is not promising a stable API surface, so treat a Promxy upgrade as a deliberate event with a config review attached, not as a routine dependency bump. The licence is MIT, in the LICENSE file at the root of the repository.

Editorial conclusion

promxy is worth adopting if you run more than one Prometheus server with the same configuration, because it removes the Grafana datasource sprawl and turns a per host question into a global one without touching the servers themselves. It is the wrong choice for a single Prometheus installation, and it is the wrong choice if your monitoring estate is heterogeneous, since the merge logic assumes the hosts in a ServerGroup are configured the same. Before you deploy it, read the error handling section of the documentation carefully, because the default of failing the whole query when a ServerGroup is down is a deliberate choice you may want to change with the ignore_error option, and note that the current release is v0.0.97 from 2026-09-04 while the repository was pushed to on 2026-10-12.

Frequently asked questions

Official sources

1. Issues
2. jacksontj/promxy on GitHub
3. License: MIT
4. README
5. Releases