Prometheus Pushgateway: a metrics cache for batch and ephemeral jobs

The gap Pushgateway fills: jobs that die before the scrape

Prometheus pulls metrics on a schedule. A cron job, a database migration, a nightly report generator, or a CI step may finish in seconds and never be running when the scrape interval comes around. Those jobs still have numbers worth keeping: rows processed, duration, exit status, retry count. The Pushgateway exists so those jobs can push their metrics to a long-running HTTP endpoint, and Prometheus then scrapes that endpoint like any other target. The README states the intent plainly: ephemeral and batch jobs "may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway."

The audience is narrow by design. Service-level metrics from batch work are the target. The README says machine-level metrics belong in the textfile collector of the Node exporter instead, which writes files that node_exporter reads on the host, with no network hop and no shared server to operate. If your workload runs long enough to be scraped directly, you do not need this component at all.

How the metrics cache and grouping key actually work

Pushed metrics are stored in groups. A group is identified by a grouping key, which is any number of labels, and the first label must be job. The URL path encodes that key: /metrics/job/some_job creates or replaces the group for job="some_job", while /metrics/job/some_job/instance/some_instance addresses a separate group. Those two groups are distinct even though they share a job value, which matters when you delete one and expect the other to disappear.

The README is direct about what the component is not. It is "not an aggregator or distributed counter but rather a metrics cache." It does not have statsd-like semantics. The metrics you push are exactly the metrics you would expose for scraping in a program that ran forever. That framing explains the storage model: the Pushgateway keeps the last payload per group and serves it back to Prometheus on every scrape, so a counter that was pushed once keeps being reported at the same value until something overwrites or deletes it. The README points readers who need distributed counting toward statsd with the Prometheus statsd exporter, or toward prom-aggregation-gateway, and notes that a native solution might come from the Prometheus project one day.

Internally the repository separates concerns into handler/, storage/, api/, and asset/ directories, with main.go wiring flags and the HTTP surface. The binary embeds its web assets, which is why the Makefile has an assets target that runs go generate in the asset directory. None of this is configuration surface you touch as an operator, but it tells you the project is a single Go binary with an embedded UI and a pluggable storage layer rather than a service with external dependencies.

Installing Pushgateway and pushing your first metric

The README gives two paths. Download a binary release for your platform from the release page and unpack the tarball, or compile from source with a working Go setup and the provided Makefile by typing make. For the most basic setup, just start the binary. The listen address is controlled by --web.listen-address, for example "0.0.0.0:9091" or ":9091". By default the Pushgateway does not persist metrics; --persistence.file names a file where pushed metrics are written so they survive restarts.

The Docker path is shorter. The image is prom/pushgateway, and the README shows pulling it and running it with port 9091 published:

docker pull prom/pushgateway

docker run -d -p 9091:9091 prom/pushgateway

The Dockerfile in the repository confirms the exposed port is 9091, that the process runs as user 65534 (nobody), and that the working directory is /pushgateway. Expect the container to come up listening on that port with no persistence configured, since no flag is passed.

Pushing a metric needs no special client. The README notes that the Prometheus text protocol makes this easy enough that no separate CLI is provided, and suggests curl. A single sample goes to a group keyed only by job:

echo "some_metric 3.14" | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job

Because no type information was sent, the README says some_metric will be of type untyped. For anything more complex, include TYPE and HELP lines, which the README calls optional but strongly encouraged:

cat <<EOF | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
# TYPE some_metric counter
some_metric{label="val1"} 42
# TYPE another_metric gauge
# HELP another_metric Just an example.
another_metric 2398.283
EOF

After that, the group appears in the web interface, which the README describes as the easy way to inspect groups. Deleting a group is a DELETE against the same path:

curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

Wiping everything requires the admin API, which is off unless you enable it with --web.enable-admin-api, and the call is a PUT rather than a DELETE:

curl -X PUT http://pushgateway.example.org:9091/api/v1/admin/wipe

One protocol detail catches people writing their own HTTP client: in the text protocol, each line must end with a line-feed. Ending a line with CR, CRLF, or the end of the packet produces a protocol error. Windows users get a Powershell example in the README, where the metric text is enclosed in double quotes and a backtick precedes the n at the end of the body.

The labels trap: honor_labels and what it changes

The Pushgateway must be configured as a scrape target in Prometheus, and the README says you should always set honor_labels: true in that scrape config, referring to its own section on the job and instance labels for the explanation. The reason is structural. When a job pushes a metric, the labels in the pushed payload are the ones the job chose, and the grouping key in the URL is also part of the identity. Without honor_labels, Prometheus resolves conflicts between the labels it would attach at scrape time and the labels already present in the exposed series by renaming, which produces series that no longer carry the job and instance values the pushing job intended. Setting honor_labels: true keeps the pushed labels authoritative.

This is the kind of detail that turns a working demo into a confusing production incident, and it is a scrape-side setting, not a Pushgateway flag. Nothing in the Pushgateway will warn you that you forgot it.

No TTL, no event log, and stale counters that look alive

The README states that the project decided not to implement a timeout or TTL for pushed metrics, on the grounds that almost all proposed use cases turned out to be anti-patterns. The practical consequence is that a group pushed once stays visible indefinitely. If a batch job stops running, its last values keep being served to Prometheus, and a counter that never moves looks like a stalled service rather than a retired job. Detecting that is left to you, through a timestamp metric or a deletion step in the job itself.

Two other boundaries are stated outright. The Pushgateway is not an event store; the README says tracking release events has to happen with an event-logging framework, even though Prometheus can serve as a data source for Grafana annotations. And it is not a way to make Prometheus push-based in general. The component is a cache in front of a pull system, and the README links to the Prometheus documentation page on when to use the Pushgateway for the fuller argument.

A quieter limitation sits in the grouping key itself. Because groups are separate per key, deleting /metrics/job/some_job does not remove metrics under /metrics/job/some_job/instance/some_instance, even when the job label matches. Any cleanup script has to enumerate the keys it means to remove.

Pushgateway versus remote write, and other alternatives

The most common comparison is with Prometheus remote write, and the difference is architectural rather than cosmetic. Remote write is a path out of Prometheus: a running Prometheus sends samples it has already scraped to another storage system. The Pushgateway is a path into Prometheus: something that is not a scrape target yet deposits metrics at an HTTP endpoint, and Prometheus scrapes that endpoint on its normal schedule. Remote write does not help a cron job that never gets scraped, and the Pushgateway does not ship your existing time series anywhere. If your problem is long-term storage or a second backend, remote write is the relevant tool and the Pushgateway is not.

For distributed counting, the README names two options directly: statsd with the Prometheus statsd exporter, or prom-aggregation-gateway. Both accept increments from many sources and aggregate them, which is precisely the behaviour the Pushgateway refuses to provide. Choosing the Pushgateway for a counter that many workers increment independently will produce last-write-wins behaviour per group, not a sum.

For machine-level metrics, the Node exporter textfile collector is the alternative the README recommends, and the difference is local files versus a network service: the collector reads files on the host it is already running on, so there is no extra server to run, no grouping key to manage, and no shared endpoint where one bad push can overwrite another team's data.

Maintenance, licensing and upgrade cost

The repository is not archived, and the last push was on 2026-09-01. Releases are infrequent and small: v1.11.3 on 2026-05-27, v1.11.2 on 2025-10-30, and v1.11.1 on 2025-04-09. That cadence fits a component whose behaviour is deliberately frozen; the non-goals section reads as a list of features that will not arrive. Upgrades are therefore mostly about the Go toolchain and dependency bumps, not about migrating pushed data. The persistence file is the one piece of state to think about during an upgrade, since it is the only thing that carries across a restart.

The licence is Apache-2.0, and the LICENSE file is at the top level of the repository alongside NOTICE. Apache-2.0 permits commercial use and modification and includes an explicit patent grant; it also requires that you keep the licence and notice files and state significant changes. That is a summary of the licence text, not legal advice, and if you redistribute the binary inside a product you should read LICENSE and NOTICE yourself. The container image is published on Quay as prom/pushgateway and on Docker Hub as prom/pushgateway, per the badges and the README's Docker section.

Operationally, the cost is the endpoint itself. It is a single Go binary with no external dependencies, which keeps the footprint small, but it is also a shared mutable surface: any client that can reach port 9091 can overwrite a group, and the admin wipe endpoint exists behind --web.enable-admin-api. The README does not document authentication or rollback for the admin API, so network placement is the control you have.

Editorial conclusion

Adopt the Pushgateway when a job finishes before Prometheus can scrape it and you accept that its last pushed values stay visible until they are overwritten or deleted. Do not adopt it for distributed counting, event logging, or machine-level metrics, and do not expect a TTL, which the project decided not to implement. Before putting it in production, verify that your Prometheus scrape config sets honor_labels: true, decide whether you need --persistence.file so pushed data survives a restart, and confirm that whatever deletes stale groups is wired up, since the README documents no automatic expiry.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. prometheus/pushgateway on GitHub
4. README
5. Releases