network_exporter: a Prometheus exporter for ICMP, MTR, TCP and HTTP probes

Four probe types behind one exporter

network_exporter gathers either ICMP, MTR, TCP Port or HTTP Get stats and exports them via HTTP for Prometheus consumption. It is written in Go, licensed under Apache 2.0, and the module is github.com/syepes/network_exporter with go 1.27.1 in the go.mod.

The four probe types are what distinguish it from the usual single-purpose ping exporter, and they are not variations on a theme. ICMP and MTR are both ICMP based but they answer different questions, one asks whether a host answers and the other asks what the path to it looks like. TCP Port is a connect test, and HTTP Get is an application level check that returns a status and a duration.

The dependency list tells you how each is implemented. The Prometheus client library and the exporter toolkit are the obvious ones. github.com/felixge/fgprof is a profiling library, which explains the readme's promise of high-performance optimizations, and github.com/mdlayher/socket and github.com/mdlayher/vsock appear as indirect dependencies, which is the machinery behind the raw socket work that ICMP and MTR need. github.com/alecthomas/kingpin/v2 is the command line parser and github.com/creasty/defaults fills in default configuration values, so a target that omits a field gets a sane one without you writing it out.

Because ICMP needs raw sockets, the container has to be privileged in a specific way, and the Dockerfile in the repository shows how. It builds from golang:alpine, produces a static binary with CGO disabled, and the final alpine:latest image installs ca-certificates, libcap and iproute2, then runs setcap with cap_net_raw,cap_net_admin+eip on the binary. Giving the binary file capabilities rather than running the container as root is the right way to grant raw socket access, and it exposes port 9427, which is the conventional port for a node exporter style service.

The Makefile builds multi-architecture images with buildx for linux/amd64, linux/arm/v7 and linux/arm64/v8, which is what you need if your probes run on Raspberry Pi class hardware or on ARM servers.

Reloading, DNS and the operational feature list

The feature list in the readme is short but each entry solves a specific operational annoyance.

IPv4 and IPv6 support comes first, and it matters for an exporter because a network monitoring tool that cannot reach a host on one of them is a tool with a blind spot you will discover in production.

Configuration reloading has three mechanisms. It can happen on a refresh interval, it can be triggered by SIGHUP on Linux and macOS, and it can be triggered by a POST to the /-/reload endpoint on any platform including Windows. The third one is the interesting one, because SIGHUP does not exist on Windows and a container orchestrator that reconfigures a mounted ConfigMap wants a way to say so without restarting the process.

Targets can be added or removed dynamically without affecting the tests that are currently running, which is what makes the reload path safe rather than disruptive.

The rest of the list is the kind of detail that shows the author has run this in anger. The target IP is updated automatically when DNS resolution changes, so you point a target at a hostname and the exporter follows it. Targets can be executed on all hosts or on a list of specified ones through a probe key, which is how you handle a network where the exporter runs in several places and only one of them has the right route. You can attach extra labels when defining a target. The logging level and format are configurable, text or JSON, which is what you want when the logs are shipped to a collector. There is a configurable DNS server and a configurable source IP per target, with the caveat that the IP has to be configured on one of the instance's interfaces, so the exporter can originate a probe from a particular address.

That last one is the feature that matters most in practice. A host with several addresses, or a monitoring host behind a management interface, often needs to probe from a specific source address, and the source IP field is how you do that without running four separate exporters.

There is also startup jitter to prevent a thundering herd, which is the detail that shows careful engineering. Without it, every exporter in a fleet starts its probes at the same moment and the first scrape interval of the day carries a spike that has nothing to do with traffic.

Concurrency is per target, not global

This is the part of the readme that anyone operating this at scale needs to read twice, and the second reading changes how you set the flag.

In steady state each target runs exactly one probe per its configured interval. The --max-concurrent-jobs parameter caps how many probe cycles may overlap for a single target, and overlap only happens when a probe takes longer than that target's interval to finish. Once the ceiling is reached the per target run loop applies back-pressure until an in-flight probe completes.

So the flag is a back-pressure valve, not a throughput dial. Raising it does not make probes run faster. It only lets a slow target keep sampling instead of queueing behind itself.

The readme is explicit that this is a per-target limit, not a total-system one, and that there is currently no global concurrency cap. That is the sentence to remember. The worst-case ceiling is targets multiplied by max-concurrent-jobs, and it is reached only if every probe consistently outruns its interval. Under normal conditions the real load is close to one in-flight probe per target.

The tradeoff is stated in both directions. A higher per-target overlap allows more concurrent samples when a probe is slower than its interval, at the cost of a higher worst-case resource ceiling. A lower per-target overlap bounds worst-case resource use more tightly, at the cost of back-pressuring a target whose probes are slower than its interval.

The guidance that follows is that leaving the setting at the default of 3 is right for most deployments, and that you should change it only when the internal metrics show probes routinely overlapping. The suggested values scale down as the target count grows:

# Default: up to 3 overlapping probe cycles per target
./network_exporter --max-concurrent-jobs=3

# Small deployments (<100 targets): allow more overlap per target if probes can exceed their interval
./network_exporter --max-concurrent-jobs=5

# Medium deployments (100-1000 targets): keep the default
./network_exporter --max-concurrent-jobs=3

# Large deployments (1000-5000 targets): keep the default and watch the internal metrics
./network_exporter --max-concurrent-jobs=3

# Very large deployments (>5000 targets): lower the per-target ceiling to bound worst-case resource use
./network_exporter --max-concurrent-jobs=2

The shape of that advice is the useful part. Smaller deployments can afford more overlap because the absolute numbers are small. Large deployments tighten the ceiling because the multiplication is what hurts.

Limits that differ by an order of magnitude

The recommended per-type target limits are the second piece of information you need before sizing a deployment, and they are not close together.

For PING, the recommended limit is 10,000 to 15,000 targets, limited by the ICMP ID counter with roughly 65,500 concurrent operations available. That last number is the important detail: ICMP packets carry an identifier, and the operating system has a limited pool of them, so there is a hard ceiling on how many ICMP probes can be in flight at once regardless of how much memory or file descriptors you have.

For MTR the recommended limit drops to 1,000 to 1,500 targets, because MTR uses multiple ICMP IDs per operation. A traceroute is not one probe, it is a sequence of probes with increasing TTLs, so it consumes the same scarce resource several times over. If you plan to run MTR against thousands of hosts, that is the constraint that will bite first.

TCP gets the highest range, 15,000 to 25,000 targets, helped by short-TTL DNS caching through conf.nameserver_cache_ttl, which reduces resolver load. That setting is worth knowing about, because at twenty thousand targets an uncached resolver is a very large amount of traffic to your own DNS server for no benefit.

HTTPGet comes in at 10,000 to 15,000 targets, bounded by per-probe connection setup and file descriptors, with the note that cross-probe keep-alive reuse is not yet enabled. That is a specific limitation rather than a vague one. Every HTTP probe opens a fresh connection, so the file descriptor cost per target is higher, and the readme tells you that connection reuse across probes is a thing that does not exist yet.

The practical consequence is that a deployment mixing probe types does not have one limit. A fleet with a thousand MTR targets and twenty thousand TCP targets is bounded by the MTR count, and the concurrency setting has to be chosen with the MTR behaviour in mind even though the TCP probes dominate the target count.

Tune from the metrics, not from a guess

The readme goes out of its way to tell you how to observe the scheduler, and that is unusual and welcome.

The exporter publishes internal network_exporter_ metrics that let you tune concurrency from measured behavior, and the bundled Grafana dashboard visualizes all of them in an Internal Metrics row.

network_exporter_probe_duration_seconds is compared against the configured interval. When the p95 duration approaches or exceeds a target type's interval, probes of that type will start to overlap, which is the precise condition under which the concurrency ceiling starts to matter. network_exporter_probe_inflight is the live number of overlapping probes per type, which you can compare directly against --max-concurrent-jobs.

network_exporter_probe_queue_wait_seconds measures how long a scheduled probe waited for a slot, so sustained non-zero values mean probes are hitting the per-target ceiling and are being back-pressured. That is the metric that tells you whether the ceiling is set too low, because a non-zero wait is a target whose sampling has been delayed.

The expression time() minus network_exporter_probe_last_completion_timestamp_seconds gives the age of the last completion, and a value that keeps climbing indicates the scheduler has stalled for that type. This is the one to alert on, because a stalled scheduler produces stale data rather than an error, and stale metrics that look healthy are the failure mode that monitoring tools are supposed to prevent.

network_exporter_collector_scrape_duration_seconds measures how long each /metrics scrape takes, and that is the signal to watch as the number of targets grows, since a scrape that takes too long starts to look like a Prometheus target down.

One metric is explicitly not useful yet. network_exporter_probe_skipped_total is currently always zero and is reserved for a future global scheduler, so it is not a live tuning signal. Mentioning it and saying so is better than shipping a metric that always reads zero and letting people wonder.

Resource ceilings and the file descriptor maths

The resource requirements section gives rough guidance that is short enough to be useful.

Memory is around 50 to 100 megabytes as a baseline plus a few kilobytes per target. CPU is mostly I/O bound, which follows directly from the nature of the work: waiting on a ping reply is waiting, not computing.

The file descriptor requirement is the one with a formula in it. Set ulimit -n to at least the product of targets and max-concurrent-jobs, plus 1000, which bounds the worst case where every target's probes overlap up to the ceiling. The extra thousand is the buffer for the exporter's own listeners, the scrape endpoint and the standard descriptors.

The examples are worked through for two sizes:

# Worst-case file descriptors: 5,000 targets × 2 + buffer
ulimit -n 20000

# Lower the per-target overlap ceiling for large scale
./network_exporter --max-concurrent-jobs=2

And the larger case uses 40,000 descriptors for 15,000 targets at a ceiling of two. The numbers are simple arithmetic, which is the appeal. There is no mysterious factor, just targets times the ceiling plus a thousand, and the readme gives you the worked example for the size you are most likely to run.

The Dockerfile makes one more operational point indirectly. The final image installs iproute2 alongside ca-certificates and libcap, and iproute2 is the package that carries the ping and traceroute binaries. Their presence suggests the exporter shells out or otherwise depends on the traditional network utilities for some of its probe paths, which is worth knowing if you are building a minimal image of your own, because skipping that package may cost you the MTR or TCP path rather than just the ping one.

The configuration itself is YAML, since gopkg.in/yaml.v3 is in the dependency list, and the repository ships two example configurations at the top level: network_exporter.yml and network_exporter_web.yml. The separate web variant is a good hint at the shape of a deployment, since the web configuration is presumably the one that exposes the exporter's own web interface alongside the metrics endpoint.

Editorial conclusion

Adopt network_exporter if you need network reachability as first class Prometheus metrics across many targets, especially when a plain ping exporter would not give you MTR path data or per target source addresses. Do not adopt it expecting a single global concurrency knob, because the concurrency control is per target and there is currently no global cap, so a slow probe type can still stack up. Before you commit, size ulimit -n using the worst case formula in the readme, and set the per type limits rather than the global default, since the recommended limits differ by an order of magnitude between MTR and TCP.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. syepes/network_exporter on GitHub