eMQTT-Bench: an Erlang benchmark tool for MQTT connections and subscriptions

Two commands: connections and subscriptions

eMQTT-Bench is described as a lightweight MQTT v5.0 benchmark tool written in Erlang, and the README says you can download pre-built binary packages from the project's releases page.

The tool is organised around two subcommands. The conn command establishes connections to a broker, and the sub command covers the publish and subscribe side. The usage output for conn is long, and the options that matter for a first run are the ones that shape the load.

The example in the README creates 50K concurrent connections at an arrival rate of 100 per second:

./emqtt_bench conn -c 50000 -i 10

-c is the max count of clients, defaulting to 200, and -i is the interval of connecting to the broker, defaulting to 10. There is also -R, the connection rate per second, which defaults to 0 and falls back to using the interval instead. So the load is described either as a rate or as a delay between connection attempts, and the tool picks whichever you give it.

That distinction matters when you are trying to answer a question. A fixed number of clients tells you what the broker does with fifty thousand sessions held open, while a connection rate tells you what it does when sessions arrive quickly, and the two put very different pressure on the accept path and on the session table.

Protocol version, client identifiers and the collision problem

Three options decide how a client looks to the broker, and they are the ones worth reading before the first run.

The -V option sets the MQTT protocol version, accepting 3, 4 or 5, and it defaults to 5. So a broker that does not speak v5 needs the flag, and the default is a modern choice rather than a safe one.

The --prefix option sets the client identifier prefix, and when it is not provided the identifier defaults to $HOST_bench_(pub|sub)_$RANDOM_$N, where $HOST is the host name or the address given in --ifaddr, $RANDOM is a random number and $N is the sequence number assigned to each client. If a prefix is provided, the random suffix is not added. The --shortids option shortens that further, using the assigned sequence number as the identifier when no prefix is given.

The -n option, the start number, exists for a specific operational reason. It is the start point when assigning sequence numbers to clients, and the README says it is useful when running multiple emqtt-bench instances against the same broker, so the start numbers can be planned to avoid client ID collisions. That is a real constraint of the MQTT protocol, since the broker has to keep sessions distinct by identifier, and it is a detail that only bites when you scale out the load generator itself.

The remaining connection options in that group are about robustness rather than load: --num-retry-connect sets how many times to retry establishing a connection before giving up, and --reconnect sets the max retries of reconnects, where 0 disables it.

TLS, WebSocket and QUIC are all in the same tool

The transport options are where the tool is more than a connection generator, and they are the reason the QUIC option can be turned off at build time.

For TLS, -S enables an SSL socket, --ssl-version enforces a version and implies SSL is enabled with the values tlsv1.1, tlsv1.2, tlsv1.3 or tlsv1.3_nocompat. There is a full certificate set for mutual TLS, with --cacertfile for server verification, --certfile and --keyfile for client authentication when the server requires it, and --ciphers for a comma separated cipher suite list.

TLS 1.3 gets its own controls, because that version moved things around. --signature-algs sets the signature algorithm for tlsv1.3 connections only, --keyex-algs sets the key exchange algorithm, also tlsv1.3 only, and the README gives example values for both. The separate --quic and --ws options select the QUIC and WebSocket transports, both defaulting to false, and --load-qst loads quic session tickets from a dets file.

The TLS options are unusually complete for a benchmark tool, which is the point. If you are deciding which transport your broker should be used over, you want the same load profile measured over each one, and that means the tool has to be able to speak all of them.

The build reflects that too. The README notes that you can disable QUIC support if you have problems compiling, with BUILD_WITHOUT_QUIC=1 make, and the Dockerfile sets exactly that variable for its builder stage, which tells you the QUIC dependency is the most fragile part of the build.

Low memory mode, QoE tracking and a Prometheus endpoint

A benchmark tool that runs fifty thousand connections has a memory problem, and the tool addresses it directly.

The -l option enables a low memory mode, and the README is explicit about the trade: it uses more CPU. That is the classic benchmark dilemma, since the process running the load consumes resources that the machine under test needs, and any measurement is distorted by that. The -Q option turns on QoE tracking, and it can be set to dump the QoE disk log to CSV and then exit, with --qoelog writing the QoE event logs to the disk log file for post processing. The --force-major-gc-interval option sets the interval in milliseconds in which a major garbage collection is forced on the bench processes, with 0 meaning disabled, and the README notes it only takes effect together with --lowmem.

The forced garbage collection is a giveaway about the implementation. Erlang's garbage collector is per process, and forcing a major collection at an interval keeps memory bounded when you are running tens of thousands of client processes.

For observability there is --prometheus, which enables metrics collection via Prometheus and is usually used with --restapi. The --restapi option enables a REST API for monitoring and control, which for now only serves /metrics, and it can be set to IP:Port to listen on a specific address and port, or just a port to listen on all interfaces on that port. The --log_to option controls where log output goes, with console writing directly to the console and null quietly disabling logs entirely, which is the setting that matters when the tool's own output is competing with the measurement for CPU.

Building it: Erlang 27.2, libatomic and rebar3

The build requirements are short. Building from source requires Erlang/OTP 27.2 or later, and the tool needs libatomic, which is installed with the platform package manager:

# centos 7
sudo yum install libatomic

# ubuntu 20.04
sudo apt install libatomic1

The source build is three commands:

git clone https://github.com/emqx/emqtt-bench.git
cd emqtt-bench
make

The Makefile shows how that works. The rebar3 binary is bootstrapped by a script, scripts/ensure-rebar3.sh, and then the default target runs release, which compiles and produces an escript archive named emqtt_bench with the tar format. The release target then runs three scripts in sequence, macos-sign-binaries.sh, macos-notarize-package.sh and rename-package.sh, which is the signing and notarisation pipeline for macOS binaries.

Other targets are worth knowing. pre-release builds a release format archive instead of a tarball, xref runs the cross reference check over the source, and the docker target builds the container image with a tag taken from git describe. The clean and distclean targets remove the build directory and the crash dumps, and the distclean target also removes rebar.lock and the emqtt_bench script.

The build system files in the repository root are rebar.config and rebar.config.script, and the source is under src with a bin directory, a scripts directory and a CHANGELOG.md. There is also a .tool-versions file, which is the asdf style pin for the Erlang and OTP versions, and topic_spec.md and topic_spec.json, which are the MQTT topic specification files the tool uses to generate publish and subscribe traffic.

The container image builds without QUIC and runs as an unprivileged user

There is a Dockerfile in the repository, and a separate README_FOR_DOCKER.md, so the container route is a first class part of the project rather than an afterthought.

The build stage uses a builder image, ghcr.io/emqx/emqx-builder, at a pinned tag, copies the source into it and runs make release. Notably it sets BUILD_WITHOUT_QUIC=1 for that stage, so the container image is built with the QUIC support switched off, which is the same escape hatch the README documents for anyone whose compiler struggles with the QUIC dependency.

The runtime stage is debian:13-slim. It copies the release tarball out of the builder, installs ca-certificates, procps and curl, creates a group and a user both with id 1000, extracts the tarball into the directory and chowns it to that user, so the container runs unprivileged.

The entrypoint is the binary itself:

ENTRYPOINT ["/emqtt_bench/bin/emqtt_bench"]
CMD [""]

The empty CMD is a small detail with a real purpose. The tool takes its subcommand as an argument, so the image is used as emqtt_bench conn ... and the default command has to be empty for that to work without the image overriding it.

For a benchmark tool, the container is arguably the more sensible deployment than a binary on a laptop, since the machine running the load should be as free of other work as possible and the image is reproducible.

Editorial conclusion

Adopt eMQTT-Bench if you need to put a known number of MQTT connections or subscriptions against a broker and read the results, particularly if your broker speaks MQTT v5 or you need to compare WebSocket and QUIC transports under load. Do not adopt it if you need protocol conformance testing or a general MQTT client, since it is a load generator with a narrow command set rather than a test suite. Verify first which MQTT protocol versions your broker accepts, since the tool defaults to version 5, that you can install libatomic and Erlang/OTP 27.2 or newer before you build, and how the client count relates to the memory of the machine you run it on, since fifty thousand connections is a configuration choice rather than a free setting.

Frequently asked questions

Official sources

1. emqx/emqtt-bench on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases