nvbandwidth: measuring real copy bandwidth between NVIDIA GPUs

A measurement tool for copy paths, not a synthetic kernel benchmark

nvbandwidth measures bandwidth for memcpy patterns across different links on an NVIDIA GPU, and it does so with two mechanisms: the copy engine or a kernel based copy. That distinction is the whole point of the tool. A copy engine is dedicated DMA hardware that moves bytes without involving the SMs, while a kernel copy does the work with the streaming multiprocessors. The same transfer can therefore behave very differently depending on which path drives it, and nvbandwidth reports both.

The framing is deliberately about the machine in front of you. The project states that nvbandwidth reports the current measured bandwidth on your system, and that additional system-specific tuning may be required to achieve maximal peak bandwidth. That sentence is a useful warning against a common misreading: the numbers in the output are not a datasheet figure for the GPU model, they are what this particular host achieved with these particular settings.

The relevant users are people with hardware questions that a datasheet cannot answer. Someone validating a new multi-GPU node, confirming that an interconnect is healthy after a cable or switch change, or comparing a kernel copy against a DMA copy for a data loader all need the same instrument.

The top-level source layout matches that scope. There are separate translation units for memcpy.cpp, multinode_memcpy.cpp, kernels.cu, testcases.cpp and json_output.cpp, plus common.h and inline_common.h headers. There is no GPU-accelerate library dependency; the tool talks to the driver directly and prints tables.

Building with CMake requires CUDA 11.x, C++17 and CMake 3.20

Linux is the primary documented platform. The build needs the CUDA Toolkit at version 11.x or above, a C++17 compiler (GCC 7.x or above on Linux, MSVC 2019 or above on Windows), and CMake 3.20 or above, with 3.24 or newer encouraged. On Linux the path to the nvcc binary has to be on $PATH, and the machine needs a CUDA-enabled GPU plus an NVIDIA display driver compatible with the toolkit used for the build.

The single-node build is two commands in the source directory:

cmake .
make

On Ubuntu and Debian a helper script installs the required software components and builds the project in one step:

sudo ./debian_install.sh

Windows builds are possible but need extra setup, and the project asks that MSVC and the CUDA Toolkit are on the PATH before CMake runs.

Two files in the repository show where the platform assumptions live. cuda_version_check.cpp and cuda_version_check.h handle toolkit and driver compatibility at runtime, and detect_cuda_arch.cmake handles the architecture detection during the CMake configure step, which is what allows a single build tree to target the GPU that is actually installed. There is also a troubleshooting.md document at the top level, and a CHANGELOG.md, both of which the README points at for details it does not repeat.

The multinode build has a stricter requirement: a 12.3 toolkit and a 550 driver or above. That constraint comes from the MPI based multinode path, and it is worth checking against your environment before you plan a two-node run.

Testcases are named, and the tool lets you select them by name, index or prefix

Running the tool with no arguments runs every testcase. The rest of the interface is selection and output control, and it is worth reading the option list as a description of what the project considers interesting to measure.

The two central switches are -t and -p. The -t, --testcase flag runs a testcase by name or index, and -p, --testcasePrefixes runs every testcase matching a prefix, so a whole family such as device_to_device can be selected in one go.

./nvbandwidth -l

lists what is available, which is the first thing to run on a new machine.

./nvbandwidth -t device_to_device_memcpy_write_ce

runs one named case. The naming convention tells you what each case does. The _ce suffix marks a copy engine implementation and _sm marks a kernel implementation, while write and read distinguish direction of the copy.

Measurement controls are separate from selection. -b, --bufferSize sets the memcpy buffer size in MiB with a default of 512, -i, --testSamples sets the number of iterations with a default of 3, and -m, --useMean switches the reported statistic from the median to the mean. -s, --skipVerification skips the data verification that happens after the copy, and -H, --useHugePages switches host allocations to huge pages. CPU affinity control is applied automatically and can be turned off with -d, --disableAffinity.

For machine readable output, -F, --format accepts text, json or perf, with text as the default.

./nvbandwidth --format json

The perf format and the json_output.cpp source file suggest the output was designed to feed other tools rather than only to be read by a person.

The matrix output shows a full grid, then sums and a coefficient of variation

A device to device run prints a table where rows and columns are GPUs, and each cell holds the bandwidth in GB/s for that ordered pair. The diagonal is zero, since a device does not copy to itself. The example in the project notes shows a single node where every off-diagonal entry sits near 276 GB/s, which is the point of the layout: consistency across the grid is visible at a glance, and a single slow cell stands out.

The header of that table names the direction of the copy, for example memcpy CE GPU(row) <- GPU(column) bandwidth (GB/s). Reading the arrow matters, because the direction is part of the measurement rather than a formatting detail.

When a run is narrowed to a specific pair of GPUs, the output changes shape. Instead of a full grid you get the two relevant entries, and the summary lines are named after the testcase:

./nvbandwidth -t device_to_device_memcpy_read_sm \
    --pair <GPU-UUID-0> <GPU-UUID-1>

SUM device_to_device_memcpy_read_sm <SUM>
COEFFICIENT_OF_VARIATION device_to_device_memcpy_read_sm <CV>

The sum and the coefficient of variation are the two numbers that matter when you are comparing a specific link against expectation. A high sum with a low coefficient of variation says the link is fast and stable. A healthy average with a high coefficient of variation points at something unstable, such as a path where a single copy is much slower than the rest, and that pattern is exactly what a pairwise test exists to isolate.

Note that the pairwise form only fills the two cells you asked about and leaves the rest of the grid as N/A, so a pairwise run and a full sweep are not directly comparable output shapes.

The --pair option needs GPU UUIDs, and finding them across nodes means MPI

The --pair option restricts device to device tests to one explicitly specified pair of GPUs, identified by UUID, and it applies to the device_to_device_memcpy_* families, the bidirectional variants, the device_to_device_latency_sm and device_to_device_latency_tma families, and the multinode equivalents. The stated reason is to isolate and measure a specific link without running the full all-pairs sweep.

A UUID has the form GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with a required GPU- prefix, for example GPU-9a6efb31-7320-ce3c-b72c-75d85b6d0cea. Getting them on one node is straightforward, since nvidia-smi -L lists every GPU with its UUID. Across nodes, the project points at mpirun with a hostfile, mapping one process per node with --map-by ppr:1:node and raising the count to match the number of GPUs per node.

mpirun --hostfile /etc/nvidia-imex/nodes_config.cfg \
    --map-by ppr:4:node -n 8 \
    nvidia-smi -L

One caveat is called out: output from different nodes may be interleaved, and per-host separation is obtained by redirecting with --output-filename.

For the multinode pairwise test itself, the rule is exact. Run with exactly 2 MPI ranks, one per GPU, each rank's UUID must be the GPU that rank will use, and an MPI hostfile pointing at the two nodes must already be configured.

mpirun -n 2 ./nvbandwidth -t multinode_device_to_device_memcpy_read_sm \
    --pair <GPU-UUID-0> <GPU-UUID-1>

The buffer size choice interacts with this. The notes are explicit that a small buffer such as -b 1 is enough to verify connectivity between two GPUs quickly, but that the default 512 MiB buffer should be used when measuring peak bandwidth, because smaller buffers may underreport achievable throughput. A one-megabyte test that returns a result is a connectivity check, not a bandwidth measurement.

Multinode measurements need a hostfile, a newer toolkit and per-rank GPU binding

The multinode feature set is visible in the source layout rather than fully spelled out in the visible portion of the project notes. multinode_memcpy.cpp, multinode_memcpy.h and multinode_testcases.cpp hold the cross-node paths, and the testcase families for them carry a multinode_ prefix.

The operational requirements are clear enough to plan around. The multinode version requires a 12.3 toolkit and a 550 driver or above. A hostfile must be configured for MPI, in the example the path /etc/nvidia-imex/nodes_config.cfg, which is the NVIDIA imexec configuration file. The number of MPI ranks has to match the GPUs in play, exactly 2 for a pair test, and each rank must be bound to the GPU whose UUID you passed to it.

That last requirement is where multinode setups most often go wrong, because a rank that lands on the wrong GPU still runs and still produces numbers. The pairwise output for a multinode run prints a two by two grid where the diagonal is N/A and the two off-diagonal cells carry the bandwidth, with the same SUM and COEFFICIENT_OF_VARIATION summary lines as the single node case.

Because the tool prints measured values rather than theoretical ones, an unexpected result on a two-node fabric is the expected use case for this tool, and the coefficient of variation is the first number to look at when a single cell is an outlier. The project also ships a troubleshooting.md at the top level, which is the place to look when the numbers do not resemble the hardware you are holding.

Versioned releases under Apache-2.0, with the last push in July 2026

The repository is not archived, and the last push was on 2026-07-28. The release history shows a version that moves regularly enough to be worth pinning: v0.10.0 on 2026-07-21, v0.9 on 2026-04-08, and v0.8 on 2025-04-15. The gap between v0.8 and v0.9 is longer than the one between v0.9 and v0.10.0, so the cadence is uneven rather than fixed.

The version number is generated rather than hand-written. The source tree contains nvbandwidth_version.h.in, a template processed at build time, and the tool exposes a --version flag that prints the release version and exits. The presence of a CHANGELOG.md at the top level means the project records what changed between versions, which is the file to read before moving a pinned binary to a new release.

Licensing is Apache-2.0, with both a LICENSE file and a Licenses.txt in the repository root. That is a permissive licence with an explicit patent grant, which for a measurement tool means you can build it, run it internally, and ship the executable as part of an internal toolchain without the copyleft obligations that would come with a GPL codebase. The notices and attribution requirements in Apache-2.0 still apply if you redistribute it.

Since the tool produces numbers that people put into capacity plans, the version matters more than usual. A bandwidth figure recorded from v0.8 and compared against one from v0.10.0 is a comparison between two tool versions as much as between two machines, and the changelog is where you would look to know whether the measurement code changed between them.

Editorial conclusion

nvbandwidth suits engineers validating a multi-GPU or multi-node machine, sizing a data pipeline against the bandwidth their hardware really delivers, or chasing a link that underperforms its specification. It does not suit someone looking for theoretical peak figures, and the project itself says additional system-specific tuning may be needed to reach them. Before you rely on a number it prints, check three things yourself: that the nvcc path is visible to the shell you build in, that the buffer size you chose is the one you want to report, since small buffers understate throughput, and that the driver and toolkit combination matches the CUDA compatibility matrix, because the multinode path additionally requires a 12.3 toolkit and a 550 driver or above.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. NVIDIA/nvbandwidth on GitHub
4. README
5. Releases