NVIDIA GDRCopy for low-latency GPU memory access from Linux

GDRCopy's small-copy latency problem

GDRCopy addresses a specific gap between ordinary CUDA copies and direct access to GPU memory. Its documentation says that GPUDirect RDMA APIs can also be used to create valid CPU mappings of GPU memory. Once such a mapping exists, a program can manipulate the mapped region as host memory, subject to the caching and PCIe behavior documented for the active backend. This is not intended to turn GPU memory into ordinary DRAM. The mapping still communicates through the GPU BAR and PCIe, and its useful characteristics differ by direction.

The intended users are engineers building Linux software with low-latency CPU and GPU interactions. A repeated sequence of small transfers is the clearest case. The README presents CPU-driven copies as a way to reduce call overhead, and cites a 6-7 microsecond overhead for `cudaMemcpy` as a reference. It also reports a pinning cost of 10 microseconds to 1 millisecond, depending on buffer size. Those figures come from the project documentation rather than a verified evaluation, but they explain the design trade-off: setup can cost more while subsequent operations avoid the overhead of the conventional CUDA path.

GDRCopy should not be treated as a general-purpose replacement for every memory copy routine. The README explicitly characterizes host-to-device copies as fast because of write combining, while device-to-host copies are slow because the GPU BAR cannot be prefetched and burst reads are not generated through PCIe. NUMA effects can also change host-to-device bandwidth. Applications dominated by large D2H transfers therefore miss the central reason to choose this library.

Inside gdrdrv and dma-buf memory mappings

The repository is primarily C and exposes a small user-space library around NVIDIA GPU memory mapping operations. Based on the repository layout, `include/` contains the public API, `src/` contains the implementation, `tests/` contains functional and measurement programs, and `scripts/`, `packages/`, and `insmod.sh` support deployment. The kernel component is named `gdrdrv`; installing or loading it requires root privileges.

There are now two backend paths. With `gdr_open()`, GDRCopy first tries `gdrdrv`. If that module is absent or cannot be opened, the library can fall back to exporting GPU memory as a Linux dma-buf through the CUDA driver and mapping that file descriptor with `mmap()`. Setting `GDRCOPY_USE_DMABUF_MMAP=1` before `gdr_open()` forces the fallback backend even when `gdrdrv` is available. This is useful for environments where the kernel module is unavailable, although it imposes additional driver and resource constraints.

The two paths do not have identical capabilities. Dma-buf mappings are always persistent, and `GDR_ATTR_USE_PERSISTENT_MAPPING` consequently returns 1. Each pinned buffer consumes one file descriptor until `gdr_unpin_buffer`, so an application with many pinned buffers can hit the process descriptor limit. `gdr_get_info_v2` reports zero for both `tm_cycles` and `cycles_per_ms` in this backend. The invalidation callback flag also remains unavailable through `gdr_get_callback_flag`. Standard GDRCopy calls retain their API shape, but callers should not assume every backend-specific observation is populated.

Caching and PCIe constraints for GDRCopy mappings

GDRCopy's mapping type is a central engineering constraint, especially on the dma-buf path. For dma-buf, CPU cacheability is selected by the CUDA driver when memory is pinned and cannot be changed later. On a coherent platform, the default pin operation produces `GDR_MAPPING_TYPE_CACHING`. On a non-coherent platform, it produces `GDR_MAPPING_TYPE_WC`. Passing `GDR_PIN_FLAG_FORCE_PCIE` produces `GDR_MAPPING_TYPE_WC` on either kind of platform.

The application cannot simply request whichever mapping it would prefer later. The dma-buf backend accepts only the mapping type generated by pinning. A request through `gdr_map_v2` that uses `GDR_MAP_FLAG_REQ_CACHE_MAPPING`, `GDR_MAP_FLAG_REQ_WC_MAPPING`, or another explicit cache flag when it differs from the default returns `EINVAL`. This makes deployment behavior depend on both the application and the coherence characteristics of the platform.

For the traditional kernel-module path, the README still frames mappings as usable like host memory with caveats. It does not provide a portable guarantee that all mapped operations have identical cache semantics. Given the documented asymmetry between fast H-D copies and slow D-H copies, direction-specific validation is necessary for any application. A design that exchanges many small messages in both directions should not infer performance from a single H-D bandwidth measurement.

Installing GDRCopy packages on RHEL

The README lists supported architectures as Linux x86_64, ppc64le, and arm64. Its documented distributions are RHEL 8, RHEL 9, Ubuntu 20.04, Ubuntu 22.04, SLE-15 with any service pack, and Leap 15.x. CUDA and the NVIDIA display driver must be installed before building or installing GDRCopy. Driver headers are also required. CUDA 6.0 or newer is required by the library and tests, while the dma-buf backend specifically requires CUDA driver 13.3 or newer.

For an RHEL package build, start by installing the documented build dependencies. The commands below install DKMS and the RPM build prerequisites. The final `yum` command needs to be run with root privileges.

sudo yum groupinstall 'Development Tools'
sudo yum install dkms rpm-build make

Build the RPM files from the repository's `packages` directory. `CUDA` must point at the CUDA installation's top-level directory.

cd packages
CUDA=<cuda-install-top-dir> ./build-rpm-packages.sh

The script produces a DKMS kernel module package by default, followed by the runtime and development RPMs. Install them in that order. The version and platform suffixes shown here are placeholders from the README, so package filenames must be matched to files produced by the local build.

sudo rpm -Uvh gdrcopy-kmod-<version>dkms.noarch.<platform>.rpm
sudo rpm -Uvh gdrcopy-<version>.<arch>.<platform>.rpm
sudo rpm -Uvh gdrcopy-devel-<version>.noarch.<platform>.rpm

On RHEL and SLE, the project also permits building a prebuilt `kmod` package by passing `-m` to `build-rpm-packages.sh`. That package is tied to the NVIDIA driver and Linux kernel versions present at build time. It avoids DKMS rebuilding for every kernel but makes kernel and driver upgrades an explicit package-management concern.

Checking the first GDRCopy backend selection

The repository includes `gdrcopy_sanity`, `gdrcopy_copybw`, `gdrcopy_copylat`, `gdrcopy_apiperf`, and `gdrcopy_pplat`. Their documented purposes cover unit checks, buffer-size bandwidth measurement, copy latency, API-call latency, and GPU-to-CPU ping-pong latency. The README excerpt does not provide commands that build and run these programs, so it would be misleading to invent executable names or flags beyond those source names.

For an existing C integration, the first backend-specific check is to inspect `GDR_ATTR_USING_DMA_BUF_MMAP` after opening the device. The following is the exact query form given in the README. `gdr` must already refer to an opened GDRCopy handle, and `gdr_get_attribute` must return successfully before the result is interpreted.

int using_dmabuf;
gdr_get_attribute(g, GDR_ATTR_USING_DMA_BUF_MMAP, &using_dmabuf);

A nonzero `using_dmabuf` means the dma-buf mmap backend is active. Applications can force that choice by setting the following environment variable before calling `gdr_open()`.

GDRCOPY_USE_DMABUF_MMAP=1

That assignment is only useful when exported or passed in the process environment before opening GDRCopy. It does not install dependencies and does not bypass the documented requirement for CUDA driver 13.3 or newer. The repository's benchmark programs are the more appropriate starting point for evaluating copy latency or bandwidth once the documented build and installation procedures have produced them.

GDRCopy versus CUDA copies and maintenance cost

The most direct alternative is a CUDA copy operation such as `cudaMemcpy`, which the README itself uses as the overhead reference. The difference is not that GDRCopy transfers data over an unrelated interconnect. Both approaches move data between host-accessible views and GPU memory, but CUDA offers a general driver-managed copy API, while GDRCopy creates a user-space mapping and lets CPU code access it. GDRCopy can reduce overhead for suitably small operations. A general CUDA memory path is easier to reason about when application code does not need direct mapping and the workload is dominated by larger transfers.

GDRCopy also introduces system integration cost. Its kernel-module package depends on DKMS and compatible NVIDIA driver headers, while installation requires root privileges. A prebuilt `kmod` package moves part of that burden to explicit compatibility: the binary is specific to an NVIDIA driver version and Linux kernel version. The dma-buf path removes the requirement to install or load `gdrdrv`, but it demands a newer CUDA driver and introduces descriptor usage, persistent mappings, missing timing fields, and a missing invalidation callback.

The project is not archived, and its last push was on 2026-09-15. Version 2.6 was released on 2026-06-13, after version 2.5.2 from 2026-03-12. That is evidence of recent repository activity, but it does not by itself establish hardware compatibility or remove upgrade work. The MIT license permits use, modification, distribution, and sublicensing under its notice and conditions. Integrating with a proprietary product remains a separate policy and compatibility decision, particularly because GDRCopy sits close to kernel modules and NVIDIA drivers.

Editorial conclusion

GDRCopy is appropriate for Linux applications that repeatedly move small GPU buffers between CPU and device code and can meet its platform, driver, and installation constraints. It is a poor fit for bidirectional throughput-oriented transfers, unsupported GPU families, or deployments that cannot satisfy the requirements of either backend. Before adoption, verify the target GPU generation, NVIDIA display driver, CUDA version, supported distribution, NUMA placement, and selected backend against the documented requirements.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. NVIDIA/gdrcopy on GitHub
4. README
5. Releases