UCX and the four layers between MPI and InfiniBand

The four layers that turn tags into fabric traffic

UCX is a C framework that exposes abstract communication primitives so the layers above it can use whatever the machine underneath offers. Two wiki pages cover installing OpenMPI and OpenSHMEM with UCX, and a third covers MPICH, which places the library under a messaging runtime instead of beside it.

The work is split across four components. UCP is the protocol layer, implementing high-level abstractions such as tag-matching, streams, connection negotiation and establishment, multi-rail, and handling different memory types. UCT is the transport layer, implementing low-level communication primitives such as active messages, remote memory access, and atomic operations. UCS is a collection of data structures, algorithms, and system utilities for common use. UCM intercepts memory allocation and deallocation events, and those events drive the memory registration cache.

The example directory makes the split visible rather than theoretical. `ucp_hello_world.c` and `ucp_client_server.c` drive the protocol layer, while `uct_hello_world.c` talks to a transport on its own, which is the fastest way to see what a fabric can do without a message layer in between.

Eight transports behind one set of abstract primitives

A runtime written against UCP does not carry an InfiniBand code path and a separate TCP code path. The transport list spans Infiniband, Omni-Path, RoCE, Cray Gemini and Aries, CUDA, ROCm, shared memory and TCP/IP, and the choice between them happens during connection negotiation and establishment, after configure has already decided which transports are compiled in.

Shared memory alone has five backends, posix, sysv, cma, knem and xpmem, and xpmem lives in its own repository. CUDA and ROCm are named as transports in their own right, right next to UCP's remit of handling different memory types, so a GPU buffer is not a special case bolted on afterwards.

The processor list is much shorter than the fabric list: x86_64, Power8/9 and Arm v8, with nothing else named. That asymmetry is the first filter for anyone planning a port, since the network side is broader than the CPU side by a wide margin.

Three commands install UCX from an unpacked tarball

From the top of an unpacked tree, installing the system is a configure run with a prefix, a parallel build, and an install step.

$ ./contrib/configure-release --prefix=/where/to/install
$ make -j8
$ make install

Replace the prefix with the install directory your MPI build is linked against. A clone of the git repository needs one line in front, because the configure script is generated rather than shipped in the tarball:

$ ./autogen.sh
$ ./contrib/configure-release --prefix=/where/to/install
$ make -j8
$ make install

Compiling support for various networks or other specific hardware may require additional command line flags when running configure, so a build that lacks a fabric is a configure decision rather than a compile error later. The repository layout follows that sequence: `configure.ac`, `buildlib/`, `contrib/` and `Makefile.am` sit at the top level, with `src/` holding the libraries and `test/` the gtest suite.

Developer builds and package builds take different scripts

A developer build configures into a prefix inside the checkout instead of a system directory, and the directory name gives the purpose away:

$ ./autogen.sh
$ ./contrib/configure-devel --prefix=$PWD/install-debug

The warning attached to it matters more than the path. Developer builds of UCX typically include a large performance penalty at run-time because of extra debugging code, and no transport list is given for that build, so any measurement taken from it describes the debug path and nothing else.

Packaging runs off the same tree, through a shell script for RPMs, the Debian rebuild path, and a top-level Doxygen target:

$ contrib/buildrpm.sh -s -b
$ dpkg-buildpackage -us -uc
$ make docs

Run one packaging path or the other for a given node image, since both are driven from the same checkout and no separate staging step is mentioned for either.

ucx_perftest needs two processes on two different cores

The first thing to run after install is the bundled performance test, and it is a two-process job. One side is the server:

$ ./src/tools/perf/ucx_perftest -c 0

The other side connects and asks for tag latency:

$ ./src/tools/perf/ucx_perftest <server-hostname> -t tag_lat -c 1

The `-c` flag sets CPU affinity, and that is the detail people trip over. If both commands run on one host, set the affinity to different CPU cores, because two processes sharing a core measure the core and not the transport.

Internal checks come from the same build tree and need no arguments:

$ make -C test/gtest test

A glibc version window that hangs UCX threads in ucs_rcache_get

Hangs are harder to read than corruption, and UCX names its own: UCX may hang with glibc versions 2.25-2.29 due to known bugs in the pthread_rwlock functions. When such hangs occur, one of the UCX threads gets stuck in pthread_rwlock_rdlock, which is called by ucs_rcache_get, even though no other thread holds the lock.

That final clause is the useful part for anyone with a wedged job. No writer exists, the holder list is empty, and the stack still names a lock acquisition inside the register cache, so a hang there points at the runtime library rather than at a UCX data structure.

Both remedies stay outside UCX: use the glibc version provided with your OS distribution, or build glibc from source using versions less than 2.25 or greater than 2.29. The related glibc report is filed as Bug 23844.

Where 1.8.0 corrupts data and rdma-core v22 removes DC

UCX version 1.8.0 has a bug that may cause data corruption when the TCP transport is used in conjunction with the shared memory transport. Versions released before 1.8.0 do not have it, and the advice is to move to 1.9.0 and above, which leaves 1.8.x as the narrow band where running both transports together is unsafe rather than absent.

A second failure comes from the RDMA stack rather than from UCX. Due to a compatibility flaw with rdma-core v22, the setting below is unsupported and makes the DC transport unavailable, taking it out of service instead of degrading it:

UCX_DC_MLX5_RX_INLINE=0

That was fixed in rdma-core v24 and backported to the rdma-core-22.4-2.el7 rpm, tracked in ucx issue 5749. Routing misreads produce the other classic symptom, peers reported as unreachable, and the documented response is again an environment variable rather than a code change:

UCX_IB_ROCE_REACHABILITY_MODE=all

Where UCX is the wrong tool, and what to check first

UCX is a library you compile into something, not a program you launch, so the first question is whether you have a component that speaks UCP or UCT. The repository points at three integration targets: OpenMPI with OpenSHMEM, and MPICH.

The second question is your fabric stack, and here the answer can be a hard stop. UCX 1.12.0 and above require rdma-core 28.0 or later, or MLNX_OFED 5.0 or later, for InfiniBand and RoCE support, so an old RDMA stack on the node rules out current builds rather than something you configure around. CPU coverage outside x86_64, Power8/9 and Arm v8 is unlisted.

Upgrade cost runs in one direction only. The 1.8.0 corruption note points forward to 1.9.0 and above, and no downgrade or rollback procedure appears anywhere in the README. Check the licence text before shipping anything built on it: GitHub's licence field reads Other, a custom licence the platform cannot classify, and points to the LICENSE file, while the README lists the project as BSD3.

Editorial conclusion

Adopt UCX when you are wiring an MPI, OpenSHMEM or PGAS runtime onto RDMA hardware and want one API across fabrics instead of per-fabric code. Skip it for a two-node job on shared memory, for any processor outside x86_64, Power8/9 and Arm v8, or for a cluster whose rdma-core is older than 28.0, because 1.12.0 and above drop InfiniBand and RoCE there entirely. Before committing, confirm the node's glibc sits outside the 2.25 to 2.29 hang window, and run the perftest with both processes pinned to separate cores so the number means something.

Frequently asked questions

Official sources

1. Issues
2. openucx/ucx on GitHub
3. Project website
4. README
5. Releases