CXLMemSim: studying CXL memory without the hardware

Two pieces, one experiment platform

CXLMemSim is a software framework for studying CXL memory systems without requiring complete CXL hardware. The repository contains two related pieces, and the README names them plainly.

One is a CXL memory latency, bandwidth, topology and coherency simulator. The other is a QEMU-integrated CXL emulation stack for Type 3 memory devices, distributed memory pooling and experimental Type 2 accelerator and GPU support.

The stated intent is what makes the two pieces one project rather than two. The implementation is intended for full-system experiments where guest software talks to a realistic CXL device interface while the host side records and controls protocol-level behaviour such as latency, routing, coherency state, directory pressure and memory placement.

That last list is the interesting part, because it is the set of things you cannot fake with a latency injector. A directory that fills up, a coherence protocol that has to evict, a placement policy that migrates a page, those are the behaviours that decide whether a memory architecture works, and they are the behaviours this framework lets you change on the host while the guest runs unmodified.

The primary language is C, the build is C++20 with CMake, the licence is BSD-3-Clause, and the homepage is a DOI rather than a website, which tells you the project is presented as a research artefact with a paper behind it. The last push was on 2026-10-06 and the repository is not archived.

The four layer stack

The implementation overview describes the stack as four layers, and the diagram is the clearest description of the project in the repository.

Guest applications
  -> guest driver/runtime shim
  -> QEMU CXL device model
  -> CXLMemSim server and host backends
  -> host memory, shared memory, RDMA/TCP transport, or host GPU

Reading it from the bottom: the simulated device can be backed by ordinary host memory, by a shared memory segment, by an RDMA or TCP transport, or by a host GPU. The server and the host backends sit above that, the QEMU CXL device model sits above them, and the guest sees a device that behaves like a CXL device rather than like a file-backed emulator.

The repository layout maps onto those layers. The src/ directory holds the CXLMemSim server, the controller, the coherency code and the HDM decode, include/ has the public headers for the simulator and server, microbench/ has the microbenchmarks, and tests/ has the server and distributed mode tests.

The integration directories are the two that make the stack work. qemu_integration/ holds the launch scripts and the guest side integration, including guest_libcuda/ which is the guest CUDA Driver API shim for the CXL Type 2 GPU path. And lib/qemu/ is a QEMU tree with the CXL device changes, with cxl_type2.c as the QEMU CXL Type 2 device model, cxl_hetgpu.c as the host GPU backend bridge and include/hw/cxl/ holding the protocol headers.

There is also a gem5_integration/ directory, with a gem5-CXL launch wrapper and O3CPU trace setup, which suggests the simulator can also be driven from a gem5 model rather than only from QEMU.

Type 3 memory and Type 2 accelerators take different paths

The two device types are handled by different mechanisms, and the README is clear about which is which.

For Type 3 memory experiments, QEMU forwards memory operations to cxlmemsim_server. The server owns the simulated memory pool, applies latency and topology policies, and tracks coherency metadata. So the emulator is a real server process talking to QEMU over one of several transports, and the policies live on the server side.

For Type 2 accelerator experiments, the guest uses a CUDA-compatible shim library. CUDA Driver API calls are translated into MMIO commands on a QEMU CXL Type 2 PCI device. QEMU then forwards those commands to the host-side hetGPU backend, which can call the real NVIDIA CUDA driver through dlopen() and dlsym().

That is a much more interesting design than emulating a GPU would be. Rather than writing a fake CUDA implementation, the guest believes it is talking to a GPU, the call is turned into device commands, and the host actually executes it against the real driver. The emulation is in the transport, not in the semantics, which means the fidelity is the fidelity of the real driver with the timing characteristics of the modelled device.

The repository layout backs this up with the specific files: lib/qemu/hw/cxl/cxl_type2.c is the device model, cxl_hetgpu.c is the host bridge, and the protocol headers are in lib/qemu/include/hw/cxl/.

The core simulator: controller, expander and a Newick topology

The original CXLMemSim path models CXL.mem behaviour from the CPU perspective. It accounts for target DRAM latency, CXL fabric latency, bandwidth, topology, ROB effects and cache-line states when estimating the application-visible memory penalty.

The controller implementation is centred on a small set of components. CXLController owns the topology, the endpoints, the policies and the latency model. CXLMemExpander represents CXL-attached memory capacity with separate read and write bandwidth and latency, which matters because memory attached over CXL is not symmetric in practice. Alongside them are the allocation, migration, paging and caching policies the controller uses, and a Newick-style topology parser, with the example syntax (1,(2,3));.

The topology the parser accepts is a tree, and the README draws it:

          endpoint 1
        /
 host -- switch -- endpoint 2
        \
          endpoint 3

which is expressed as (1,(2,3));.

The accounting layer is an LRU cache and a last-branch-record based path for application memory behaviour, which is how the simulator estimates what the application actually suffers rather than what the device does in isolation.

The microbench/ directory holds the workloads used to drive this, and the workloads/ directory in the repository root holds the application level benchmarks that the PMU sampling is used on.

The server, its request classes and its transports

The server stack is where the memory model becomes a service that QEMU and test clients can talk to.

The top-level build always enables SERVER_MODE and builds cxlmemsim, a static library with the core simulator, cxlmemsim_server, the Type 3 memory server, cxlmemsim_latency, a latency calculator, built test binaries for unit and integration checks, and CTest targets for DCD and GFAM behaviour, for the ROB, and for memory-stall unit behaviour.

The entry point is src/main_server.cc, and the README describes what it does in one sentence: it creates a CXL controller, adds a CXL memory expander endpoint, loads the topology, initializes a shared memory pool, and then serves requests from QEMU or test clients.

The request classes it understands are the interesting surface. They are cache-line reads and writes, shared-memory information queries, atomic fetch-and-add, atomic compare-and-swap, memory fences, Label Storage Area reads and writes, the add, release and query operations of the Dynamic Capacity Device, and the GFAM host map, unmap, access-check and query operations.

The communication modes are a table of four. The tcp mode is socket-based communication between QEMU and the server, shm is a shared memory ring buffer through /dev/shm, pgas-shm is a PGAS-style shared memory protocol used by cxl_backend.h clients, and distributed is a multi-node memory server mode with SHM, TCP, RDMA or a hybrid transport.

The memory pool is managed by SharedMemoryManager, which can use POSIX shared memory or a regular file as a backing store. The shared memory header records a magic value, a format version, the total size, the data offset, the base address and the cache-line count, and the default cache-line data area is mapped with mmap() and is reused when the backing object already exists.

The CLI parsing is handled in-tree, using local parse-result structs in src/main_server.cc, src/main.cc and src/rob.cc, which is a small choice that keeps the server free of a command line dependency.

SSD streaming mode and the performance counter reality

Two advanced features are worth their own paragraphs, because both are about the gap between a simulation and a measurement.

The first is the SSD streaming backing mode. When include/ssd_streaming_backend.h and src/ssd_streaming_backend.cpp are present, the server also builds an SSD streaming backing mode, which keeps the CXL address space persistent in a backing file or a raw block device and routes cache-line reads and writes through SsdStreamingBackend instead of exposing a direct mmap pointer. The MVP backend uses 4KB page metadata by default, 64KB read-ahead at 16 pages of 4KB, a resident page cache sized by --ssd-cache-mb, CLOCK-like eviction in the backend, Linux io_uring with a registered fixed file and buffer when available, O_DIRECT-safe aligned page buffers, and placeholder hint plumbing for prefetch, pin and unpin, evict-after and streaming ranges.

The second is CPU PMU compatibility, and the README is refreshingly honest about it. The application-level simulator uses Linux perf_event_open() for sampling. Intel systems use PEBS for load-miss samples and the Intel CHA PMUs when the CPU model has a known uncore path. AMD systems use the AMD IBS op PMU when it is available, and then fall back to generic hardware cache-miss sampling if IBS or physical-address sampling is unavailable. Intel CHA counters and LBR accounting are optional on non-Intel systems, and unsupported PMU paths are logged and disabled instead of aborting the run.

That last behaviour is the important one. A simulator that refuses to start because a counter is missing is a simulator that stops working on a new laptop.

If the monitor setup fails, the README tells you to check the CPU PMU support exposed by the kernel and the perf permission level:

cat /proc/sys/kernel/perf_event_paranoid
ls /sys/bus/event_source/devices/

And the macOS note closes it: on macOS, CXLMemSim cannot rely on Linux PEBS, LBR or the physical-address PMU.

Editorial conclusion

Adopt CXLMemSim if you are working on CXL memory and need to run full-system experiments before any CXL hardware is available, or if you want to reproduce the latency and coherency behaviour of a topology on a normal machine. Do not adopt it if you only need a memory model in your own simulator, since this is a whole system with QEMU, a server and a guest shim rather than a library you link against. Before you start, read the PMU section, because the application level simulator needs perf_event_open and the right perf permission level on your kernel, and check the transport mode you intend to use, since distributed mode with RDMA is a different operational shape from the shared memory mode.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. SlugLab/CXLMemSim on GitHub