pmpp: worked CUDA solutions for the fourth edition of the GPU programming textbook

Twenty chapters of exercises, each with the same shape

The repository covers all the exercises in the fourth edition of Programming Massively Parallel Processors by David Kirk and Wen-mei Hwu, and each chapter is organised the same way, so a reader who has understood one chapter knows how to read the next. A chapter directory, named chapter-XX, holds a code directory and a README:

├── code/
│   ├── *.cu          # CUDA implementations
│   ├── *.py          # Python alternatives  
│   ├── Makefile      # Build configuration
│   └── ...
└── README.md         # Theory + Exercises + Solutions

The README is the part that carries the argument, since it holds the theory alongside the exercises and the solutions, and the code directory holds the implementations, the Python alternatives, a Makefile for the build and anything else the chapter needs. The project describes itself as theoretical explanations, working implementations and performance analysis together, and the primary language recorded for the repository is Jupyter Notebook, which is a hint about how the analysis is delivered. The licence is MIT, and the last push was on 2026-09-07.

The early chapters are the ones the whole book rests on

The chapter list runs from 2 to 21 and the sequence is the argument of the book in miniature. Chapter 2 is heterogeneous data parallel computing, covering vector operations, thread mapping and CUDA basics, and chapter 3 is multidimensional grids and data, about grid organisation and the thread hierarchy. Chapter 4 is compute architecture and scheduling, with GPU architecture, warps and occupancy, and chapter 5 is memory architecture and data locality, with memory types, tiling and bandwidth optimisation. Chapter 6 covers performance considerations, specifically memory coalescing and latency hiding. Those six chapters are the foundation: if you read them carefully, the rest of the repository is application rather than introduction, and the diagnostics for a slow kernel are already in the chapters you have read.

Chapters seven to thirteen are the classic parallel patterns

The middle of the book is a catalogue of patterns, and the repository follows it closely. Chapter 7 is convolution, with constant memory, caching and halo cells, and chapter 8 is stencil computation, covering two and three dimensional stencils and register tiling. Chapter 9 is parallel histogram, with atomic operations, privatization and aggregation. Chapter 10 is reduction, with tree reduction and divergence minimisation. Chapter 11 is prefix sum, or scan, with work-efficient algorithms and the Kogge-Stone and Brent-Kung variants named explicitly. Chapter 12 is merge, with the co-rank function and circular buffers, and chapter 13 is sorting, with radix sort and merge sort optimisation. Every one of these has a well known naive version and a well known optimised version, and the performance benchmarks in each chapter are what show the gap.

Sparse, graph, learning and medical chapters

The later chapters widen the scope. Chapter 14 is sparse matrix computation, with SpMV and the CSR, ELL and COO formats, and chapter 15 is graph traversal, covering BFS algorithms and frontier-based approaches. Chapter 16 is deep learning, with a CNN implementation and the GEMM formulation, which is the chapter where a lot of people arrive since it connects the patterns back to a familiar workload. Chapter 17 is iterative MRI reconstruction, described as medical imaging algorithms, and chapter 18 is electrostatic potential map, with scatter versus gather and cutoff binning. Chapter 19 is parallel programming and computational thinking, about algorithm selection and problem decomposition, and chapter 20 is heterogeneous computing cluster, with CUDA streams and MPI integration. Chapter 21 is CUDA dynamic parallelism, with recursive algorithms and quadtrees. The scatter versus gather distinction in chapter 18 is the kind of thing that looks obvious once you have read it and is easy to get wrong before you have.

Setting up, and what the hardware requirement really means

The prerequisites are an NVIDIA GPU with CUDA support, the CUDA Toolkit installed, and Python 3.11 or newer if you want the Python examples, which are marked optional. The setup is a clone and a conda environment for the Python side:

# Clone the repository
git clone <repository-url>
cd pmpp

# For Python examples (optional)
conda create -n pmpp python=3.11
conda activate pmpp
pip install -r requirements.txt

The requirements file itself is short and tells you what the analysis tooling needs: torch and torchvision, Ninja as the build tool, pre-commit, gradio, cython, and h5py, with a comment noting that it is needed for MRI. The gradio and torch entries point to the deep learning chapter and to any interactive element in the notebooks, and Ninja is what the CUDA build uses under the hood. The hardware requirement is the real constraint, since every performance number in the repository was produced on a GPU and the analytical parts of the exercises depend on measuring.

Building and running one program at a time

Each chapter's code is built on its own, which keeps the feedback loop short. For the CUDA and C examples you change into the chapter's code directory, run make, and then run the program:

cd chapter-XX/code
make
./program_name

For the Python versions the shape is the same without the build step:

cd chapter-XX/code
python script_name.py

The Makefile in each code directory is what the chapter structure promises, and the placeholder names in the commands, chapter-XX and program_name, are what you replace with the actual chapter and binary. The repository also carries a .clang-format file and a pre-commit configuration, so the C and CUDA code is formatted by a tool rather than by hand, and the contribution guidelines ask for working code with any new implementation, performance data where relevant, and the existing explanation style with clear reasoning, which is the same standard the chapters themselves are written to.

It is a companion to the book, not a replacement for it

It is worth being clear about the relationship. The repository describes itself as complete solutions to the exercises in the book, and the four parts it claims for each chapter, detailed exercise solutions with step-by-step explanations, working code implementations in both CUDA C and Python, performance benchmarks comparing different approaches, and visual diagrams for complex algorithms, are all the things a textbook normally puts in the surrounding prose. That is a genuine complement rather than a substitute, since a book on GPU programming is as much about the reasoning behind a memory access pattern as about the pattern itself, and the reasoning is what the README files carry. The book also carries the exercises themselves, so the repository is most useful next to it rather than instead of it. For a reader who has the book and wants to check their own solution, or who has worked through a chapter and wants to see how the optimized version compares, this is the useful artifact.

Editorial conclusion

pmpp suits a reader working through Programming Massively Parallel Processors who wants the derivation of each technique alongside a runnable implementation and a performance comparison, and who has a CUDA capable GPU to run the benchmarks on. It does not suit a reader without GPU hardware, since the performance sections are the point and they cannot be reproduced on a CPU, and it does not replace the book, which is where the conceptual framing comes from. Before working through a chapter, check that the CUDA Toolkit and a supported GPU are in place, and note that the Python examples are optional and need their own environment.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. tugot17/pmpp on GitHub