What is CUDA?

What the programming model actually is

A CUDA program has two halves. Host code runs on the CPU and launches work; device code runs on the GPU. A kernel is a device function with no return value, and a launch creates a grid of thread blocks, each holding a block of threads, each thread carrying a thread index that it uses to decide which elements of the data it handles. There is no loop over threads and no thread object: the hardware owns the threads and the index is how you address them.

Threads execute in lockstep groups called warps, and that single fact explains most of what makes a kernel fast or slow. Neighbouring threads reading neighbouring addresses coalesce into few memory transactions; threads reading scattered addresses do not, and the memory system becomes the bottleneck. When threads in the same warp take different branches, the paths run serially, so a kernel whose threads diverge in a loop can be several times slower than the same work arranged without divergence.

Memory is a hierarchy you choose between. Registers and shared memory are on-chip and fast but small and per-block; global memory is large and slow; there is a cache in between. Moving data into shared memory explicitly and reusing it is usually where the speedup in a hand-written kernel comes from, and it is the reason occupancy, the ratio of active warps to possible warps, is the number people end up tuning.

Work is submitted asynchronously through streams, and real overlap comes from running one kernel while the next one's data is still being copied across the bus. That requires pinned host memory, which is why a transfer from pageable memory is slower than the number of bytes suggests. Compilation goes through nvcc, which emits device code as PTX and then to the machine-specific form called SASS; a binary can carry PTX for forward compatibility, at the cost of a just-in-time compile when a newer device needs it. The toolkit at developer.nvidia.com/cuda-toolkit bundles the compiler, the runtime, and libraries such as cuBLAS, cuDNN, cuFFT and NCCL, which is where most people actually meet it.

The hardware decides the code

A kernel is written against an architecture, and the architecture decides what is possible. The general shape of the problem does not change: for most machine learning work the arithmetic units are fast and the memory system is not, so a kernel that reads its inputs more than once is a bandwidth problem wearing a compute problem's clothes. Data type, vector width, and how many elements each thread handles all follow from that.

Compute capability matters because the instruction sets differ. Some architectures have fast paths for reduced-precision matrix operations in their tensor cores, and the cost of using them is a constraint on the data layout. Our analysis of facebookincubator/AITemplate is explicit about this, describing it as a framework that renders neural networks into self-contained CUDA or HIP binaries specialised for FP16 TensorCore on NVIDIA GPUs and MatrixCore on AMD GPUs, aimed at teams willing to trade portability for kernel-level control. The same binary is not the fastest binary on a different card, which is why shipping kernels is a per-architecture problem rather than a per-language one.

The tools below mirror that. NVIDIA/cutlass is a library of CUDA C++ template abstractions for high-performance matrix multiplication and related linear algebra, plus a Python DSL family called CuTe DSL, and our analysis is clear that it is a tool for kernel authors rather than a drop-in BLAS replacement. xlite-dev/LeetCUDA is a GPL-3.0 collection of more than two hundred CUDA kernels including FlashAttention, an HGEMM implementation and a four-hundred-page book, aimed at people who want to read the actual instructions. antirez/ds4 is a C inference engine for DeepSeek V4 Flash that targets Apple Silicon Metal, NVIDIA CUDA and AMD ROCm, and our analysis notes it is deliberately narrow: it uses its own GGUF format rather than accepting arbitrary files, and targets engineers with 96 GB Macs, DGX Spark systems or Strix Halo workstations.

When writing a kernel is the right answer

Writing CUDA is justified in a short list of situations, and each one has a cheaper alternative that is usually tried first.

Where no library kernel exists. A framework gives you a set of operations, and an operation outside that set means either composing several of them, which moves more data than necessary, or writing the kernel. The second option wins when the operation is on the hot path and the composition reads memory several times.

Where fusion is the win. Several small operations that each read and write global memory can become one kernel that keeps the intermediate in registers or shared memory. The saving is memory traffic, and it is often larger than any arithmetic optimisation.

Where the memory layout is the problem. Custom layouts, prefetching, or moving data to where the consumer wants it are all kernel work.

Where the hardware is specialised. A model that maps unusually well onto one architecture's fast paths will leave the general libraries behind.

Where it is not worth it: when cuBLAS or cuDNN or a framework's own fused operator already covers the operation, because those are tuned for the same hardware by people who work on it full time. When the bottleneck is not the kernel, which in a training loop is usually the input pipeline or the host. When the same code has to run on more than one vendor's hardware, since CUDA is specific to NVIDIA and the alternatives are HIP for AMD, Metal for Apple, or a portable layer such as Triton. And when nobody will maintain it, because a custom kernel is a permanent maintenance obligation attached to whatever it is fastest in.

The mistakes that cost the most time

Five of them dominate.

Environment mismatch. The driver, the toolkit, the framework build and the compiler have to agree. The classic symptom is an error saying no kernel image is available for execution on the device, which means the binary has no code for the GPU you are running on, and the fix is a rebuild for that architecture rather than anything to do with your code.

Forgetting that the architectures differ. A kernel that is fast on one card can be slower than a naive one on another, because register pressure and cache behaviour change. A kernel that is correct everywhere is not automatically fast everywhere.

Divergence and uncoalesced access. Both are invisible in the source and obvious in a profiler. Printing from a kernel and running the memory checker are the two tools that find them, and using them costs less than guessing.

Synchronisation and lifetime errors. Kernels are asynchronous, so missing a synchronisation shows up as data that is wrong later rather than at the line you forgot. Device allocations made through one API and freed through another, or a host pointer that went out of scope while a copy was in flight, produce the same class of problem.

Compile time. Template-heavy kernel libraries such as CUTLASS can take minutes per configuration, which turns an experimentation loop into a waiting room. Budget for it before you choose that approach, not after.

How the projects divide

The repositories here are five different relationships to the programming model.

The kernel library. NVIDIA/cutlass is the one to read if you want to write high-performance matrix multiplication: template abstractions over the hardware's fast paths, with a Python DSL alongside. It is a foundation, not something you add.

The compiler. facebookincubator/AITemplate sits one level up: you describe a network in Python and it emits a self-contained binary, fusing operations horizontally, vertically and in memory. Our analysis of AITemplate calls those fusions its main selling point and frames the trade as portability given up for kernel-level control.

The reference code. NVIDIA/cuda-samples is the official CMake-based collection of toolkit examples, and our analysis describes it as reference code for developers who already have a GPU and a toolkit, from deviceQuery to bandwidthTest, rather than a library to add to a project. It is also the fastest way to find out what a feature is called.

Bindings. LWJGL/lwjgl3 is a set of Java bindings to OpenGL, Vulkan, OpenAL, OpenCL and XR APIs distributed as optional modules, and our analysis of it says it gives you the native calls and nothing else, which is both the point and the main reason to look elsewhere when you need more. hybridgroup/gocv is the same idea for computer vision from Go, where our analysis says the install is the hard part because you must match OpenCV yourself.

Consumers. Several projects here simply run on CUDA rather than program it: antirez/ds4 as a local inference engine, ace-step/ACE-Step-1.5 as a local music generation stack that our analysis says pairs a language-model planner with a diffusion transformer and runs on CUDA, ROCm, Intel XPU, Apple Silicon or CPU, nerfstudio-project/gsplat as a CUDA-accelerated rasterizer for 3D Gaussian Splatting that our analysis says is built to be embedded in a training loop rather than used as an app, and chrxh/alien as a CUDA-powered artificial life simulation running a two-dimensional particle engine on GPU for real-time simulations with millions of particles. The last one is worth keeping in mind as a category: CUDA is not only a machine learning tool, and the projects that are not machine learning are often the ones that make the programming model easiest to learn from.

In practice

CUDA is a programming model you take responsibility for: the memory hierarchy, the synchronisation, the per-architecture binaries and the permanent maintenance of whatever you optimise. Most work should start inside a framework's existing operators, and the right time to write a kernel is when the operation is hot, absent from the libraries, and reads its inputs more than once. If you do write one, compile for the exact architecture you deploy on, check it with the memory checker and a profiler before you believe it, and read NVIDIA/cutlass and xlite-dev/LeetCUDA to see what other people had to do to make the hardware behave. If the code has to run on more than one vendor, choose a portable layer such as Triton or HIP instead, and the maintenance argument decides it for you.