KernelBench: a benchmark for LLM-written GPU kernels, and the scaffolding you have to build yourself

The gap KernelBench fills: correctness is not the same as speed

Most code-generation benchmarks stop when the output compiles and returns the right tensor. KernelBench treats that as the entry fee. The task is to transpile an operator written in PyTorch into a CUDA kernel, and the README is explicit that the model may choose whatever level of granularity it wants. The scoring then asks a second question: is the generated kernel faster than the PyTorch operator it replaces?

The audience is narrow and identifiable. Researchers who want a comparable number across models, and engineers who want to know whether a given model can write a kernel worth keeping. Level 1 covers single-kernel operators such as convolutions, matrix multiplies and layer normalization, 100 problems in total. Level 2 is 100 fusion patterns where one fused kernel should beat two separate ones, for example Conv + Bias + ReLU. Level 3 holds 50 full model architectures including MobileNet, VGG, MiniGPT and Mamba. Level 4 is described as optimizing whole model architectures pulled from HuggingFace. That progression matters: a model that handles pointwise fusion can still fall apart on an end-to-end architecture, and the level split is what lets you see where.

fast_p: why a single accuracy number hides the interesting result

The overall metric is fast_p, the fraction of tasks whose generated kernel is both correct and exceeds a speedup threshold p. Speedup is the ratio of PyTorch reference wall-clock time to generated kernel time. The README spells out three useful readings: fast_1 is the fraction that is correct and faster than the PyTorch baseline, fast_2 requires at least 2x, and fast_0 collapses to plain correctness because any positive speedup clears a threshold of zero.

This is a better design than a leaderboard of correctness rates, and it is also where the metric gets uncomfortable. Raising p does not just make the task harder, it changes what you are measuring. A model with a high fast_0 and a near-zero fast_2 is producing kernels that run but do not earn their place, which is exactly the failure mode a correctness-only benchmark would report as success. The cost is that fast_p is a wall-clock measurement, so it inherits every problem wall-clock measurements have. The README points to src/eval.py for how correctness and timing are implemented and to EVAL.md for evaluation and benchmarking guidelines, but it marks EVAL.md as work in progress. That is the thinnest part of an otherwise well-specified project, and it is the part you most need if you intend to publish numbers.

What the repository actually contains, and what it deliberately does not

The directory layout is a benchmark, not a product. KernelBench/ holds the dataset files. src/kernelbench/ holds the logic, with unit_tests/ and prompts/ underneath it. scripts/ holds the runnable entry points, results/ holds baseline times across hardware, runs/ is where your own runs land, and notebooks/ holds analysis examples. Dependency management moved to pyproject.toml with uv as the tool.

The README draws a line that is worth reading twice: the repo is not intended to provide complex agentic scaffolds that solve this task, and the recommendation is to clone and modify it or use it as a git submodule. So the deliverable is a measurement harness plus a fixed problem set. If your actual goal is a pipeline that generates, validates and ships kernels, KernelBench gives you the scoring half and leaves the generation loop to you. That is a reasonable scope decision, but it means any comparison you read between two KernelBench numbers is only as meaningful as the scaffolds behind them, and the repo does not standardize those.

Getting a single problem to run: uv, litellm and the one command that matters

Setup assumes uv. The base install is uv sync, which the README notes works without a local GPU. Local GPU evaluation needs uv sync --extra gpu. AMD users are told to remove torch and re-add it against the ROCm 7.1 index, and the README recommends Docker for that path because of ROCm setup complexity. A conda route with python=3.10 and requirements.txt still exists as a fallback.

API keys go through litellm, configured by copying .env.example to .env. Running and profiling kernels requires a GPU, so without one you either use Modal (modal token new after creating an account, then the generate_and_eval_single_sample_modal.py script) or the tutorial notebook, which the README links as a Colab notebook and also ships at notebooks/tutorial.ipynb.

The single-problem command is the fastest way to see whether the harness works on your machine:

uv run python scripts/generate_and_eval_single_sample.py dataset_src=huggingface level=2 problem_id=40 server_type=google model_name=gemini/gemini-2.5-flash

dataset_src accepts local or huggingface, and adding .verbose_logging increases output. The README also flags gpu_arch as something you will likely need to adjust for your card. For one sample against a reference, scripts/run_and_check.py checks correctness and computes speedup, with eval_mode=local or eval_mode=modal. Aggregate scoring runs through scripts/greedy_analysis.py.

Where the harness will fight you

The GPU requirement is not incidental. Without a local card you are on Modal or Colab, and that changes the timing environment, which matters because the metric is wall-clock. The README does not claim the comparison is hardware-independent, and it ships results/ as baseline times across hardware, which implies you should be checking which hardware a given baseline belongs to rather than assuming one number travels.

gpu_arch is the second friction point. It is listed under what you might need to modify, not as an optional tweak, so a first run that produces nonsense timings is more likely a misconfigured arch than a broken model. Third, the version situation is genuinely confusing: main is the latest stable branch, v0.1 and v0 are separate branches, and the HuggingFace dataset is described as updated to v0.1. If you pull from HuggingFace while reading code on main, confirm which problem set you actually received. Fourth, the licence is reported as NOASSERTION, which means GitHub could not map the file to a known identifier. That is not a statement about what the licence permits. Read the licence file yourself before you build anything commercial on it, and treat the repository as the only authority.

The alternative: Triton and torch.compile, and why they answer a different question

If the goal is faster kernels rather than evidence about a model, the comparison is not another benchmark. It is Triton, where you write the kernel in a Python-embedded DSL and the compiler handles much of the low-level scheduling, and torch.compile, where you write nothing new at all and let the compiler fuse and lower your existing PyTorch. Both change the unit of work. KernelBench asks a model to emit raw CUDA, and the README notes the project is actively extending to other DSLs beyond cuda, so the authors are aware the target language is a variable rather than a constant.

That is the real difference in approach. With torch.compile you are trusting a compiler to find the fusion; with Triton you are writing the kernel yourself in a higher-level language; with KernelBench you are measuring whether a model can write the low-level kernel and whether it is worth running. If your answer to the speedup question is that you do not need a model in the loop, KernelBench is the wrong tool and the README's own framing says as much by positioning the repo as a benchmark and environment.

Maintenance and the cost of staying current

There are no retrieved releases, so there is no versioned artifact to pin against. The README says the latest stable version is on main and that the repo continues to be updated, with v0.1 and v0 pointing at earlier states. Practically, that means tracking main is the supported path, and a run you did last month may not be reproducible against main today without checking out a branch.

The dependency surface is where upgrade cost concentrates. Torch is pinned through uv against a specific index, and the ROCm path requires ROCm 7.1 or newer, which the README itself calls complex enough to warrant Docker. Moving to a new ROCm or CUDA release is therefore an environment migration, not a version bump. For a benchmark this is the cost that matters most, because a number is only comparable to another number produced under the same environment. The absence of releases makes that harder to guarantee, not easier.

Editorial conclusion

Adopt KernelBench if you are measuring kernel-generation quality and want a fixed 250-problem task set with a speedup-aware metric rather than a pass/fail score. Do not adopt it if you need a working agent that produces kernels: the README states the repo is not intended to provide complex agentic scaffolds and recommends cloning and modifying it. Before committing, verify three things: whether the HuggingFace dataset you pull is on v0.1 or the older v0 branch, what gpu_arch value matches your card, and whether your generated kernels are being timed against the baseline in results/ for your hardware or a different one. If you have no local GPU, confirm your Modal token works by running scripts/generate_and_eval_single_sample_modal.py before building anything on top.

Official sources

1. Issues
2. Project website
3. README
4. ScalingIntelligence/KernelBench on GitHub