NATTEN runs neighborhood attention as a fused CUDA kernel

Locality in self attention, borrowed from convolution

The idea the project builds on is stated in one sentence. Neighborhood Attention introduces locality and sparsity into self attention in a manner similar to convolution, and the consequence is that for any self attention problem you can specify a `kernel_size`, a `stride` and a `dilation`, and because it is attention you can also toggle causal masking. That is a different contract from a windowed convolution, because the values inside the window are still computed with the attention formulation, so a token still attends to the others in its neighbourhood with learned weights rather than a fixed kernel. The paper lineage is spelled out in the README. The original neighborhood attention paper introduced the mechanism and NATTEN itself, the dilated paper introduced `dilation` for sparse global context, and the fused paper introduced the multi-dimensional kernels, causal neighborhood attention and parameters that vary across dimensions.

The parameter space, dimension by dimension

What makes the library more than one operator is that the parameters are per dimension rather than global. The README says NATTEN is dedicated to multi-dimensional layouts of tokens, with 2-D and 3-D feature maps linked from the operations documentation, and that users have the freedom to explore the massive parameter space in which the attention span in any dimension or axis of the input can be controlled with its respective `kernel_size`, `stride`, `dilation` and `is_causal` parameters. The visualisation table in the README makes the point with four examples at the same window size, showing plain neighborhood attention, the same window with `dilation=(2,2)`, the same window with `is_causal=(True,True)`, and the same window with `stride=(2,2)`. The latest addition in the generalisation paper is strided neighborhood attention, which is what makes a strided window a first class option rather than a trick.

The kernels, and the GPUs they cover

This is a CUDA project wearing a Python hat, and the README is clear about the hardware. NATTEN provides Fused Multi-Headed Attention, FMHA, and Fused Neighborhood Attention, FNA, training and inference kernels for all NVIDIA architectures since Maxwell, which it names as SM50 and later. It also ships Hopper, SM90, and Blackwell, SM100 and SM103, native kernels, offering speedups proportional to the reduction in FLOPs over cuDNN and Flash Attention 3. That proportionality is the honest framing to keep in mind. The gain comes from not computing the attention pairs you do not want, so the smaller your window relative to the full sequence, the larger the saving, and a model that needs a large receptive field will see less benefit.

The build is a compiled extension against torch

The package is a Python distribution that compiles a C++ extension, and the build metadata says so plainly. The `pyproject.toml` names the project NATTEN, a Neighborhood Attention Extension, with a dynamic version, requires Python 3.9 or later, and a build system that requires setuptools 64 or later, torch and build 1.3.0 or later. The `setup.py` imports `torch.utils.cpp_extension` and defines a custom `build_ext` command, which is the mechanism by which the CUDA sources under `csrc/` are compiled, and it detects whether libtorch was built with the C++11 ABI, which is the kind of check that saves an afternoon when linking fails. The runtime requirements file is short, listing only torch, since the compiled extension is the dependency. The top level directory also carries a `Makefile`, a `Dockerfile`, a `WindowsBuilder.bat`, `third_party/`, `scripts/`, `tests/` and `docs/` with an `mkdocs.yml` for the documentation site.

The container path, and the architecture list you build for

The Dockerfile is the documented route to a working build, and its build arguments are the interesting part. The base image defaults to an NGC PyTorch image, and the comment at the top says it builds NATTEN for a range of architectures with a number of workers, and that you can customise both. The arguments that exist are `NATTEN_VERSION`, `NATTEN_VERBOSE`, `NATTEN_CUDA_ARCH` with a default list covering 7.5 through 12.0, `NATTEN_AUTOGEN_POLICY` with the choices default, coarse and fine, and `NATTEN_N_WORKERS`, which defaults to 4 with the comment that coarse is used by default since the default number of workers is only 4. The comments give the two build commands:

docker build --build-arg BASE_IMAGE=nvcr.io/nvidia/pytorch:25.04-py3 ...
docker build --build-arg NATTEN_N_WORKERS=128 --build-arg NATTEN_CUDA_ARCH="9.0;10.0" ...

The install step in the image pulls a released version from PyPI rather than compiling from source, and the commented-out block below it shows how to clone the repository and run `make` instead, with the comment noting that building from source is not recommended unless you want unreleased features or custom branches.

Licence, citations and a fast release line

The licence is MIT, and the package metadata names Ali Hassani as author and maintainer, with the homepage and documentation both at natten.org and the PyPI project named NATTEN. The README asks users who found NATTEN or neighborhood attention useful to cite the appropriate papers, and lists four in a progression that maps onto the features. The original neighborhood attention paper, Neighborhood Attention Transformer, appeared at CVPR 2023. The dilated paper introduced `dilation` for sparse global context. The fused paper, Faster Neighborhood Attention, appeared at NeurIPS 2024 and introduced the multi-dimensional kernels, causal neighborhood attention and parameters that vary across dimensions. The generalised paper, from 2025, added even-sized windows, strided neighborhood attention, block-sparse forms, a NATTEN Simulator and the new Hopper and Blackwell FNA kernels implemented with out-of-kernel token permutation. The release history is steady, with v0.21.5 in February 2026, v0.21.6 in April and v0.21.7 on 2026-07-26, which is also the date of the last push.

Editorial conclusion

Adopt NATTEN when your model needs sliding window attention over image or volumetric tokens and you want the locality of a convolution with the formulation of attention, since the fused kernels avoid building the full attention matrix and the parameter space covers the common variants. Do not adopt it as a drop-in replacement for dense attention, because the window structure is a modelling choice that changes what the network can see, and it is not a general purpose sparse attention library. Verify first that your GPU architecture is covered and that the build has the CUDA toolchain you need, since the package compiles a C++ extension against torch and the container path is the documented way to get a working build.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. SHI-Labs/NATTEN on GitHub