AITER: AMD's operator library for ROCm, and the default behind vLLM on AMD GPUs

An operator library, not a framework

AITER stands for AI Tensor Engine for ROCm, and the README describes it as AMD's high-performance AI operator library providing optimized GPU kernels for inference and training workloads on ROCm. The framing that matters is that it serves as a unified collection of production-ready operators that framework developers can integrate directly into their stacks. That is a library relationship, not a framework relationship. AITER does not serve requests, schedule batches or manage a model; it provides the individual pieces, attention, MoE, GEMM, normalization, quantization, communication, that a serving stack needs, and a serving stack decides which ones to use. The key features list makes the same point in four bullets. There are C++ and Python APIs, so you can call an operator from either level. There are multiple kernel backends, namely Triton, Composable Kernel and hand-tuned assembly. There is inference and training support, not just serving kernels but also training and fused GEMM and communication kernels. And it is framework-agnostic, integrable into vLLM, SGLang or any custom framework.

Where it is actually used, and what each framework takes from it

The ecosystem section makes the strongest claim in the README, that AITER is the default kernel backend for large language model inference on AMD GPUs, integrated into the major serving frameworks and powering production workloads at scale. The framework table gives the detail. vLLM uses it as the default attention backend on ROCm, in production, taking MHA, MLA, Paged Attention, Fused MoE, GEMM, RMSNorm and RoPE with KV cache. SGLang uses it as the default in its ROCm Docker image, also in production, using attention, Fused MoE, block-scale GEMM, all-reduce and RMSNorm. ATOM is built natively on AITER and is in active development, using all of the AITER operators including attention, MoE, sampling and communication. JAX has an XLA FFI bridge with no PyTorch dependency, and is marked experimental with MHA and FMHA, RMSNorm and BF16 GEMM. Various customer proprietary inference engines take a kernel-level integration, in production, with attention, MoE, GEMM and quantization.

The performance table, and the end-to-end number that matters

The README lists operator-level speedups and one end-to-end figure. The MLA decode kernel is up to 17 times faster than the comparison, the MHA prefill kernel up to 14 times, block-scaled Fused MoE up to 3 times and block-scaled GEMM up to 2 times. The end-to-end row is the one to look at, because it is the only one that says what a user of a serving framework sees, and it is the DeepSeek-R1 run through SGLang going from 6,484 to 13,704 tokens per second, which is 2.1 times. The JAX AITER attention figure on an MI350 is quoted as a 4.39 times median. The README points to the ATOM benchmark dashboard for detailed benchmarks rather than presenting these as the whole story. The comparison baseline is not named in the table, so the sensible reading is that these are against the default kernel implementations in the upstream frameworks, which is the comparison the surrounding text describes.

Supported hardware, and what the RDNA footnote actually says

The hardware table separates what is done from what is in progress. The Instinct MI300X and MI325X on gfx942 with CDNA3 are fully supported, and the Instinct MI350 and MI355X on gfx950 with CDNA4 are listed as supported. The RDNA parts are different. The AMD Pro W7900 on gfx1100, the AI Max and Max Pro 400 and 300 series on gfx1151, and the Radeon AI PRO R9700 on gfx1201 are all marked experimental, and the footnote is worth reading properly rather than skimming. On RDNA, Triton and most FlyDSL kernels run, as do most HIP kernels covering norm, RoPE, quantization and activation plus some GEMM and attention, but most Composable Kernel and assembly kernels are CDNA-only. The footnote also says experimental does not mean suboptimal, since AITER often ships faster kernels than the defaults in upstream frameworks such as vLLM, and that installing it is strongly recommended on those parts, with ongoing RDNA optimisation work landing there first.

Running the operator tests directly

Each operator has unit tests under op_tests, and the README shows how to run one.

# Example: run a single operator test
python3 op_tests/test_mha.py
python3 op_tests/test_mla.py
python3 op_tests/test_moe.py
python3 op_tests/test_gemm_a8w8.py
python3 op_tests/test_rmsnorm2d.py

# See all available operator tests
ls op_tests/test_*.py

The test file names double as an index of what is implemented, with separate scripts for multi-head attention, multi-head latent attention, mixture of experts, an eight-bit by eight-bit GEMM and a two-dimensional RMSNorm. That is a useful way to scope an evaluation, because you can find out whether the kernel you care about exists before you spend a day on an integration. The repository layout supports the same picture, with a csrc directory for the native kernel sources, an aiter directory for the Python package, a 3rdparty directory, a gradlib directory, an hsa directory for the hardware abstraction layer, a docs directory and a githooks directory, and the aiter_worker_limits.py module at the root handling the parallel build budget.

The build has more switches than a kernel library usually does

The setup.py explains the build in terms of environment variables, and the set is worth reading before you build from source. The package is published on PyPI as amd-aiter, and the build target comes from the BUILD_TARGET environment variable, defaulting to auto. PREBUILD_KERNELS controls whether kernels are compiled ahead of time, and AITER_PY3_WHEEL opts into a wheel tagged without compiled modules for a Python-agnostic build. ENABLE_CK, on by default, turns on the Composable Kernel backend, and AITER_TRITON_ONLY is a single gate that skips the C++ and HIP build. That last one has a platform split, because the HIP SDK is a separate optional install on Windows and a Triton-only install is the supported fallback there, while Linux keeps failing loudly instead, on the grounds that ROCm is a hard prerequisite and its absence is a broken install. The build requirements pull in pybind11, setuptools, setuptools_scm, wheel, packaging, psutil, ninja, pandas and a pinned flydsl, with the flydsl pin skipped on Windows because it publishes Linux-only wheels.

Fortnightly releases, and a versioned dependency policy

The release plan is fixed cadence. AITER publishes a scheduled release every two weeks, and each one uses a release branch named after the target version, such as release/v0.1.20, with a matching release tag. Automation creates any missing branch and matching tag for the scheduled releases. The recent tags follow that, with v0.1.26 on 2026-10-19, v0.1.25 on 2026-10-12 and v0.1.24.post1 on 2026-10-02, and the last push was on 2026-10-20. The requirements file documents its own pinning philosophy, with runtime and CI dependencies version-pinned for reproducible builds and to reduce the supply chain risk of a typo-squatted or compromised upstream package. It also explains the environment markers for packages whose pinned release drops an older Python, and notes that torch and triton are intentionally not pinned there because they come from the base image as ROCm-specific builds that are not published on PyPI. The licence is MIT, with the copyright held by Advanced Micro Devices.

Editorial conclusion

AITER fits a team serving large language models on AMD Instinct hardware through vLLM or SGLang and wants the ROCm path to be the fast one rather than the fallback one. It does not fit a team on Nvidia hardware, or one on the RDNA consumer parts where the README still calls support experimental, and the operators have to be checked per GPU generation. Verify first that your Instinct card maps to a supported architecture, that the operator test for the kernel you care about passes on your machine, and read the RDNA footnote before you believe the experimental parts are equivalent.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. ROCm/aiter on GitHub