flash-linear-attention: a kernel library that tracks the linear attention literature paper by paper

A news log that doubles as the project's feature list

The first thing you notice about the README is that it leads with dates. Entries run from 2026-07 backwards through 2026-05, 2026-04, 2026-03, 2025-10, 2025-09, 2025-08, 2025-07, 2025-06, 2025-05, 2025-04, 2025-03, 2025-02, 2025-01, 2024-12, 2024-11, 2024-10, 2024-09, 2024-05, down to the 2023-12 launch.

Each entry names an architecture and links its paper: Kimi Delta Attention, Log-Linear Attention, MoBA and FlashMoBA, MLA, YOCO, DeltaFormer, Mamba3, Gated DeltaNet 2, Preconditioned Gated DeltaNet and PKDA, Parallax, Wall attention, CAT, Raven, and more. Older entries fold into a collapsible section that still runs back to the original DeltaNet implementation in 2024-05 and the initial fla release in 2023-12.

That structure tells you what kind of project this is. It is a running record of the sequence modelling literature rendered as code, where each addition corresponds to a publication rather than to a user request. The topic tags on the repository name large language models, machine learning systems, natural language processing and sequence modelling.

One entry is worth pulling out because it is a change in behaviour rather than a new kernel. The 2025-03 note records that the initializer_range was changed and then rolled back to the default value of 0.02, with a suggestion to try both for actual training. A project willing to publish that kind of churn is showing you its training defaults are unsettled, which is useful to know before you spend a week on a run.

Installing means picking a backend extra and an index URL

This is the part of the project that will cost you an afternoon if you skip it. The package name on PyPI is flash-linear-attention, the import name is fla, and the base dependencies are deliberately thin: transformers at 4.45.0 or newer, plus einops.

Torch and triton are not in the base dependencies. pyproject.toml carries a comment saying they are not in base deps and that you should pick a backend extra plus the matching extra index URL. The extras cover cuda, rocm, xpu, npu and cpu, so the accelerator choice is made at install time rather than discovered at runtime.

[project.optional-dependencies]
rocm = ["torch>=2.7.0"]
xpu  = ["torch>=2.7.0"]

The asymmetry between extras is worth noting. cuda and cpu both pin triton at 3.3 or newer alongside torch 2.7.0. rocm and xpu intentionally do not pin a triton flavour, and a comment in pyproject.toml explains why: torch sourced from the matching PyTorch wheel index pulls the right one transitively, whether that is pytorch-triton-rocm on stable or triton-rocm on nightly. The npu extra is the most specific, pinning torch at exactly 2.9.0, torch_npu at 2.9.0.post6, torchvision at 0.24.0 and triton-ascend at 3.2.2.

The minimum Python version is 3.10. Version numbers for the package itself are read out of the fla package at build time rather than hardcoded, since setup.py searches fla/__init__.py for the __version__ assignment. There are also extras for tilelang, causal-conv1d, benchmark tooling and pytest, so a development install is a separate decision from a runtime one.

Kernels, layers and models are three different things here

The README describes the project as bringing together hardware-efficient building blocks, training-ready layers, and components for modern sequence models, spanning linear attention, sparse attention, state space models and hybrid LLM architectures. Those three tiers map onto the directory layout, and the difference between them decides which part you actually need.

The fla/ops directory holds the operations, and the README links into individual op directories when it announces an addition, for example fla/ops/attnres for the fused AttnRes support added in 2026-05 and fla/ops/gated_delta_rule for Gated DeltaNet, which gained a FlashQLA backend from QwenLM in 2026-07. An op is a Triton kernel with its own numerics.

Above that sit training-ready layers, and above those the model implementations, which the README points at with a Models section covering architectures from ABC in 2022 through RetNet, GLA, Mamba, HGRN, HGRN2 and RWKV6. Model implementations are the part that saves the most work if your architecture is on the list, and the part that is most likely to need adjustment if it is not.

The third tier is where training actually happens, and the project moved it out. A 2025-01 entry says fla is migrating to a torchtitan-based training framework, with the details in a separate flame repository described elsewhere in the log as a minimal and scalable framework for training fla models. So the kernels and layers stay here and the training loop moved, which is worth knowing before you plan to train a model with this library alone.

The library changes shape under you, and the news log says so

Several entries describe breaking changes rather than additions, and they are the strongest argument for reading the log before pinning a version.

A 2024-11 entry states that inputs switched from head-first to seq-first format. That is a silent semantic change: code written for the old layout still runs and produces wrong shapes or wrong numbers. A 2024-12 entry announces that fla officially supports kernels with variable-length inputs, and a 2024-11 entry notes that fla now provides a flexible way for training hybrid models.

The 2025-01 migration to torchtitan is the third instance of the same pattern, where the surrounding framework moves while the kernels stay. Taken together these suggest treating the library as something that tracks upstream papers closely, with the corresponding cost that layouts and entry points do change.

There is also a legacy/ directory in the tree, which is the usual signal that older code paths are retained for compatibility. If you are porting from an earlier version, that directory and the CHANGELOG-style history in the README are where the differences will show up.

The good news is that the surrounding documents exist separately rather than being folded into the README. INSTALL.md, ENVs.md, FAQs.md, CONTRIBUTING.md, AGENTS.md and CITATION.cff are all in the repository root, so the detail-heavy material has a home of its own.

Hardware claims cover three vendors, and the benchmarks live in-repo

The README states that all implementations are platform-agnostic and verified on NVIDIA, AMD and Intel hardware. Platform-agnostic is doing some work in that sentence, because the extras list is where the real per-vendor detail lives, with npu getting its own pinned stack. But three vendors named as verified is a meaningfully broader claim than a CUDA-only library makes.

The repository carries a benchmarks/ directory and a benchmark extra that pulls matplotlib, pandas and datasets. A tests/ directory is present too, and the pyproject test configuration defines markers including a smoke tier described as core tests running on every pull request push, with pytest-xdist available for parallelism. You can read how a claim in the README was produced rather than taking it on trust.

There is an evals/ directory as well, which matters because for a library whose selling point is matching published results, evaluation is the part that distinguishes a real implementation from a plausible one.

The honest limit is that none of this is a substitute for running the numbers on your own hardware. Different accelerators, different torch versions and different triton builds produce different throughput, and the extras list makes clear that these are genuinely different stacks rather than one stack with a compatibility shim.

Where this sits against building the layer yourself

The alternative is not another library so much as writing the Triton kernel yourself from the paper, which is what most of these architectures would have required before this project existed. For a paper with no other maintained implementation, that is the comparison, and it is not close.

The comparison that is closer is against the model author's own reference code. Many of the architectures here were published alongside an official implementation, and the reference code is the ground truth for whether the layer computes what the paper says. A library kernel that is fast and numerically close is doing a different job from a reference implementation that is slow and exact.

Where a library like this earns its keep is breadth of backends and the fact that the layers are training-ready rather than inference-only. A reference implementation written for one paper is usually tuned for one machine, and porting it to AMD or to a distributed sequence dimension is real work. This project maintains those paths as a matter of course, which is the actual product.

The cost is that you inherit a fast-moving surface. The advantage over a reference implementation is also its risk: when a paper gets a new kernel, the surrounding conventions can move too, and the news log is where you find out. For a research codebase that trade is usually worth taking. For anything on a release schedule, pinning a version and reading the log for that version is not optional.

Editorial conclusion

flash-linear-attention is worth adopting when you are working with one of the architectures in its news log, usually because a model you need to train or reproduce has no other maintained implementation. What you are adopting is a research library, not a product: it adds a new paper roughly monthly, the dependency tree asks you to pick a backend extra and match an extra index URL, and torch and triton are deliberately kept out of the base dependencies so that choice stays explicit. Version v0.5.2 shipped on 2026-07-27 and the last push was on 2026-09-21, so the pace is fast and will not slow down for you. Read INSTALL.md and ENVs.md, which exist as separate documents for a reason, before installing anything, and confirm your accelerator is one of the three vendors the README names.

Frequently asked questions

Official sources

1. fla-org/flash-linear-attention on GitHub
2. License: MIT
3. Project website
4. README
5. Releases