RWKV-LM: An Attention-Free RNN With Constant-Size State
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
At a glance
- What is it?
- RWKV-LM is the reference training and inference repository for RWKV-7 "Goose", a 100% RNN architecture with linear-time, constant-space decoding and no KV cache. The core judgement: the training code is deliberately not a drop-in PyTorch layer, and the README says so.
- Who is it for?
- Adopt RWKV-LM if you need constant VRAM decoding, a state you can carry across sessions, or a 1-GPU training entry point, and you are willing to work from the train_temp reference implementation rather than a packaged layer. Do not adopt it if you need a drop-in nn.Module to swap into an existing transformer training stack, or if you depend on Hugging Face Trainer-style abstractions.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 13 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The KV cache is the thing RWKV-LM removes
Standard transformer inference keeps a key-value cache that grows with every generated token. Memory rises with context length, and throughput per sequence degrades as the batch fills that cache. RWKV-LM is built around an architecture where that cache does not exist. The README describes RWKV-7 as linear-time and constant-space, with no kv-cache, and it publishes a state size comparison against a transformer of the same width. For a 24-layer, 1024-dimension model, the README gives RWKV-7 state_params as L*(2*D+64*D) = 1.622016 M, against a Qwen3.5 figure of 5.050368 + 6.144*(T/1000) M, where T is the context length. That is the whole argument in one line: one number versus a number that grows with T. The intended audience is people serving long-context or high-batch-count inference, and people who want a fixed-size recurrent state they can persist between requests rather than recompute from a prompt.
How RWKV-7 differs from a transformer in the training loop
The README makes a claim that is unusual for a model repository: RWKV-7 is described as a meta-in-context learner that performs in-context gradient descent on its state at every token. Architecturally it is attention-free and 100% RNN, yet the README states it can be trained like a GPT transformer in parallel. That combination is the reason the repository exists as training code rather than as a model definition file. The README is explicit that RWKV-7 is not just an attention replacement. It says the model depends on carefully set initialization, weight decay and learning rate for each parameter, and that the price of this is that there is no good simple RWKV-7 layer, because a PyTorch layer cannot guarantee it is using the correct init and hyperparameters. The README also notes that FLA RWKV-7 is not aligned with the reference implementation and performs worse, and links a comparison repository. So the data flow is: you take the reference training script, and the correctness of the model lives in the script's hyperparameter choices, not in a module boundary.
Getting the reference implementation running
The README directs users to the RWKV-v7/train_temp directory as the RWKV-7 reference implementation, and states the default config requires 1 GPU with 7G VRAM, with the option to reduce bsz if you have less. A simplified training demo lives at RWKV-v7/train_temp/rwkv7_train_simplified.py, and the README says all of its important settings are shown in that file. Three of those settings are stated plainly. Use PreLN LayerNorm instead of RMSNorm, which the README ties to a better initial state given that no trainable initial state is used. Apply weight decay only to large matrix parameters (basically projections), which the README flags as very important. Use correct initialization. The fastest CUDA kernels are pointed to at RWKV-v7/train_temp/cuda and at Albatross's rwkv7_fast_v3a.py. A second kernel set, described as vanilla, state-tuning and state-passing infctx, simpler but slower, is at RWKV-CUDA/rwkv7_fast_fused. For inference there is a separate repository, BlinkDL/Albatross, and a numpy reference at RWKV-v7/rwkv_v7_numpy.py.
The limitation the README states outright
Most model repositories bury the awkward part. This one puts it in bold. There is no good simple RWKV-7 layer, because a PyTorch layer cannot ensure it is using the correct init and hyperparameters. If your workflow is to import a module, wrap it in your own training harness, and let your existing optimizer and scheduler handle everything, RWKV-LM does not support that path today. The README's instruction is to study the train_temp code, which it describes as only several hundred lines, and change it to suit your task. That is a real cost. It also means the FLA port is not a substitute: the README states FLA RWKV-7 is not aligned with the reference implementation yet and performs quite worse. A second constraint is hardware. The README notes that the current RWKV7 kernel gets faster as you increase Bsz*HeadCount, which means small-batch training on a single modest GPU may not see the throughput the headline numbers describe. Those numbers (270k tokens/s for a 7.2B bf16 run on 4x8xH100 at ctx10240 with DeepSpeed zero2+gradcp) are published in the README as the author's configuration, not as a portable benchmark.
What the constant state buys you at inference
The README's Albatross section lists decoding throughput for RWKV-7 7.2B fp16 on an RTX5090: 145+ token/s at bsz1, 10250+ token/s at bsz960, 9650+ token/s at bsz320, and 11289 token/s prefill at bsz1. Each line carries the same parenthetical: always const speed and vram. That qualifier is the point. A transformer's batch-960 decode would be bounded by a KV cache sized for 960 sequences times context length; here the memory figure does not move with batch or context. The README frames this as making RWKV-7 ideal for large bsz inference because the state size is particularly small. If you are provisioning for a workload where many concurrent sequences share one GPU, this is the property to evaluate. If you are running a single short chat session, the advantage is much smaller and a well-optimized transformer runtime may be competitive.
Alternatives and where they actually differ
The obvious comparison is a transformer implementation you already run, and the difference is not subtle: it keeps a KV cache that scales with context, RWKV-7 keeps a fixed recurrent state. The more interesting comparison is FLA, the Flash Linear Attention library. Both target linear-attention style models, so a reader might assume they are interchangeable. The README says they are not: FLA RWKV-7 is not aligned with the reference implementation and performs worse, and it links the RWKV-FLA-comparison repository as evidence. The reason given circles back to the design: RWKV-7's behaviour depends on per-parameter init, weight decay and learning rate, which a generic linear-attention layer does not carry. So the choice is not "library versus reference code" on convenience alone. It is whether you want a composable layer that measurably underperforms the reference, or the reference script that you fork. The README also mentions johanwind/wind_rwkv as an alternative kernel implementation, and RWKV-PEFT for LoRA-style fine-tuning, which matters if you intend to adapt released weights rather than train from scratch.
Licence, maintenance and upgrade exposure
RWKV-LM is Apache-2.0, and the README states RWKV is a Linux Foundation AI project, so it is totally free. Apache-2.0 is permissive and includes an explicit patent grant, which matters for a model architecture that has drawn attention from commercial runtimes; this is a description of the licence text, not legal advice, and you should read the LICENSE file and any model weight terms separately, since weights are distributed on Hugging Face and may carry their own conditions. On maintenance: the repository is actively pushed, and the release history is sparse by design. The listed releases jump from 2.00 in March 2022 to 4.00 in December 2022 to 5.00 in December 2023, while the README describes work at RWKV-7 with RWKV-8 material already present in the tree. If you pin to a release tag, you are pinning to something well behind the documented state of the project. The practical upgrade cost is that your fork of train_temp is the artifact you maintain, and upstream changes to init, weight decay or kernel layout will not merge cleanly into a modified copy.
Who should start here, and what to check first
Start with RWKV-LM if you are training from scratch or continuing pretraining, if you need a recurrent state you can save and reload instead of a prompt prefix, or if you are serving many concurrent sequences and want VRAM that does not track batch size or context. The 1-GPU, 7G VRAM default config makes the first experiment cheap. Stay away if you need an off-the-shelf module to drop into an existing transformer training pipeline, if you rely on a linear-attention library for composability, or if your workload is single-stream short-context generation where constant state buys little. Before you commit engineering time, confirm that the train_temp config runs on your hardware at your intended bsz and ctx10240, and that the kernels under RWKV-v7/train_temp/cuda compile in your environment. The README's own instruction, to study the several-hundred-line reference and adapt it, is the honest description of the work involved.
Editorial conclusion
Adopt RWKV-LM if you need constant VRAM decoding, a state you can carry across sessions, or a 1-GPU training entry point, and you are willing to work from the train_temp reference implementation rather than a packaged layer. Do not adopt it if you need a drop-in nn.Module to swap into an existing transformer training stack, or if you depend on Hugging Face Trainer-style abstractions. Before committing, verify two things yourself: that the train_temp config fits your GPU at your target bsz and ctx10240, and that the CUDA kernels under RWKV-v7/train_temp/cuda build on your toolchain, since the README points at them as the fastest path and does not describe a fallback build matrix.
Community notes