ChatRWKV: Running RWKV Language Models as a Chatbot in Python
ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.
At a glance
- What is it?
- ChatRWKV is the chat-oriented front end for the RWKV language model family, a 100 percent RNN architecture that keeps state instead of recomputing attention over a growing context. It is aimed at developers who want to inspect and control inference rather than call a hosted API.
- Who is it for?
- Adopt ChatRWKV if you want to read the inference path, control the prompt format, and run RWKV weights locally on your own hardware, and if you are comfortable working from example scripts rather than a stable command line. Do not adopt it if you need a packaged application, a versioned release with a changelog, or a supported installer; the repository has no releases and the README points elsewhere for GUI and mobile use.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 58 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What ChatRWKV is for, and who should read the code
ChatRWKV is the chat-facing part of the RWKV project. The README describes it as "like ChatGPT but powered by my RWKV (100% RNN) language model", and the repository is a set of Python scripts around that model rather than a product. The intended reader is a developer who wants to run an RWKV checkpoint locally, see the token-level mechanics, and shape the prompt themselves.
The design assumption is that you already have weights. ChatRWKV does not ship a model. The README links to Hugging Face for the raw weights, for GGUF conversions, and for HF-compatible versions, and it points at the RWKV-LM repository for training and fine-tuning. If your goal is to chat with a hosted assistant, this project is the wrong layer: it is the inference and prompting layer, and everything above it (a UI, a server, a mobile app) lives in separate projects that the README lists.
How RWKV state replaces the transformer KV cache
The mechanism that makes ChatRWKV different from a typical transformer chat stack is visible in the README's short example. The model is loaded once with a strategy string, and then `forward` takes a list of token IDs plus a state, and returns logits plus a new state. The README demonstrates that feeding tokens in two calls and passing the returned state produces the same logits as feeding them in one call. That is the whole architecture in miniature: the model is an RNN, so the entire conversation history is compressed into a fixed-size state tensor rather than an attention cache that grows with every turn.
The README gives one operational instruction about that state, and it is worth taking literally: "Never call raw forward() directly. Instead, put it in a function that will record the text corresponding to the state." State and text must stay in sync, because a state that has drifted from the text you think it represents will produce plausible-looking but wrong continuations. The repository ships `rwkv_state_merger.py`, which is the kind of tool you need once state is a first-class object you can save, combine, or branch.
Strategy strings are the other half of the mechanism. The README mentions `strategy='cuda fp16'` in the example and states that ChatRWKV v2 adds "stream" and "split" strategies plus INT8, with the claim that "3G VRAM is enough to run RWKV 14B". There is a `v2/convert_model.py` script for converting a model to a strategy, which the README says makes loading faster and saves CPU RAM. That conversion step is the price of the memory savings.
Installing ChatRWKV and running a first generation
The repository's `requirements.txt` lists two direct dependencies, `tokenizers>=0.13.2` and `prompt_toolkit`, and the README separately points at the `rwkv` pip package, telling readers to "always check for latest version and upgrade". There is no setup script and no packaged entry point, so installation means cloning the repository and installing those pieces yourself.
pip install tokenizers>=0.13.2 prompt_toolkit
pip install rwkvAfter that, the README's own example is the smallest real use. It sets two environment variables, imports the model class, loads a checkpoint with a strategy, and calls `forward`. Note that the model path in the README is the author's own filesystem path, so you must substitute the location of a checkpoint you have downloaded.
os.environ["RWKV_JIT_ON"] = '1'
os.environ["RWKV_CUDA_ON"] = '0' # if '1' then use CUDA kernel for seq mode (much faster)
from rwkv.model import RWKV # pip install rwkv
model = RWKV(model='/fsx/BlinkDL/HF-MODEL/rwkv-4-pile-1b5/RWKV-4-Pile-1B5-20220903-8040', strategy='cuda fp16')
out, state = model.forward([187, 510, 1563, 310, 247], None)
print(out.detach().cpu().numpy())For an actual chat loop, `chat.py` is the reference implementation, and the README points developers at `src/model_run.py` as the easier-to-understand starting point because `chat.py` uses it. For a minimal scripted chat, `API_DEMO_CHAT.py` is the demo the README calls out for developers.
If you turn on `RWKV_CUDA_ON`, the README states that a CUDA kernel is built, which is faster and saves VRAM, and it requires `ninja` first. On Linux the README's instructions are to export `PATH` and `LD_LIBRARY_PATH` pointing at the CUDA toolkit before running `v2/chat.py`; on Windows it says to install VS2022 build tools with the Desktop C++ workload, reinstall CUDA 11.7 with the VC++ extensions, and run `v2/chat.py` from an "x64 native tools command prompt".
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATHThe chat format is not optional
The README spends more words on prompt formatting than on any other single topic, which tells you where new users get burned. The stated best format is a strict alternation with no space after the final speaker label:
```Bob: xxxxxxxxxxxxxxxxxx\n\nAlice: xxxxxxxxxxxxx\n\nBob: xxxxxxxxxxxxxxxx\n\nAlice:```
The model continues from that trailing "Alice:", and the README notes that the generation will begin with a space, which you strip. It also warns against blank lines inside a turn and gives the normalization it expects: `xxxxx = xxxxx.strip().replace('\r\n','\n').replace('\n\n','\n')`. Speaker names depend on the checkpoint: v4-raven models use Bob and Alice, while v4, v5 and v6 world models use User and Assistant. Getting this wrong does not raise an error. It produces a model that answers in the wrong voice or drifts out of the conversation, which is why the README's advice to keep text tied to state matters in practice.
Where ChatRWKV is the wrong tool
The repository has no releases. There is no versioned artifact to pin, no changelog to read before upgrading, and no compatibility promise between the scripts and a given checkpoint. The README itself instructs readers to "always check for latest version and upgrade" for the `rwkv` package, which is a maintenance instruction aimed at the user, not a guarantee from the project.
There is also a hard dependency on the Python and PyTorch path. The README links to `rwkv.cpp` for CPU inference with int4, int8, fp16 and fp32, to `rwkv-cpp-cuda` for GPU inference without Python or PyTorch, and to `ai00_rwkv_server` for a Vulkan inference API. Those exist precisely because ChatRWKV is not the right answer when you want a small binary, a language runtime other than Python, or an HTTP endpoint. If your deployment target is a phone, the README points at RWKV_APP instead.
The checkpoint situation is a second limitation. The README's headline example uses a 1B5 Pile model with a 2022 date in its name, while the current generation it promotes is RWKV-7, described as preview models hosted under a "temp" path on Hugging Face. Preview weights are not a stable interface. If you need a frozen model for a long-lived product, this is a moving target and the README does not pretend otherwise.
Alternatives and the actual difference in approach
The most direct alternative in the same ecosystem is `rwkv.cpp`, linked from the README as fast CPU inference using ggml with int4, int8, fp16 and fp32 support. The difference is not just speed: `rwkv.cpp` is a C++ implementation with quantized weights, aimed at running without a Python and PyTorch stack, while ChatRWKV is the Python reference path where you can read and modify the inference loop. If you want to change how state is handled, ChatRWKV is where that change is legible; if you want a small dependency footprint on a CPU-only box, `rwkv.cpp` is the one the README points to.
A second alternative is `ai00_rwkv_server`, which the README describes as the fastest GPU inference API with Vulkan and as good for NVIDIA, AMD and Intel hardware. That is a server with an HTTP surface, and `web-rwkv` is named as its backend. Choosing between them is choosing between a library and a service: ChatRWKV gives you `model.forward` and a state tensor, while `ai00_rwkv_server` gives you an endpoint and takes the state management away from you. Neither is a superset of the other.
Maintenance, licensing, and what upgrades cost
The repository is not archived, and the last push was on 2026-07-19. That is recent enough that the code is being touched, but the absence of releases means "upgrading" here is pulling the default branch and re-reading the scripts you depend on. The README's own framing reinforces this: it tells you to always check for the latest `rwkv` package version, and it directs anyone building an inference engine to start from `src/model_run.py` rather than treating `chat.py` as a stable API.
ChatRWKV is licensed under Apache-2.0, which is a permissive license with an explicit patent grant and a requirement to preserve notices. That covers the code in this repository. It does not automatically cover the model weights, which are distributed separately on Hugging Face under whatever terms those repositories state, and the README's links to GGUF and HF-compatible conversions point at third-party accounts. Check the license on the specific checkpoint you download; the Apache-2.0 file at the repository root says nothing about it. This is a description of what the license text and the README say, not legal advice.
The practical upgrade cost is the strategy string and the conversion step. If you converted a model with `v2/convert_model.py` for a given strategy, a change in the inference code or the `rwkv` package can invalidate that conversion, and the README does not document a rollback path. Budget for re-downloading or re-converting weights when you move between versions.
Editorial conclusion
Adopt ChatRWKV if you want to read the inference path, control the prompt format, and run RWKV weights locally on your own hardware, and if you are comfortable working from example scripts rather than a stable command line. Do not adopt it if you need a packaged application, a versioned release with a changelog, or a supported installer; the repository has no releases and the README points elsewhere for GUI and mobile use. Before committing, verify that the strategy string you plan to use matches your GPU memory, that you can build the optional CUDA kernel with ninja if you enable RWKV_CUDA_ON, and that your prompts follow the Bob/Alice or User/Assistant format the models were tuned on.
Frequently asked questions
What is ChatRWKV and how is it different from ChatGPT?
ChatRWKV is a Python chat front end for the RWKV language model, which the README describes as a 100 percent RNN that can match transformers in quality and scaling while being faster and using less VRAM. Unlike a hosted service, you supply the weights and run inference yourself.
How do I install ChatRWKV?
There is no installer. The repository's requirements.txt lists tokenizers>=0.13.2 and prompt_toolkit, and the README points at the rwkv pip package, which it says you should always check for the latest version and upgrade.
How much VRAM does ChatRWKV need to run a 14B model?
The README states that ChatRWKV v2, with stream and split strategies and INT8, needs 3G VRAM to run RWKV 14B. The memory you actually use depends on the strategy string you pass when loading the model.
What chat format should I use with ChatRWKV?
The README's stated best format alternates speakers with no space after the final label, such as Bob: ... Alice: ..., and it says there should be no blank lines inside a turn. For v4-raven models use Bob and Alice; for v4, v5 and v6 world models use User and Assistant.
Why does ChatRWKV tell me not to call forward() directly?
Because the model is an RNN and carries state, the README says to wrap forward in a function that records the text corresponding to the state. If state and text drift apart, the model continues from a context you did not intend.
Is ChatRWKV available for CPU-only or mobile use?
ChatRWKV itself is the Python and PyTorch path. The README links to rwkv.cpp for CPU inference with int4, int8, fp16 and fp32, and to RWKV_APP for local inference on Android and iOS.
Community notes