AI00 RWKV Server: an OpenAI-compatible runtime for RWKV on Vulkan
The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.
At a glance
- What is it?
- AI00 RWKV Server wraps the web-rwkv engine behind OpenAI-shaped HTTP endpoints so you can serve RWKV models on AMD, Intel or integrated GPUs without CUDA. The trade-off is a narrow model format and a small, single-maintainer surface area.
- Who is it for?
- Adopt AI00 RWKV Server if you already run RWKV weights and your hardware is Vulkan-capable but not CUDA-capable, and you want an OpenAI-shaped endpoint without a PyTorch install. Do not adopt it if you need a broad model zoo, multi-user isolation guarantees, or a project with a large maintainer bench.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 99 days ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What AI00 RWKV Server is actually for
This is an inference server for RWKV language models, built on the web-rwkv engine and written in Rust. The README states the goal plainly: it supports Vulkan parallel and concurrent batched inference and runs on GPUs that support Vulkan, with no requirement for Nvidia hardware. That sentence is the whole pitch. If you have an AMD card, an Intel Arc, or a recent integrated GPU, and you want to serve a local model without installing CUDA, this project exists for that situation.
The secondary audience is anyone who wants an OpenAI-compatible HTTP surface in front of a non-OpenAI model. The server exposes the familiar paths under /api/oai, so a client that already speaks the OpenAI API can be pointed at http://127.0.0.1:65530/api/oai and, in the documented Python example, work with only an api_base change. The README lists chatbots, text generation, translation and Q&A as intended uses, which is the standard list and not a differentiator. The differentiator is the hardware path.
The mechanism: web-rwkv, Vulkan, and an OpenAI-shaped facade
The architecture has three visible layers. At the bottom is web-rwkv, the inference engine the README credits as the basis of the project. Above it sits the Rust server, which owns model loading, batching and the HTTP surface. On top is a compatibility layer that maps OpenAI request and response shapes onto RWKV generation. The README says the data input and output format follow the OpenAI API specification, and that chat and completions carry additional optional fields for advanced functionality.
Those extra fields are the interesting part, because they leak RWKV-specific concepts into an OpenAI-shaped request. The Python example passes system_name, user_name and assistant_name, plus half_life, alongside conventional parameters like max_tokens, top_p, temperature, presence_penalty and frequency_penalty. half_life has no OpenAI equivalent. It is an RWKV-side control over how quickly older context decays, and its presence in the request body is the clearest sign that this is a compatibility shim rather than a reimplementation. The example also sets stop to ['\x00','\n\n'], which suggests the model's native output stream needs explicit terminators rather than relying on the server to trim them.
Batching is claimed as parallel and concurrent, which matters because RWKV's state-based design allows sequence-level parallelism in ways that transformer KV-cache serving does not. The README does not describe the scheduler, queue depth, or how concurrent requests share GPU memory, so treat the concurrency claim as a capability statement, not a capacity number.
Getting it running: binaries, config, and the .st requirement
The documented path is short. Download a release, place a model file under assets/models/, edit assets/configs/Config.toml if you need to change the model path or quantization layers, then run ./ai00_rwkv_server. The WebUI is served at http://localhost:65530, or https://localhost:65530 when tls is enabled in the config. Building from source follows the usual Rust sequence: git clone, cargo build --release, cargo run --release.
Three command-line arguments are documented: --config for the config file path (default assets/configs/Config.toml), --ip for the bind address, and --port for the listening port. Everything else, including the model path and quantization settings, lives in the TOML file rather than on the command line. That split is worth noting if you plan to run several models: you either edit the file or pass a different --config per instance.
The constraint that shapes deployment is model format. The README states it only supports Safetensors models with the .st extension. Torch .pth checkpoints must be converted first, either with the Python script assets/scripts/convert_safetensors.py (requiring torch and safetensors), with the prebuilt converter binary from the releases page, or from source via cargo run --release --package converter -- --input /path/to/model.pth --output /path/to/model.st. If your weights are already .st, this step disappears. If they are not, you have added a Python dependency or a conversion step to your pipeline.
The API surface is narrower than the OpenAI name suggests
The documented endpoints are models, chat/completions, completions and embeddings, each available with and without the /v1 prefix. That is a working subset, not the full OpenAI surface. There is no mention of files, fine-tuning, moderations, assistants or image endpoints, which is unsurprising for a local runtime but worth stating because the README's framing invites the assumption of drop-in parity.
The embeddings endpoint is listed but the README gives no example of it and does not explain which RWKV model produces the vectors or how dimensionality is determined. The repository description mentions embed, RAG and AI agents, but the README body does not document a RAG pipeline or an agent framework. Those features exist in the project's self-description without corresponding documentation in the material available here. If RAG or agent orchestration is why you are looking at this project, the README does not tell you how to use them.
The schema is served locally at http://localhost:65530/api-docs, which is the right place to resolve these gaps. The README points there for the optional fields on chat and completions rather than enumerating them, so the authoritative parameter list lives in the running server, not in the repository documentation.
Where it breaks down
The first limitation is model lock-in. This server runs RWKV and only RWKV. The OpenAI-compatible endpoint does not mean it can load a GPT-architecture checkpoint, a Llama derivative, or anything else in Safetensors form. If your evaluation involves comparing several model families, this server covers one of them, and you will need a second runtime for the rest.
The second is the .st-only rule combined with a conversion script that depends on torch. The prebuilt converter binary softens this, but the README does not state which .pth variants it handles or whether conversion is lossless for quantized checkpoints. It also does not describe what happens when a model file is present but incompatible: whether the server fails at startup with a clear error or at first request. That is the kind of failure mode you want to know before putting it behind a load balancer.
The third is operational opacity. There is no documented health endpoint, no metrics surface, and no discussion of how concurrent requests are queued or rejected under memory pressure. The config file is described only as covering the model path and quantization layers, with no full key list in the README. For a single-user local setup this is fine. For anything multi-tenant, the documentation does not give you enough to reason about isolation or backpressure.
How it differs from llama.cpp and Ollama
The closest comparison is llama.cpp's server, which also targets CPU and non-CUDA GPU inference and also exposes an OpenAI-compatible endpoint. The difference is the model architecture and the acceleration backend. llama.cpp is built around GGUF-quantized transformer models and supports a broad family of architectures; AI00 RWKV Server is built around web-rwkv and runs RWKV checkpoints in Safetensors form on Vulkan. If your model is a Llama or Mistral derivative, llama.cpp is the direct answer and this project is not. If your model is RWKV, llama.cpp is not the tool.
Ollama is the other comparison, and the difference is packaging philosophy. Ollama manages model downloads, a registry, and a background daemon with a CLI in front. AI00 RWKV Server ships as a single executable plus a TOML file and a model you place yourself under assets/models/. There is no registry and no model pull command in the documented workflow. That is less convenient and more predictable: you control exactly which file is loaded and where it lives. The cost is that upgrades, model swaps and multi-model setups are manual file operations rather than one-line commands.
Maintenance, licensing, and what the release cadence tells you
The licence situation is stated inconsistently in the source material. The repository metadata says MIT, the README badge says MIT/Apache-2.0, and the README body says 100% open source and commercially usable, under the MIT license. Those are not the same claim. Before shipping anything commercial, read the actual LICENSE file in the repository rather than the badge, and note that a dual MIT/Apache-2.0 grant and a pure MIT grant carry different patent clauses. This is a factual discrepancy, not a legal opinion.
Release cadence is uneven. v0.6.1 landed in September 2025, v0.6.2 in October 2025, and v0.7.1 in June 2026, an eight-month gap. The README credits eight contributors, and the project is not archived. That combination suggests active but low-frequency maintenance, which is normal for a runtime that tracks an upstream engine. The practical upgrade cost is tied to web-rwkv: when the engine changes its model format expectations, this server has to follow, and your .st files may need reconversion. The README does not describe a migration policy or a compatibility matrix between server versions and model versions, so pin your server version alongside your model file and treat both as a unit when upgrading.
Editorial conclusion
Adopt AI00 RWKV Server if you already run RWKV weights and your hardware is Vulkan-capable but not CUDA-capable, and you want an OpenAI-shaped endpoint without a PyTorch install. Do not adopt it if you need a broad model zoo, multi-user isolation guarantees, or a project with a large maintainer bench. Before committing, verify three things yourself: that your GPU's Vulkan driver actually loads the model, that your weights are already in .st form or that the converter runs cleanly on them, and that the fields you depend on (half_life, the name fields, the extra stop handling) behave as the Python example implies. Those three checks decide the adoption, not the README.
Community notes