rkllama: an Ollama-style LLM server that runs on the Rockchip NPU

What rkllama is and what the NPU buys you

rkllama is a server and client for running and interacting with large language models on Rockchip platforms, with the RK3588, RK3588S and RK3576 as the targets. Its pitch is stated in the README as the difference from software of the same kind such as Ollama or llama.cpp: rkllama allows the models to run on the NPU. On a board with a 6 TOPS NPU, that is the whole argument, since the alternative is a CPU inference loop on a Cortex-A76 cluster that is far slower for transformer workloads.

The project pins its native runtime versions: rkllm-runtime 1.3.0 and rknn-runtime 2.3.2. The C++ rkllm and rknn libraries live in the lib directory and are used for inference and for fixing the platform frequency, and the server entry point is app.py, a Flask REST API, with client.py as the companion client. The models directory holds your rkllm models, and for multimodal models the matching rknn models as well.

This is a single-board-computer project in practice. The README lists the tested hardware as the Orange Pi 5 Pro, Orange Pi 5 Plus and Orange Pi 5 Max, each with an RK3588S and a 6 TOPS NPU with 16GB of RAM, and the Radxa Rock 4d with an RK3576 and a 6 TOPS NPU. The tested operating systems are Ubuntu 24.04 arm64 and Armbian Linux 6.1.99-vendor-rk35xx, based on Debian stable bookworm, version 25.2.2. Supported Python versions are 3.9 through 3.12. Version 0.0.76 is the current release, and the numbering tells you that this is still a project in active churn rather than a stable platform.

The API surface: Ollama endpoints, partial OpenAI endpoints and tool calls

The compatibility story is the feature that lets existing clients work unchanged. The Ollama-compatible side covers the endpoints a local-model client actually uses.

Those are chat, generate, ps, tags, embed with the legacy embeddings route, version and pull. Covering ps and tags means a client can ask which models are resident and which are available, and covering pull means a model can be fetched from HuggingFace without leaving the API conversation. The project also ships a dedicated compatibility guide for the Ollama API and a naming convention document, which matters because rkllama lets you refer to models with familiar names such as qwen2.5:3b.

The OpenAI-compatible side is explicitly partial, and the README is upfront about which parts work. Completions, chat completions and embeddings are covered, and so are images generations, audio speech, audio transcriptions and audio translations, the last one English only for now, the way OpenAI handles it with Whisper models. Image generation runs LCM Stable Diffusion 1.5 RKNN models through the OpenAI images endpoint, text to speech uses Piper TTS with an ONNX encoder and an RKNN decoder or MMS-TTS with RKNN, and transcription uses omniASR-CTC or Whisper running as RKNN models.

Tool and function calling is described as complete, with support for tool calls across multiple LLM formats including Qwen and Llama 3.2 and later, and there is a dedicated guide for it. The server also supports streaming and non-streaming modes, keeps message history, and lists the available models. Documentation is published in English and French, with separate files for the REST API, the Ollama compatibility, the naming convention and the tools guide.

Model residency: parallel models, automatic unloading and the prompt cache

Memory on a board with 16GB of RAM is the constraint that shapes the design, and rkllama's answer is a residency manager rather than a single loaded model. Multiple RKLLM and RKNN models can be resident at the same time, with parallel executions between distinct models in streaming mode and first-in first-out behaviour when the request is not streaming.

Models are loaded on demand. A request for a model that is not in memory triggers a load, and a model is unloaded after it expires through inactivity, with the default at 30 minutes. When a new model has to be loaded and there is not enough memory, the oldest model in memory is unloaded first. Automatic detection of the platform, RK3588 or RK3576, is included, along with a simplified custom model naming scheme and an optional debug mode behind a --debug flag.

The prompt cache is the feature that matters most for agent workloads. The server automatically saves a prompt cache file for each chat session of the same model, so a long context session can be resumed quickly when you switch back to it, which the README describes as useful with large-context chat sessions and with agents or tools such as OpenWebUI that use the same model. It also allows a previous session to be restored for fast inference even after the model was unloaded from memory. The cache files are kept for 7 days by default, the value is configurable, and they are deleted automatically if unused.

Beyond text: multimodal models and the fork-based GGUF path

The multimodal story covers the vision-language models in the QwenVL family, Qwen2VL, Qwen2.5VL and Qwen3VL, plus MiniCPMV4, MiniCPMV4.5 and InternVL3.5, so that a request can ask questions about an image. Images can be supplied as base64, as a local file or as a URL, and more than one image is allowed in the same request.

The .GGUF path is where the project depends on other people's work, and it needs careful reading. GGUF models are supported on the NPU through a llama.cpp fork by the user @invisiofficial, at the repository rk-llama.cpp. The README notes that the Dockerfile currently uses a more updated version of that fork in order to get the last features of llama.cpp, such as MTP, and it points at other forks it considers interesting, including one by @Mojo24x7.

That arrangement is the main fragility in the whole project. A model format that many users already have is supported through a chain of community forks rather than through an upstream llama.cpp release, and the Dockerfile pins a fork rather than a release. When the fork moves, the compatibility surface moves with it.

Installing rkllama from source or as a Docker image

The recommended path is a standard clone followed by a pip install, inside a virtual environment such as conda, uv or venv.

git clone https://github.com/notpunchnox/rkllama
cd rkllama

python -m pip install .

The Docker route avoids the local Python and runtime setup. The image is published on GitHub Container Registry and the run command mounts a local models directory into the container and maps the port.

docker run -it --privileged -p 8080:8080 -v <local_models_dir>:/opt/rkllama/models ghcr.io/notpunchnox/rkllama:main

The container needs privileged mode, which is worth noting as a security decision rather than a detail. The compose file in the repository makes the same choice, running privileged, mounting the models directory, mapping 8080 to 8080 and additionally mapping 11432 to 8080, which makes the service reachable on the port that Ollama conventionally uses, and restarting the container unless it is stopped.

docker compose up --detach --remove-orphans

The Dockerfile is Ubuntu 24.04 based, and its comments explain a portability trap that is worth reading before you build your own image. The RK3588 core is ARMv8.2-A and does not support Pointer Authentication, which arrived in ARMv8.3-A, and Debian Trixie ARM64 Python builds use PAC guards, so the kernel returns ENOEXEC and the container fails at startup with an exec format error. Ubuntu 24.04 ARM64 packages avoid PAC and run correctly on that hardware. The image also builds the rk-llama.cpp fork with the RKNPU2 and KleidiAI CPU options enabled and GCC 14 as the C compiler, and it creates a virtual environment to work around the externally-managed-environment restriction of modern Debian packaging.

Running the server and the limits of the approach

The server is started with the models directory and the path to a build of the llama.cpp fork. Starting it also handles the environment the README describes, starting the conda virtualisation automatically and setting the NPU frequency. The same command with the --debug flag turns on the server's optional debug mode.

rkllama_server --models <models_dir> --llamacpp <bin_folder_of_build_llama.cpp_fork>

The first limitation is hardware lock-in, and it is absolute. The NPU path exists for Rockchip silicon, the runtimes are Rockchip's own rkllm and rknn libraries, and none of this changes if you would rather run the same model on a CPU or a GPU. The comparison with Ollama and llama.cpp is a comparison of execution targets, not of features, and choosing rkllama means accepting that the deployment story only applies to a board family.

The second limitation is the version treadmill. The project is at 0.0.76, the release history in the repository shows versions from 0.0.1 onward, and the feature list is still growing quickly across multimodal, image generation and speech. The dependency list in pyproject.toml is pinned hard, Flask 2.3.2, transformers 4.57.6, torch 2.8.0, numpy 1.26.4, and the rknn-toolkit-lite2 wheels are shipped as platform-specific binaries for Python 3.9 through 3.12 on aarch64 only. That set of wheels is the practical bound on the Python versions you can use, and it is narrower than the versions the README claims to support.

The alternative to weigh is CPU inference with llama.cpp on the same board, or a bigger machine with a GPU. The CPU route is slower but has none of these constraints and keeps .GGUF support in upstream hands. Moving to a desktop with an accelerator is more expensive but is where the models actually get fast. rkllama makes sense when the board is the constraint and the NPU is otherwise idle.

Editorial conclusion

rkllama is the practical choice for putting a local model on an Orange Pi 5 or Radxa Rock 4d and serving it over the Ollama or OpenAI shapes of API, especially when the NPU has to do the inference. It is not a substitute for a datacentre GPU and it does not help on hardware without an RKLLM path. Verify first that your board is RK3588 or RK3576, that the rkllm-runtime and rknn-runtime versions match the ones the project pins, and that your models are in a format the NPU runtime can load, since .GGUF support still depends on a community fork.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. NotPunchnox/rkllama on GitHub
4. README
5. Releases