RunAnywhere: One C++ Core, Eight SDKs, and the Reality of On-Device AI

What RunAnywhere Actually Solves

RunAnywhere addresses the fragmentation of on-device AI inference. Developers face a choice between llama.cpp, MLX, Core ML, ONNX, and vendor NPU SDKs, each with different APIs and hardware targets. RunAnywhere wraps these into one semantic API exposed through eight SDKs: Swift, Kotlin, C++, React Native, Flutter, web, and others. The core is written in C++, and the project claims production readiness with recent releases like cpp-desktop-v0.20.37. The target user is a team building mobile, desktop, or browser apps that need LLM chat, vision, speech, RAG, or image generation without sending data to a server. The README emphasizes privacy by default and offline operation, which matters for regulated industries or consumer apps with connectivity constraints. The value proposition is that your code rarely picks hardware; the capability registry does that for you.

Architecture: Capability Registry and Engine Routing

The architecture diagram in the README shows eight SDKs sitting over one C++ core. That core includes a capability registry that routes every call to the best engine available on the device. The engines are QHexRT for the Snapdragon Hexagon NPU, MLX for Apple silicon, llama.cpp for general CPU and GPU (Metal on Apple, CUDA on NVIDIA as an opt-in build, WebGPU in the browser), sherpa plus ONNX for speech and embeddings, and Core ML for diffusion. The registry does not assume an engine is present just because the enum exists. The README explicitly warns that enum presence alone does not mean an engine is installed, and it instructs developers to call RunAnywhere.capabilities() (v4) to discover what the current package and device can execute. This is a critical design point: the API is static, but the runtime capability is dynamic and hardware-dependent. The routing logic picks the highest-priority engine that fits the device, which means the same code path can run on a phone NPU, a laptop GPU, or a browser without changes.

Getting Started: Commands and Config Keys

The README provides a Python quick start that is straightforward. Install with pip install runanywhere, then import, initialize, and generate. The example uses ra.initialize() and ra.llm.generate with an LlmOptions object that takes a model parameter, such as model="qwen2.5-0.5b". The comment notes that downloads happen on first use. For a terminal-based workflow, there is a separate CLI repository called RCLI, installed via brew install runanywhereai/tap/rcli or a curl script. The CLI command is rcli run qwen3 "Explain on-device AI in one sentence.". For Swift on iOS or macOS, the steps are more explicit: import RunAnywhere and LlamaCPPRuntime, call LlamaCPP.register(), then RunAnywhere.initialize(). Loading a model requires a RAModelLoadRequest with fields like modelID, category (.language), and framework (.llamaCpp). Generation uses RALLMGenerateRequest with a prompt, and the result text is returned asynchronously. The README also shows adding the MLX backend by importing RunAnywhereMLX and calling MLX.register(). These are real commands and config keys, but the material does not include Kotlin, React Native, or web examples.

Supported Capabilities and Their Caveats

The README lists a broad range of capabilities: LLM chat with token streaming, structured output with schema-validated JSON, tool calling with an agent loop, vision (VLM), computer-use action parsing (CUA), speech-to-text, text-to-speech, voice agents, embeddings, RAG, and image generation. Each has a caveat. Structured output relies on constrained decoding where the engine supports it, so not every backend will enforce the schema. Tool calling supports parallel calls only when the engine and capability report support. CUA is explicitly not a full autonomous agent framework; it only parses Fara1.5-style action strings into viewport-scaled coordinates. Voice agents do not implement wake-word detection. Image generation is platform-gated: Stable Diffusion runs on Core ML, and inpainting on the Hexagon NPU. The README also notes that LiteRT and ExecuTorch are reserved framework values only, not integrated runtimes. This is a clear honesty about what is not done, which is refreshing, but it also means the marketing phrase "every capability" must be read with those caveats in mind.

Real Limitations and Wrong-Tool Cases

The most concrete limitation is the gap between the advertised engine list and actual integration. LiteRT and ExecuTorch are named but not integrated, so if you were hoping to use TensorFlow Lite or PyTorch ExecuTorch for a specific model, RunAnywhere will not route to them. Wake-word detection is absent from voice agents, which is a dealbreaker for any always-on assistant use case. The README also states that the capability registry is the only way to know what runs, which implies that a model that works on one device may fail on another without a clear error at compile time. Image generation is limited to Core ML and Hexagon NPU, so on a generic x86 Linux server you may have no diffusion backend at all. The wrong tool scenario is a team that needs a single, predictable inference stack across all their devices; RunAnywhere's dynamic routing introduces variability that could be hard to debug in production. Also, the license is listed as NOASSERTION on GitHub, with a custom "RunAnywhere License" badge, which is a red flag for enterprises that require standard open-source licenses like Apache-2.0 or MIT.

Alternatives: llama.cpp and MLX Are the Building Blocks

The most direct alternative is to use the underlying engines directly. llama.cpp is a standalone C++ library for LLM inference that runs on CPU, Metal, CUDA, and WebGPU, and it is the engine RunAnywhere uses as its fallback. If you only need LLM chat on desktop or mobile, you can integrate llama.cpp yourself and avoid the abstraction layer. Similarly, MLX is Apple's array framework for machine learning on Apple silicon, and it provides its own Swift and Python APIs for LLMs and diffusion. The difference in approach is that RunAnywhere offers a unified API and a routing layer, while direct use of llama.cpp or MLX gives you fine-grained control over model loading, quantization, and engine-specific features like LoRA adapters. The trade-off is that you must write platform-specific code for each target, which is exactly the problem RunAnywhere tries to solve. For teams with a single platform, direct integration is often simpler and more predictable. For multi-platform teams, RunAnywhere's registry could save time, but you must trust its routing logic and keep up with its release cycle.

Maintenance, Upgrades, and Licensing

The repository shows active maintenance with releases almost weekly, such as v0.20.35, v0.20.36, and v0.20.37 in early September 2026. That cadence suggests the project is under active development, but it also means you will need to track frequent updates to the SDKs and the C++ core. The README mentions a RunAnywhere Console that deploys models and collects telemetry, which implies a cloud component, but the material does not specify whether the console is self-hosted or a paid service. The license is not open source in the conventional sense. The GitHub license field is NOASSERTION, and the badge says "RunAnywhere License". That means you cannot assume you have the right to redistribute or modify the SDKs under standard terms. The README does not include a copy of the license text, so you must request it from the project or check the LICENSE file in the repository before using it in a commercial product. If your organization has legal requirements for permissive licenses, this is a blocker. Also, the SDKs are distributed for multiple platforms, but the material does not list any versioning or compatibility guarantees between the C++ core and the SDKs, so upgrading one may require upgrading all.

Who Should Adopt It and What to Verify

RunAnywhere is a serious attempt to unify on-device AI, but it is not a drop-in replacement for a single engine. Adopt it if you are building a cross-platform app that needs LLM, vision, and speech, and you want to avoid writing separate integrations for each platform. The Python quick start and CLI make it easy to prototype. Do not adopt it if you are shipping a product with a strict open-source license requirement, or if you need wake-word detection or support for LiteRT or ExecuTorch. Before you commit, verify which engines are actually available on your target devices by calling RunAnywhere.capabilities() on each. Test the model download and inference paths on at least one mobile device and one desktop. Check the LICENSE file in the repository and clarify whether the Console is required for production. The project's frequent releases are a sign of life, but they also mean you should pin versions and plan for upgrade testing. The architecture is sound, but the real test is whether the routing logic selects a working engine on your specific hardware, and that is something only your own tests can confirm.

Editorial conclusion

Adopt RunAnywhere if you need a single semantic API across iOS, Android, web, and desktop for on-device LLM, vision, and speech, and you accept its proprietary license and the need to verify which engines are actually installed on each target. Do not adopt it if you require a fully open-source stack or if you need wake-word detection, LiteRT, or ExecuTorch support, as these are explicitly not implemented. Before committing, verify that the capability registry reports the engines you need for your specific devices, test the CLI and Python quick start with your models, and review the RunAnywhere License terms for redistribution and commercial use. The project's active release cadence and clear architecture are real strengths, but the gap between advertised capabilities and actual engine integration is the key risk to assess first.

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. RunanywhereAI/runanywhere-sdks on GitHub