callstackincubator/ai: On-device LLM primitives for React Native with Vercel AI SDK compatibility
On-device LLM execution in React Native with Vercel AI SDK compatibility
At a glance
- What is it?
- A TypeScript toolkit that wraps Apple Foundation Models, llama.rn and MLC LLM behind the Vercel AI SDK surface. It removes the server from the inference path, but the platform floor is high and the provider matrix is uneven.
- Who is it for?
- Adopt it if you are already on the Vercel AI SDK and want a chat, embedding or transcription call to resolve to a local model instead of an HTTP endpoint. Do not adopt it if you need a uniform feature set across iOS and Android, or if your iOS deployment target is below 26 for text generation.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 71 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: inference that never leaves the device
Most React Native AI integrations end at a fetch call to a hosted endpoint. That works until the requirement is that the prompt and the response never leave the handset, or until the network round trip is the dominant cost in a typing indicator. This repository packages the other option: model execution inside the app process, exposed through the same API surface developers already use for remote models.
The target reader is a React Native engineer who has already written `generateText` or `streamText` against a hosted provider and now needs a local one. The README frames the value as privacy-preserving, low-latency inference without server costs. The AI SDK compatibility table is the more concrete claim: version 0.11 and below maps to AI SDK v5, version 0.12 and above maps to AI SDK v6. That table is the single most important line in the README, because it tells you the upgrade is a breaking change across the SDK major version, not a patch.
Three providers, three very different runtimes
The project is not one inference engine. It is a dispatch layer over three, and the differences matter more than the shared API.
Apple is the only built-in provider. It uses Apple Foundation Models for text generation, NLContextualEmbedding for 512-dimensional vectors, SpeechAnalyzer for transcription, and AVSpeechSynthesizer for speech output. Nothing is downloaded. The documentation states text generation requires iOS 26 or later on an Apple Intelligence device, embeddings require iOS 17, transcription requires iOS 26, and speech synthesis works from iOS 13 with Personal Voice needing iOS 17 or later. Those numbers are a hard floor, not a suggestion.
Llama wraps llama.rn and runs GGUF files pulled from HuggingFace. The model ID format is `owner/repo/filename.gguf`, for example `ggml-org/SmolLM3-3B-GGUF/SmolLM3-Q4_K_M.gguf`. It supports text generation, embeddings and speech through a vocoder model. Installation pulls in three packages: `@react-native-ai/llama`, `llama.rn` and `react-native-blob-util`.
MLC wraps the MLC LLM runtime and, per the README, requires the Increased Memory Limit capability in Xcode. That one line is worth pausing on. It means the runtime expects to exceed the default per-process memory ceiling, which is a real constraint on which devices and which model sizes are viable.
The data flow: download, prepare, generate, unload
The Llama example in the README makes the lifecycle explicit, and it is not the lifecycle of a hosted model. You construct the model instance, call `download` with a progress callback, call `prepare` to load weights into memory, run generation, then call `unload`.
That sequence is the whole architecture in miniature. A hosted provider has no analogue for `download` or `prepare`, and no analogue for `unload` either. On-device inference turns model weights into a resource you allocate and release, like a camera session or an audio context. The README logs download progress as a percentage, which implies the download is long enough to warrant a progress UI. For a multi-gigabyte GGUF file over a mobile connection, that is a fair assumption.
The Apple path skips all of it. There is no download step and no prepare step in the example, because the weights are already on the device as part of the OS. The trade is control: you cannot choose the model, you cannot quantize it differently, and you cannot run it on a device that does not have it.
Getting it running: packages and the calls that matter
For Apple, the README gives a single install line and states that no additional linking is needed because the package is autolinked. The usage example imports `apple` from `@react-native-ai/apple` and then calls `generateText` with `model: apple()`, `embed` with `model: apple.textEmbeddingModel()`, `experimental_transcribe` with `model: apple.transcriptionModel()`, and `experimental_generateSpeech` with `model: apple.speechModel()`.
Note the `experimental_` prefixes on transcription and speech. Those are not cosmetic. They signal that the upstream AI SDK treats those entry points as unstable, and a rename or signature change there is a plausible source of breakage on an SDK upgrade.
For Llama the install is `npm install @react-native-ai/llama llama.rn react-native-blob-util`. The model ID string is the configuration surface: `llama.languageModel('ggml-org/SmolLM3-3B-GGUF/SmolLM3-Q4_K_M.gguf')`. There is no config file and no provider registry. The repository layout puts an Expo demo app under `apps/expo-example`, which is the reference wiring if autolinking or native module registration misbehaves.
For MLC, the install is `npm install @react-native-ai/mlc` plus the Xcode memory capability. The README truncates the MLC getting-started section, so the exact model download API for that provider is not visible in the supplied material.
The DevTools plugin and the stale-session failure
`@react-native-ai/dev-tools` is a separate package that captures OpenTelemetry spans from Vercel AI SDK requests and renders them in Rozenite DevTools. The README states DevTools are runtime agnostic, so the same panel works whether inference is local or remote. Rozenite has to be installed and enabled in the app first.
The interesting part is a documented failure mode. If the AI SDK Profiler panel appears but stays empty after you send a chat message, the README says to close the React Native DevTools window and open a fresh one, because a stale debugger session can keep the Rozenite panel mounted without receiving the current app's telemetry stream. That is a concrete, reproducible-sounding bug with a one-step workaround, and it is more useful than a feature list. It also hints at the debugging experience on this stack generally: the tooling layer has its own state that can drift from the app's.
Where this is the wrong tool
The clearest limitation is platform asymmetry. Apple text generation needs iOS 26 or later on an Apple Intelligence device. There is no Android equivalent in the built-in tier. If your product ships to both platforms and you want one behavior, you are pushed to Llama or MLC, which means shipping model weights, which means a download step and a memory budget.
The second limitation is model size against device memory. The README's own MLC note about the Increased Memory Limit capability is the tell. On-device inference competes with the rest of your app for RAM, and a mid-range Android device will not run the same GGUF file a flagship will. The project gives you no sizing guidance in the supplied material, and that is a gap.
The third is the version coupling. The compatibility table ties the library major to the AI SDK major. Upgrading the AI SDK from v5 to v6 forces a library upgrade from 0.11 to 0.12. That is a coordinated migration across two dependency trees, and the `experimental_` APIs make it riskier than the semver suggests.
Finally, if your task is classification or retrieval over a small fixed corpus, a local embedding model plus a lookup may be cheaper in every sense than running a generative model on the handset.
Alternatives and how they differ
The most direct alternative is calling the underlying runtimes yourself. `llama.rn` is a dependency of this project, not a competitor to it, and you can use it directly. The difference is the API shape. `llama.rn` gives you a context and token-level control; this library gives you `generateText` and `streamText` with a model object. If you need custom sampling, logit manipulation or a token stream you post-process, the direct binding is the shorter path. If you want the same call site to work against a hosted model in one build and a local model in another, the abstraction is the point.
On the Apple side, the alternative is writing Swift against Foundation Models and bridging it. That gives you access to APIs this wrapper may not surface, at the cost of maintaining the bridge and the TypeScript types yourself. The library's value is that the bridge already exists and is autolinked.
The third alternative is a hosted inference endpoint with a local cache. That inverts the trade: you keep a uniform feature set and small binaries, and you give up the privacy and offline properties. For a chat feature where the prompt contains user data you would rather not transmit, that inversion is the wrong direction.
Maintenance, licence and what to check before adopting
The repository is MIT licensed, which permits commercial and closed-source use. That covers this project's own code. It does not cover the model weights you download. GGUF files on HuggingFace carry their own licences, and the README's model list mixes vendors with different terms. The licence of `SmolLM3` and the licence of `Llama-3.2-3B-Instruct` are not the same document. Checking the model card is your responsibility, and it is not something this library can do for you. Nothing here is legal advice.
On maintenance cost, the release cadence visible in the material is roughly one minor release every three to four months across 0.10, 0.11 and 0.12, with the last push in July 2026. The 0.11 to 0.12 jump carried the AI SDK v5 to v6 migration, which is the kind of change that lands on you as a dependency bump and a set of type errors. Budget for reading the release notes on each minor, not just the patch.
The upgrade cost is asymmetric by provider. The Apple path tracks OS releases: when iOS 26 becomes the floor for text generation, your app's deployment target follows. The Llama and MLC paths track the upstream runtimes and the model files you chose, so a regression can come from `llama.rn`, from the GGUF file, or from this wrapper.
Before adopting, verify three things against your own targets. Confirm the iOS version and Apple Intelligence hardware requirement for text generation, since that is the narrowest gate. Confirm the memory headroom on your lowest-spec Android device against the quantized model you intend to ship. And pin the AI SDK version in your lockfile before you touch this library, because the compatibility table makes the two move together.
Editorial conclusion
Adopt it if you are already on the Vercel AI SDK and want a chat, embedding or transcription call to resolve to a local model instead of an HTTP endpoint. Do not adopt it if you need a uniform feature set across iOS and Android, or if your iOS deployment target is below 26 for text generation. Verify first that your target devices satisfy the Apple Foundation Models requirements, and that the model file you intend to ship fits the memory budget of the Llama or MLC runtime you pick.
Community notes