SmolChat-Android: GGUF Inference on the Phone, Without a Server
Running any GGUF SLMs/LLMs locally, on-device in Android
At a glance
- What is it?
- SmolChat-Android is a Kotlin Android app that loads GGUF models through llama.cpp compiled with the NDK. Its scope is narrow and its setup path is a git clone plus an Android Studio build; the interesting part is the JNI layer, not the chat UI.
- Who is it for?
- Adopt SmolChat-Android if you want a working Android front end over llama.cpp and are willing to build it yourself with the submodule initialised. Do not adopt it if you need a stable API surface, a headless inference library, or something you can drop into another app without touching the smollm module.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 87 days ago.
- What is it written in?
- Mainly Kotlin, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem is not chat, it is the C++ boundary
Running a language model on an Android device is mostly a packaging problem. llama.cpp already executes GGUF models, and the README notes it is written in pure C/C++, which makes it straightforward to compile for Android targets using the NDK. What is missing between llama.cpp and a Kotlin app is a bridge: a JNI layer that owns the model handle, feeds tokens in, and pulls generated text back out. SmolChat-Android exists to fill that gap and to put a usable interface on top of it. The stated project goals are a usable UI for local SLMs, the ability to add and remove GGUF models and edit their system prompts and inference parameters such as temperature and min-p, quick creation of downstream tasks, and a codebase that is simple enough to understand and extend. That last goal is the one that shapes the repository. The code is split so that the C++ and the Kotlin are separable, and the README points readers directly at the C++ source files and the SmolLM class rather than treating the app as a black box. The audience is Android developers who want to experiment with on-device inference and are comfortable reading a JNI binding, not end users looking for a finished product.
How the smollm module moves a prompt to tokens and back
The architecture has three layers. At the bottom sits llama.cpp, vendored as a git submodule. Above it, the smollm module contains llm_inference.cpp, which the README describes as interacting with llama.cpp's C-style API to execute the GGUF model, plus smollm.cpp as the JNI binding. On the Kotlin side, the SmolLM class exposes the methods that call into those JNI bindings. The app module sits on top and holds the UI and application logic. The data flow for a normal chat is explicit in the README: when a new chat is opened, the app instantiates SmolLM and hands it the model file path, which is stored by the LLMModel entity. Messages with role user and system are then read from the database and pushed into the native side through LLMInference::addChatMessage. Tasks take a different route. Their messages are not persisted, and the app signals this by passing _storeChats=false to LLMInference::loadModel. That flag is the cleanest design decision visible in the material. Chat history is a database concern; a one-shot task is not, and rather than filtering persisted rows at read time the app tells the native layer not to keep them at all. Model files themselves are tracked as LLMModel entities, so the app knows where each GGUF file lives on disk. Markdown responses are rendered with Markwon, and Prism4j handles code syntax highlighting, which matters because small models frequently answer with fenced code blocks.
Setup assumes you are building the app, not consuming it
There is no library artifact here. The README offers three installation routes for the app itself. Google Play hosts it under the package io.shubham0204.smollmandroid. GitHub Releases carries APKs that you download and transfer to the device, with the README noting you may need to allow installation from unknown sources. Obtainium can track the repository for updates: add the app source URL https://github.com/shubham0204/SmolChat-Android in the app's Add App screen, and it will notify you of newer releases without going back to GitHub. For anyone modifying the code, the path is a clone with the submodule: git clone --depth=1 https://github.com/shubham0204/SmolChat-Android, then cd SmolChat-Android, then git submodule update --init --recursive. Skipping that third command leaves llama.cpp absent and the native build will not produce the JNI library. Android Studio then builds the project automatically, or you trigger Build > Rebuild Project. Running on hardware requires connecting an Android device and confirming its name appears in the top menu bar of Android Studio. The material does not document Gradle properties, CMake flags, or an abiFilters configuration, so the set of target ABIs is something you determine from the build files rather than from the README.
What the repository does not tell you about model support
The README says the app loads GGUF models and lets users add or remove them, but it does not list supported architectures, quantisation types, or a memory ceiling. That is a real gap rather than a documentation nitpick. GGUF is a container format, and llama.cpp's support for a given model family depends on the version of the submodule pinned in this repository. Because the submodule is pinned rather than floating, the set of architectures you can run is frozen at whatever commit the maintainer last updated. A model released after that commit may load, may fail at load time, or may load and produce degraded output, and the README gives you no way to predict which. The same applies to context length: llama.cpp accepts a context size parameter, and the README mentions inference parameters generally without enumerating them or stating defaults. On a phone, context length and quantisation together determine whether a model fits in RAM at all. Anyone planning to ship this in an app should test their specific GGUF files on their specific target devices before assuming the app will handle them, because the repository offers no compatibility matrix to lean on.
The GPU question is still open
The Future section lists a Vulkan backend as something to investigate, phrased as checking whether llama.cpp can be compiled to use Vulkan for inference on Android devices and use the mobile GPU. That phrasing matters. As of the material provided, GPU acceleration is not a shipped feature; it is an open question the maintainer has written down. The practical consequence is that inference runs on the CPU, and CPU inference on a phone is bounded by thermal headroom as much as by raw compute. A model that generates acceptably for the first few exchanges may slow as the device heats and the scheduler throttles. The other items on that list are similarly unshipped: automatic chat naming, a search bar in the navigation drawer for finding messages within chats, a background service using Bluetooth, HTTP or WiFi to relay queries from a desktop to the phone for inference, and integration with the author's Android-Document-QA project for on-device retrieval-augmented question answering. Treat the Future section as a statement of intent, not a roadmap with dates. Three releases landed between March and June 2026, so the project is active, but activity in the release list says nothing about which of those five items is next.
Where a different approach fits better
The obvious alternative for Android developers is Google's MediaPipe LLM Inference API, which is distributed as a Gradle dependency rather than as source you compile. The difference in approach is not performance, it is control and coupling. With MediaPipe you add a dependency, call an initialiser, and receive a stream of generated text; the model runtime is the vendor's problem. With SmolChat-Android you inherit a pinned llama.cpp submodule and a JNI layer you can edit. That means you can change the sampling code, add a parameter the UI does not expose, or read llm_inference.cpp to understand exactly what happens between prompt and token. It also means that when the underlying runtime needs updating, you update a submodule and rebuild, and when the model format you want is not yet supported, you wait for llama.cpp rather than for a vendor SDK. If your goal is to ship a chat feature in a product and you do not want to own a C++ build, the dependency-based route is the lower-cost choice. SmolChat-Android is the better fit when the inference layer itself is the thing you are working on, or when you need a model format that only llama.cpp handles.
Licence, upgrades and the cost of a pinned submodule
The project is Apache-2.0, which permits commercial use and modification, and the README's own framing of an extensible codebase is consistent with that. The licence does not resolve the dependency question: llama.cpp carries its own licence, and if you redistribute an app built on this code you are distributing a compiled llama.cpp as well. Checking that licence and its attribution requirements is your job, not something this review can settle. The upgrade cost is the submodule. Because llama.cpp is vendored rather than resolved from a package registry, updating it is a deliberate act: you move the submodule pointer, rebuild the native code, and re-test every model you care about. There is no version range to relax and no automatic patch flow. The app itself is versioned as v14, v15 and v16 across March, April and June 2026, so releases are frequent enough that tracking them is plausible, but each one may carry a submodule bump that changes which GGUF files load. If you fork this for a product, budget for a rebuild-and-retest cycle on every upstream update rather than assuming a dependency bump is enough.
Editorial conclusion
Adopt SmolChat-Android if you want a working Android front end over llama.cpp and are willing to build it yourself with the submodule initialised. Do not adopt it if you need a stable API surface, a headless inference library, or something you can drop into another app without touching the smollm module. Before committing, verify that the llama.cpp submodule commit pinned in the repository builds against your installed NDK, and confirm which GGUF quantisation the models you intend to ship are stored in.
Community notes