PrivateLM: a Flutter client that runs GGUF models on Android and iOS, with cloud APIs as fallback
A unified cross-platform AI client supporting seamless transitions between standard cloud APIs and on-device, offline execution of custom and uncensored language models.
At a glance
- What is it?
- PrivateLM wraps llama.cpp behind a Flutter plugin, adds Vulkan and Metal acceleration, and normalizes four cloud provider APIs into one interface. The interesting part is the platform split: local inference works on mobile, and the web build is cloud-only by design.
- Who is it for?
- Adopt PrivateLM if you want a Flutter codebase that already solves the messy parts of on-device inference: GPU layer offload, device tier detection, chat template selection, and a single interface over four cloud providers. Do not adopt it if you need local inference on the web, or if you are not prepared to keep one Android signing key stable across every release, because Android rejects an APK upgrade signed with a different key.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 57 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap PrivateLM fills: one client, two very different inference paths
Most chat clients pick a side. They are either thin wrappers over a cloud endpoint, or they are local runners that treat network calls as an afterthought. PrivateLM's README describes a client that holds both: GGUF models executed on the phone through llama.cpp, and OpenAI, Anthropic, Google Gemini or Kimi (Moonshot AI) when the device cannot carry the load or the platform does not support local execution. The audience is narrow and specific. It is for people who want a phone app where a conversation can stay on the device, and who also want the same app to reach a hosted model without switching tools. The README frames the motivation as control over how models run, and the architecture backs that up: chats, tasks and settings live in Hive, and the documentation states that nothing leaves the device unless cloud mode is chosen explicitly.
InferenceService, CloudService, and the conditional compile that decides what the app can do
The pipeline diagram in the README puts a GetX controller layer between the UI (ChatView, TaskView, ModelView, SettingsView) and a service layer containing InferenceService, CloudService, DownloadService, HiveService, DeviceInfoService and ExecutionService. Controllers do not talk to llama.cpp directly; they talk to services, and the services are where the platform split happens. Local inference is conditionally compiled: Android and iOS both resolve to inference_android.dart, the web build resolves to inference_stub.dart, and InferenceService exposes a supportsLocalInference flag so the UI can hide local-model controls on platforms that cannot run them. That flag is the honest part of the design. Rather than failing at generation time, the app knows up front that the web build is cloud-only. The README labels web local inference as coming soon, which is a promise rather than a shipped capability, and the table marks the web column as No.
What actually happens between tapping a model and reading a token
The local path is a five-step sequence the README spells out. First, the app detects GPU capabilities via Vulkan to decide how many layers to offload. Second, it picks a thread count from a device tier (ultra, high, mid, low). Third, it loads the GGUF file with progress streaming. Fourth, it generates tokens through generateChat(), which supports native chat templates for ChatML, Llama-3, Gemma and Phi. Fifth, if a native template fails, it falls back to manual prompt construction. That last step matters more than it looks. Chat template mismatches are a common source of garbled output in local inference, and having a manual fallback means a model with an unusual template degrades instead of breaking. The README also gives two timeout constants: idle detection at 5 seconds and a hard timeout at 180 seconds. Those numbers describe a UI that assumes generation may stall, which is a reasonable assumption on phones.
Four provider APIs behind one interface, and where the keys live
CloudService normalizes four different request shapes. OpenAI uses the standard /v1/chat/completions route. Anthropic's Messages API takes the system prompt as a separate parameter rather than a message. Google Gemini uses generateContent with images inlined as base64. Kimi is described as an OpenAI-compatible endpoint from Moonshot AI, which means it is the cheapest of the four to support. The README states that API keys are stored in Hive and never transmitted anywhere except to the provider's endpoint. That is a claim about the client's behavior, not an audit result, and it is worth treating as a design intent you would verify yourself if key handling is your concern. The four-provider abstraction is the part of this project with the broadest reuse value, since the request-shape differences are stable and well documented by each provider.
Build commands, the signing key rule, and the iOS sideload route
The prerequisites are Flutter SDK 3.3.0 or newer, Android SDK API 26 or higher, JDK 17, and the NDK bundled with the Android SDK. The debug build is two commands: flutter pub get, then flutter build apk --debug. Release builds need a signing key. The README instructs you to copy android/key.properties.example to android/key.properties, fill in the keystore values, then run flutter build apk --release --split-per-abi. It states the constraint plainly: Android accepts an APK upgrade only when it is signed with the same key as the installed APK, so the signing key must not rotate between GitHub releases, and the build number in pubspec.yaml must keep increasing. This is the single most operationally demanding instruction in the repository, and it is a distribution constraint rather than a code problem. iOS takes a different route: flutter pub get, cd ios, pod install, flutter build ios. The iPad release ships as a standalone ZIP, PrivateLM-iOS.zip, installed as an .ipa through AltStore, Sideloadly or Xcode. The README calls iPhone support experimental and recommends iPad because of RAM requirements for local models.
The platform table is also the limitation list
Three platforms, three answers. Android runs local inference with CPU offload via NEON and a minSdk of 28. iOS runs local inference with Metal GPU acceleration. Web runs cloud only. If your users are on the web, the headline feature of this project does not exist for them yet, and the README's coming soon note is the only timeline offered. The minSdk 28 floor excludes older Android devices outright. The iPhone-versus-iPad split means the iOS story is really an iPad story, and the README says so. There is also a hardware variance problem that the README documents without resolving: the image generation screenshot lists testing across Moto G71, OnePlus 10R, Pixel 6A, Poco F1 and Galaxy S23, spanning Snapdragon, MediaTek and Tensor chips. That list is evidence that the app was exercised on varied silicon, not evidence that any given model runs acceptably on all of it. The auto-configuration step detects RAM and recommends context size and token limits on first launch, which is a heuristic. Whether the recommendation is right for your model is something you find out by running it.
Where PrivateLM fits against a plain llama.cpp build
The obvious alternative is running llama.cpp directly, either through its own server binary on a desktop or through one of the mobile example apps in the upstream repository. The difference in approach is the layer each one owns. Upstream llama.cpp owns the inference engine and leaves you to build the client: model download, storage, chat history, provider switching, background execution, push notifications. PrivateLM owns the client and delegates inference to a Flutter plugin wrapping llama.cpp. If you want a phone app with persistent sessions in Hive, a task view, Firebase Cloud Messaging, and a cloud fallback when the local model is too slow, the upstream route means writing all of that yourself. If you only need to run a GGUF file on a device and read the output, the upstream route avoids an entire Flutter application and its build chain. Neither is better in the abstract; they sit at different points on the same stack.
Licence, maintenance surface, and what the release history shows
The project is MIT licensed, which permits commercial use and modification provided the licence text is retained. That is the whole of what the repository states, and it is not legal advice; if you redistribute a modified build, read the LICENSE file and the licence terms of the bundled llama.cpp and Flutter dependencies, which the README does not enumerate. On maintenance, the release history shows 1.0.3 in April 2026 and 1.0.5 in July 2026, with the release titles pairing app versions to build numbers (PrivateLM App v1.0.4 for release 1.0.3, for instance). That pairing is the mechanism behind the build number rule in the build section: the release tag and the pubspec build number are not the same thing, so you have to track both when you cut your own builds. The upgrade cost is concentrated in two places. The Android signing key must stay fixed across releases, and the llama_flutter_android plugin ties you to whatever llama.cpp revision it wraps, so a new upstream GGUF feature arrives only when that plugin is updated. The README does not describe a plugin release cadence, so that dependency is a question to ask before committing to the project.
Editorial conclusion
Adopt PrivateLM if you want a Flutter codebase that already solves the messy parts of on-device inference: GPU layer offload, device tier detection, chat template selection, and a single interface over four cloud providers. Do not adopt it if you need local inference on the web, or if you are not prepared to keep one Android signing key stable across every release, because Android rejects an APK upgrade signed with a different key. Before building, check the minSdk 28 floor against your device fleet, confirm the iOS iPad ZIP install path works for your distribution method, and verify that your target GGUF model fits the RAM tier that DeviceInfoService reports on your actual hardware.
Community notes