OlliteRT: an OpenAI-compatible LLM server running on an Android phone

What it is, and the one-line pitch

The README's framing is that you should think of it as Ollama for Android. Pick a model, tap Start, and your phone becomes an LLM server, running models on your mobile GPU or CPU through Google's LiteRT-LM runtime and serving them as a standard OpenAI-compatible HTTP API on your local network.

The next line is the whole pitch in three clauses: no cloud, no API keys, no subscriptions, just your phone.

That combination is what makes the project interesting rather than merely clever. An OpenAI-compatible API is a well-established contract, and the whole reason it exists is that a great many clients already speak it. The README states the compatibility range as Home Assistant, Open WebUI, OpenClaw, Python and curl, with the summary that if it talks to OpenAI, it works.

The badges add a second dimension to the compatibility claim. There is an OpenAI Compatible badge and an Anthropic Compatible one, and the endpoint table confirms both are real rather than aspirational: the chat completions, completions and responses endpoints on the OpenAI side, and the messages and count_tokens endpoints on the Anthropic side.

The primary language is Kotlin, the licence is Apache-2.0, the default branch is main, and the recent releases are v0.9.7-beta.1 from 2026-09-22, v0.9.6-beta.1 from 2026-06-06 and v0.9.5-beta.1 from 2026-04-30. So every published release is a beta, at a 0.x version, with a roughly quarterly cadence. The last push was on 2026-10-08 and the project is not archived.

The repository layout matches an Android application rather than a general project: the Android/ directory holds the app, assets/ the icons and screenshots, docs/ the integration guides, and model_allowlists/ a directory whose name suggests the model catalogue is filtered by what the runtime supports.

The features, and the power claim

The feature list is eight items and several of them are about operations rather than inference.

Multi-model support means one-tap download from HuggingFace, importing .litertlm files, importing model lists from local storage, or adding custom model sources via a JSON file or URL. That last option is what makes this usable in an air-gapped setup or behind a corporate proxy, and it is a small line in the list that covers a real deployment need.

Multimodal and reasoning support covers vision, audio, thinking, streaming, and tool calling, with tool calling marked experimental and the caveat that it may not always be reliable due to model limitations.

A benchmark built in lets you test and compare models on your own device to find the best fit for your hardware. That is more useful than it sounds, because phone chips vary enormously and the model table's memory requirements are the real constraint, so being able to try a model before committing to it saves a lot of downloading.

Activity logs give detailed request and response logs with search, filtering and JSON highlighting.

Always On, Low Power is the item that decides whether the phone-in-a-drawer idea is realistic. The claim is configurable auto-start on boot, and about five to ten watts against the three hundred watts or more of a GPU server, described as perfect for that old phone in your drawer. The README is careful to add a footnote that the author is not responsible for swollen batteries, crispy phones or spontaneous pocket warmers, and asks you not to run an LLM on your phone under a pillow.

Highly Configurable is the longest entry and covers per-model inference settings, choosing a GPU or CPU accelerator, idle model unloading, listener scope, client IP access rules, and bearer token auth. Idle model unloading matters for the always-on case, because leaving a model resident in memory on a phone is what actually drains the battery.

Model and server monitoring adds a live stats dashboard, an optional movable floating monitor for live background metrics, Prometheus metrics for Grafana, and a Home Assistant REST API for remote server control.

The combination of an OpenAI-compatible endpoint, low power and a client IP allowlist is what makes this usable as a backend for other devices on a home network rather than only as a toy.

Quick start, and the hardware floor

The quick start is four steps and three of them are things you do on the phone.

Download and install the APK from the GitHub releases page. Download a model, with Gemma 4 E2B recommended for most devices at 2.4 GB running on 8 GB of RAM. Start the server by tapping the Start Server button on the downloaded model card. And configure your client with the endpoint shown on the Status screen, for example http://PHONE_IP:8000/v1, using any OpenAI-compatible client.

The important line is the fourth one. There is no configuration file, no port mapping and no reverse proxy. The phone serves on port 8000 on the local network, the client points at the phone's IP address, and the app's Status screen is what tells you the address to use. That simplicity is a deliberate consequence of the target being a phone with a touchscreen rather than a server.

The requirements note is worth reading carefully because it is more restrictive than the general idea of an Android phone implies: Android 12 or newer, an arm64-v8a device, 6 GB of RAM as a minimum, and 8 GB or more recommended for multimodal models.

The arm64-v8a requirement is the one that will rule out devices. It excludes older 32-bit-only phones and most x86 devices, including the emulator images that most Android developers have on their machines. If you are going to try this, the first thing to check is your phone's ABI.

The 6 GB minimum is likewise a real constraint, and it is consistent with the model table, where even the smallest listed model, Gemma 3 1B, needs 6 GB and the multimodal models need 12 GB.

The model table, and what the memory column tells you

The available models table is the practical core of the project, because it tells you what runs on what hardware.

Seven models are listed with size, minimum RAM, context length and capabilities. Gemma 4 E2B is the recommended one for most devices, at 2.4 GB and 8 GB of RAM with a 32K context, and it supports text, vision, audio, thinking, tools and MTP. Gemma 4 E4B is the same family at 3.4 GB and 12 GB of RAM, also with 32K context.

The 32K context on the Gemma 4 models is double that of everything else in the list, which is a meaningful difference for anyone feeding it documents.

The Gemma 3n models sit below that: Gemma 3n E2B at 3.4 GB and 8 GB of RAM with a 4K context, and Gemma 3n E4B at 4.6 GB and 12 GB, both supporting text, vision and audio but without thinking or tools.

Then the smaller text-only entries: Gemma 3 1B at 0.5 GB and 6 GB of RAM with a 1K context, Qwen 2.5 1.5B at 1.5 GB with a 4K context, and DeepSeek-R1 1.5B at 1.7 GB with a 4K context. DeepSeek-R1 appearing at 1.5 billion parameters is notable, since the R1 reasoning behaviour is normally associated with much larger models, and the model list puts a reasoning model within reach of a mid-range phone.

The size column is the download size and the RAM column is what the device needs to hold it. The gap between those two numbers is roughly a factor of three in every row, which is the memory overhead of running a model rather than storing it, and it is why a 0.5 GB model still needs 6 GB of RAM.

The note under the table is that tool calling is experimental and may not always be reliable due to model limitations, which places the limitation on the model rather than on the server implementation.

The README also points to a Model Guide document for recommendations, capability details and import instructions, which is where the per-model detail lives.

The API surface, and the two protocols it speaks

The endpoint table is what makes the compatibility claim concrete.

On the OpenAI side there is the chat completions endpoint accepting both streaming and non-streaming, the completions endpoint, and the responses endpoint, plus a GET on the models path to list what is available and a GET on a model path.

On the Anthropic side there is the messages endpoint, again with streaming and non-streaming, and a count_tokens endpoint for the Anthropic input-token estimator.

Then two endpoints that are not part of either vendor's API: an audio transcriptions path for audio transcription, and the Prometheus metrics endpoint, described in the integrations section as a /metrics endpoint with 29 metrics for Grafana, Datadog and similar.

Twenty-nine metrics is a real number rather than a token one, and it is what you need if you are going to run this always-on and care about how much it actually costs in battery and heat.

The presence of both API families is the interesting engineering decision here. A client that only speaks OpenAI and a client that only speaks Anthropic can both talk to the same phone, which means you do not have to choose which ecosystem's client to standardise on. For a home assistant or an Open WebUI deployment, that is the difference between it working and not.

The Home Assistant integration is documented separately, and the README's note is a practical warning rather than a formality: Home Assistant currently requires a custom integration such as Extended OpenAI Conversation or Local OpenAI LLM, and OpenAI STT for voice commands. So Home Assistant is supported, but not with its stock conversation component.

The audio transcription endpoint is the one that sits outside both vendor APIs, and it is worth noting that a speech-to-text service on a phone is a genuinely useful thing to have on a home network given how much audio the phone's own microphone can capture.

How to evaluate it, and what the beta tag means for you

There are four things to establish before you set this up.

First, whether your phone qualifies. Android 12, arm64-v8a, and 6 GB of RAM is the floor, and the model you want to run sets a higher bar. A phone from 2019 with 4 GB of RAM is out. A mid-range phone from the last three years with 8 GB is fine for Gemma 4 E2B, which is the recommended starting point.

Second, what you will actually use it for. The always-on, low-power framing points at a background role rather than an interactive one: a home assistant that wants a summariser, a private assistant for your own tooling, a model endpoint for a client on your network that must not send data to a third party. The 5-to-10-watt claim is what makes that role practical, and it is also what makes an old phone in a drawer a reasonable host.

Third, the network exposure. The server listens on your local network, and the feature list mentions listener scope and client IP access rules as configurable, with bearer token auth available. Decide before you start whether the phone's address is reachable from your wider network, because an LLM endpoint with a permissive default is a liability, and the README does not tell you what the default scope is.

Fourth, the maturity. Every release is a beta and the version is 0.x. The project is eight months into its published history at least, with a beta in June, one in September and one in April before that, so it is moving but not moving fast, and the version numbering says the API surface may still shift.

What you get in return is a genuinely useful thing that is hard to build otherwise: a private, offline, low-power inference endpoint that speaks the API every existing tool already speaks. For a home lab, a local development stack that should not phone home, or a privacy-sensitive workload, that is a real capability rather than a novelty. The tool calling caveat and the experimental tag on the multimodal features are the two places to be careful about relying on the behaviour rather than just the endpoint.

Editorial conclusion

Adopt OlliteRT if you want a private, always-available LLM endpoint on your own network without sending prompts to a third party, if you have a spare Android 12 phone with arm64-v8a and 8 GB of RAM, and if your client already speaks the OpenAI or Anthropic API. Do not adopt it expecting the capabilities of a datacentre model, since the largest option here needs 12 GB of RAM and the recommended one has a 32K context, and do not treat tool calling as dependable, because the README marks it experimental and attributes the unreliability to model limitations. Before you commit, confirm your phone's ABI and RAM against the model table, decide the listener scope and whether to require a bearer token before the server is reachable from your network, and read the Home Assistant setup document, since that integration needs a custom component rather than the stock conversation agent.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. NightMean/OlliteRT on GitHub
4. README
5. Releases