OnPrem.LLM: A Python Toolkit for Running LLMs Over Non-Public Documents

The Problem OnPrem.LLM Targets: Documents That Cannot Be Uploaded

Most LLM tooling assumes the document can be sent somewhere. OnPrem.LLM starts from the opposite assumption. The README describes it as a toolkit for applying LLMs to sensitive, non-public data in offline or restricted environments, and says it is inspired largely by privateGPT. The audience is therefore narrow and identifiable: engineers in regulated industries, government or defence-adjacent work, legal and medical settings, and anyone whose data handling rules forbid a third-party inference endpoint. The project also supports cloud providers such as OpenAI and Anthropic, so the constraint is not enforced by the library. It is a default, not a cage. That distinction matters when you evaluate it, because the same object you instantiate for a local Ollama model can be re-instantiated against a hosted model with a different string. The library trusts you to know which of those two things your data permits.

Backends, Ingest and Ask: The Core Data Flow

The mechanism is a thin Python layer over a model backend plus a vector store. You construct an LLM object with a backend identifier string. The README shows LLM('ollama/llama3.2') for a local model and LLM('anthropic/claude-sonnet-4-5-20250929') for a cloud one. Supported backends listed in the README include llama_cpp, Hugging Face transformers, Ollama, vLLM, OpenAI and Anthropic. Retrieval works through two calls: llm.ingest(path) points at a directory and builds the index, and llm.ask(question) queries it. The README example downloads a PDF with utils.download, ingests a directory, then asks what OnPrem.LLM is. Prompting without retrieval is a separate call, llm.prompt. Structured extraction is llm.pydantic_prompt, which takes a Pydantic BaseModel and returns a populated instance; the README example extracts a value and unit from the sentence 'He was going 35 mph.' The release notes add features that sit on top of this: asynchronous prompts in v0.20.0, metadata-based query routing in v0.21.0, provider-implemented structured outputs in the same release, and improved information extraction in v0.23.0. Workflows arrived in v0.19.0 as YAML-configured pipelines, and a visual workflow builder is listed among the key features.

SparseStore: Retrieval Without a Precomputed Embedding Index

The most interesting design choice in the feature list is SparseStore. The README frames it as support for environments with modest computational resources, describing it as RAG without having to store embeddings in advance. That is a real constraint, not a marketing line: dense retrieval normally requires you to embed every chunk before you can query anything, which costs time and disk proportional to corpus size, and it locks you into a specific embedding model, since changing it invalidates the index. A sparse approach that computes term weights at query time avoids both problems and is friendlier to a laptop or a small VM. The cost is retrieval quality. Sparse retrieval depends on lexical overlap, so it handles exact terminology, identifiers and rare tokens well and handles paraphrase, synonymy and cross-lingual queries poorly. The README does not publish retrieval benchmarks, and no numbers are given for recall or latency, so the honest position is that you must measure it on your own corpus before trusting it. The project offers both paths, which is the right call, but the choice between them is a quality-versus-resource trade-off you own.

AgentExecutor and the Sandbox Claim

The AgentExecutor, introduced in v0.22.0, runs an agent that can read and write files. The README example asks it to search a directory for .md files, extract headings, count words and write an index file. It is constructed as AgentExecutor(model='openai/gpt-5-mini', sandbox=True). The word sandbox is doing a lot of work here and the README does not explain what it enforces. It does not say whether the sandbox is a container, a chroot, a restricted working directory, or a permission check on file paths. For an agent that writes files, that distinction is the whole security story. Treat sandbox=True as a claim to verify in the source before you point the agent at anything you care about. The agent extra also has to be installed separately, via pip install onprem[agent], so this is not part of a default install. There is a second, subtler issue: the README's own agent example uses a hosted OpenAI model, which sits awkwardly beside the offline-first pitch. If your environment is genuinely air-gapped, you need an agent-capable local model, and the README does not say which local backends support tool calling well enough for this to work.

Install and Configuration Surface

The install path is short. Install PyTorch first, then pip install onprem. Chroma is an extra: pip install onprem[chroma], needed if you use the default dense vectorstore for RAG rather than the sparse one. Agents need pip install onprem[agent]. llama-cpp-python is optional and the README is explicit about when you can skip it: if you use Ollama as the backend, or if you use Hugging Face transformers by supplying the model_id parameter when instantiating an LLM, or if you are pointing at an external REST API. If you do use llama-cpp-python, CPU installs are a plain pip install on that package, with extra steps documented for Microsoft Windows, and GPU installs follow separate instructions. There is no config file described for basic use. Configuration is constructor arguments and, for multi-step pipelines, YAML files consumed by the workflow feature. The library also advertises integration with tools already in your environment, naming Elasticsearch and SharePoint as examples, which means the vector store is pluggable rather than fixed to Chroma.

Where It Fits Badly: Version Churn and Scope

Two limitations are visible from the release history alone. First, the cadence is fast. Releases listed run v0.23.4, v0.23.5 and v0.23.6 within roughly six weeks, and the news items show major features landing every few months: workflows in v0.19.0, async prompts in v0.20.0, query routing and provider structured outputs in v0.21.0, AgentExecutor in v0.22.0. The version number is still 0.x. If you build a document pipeline on this and pin nothing, you will be chasing API changes. Second, the project is orchestration, not infrastructure. It does not serve models in production, it does not manage GPU scheduling, and it does not give you a hardened multi-tenant endpoint. The web UI and workflow builder are conveniences for interactive work, not deployment surfaces. The right way to read this project is as glue: it makes a local model plus a vector store plus a prompt template into a few method calls. If you already have that glue, or you need a service rather than a library, the value here is lower than the feature list suggests.

The Alternative: LangChain, and the Difference in Approach

The obvious comparison is LangChain, which also wraps backends, vector stores and retrieval chains in Python. The difference is in what each project treats as fixed. LangChain exposes a large set of composable abstractions, so you assemble a chain from parts and the framework stays out of the way of your architecture. OnPrem.LLM inverts that. It fixes the shape of the workflow (ingest a directory, ask a question, prompt with a Pydantic model, run an agent) and varies the backend underneath. You get fewer decisions and less flexibility. Concretely, the SparseStore is the clearest example of the divergence: it is a project-specific answer to the low-resource case, and LangChain's equivalent requires you to pick and wire a sparse retriever yourself. The trade is that OnPrem.LLM's opinions are also its ceiling. If your retrieval needs cross-encoder reranking, hybrid scoring or a custom chunking strategy, you will be working around the library rather than with it. privateGPT, which the README names as the inspiration, is the other reference point, but it is an application rather than a library, so it competes with OnPrem.LLM's web UI more than with its Python API.

Maintenance, Licence and What to Check First

The repository is not archived and the last push is recent, so the project is actively maintained. The licence is Apache-2.0, which permits commercial and closed-source use and includes an explicit patent grant, with the usual obligations around preserving notices and stating changes. That is a permissive licence and it is compatible with proprietary internal tooling; it is not legal advice and your counsel should confirm the notice requirements for your distribution model. The maintenance cost you should budget for is not the library itself but the backend underneath it. Model identifiers, Ollama tags and provider model names change on their own schedules, and the README's examples hardcode dated model strings such as claude-sonnet-4-5-20250929. Pin your dependencies, pin your model identifiers, and keep a note of which release introduced each feature you rely on. Before adopting, verify that your backend loads on your hardware, that the extras you need install cleanly, and that SparseStore retrieval is good enough on a sample of your own documents.

Editorial conclusion

Adopt OnPrem.LLM if your documents cannot leave your network and you want a Python API rather than a standalone application, and if you can accept that the package is a thin orchestration layer whose behaviour depends on whichever backend you point it at. Do not adopt it if you need a hardened multi-tenant service, a stable long-term API, or a tool that manages model downloads for you. Before committing, verify three things: that your chosen backend actually loads on your hardware, that the extras you need (chroma, agent) install cleanly in your environment, and that the SparseStore path retrieves acceptably on your own corpus, since that is the module the project offers for machines without room for a precomputed embedding index.

Official sources

1. amaiya/onprem on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases