llmman: agents pointed at models stored as OCI images
Run any agent on any model, models stored as OCI images
At a glance
- What is it?
- llmman wraps local inference engines and hosted providers behind one command, and stores models as standard OCI artifacts. The design is unusual and the trade-offs are real: read this before you replace your current model runner.
- Who is it for?
- Adopt llmman if your problem is distribution rather than inference: you already know which engine you want, and you need models to move between Hugging Face, your own registry and air-gapped machines without a private blob format. Do not adopt it if you want a curated model library with a single vendor's engine tuning, or if you need a stable interface.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem is distribution, not inference
The README's one-line pitch is "Run any agent on any model." Read past the slogan and the actual complaint is narrower. Getting a model file to a machine, and getting an agent to talk to it, are two separate chores that most tooling merges into one opinionated stack. llmman separates them. The agent side is an exec wrapper: llmman launch claude starts a server, loads a model, and hands the agent process an endpoint. The model side is a packaging decision: models are OCI images, so the same artifact that a container registry already knows how to store, sign and replicate is what holds your weights. The audience is engineers who already run llama.cpp or vllm and are tired of re-solving transport. It is not aimed at someone who wants a chat app.
What launch actually does, step by step
The README gives one concrete example: llmman launch claude --model qwen3.8. According to the documentation, that single command starts a local inference server, downloads a llama.cpp build matching your GPU, loads the model, and execs the agent against it. So the data flow is: model reference resolves to an OCI artifact, the artifact is pulled and unpacked into a local store, a backend process starts, and the agent inherits an endpoint pointing at that backend. The backend is not a fork. The README states llmman uses upstream llama.cpp releases, or a llama-server already on your PATH, plus vllm and mlx-lm as-is, serving unmodified GGUF and safetensors files. That choice matters more than it sounds. It means a bug in quantization or a new model architecture gets fixed by upstream, not by llmman, and it means you can inspect the store with ordinary OCI tooling because it is a standard OCI Image Layout.
Aggregation turns several machines into one endpoint
The feature worth reading twice is aggregation. Several llmman serve daemons can be named to each other, and a request to any of them runs on whichever node has the model loaded or the most room for it. The README's framing is a laptop, a workstation and a Spark looking like one endpoint. This is a scheduling concern, not a serving concern, and it is where llmman diverges most from a plain model runner. The constraint the README does not resolve is what happens when no node has the model: whether the request blocks, fails, or triggers a pull is not stated in the supplied material, and that is the first thing to test if you plan to rely on it.
Install and the first three commands
Installation is conventional. On Linux and macOS the README gives curl -fsSL https://llmmanorg.github.io/install.sh | sh, on Windows irm https://llmmanorg.github.io/install.ps1 | iex, plus brew install llmmanorg/tap/llmman and winget install llmmanorg.llmman. The Cargo path is worth flagging: the README says cargo install llmman needs Go 1.25+ as well as Rust, and LLVM on Windows. A Rust project that requires a Go toolchain to build from source is a real friction point, and it suggests parts of the stack are not Rust. Once installed, three commands cover the common cases: llmman launch claude --model qwen3.8 for an agent on a local model, llmman run qwen3.8 to chat, and llmman serve for an Ollama, OpenAI and Anthropic compatible endpoint. Running llmman launch with no arguments lists the supported agents and whether each is installed, which is the honest way to find out what you actually have.
Transfer and signing are the parts with no easy substitute
llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest copies from Hugging Face into a registry directly, and the README states no copy lands in your local store. For air-gapped or compliance-bound environments that is the whole argument for the tool, because the alternative is pull, write a Modelfile, create, push. Signing uses cosign-format signatures, with a verify command and a per-repo pull-time trust policy configured through llmman config. Treat the trust policy as the security boundary it is: a per-repo policy means a typo in a repository name is a policy hole, not a cosmetic problem. The README does not describe how keys are distributed or rotated, so verify that against your own key management before relying on it.
Diffusion support and where it stops being one tool
The README also covers image, video and audio generation. llmman run unsloth/LTX-2.3-GGUF "Draw a cat" writes a PNG, --video --seconds 2 produces an mp4 with an audio track and needs ffmpeg, and --audio --seconds 3 produces a 48 kHz stereo wav. An interactive loop accepts /set width|height|steps|seed|cfg|negative|seconds|media, and the same model answers /v1/images/generations, /v1/videos and /v1/audio/speech on llmman serve. Here the abstraction leaks. Diffusers-layout safetensors with a root model_index.json are not served by the same path at all; they go through vLLM-Omni, which you install next to vllm or reach with --ociman docker. So "any model" is really two pipelines with different dependencies, and the second one requires a separate package.
How it differs from Ollama, concretely
The README's own comparison is the clearest statement of intent. Ollama pulls from ollama.com's library over its own registry protocol and imports GGUF and safetensors through a Modelfile into a blob layout. llmman pulls from Hugging Face directly or any OCI registry, and stores unmodified files in a standard OCI Image Layout. Ollama bundles a llama.cpp fork plus its own engine; llmman uses upstream releases or your own llama-server. The practical difference is who controls the artifact. With llmman, the thing on disk is a container image, so GHCR, quay, Harbor or a self-hosted mirror can hold it, and cosign can sign it. With Ollama, the thing on disk is Ollama's. If your organisation already runs a registry, llmman fits an existing process. If you want a curated library where someone else has tuned the engine per model, Ollama's approach is the one that removes work.
Release cadence, licence and what to verify before adopting
Three releases landed on 2026-09-10: v0.1.381, v0.1.383 and v0.1.386. That is a project moving fast at a 0.1 version number, and it is the single most important fact for anyone evaluating it. The interface described here may change between the version you read about and the version you install. The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant; it also means no warranty, and you are responsible for the licences of the models you pull, which are separate from the tool's licence. Before committing, verify three things: that llmman launch finds the agent you use, that a transfer into your own registry works with your credentials, and that aggregation behaves the way you need when no node has the model loaded. This is not legal advice; check the licence text and your model licences yourself.
Editorial conclusion
Adopt llmman if your problem is distribution rather than inference: you already know which engine you want, and you need models to move between Hugging Face, your own registry and air-gapped machines without a private blob format. Do not adopt it if you want a curated model library with a single vendor's engine tuning, or if you need a stable interface. This project shipped three releases on 2026-09-10 alone, so pin a version before you build anything on top of it, and run llmman launch with no arguments first to see which agents it can actually find on your PATH.
Community notes