llm-server-docs: A Debian Build Guide for a Private LLM Stack

The gap this fills is integration, not inference

Running a model locally is a solved problem with a one-line install. Running a local model behind a chat interface that also has web search, retrieval, image generation, speech output, tool calling and remote access is a wiring problem, and the wiring is where people give up. This repository is a documentation set for that second problem. The README describes it as end-to-end documentation to set up a local and fully private LLM server on Debian, and the table of contents bears that out: it covers drivers, Docker, HuggingFace model downloads, SearXNG, three inference engines, a model server, Open WebUI, two MCP proxies, Kokoro FastAPI, ComfyUI, SSH, a firewall, Tailscale and an updating section.

The intended reader is stated plainly. The About section says it aims to be a guide for Linux beginners like me who are setting up a server for the first time. That framing matters when you evaluate the document. It is written as a sequence of steps someone actually performed, with the author's own hardware as the reference point, rather than as a reference manual with options enumerated for every case.

llama-swap sits between Open WebUI and the inference engines

The architecture is a set of containers and services on one host, joined by an OpenAI-compatible HTTP surface. Open WebUI is the chat platform and the single entry point for the user. Behind it, llama-swap acts as the model server, and behind that sit the inference engines: Ollama, llama.cpp and vLLM. The README lists llama-swap as a systemd service, which means model routing survives a reboot without anyone logging in.

The reason llama-swap exists in this stack rather than Open WebUI talking directly to Ollama is model management. Open WebUI is documented as integrating with llama-swap, and the guide gives separate integration sections for the model server and for the systemd unit. That separation is the tell: the model server is treated as infrastructure with its own lifecycle, not as a plugin inside the chat UI.

Around that core, the guide bolts on optional capability. SearXNG provides web search, with its own Open WebUI integration section. Kokoro FastAPI provides text-to-speech. ComfyUI provides image generation. mcp-proxy and MCPJungle provide MCP servers, and the README includes a comparison section between the two plus a VS Code and Claude Desktop integration section, which means the MCP layer is not confined to Open WebUI. Tailscale handles remote access, Caddy is listed as a reverse proxy and Cloudflare as a DNS provider.

Getting it running means following init.bash, not running a compose file

There is no install command in the usual sense. The README's General section covers allowing sudo permissions, updating system packages, scheduling a startup script, configuring script permissions and an optional auto-login. The startup script is named init.bash, and the guide's stated approach is to schedule it to run at boot, after setting the GPU power limit and enabling auto-login.

The Docker section then covers adding your user to the docker group, installing the Nvidia Container Toolkit, creating a network, hardening the containers, and a set of helpful commands. Model files come through the HuggingFace CLI, with subsections for managing, downloading and deleting models. Each service gets its own installation block and its own Open WebUI integration block, which is the pattern throughout: install the thing, then point Open WebUI at it.

Because the guide is a sequence rather than a configuration artifact, the exact flags and keys live in the individual sections rather than in one file you can copy. That is a real cost. You cannot diff this repository against your own setup the way you would diff a compose file. You read, you adapt, and you verify each command, which the README itself urges: check everything you execute in a terminal.

The reference hardware is two RTX 3090s and that shapes the advice

The README states the guide was built around an Intel Core i5-12600KF, 96GB of 3200MHz DDR4, a 1TB M.2 NVMe SSD and 2x Nvidia RTX 3090 with 24GB each. That is not a casual homelab box. Two 24GB cards change which models are practical, how vLLM tensor parallelism is configured, and whether the GPU power-limit step even applies to your system.

The README is candid about one hardware caveat: it notes that AMD GPU compatibility was previously an issue but that recent Ollama releases support AMD GPUs natively, and the visible text ends mid-sentence on an AMD GPU power limit note. If you are on AMD, treat the power-limit and driver portions as unverified for your card. The guide also says most Linux distros should follow a very similar process, but everything named in the stack is Debian-specific in its packaging, so a Fedora or Arch reader is translating, not following.

It is a guide, so version drift is the failure mode

The repository has no releases. The last push date is recent, but a documentation project's freshness is not the same as its correctness, and there is no version pinning visible in the material. Every component listed (Ollama, llama.cpp, vLLM, Open WebUI, ComfyUI, Kokoro FastAPI, llama-swap, mcp-proxy, MCPJungle, SearXNG, Caddy, Tailscale) ships on its own schedule. A step that installs the latest of something will eventually install a version whose flags or endpoints differ from what the guide shows.

The README acknowledges this indirectly in its Priorities section, where maintainability is defined as keeping the interactions simple enough that you can maintain them as they evolve, with the parenthetical because they will evolve. It also has an Updating section covering general updates, Nvidia drivers and CUDA, Docker services, and each of Ollama, llama.cpp, vLLM and ComfyUI individually, plus a Troubleshooting section split by component. That structure is the honest admission that upgrades are where this stack breaks.

The second failure mode is scope. The README's own note says no part of the guide was written using AI and that AI was used to format Markdown and vet for technical inaccuracies, while warning readers to check everything. A single-author guide with a stated beginner audience and a warning label is a document you follow carefully, not one you trust blindly on a production host.

Where a packaged alternative is the better call

The closest comparison is not another model server but a distribution that bundles the same components. Open WebUI itself ships as a container, and Ollama ships as a single install with its own model management and an OpenAI-compatible endpoint. If your goal is chat with a local model and nothing else, Ollama plus Open WebUI is two installs, and it removes llama-swap, the systemd unit, the Docker network and the HuggingFace CLI steps from your path entirely.

The difference in approach is what you give up. That minimal pairing has no model-swap layer managing which engine serves which model, no SearXNG for web search, no MCP proxy for tool calling, no Kokoro FastAPI for speech and no ComfyUI for images. This repository's whole argument is that those pieces belong together and that the integration steps are the hard part worth documenting. If you have already solved the integration yourself, or you only want one of the capabilities, the guide is more surface area than you need.

Updating is manual, and the licence is permissive

Upgrade cost here is human attention, not subscription fees. The Updating section lists separate procedures for general packages, Nvidia drivers and CUDA, Docker services, Ollama, llama.cpp, vLLM and ComfyUI. There is no single command that brings the stack forward, and no compatibility matrix telling you which combination of versions was known to work together. Budget an afternoon per upgrade cycle, and expect the Troubleshooting subsections to be your first stop when something stops responding.

The repository is MIT licensed. That covers the documentation and any scripts it ships, such as init.bash. It does not relicense the components the guide installs, and those carry their own terms: Ollama, llama.cpp, vLLM, Open WebUI, SearXNG, ComfyUI, Kokoro FastAPI, llama-swap, mcp-proxy and MCPJungle are separate projects with separate licences, and Tailscale and Cloudflare are commercial services with their own plans. The MIT grant on this repository tells you what you may do with the guide, not what you may do with the stack it describes. Check each component's licence before you ship anything built on it; nothing here is legal advice.

Editorial conclusion

Adopt this if you are comfortable on Debian, have an Nvidia GPU, and want a documented path to a private chat stack rather than a pile of unrelated READMEs. Skip it if you need a supported product, a Kubernetes deployment, or a single-binary install, because this is a guide whose correctness depends on your hardware matching its assumptions. Before running anything, read the init.bash startup script end to end and confirm the GPU power-limit, auto-login and sudo steps match your machine, since those are the parts most likely to differ from the author's 2x RTX 3090 reference system.

Official sources

1. Issues
2. License: MIT
3. README
4. varunvasudeva1/llm-server-docs on GitHub