llm-server-docs: A Complete Debian Setup Guide for a Private Local LLM Server

What llm-server-docs Provides and Who It Is For

llm-server-docs is a documentation repository, not a software package. It contains no installable library and no deployable container of its own. What it provides is a step-by-step guide for someone setting up a dedicated Debian machine to run local language models, along with the surrounding services needed to make that machine useful: a search engine, a chat platform, model management tooling, MCP proxy servers, text-to-speech, image generation, a VPN, a reverse proxy, and DNS configuration.

The repository description says it is aimed at Linux beginners. The About section reinforces this: the author describes the project as a guide for people setting up a server for the first time. No part of the guide was written using AI. The README states explicitly that AI was used only for formatting Markdown elements and vetting for technical inaccuracies, and it asks readers to check every command before executing it.

AMD GPU users can follow most of the guide. The README notes that AMD GPUs are now supported natively by Ollama, so the compatibility gap that previously affected AMD hardware has been addressed in that component. The one step that is skipped for AMD is GPU power limiting, because AMD has recently made that operation difficult to configure.

Ten Components, One Debian Machine

The software stack listed in the README covers ten categories. For inference, the guide documents three engines: Ollama, llama.cpp, and vLLM. For web search, it uses SearXNG, a self-hosted metasearch engine that keeps queries off commercial search providers. For model serving, it covers llama-swap alongside a systemd service approach. Open WebUI provides the chat platform. For MCP proxy servers, the guide covers two options: mcp-proxy and MCPJungle, and includes a comparison section explaining the difference. Kokoro FastAPI handles text-to-speech. ComfyUI handles image generation. For remote access, Tailscale provides the VPN layer. Caddy acts as the reverse proxy. Cloudflare provides DNS.

The README design philosophy explains the rationale for this component selection. Standard protocols are preferred over bundled solutions throughout: OpenAI-compatibility and MCP protocols are explicitly called out as the reasons why components can be swapped out when better alternatives emerge. This modularity is built in as a requirement rather than an afterthought. Each component can be replaced independently as long as the replacement speaks the same protocol.

The Priorities section lists seven goals: Simplicity, Stability, Security, Maintainability, Aesthetics, Modularity, and Open source. Aesthetics is defined as making the result look like a cloud provider's chat platform rather than something cobbled together. Open source is justified on privacy grounds: chat platforms handle personal data in natural language, and verifiable code is the mechanism for confirming that data stays on the machine.

Prerequisites and the Debian Setup Process

The hardware reference configuration documented in the Prerequisites section is an Intel Core i5-12600KF, 96GB of 3200MHz DDR4 RAM, a 1TB M.2 NVMe SSD, and two Nvidia RTX 3090 GPUs each with 24GB of VRAM. This is a high-end homelab build. The README says any modern CPU and GPU combination should work, but the guide steps around GPU power limits and driver configuration were written for that specific hardware.

The General setup section of the table of contents describes steps for allowing sudo permissions, updating system packages, scheduling a startup script, configuring script permissions, and optionally configuring auto-login. The repository relies on an init.bash script that is scheduled to run at boot via cron or a similar mechanism. The guide covers setting the GPU power limit during this initialization phase.

The Docker section covers adding the current user to the Docker group, installing the Nvidia Container Toolkit for GPU passthrough into containers, creating a Docker network for service communication, and hardening Docker containers. The HuggingFace CLI section covers model management: downloading models to local storage and deleting them. The SearXNG section covers the search engine deployment and its integration with Open WebUI, so that the chat platform can perform web searches without sending queries to a commercial provider.

Inference Engine Choices and Model Management

Three inference engines are documented, and the guide treats them as interchangeable where the OpenAI-compatible API surface is concerned. Ollama is the most straightforward to install for most users and now handles AMD GPUs natively according to the README. llama.cpp offers more direct control over quantization levels and memory layout at the cost of more manual configuration. vLLM is optimized for throughput in multi-user scenarios and supports a wider range of model formats, but it has stricter hardware requirements and a more complex setup.

The model server layer sits between the inference engine and the chat platform. llama-swap manages model switching, loading different models into GPU memory on demand. A systemd service can also fulfill this role. The guide documents both paths and their respective integrations with Open WebUI, so the chat platform receives model requests and routes them appropriately regardless of which model server is running.

HuggingFace CLI handles the download and deletion of model files. This is a deliberate separation of concerns: the CLI manages the file system while the inference engine manages runtime loading. Keeping these as separate operations makes it easier to update or remove models without stopping the inference service.

MCP Proxy Options and Secure Remote Access

The MCP Proxy Server section covers two projects: mcp-proxy and MCPJungle. The guide includes a comparison section, which is unusual for a setup guide and suggests the author encountered a genuine trade-off worth documenting. Both are integrated with Open WebUI and with VS Code and Claude Desktop, giving the user a choice based on their workflow. The README does not give a verdict on which is better; it documents both and lets the reader decide after reading the comparison.

For remote access, Tailscale provides a WireGuard-based VPN that lets the user reach the server from outside the local network without opening ports on a router. Caddy acts as the reverse proxy in front of the services, handling HTTPS termination. Cloudflare handles DNS. The SSH and Firewall sections of the guide document how to lock down direct access to the machine independently of the application-layer access controls.

The guide also covers updating each component. Separate update sections cover general packages, Nvidia drivers and CUDA, Docker services, Ollama, llama.cpp, vLLM, and ComfyUI. This explicit update documentation reflects the Maintainability goal: the components will evolve, and the guide tries to give enough knowledge to follow along when they do.

Where This Guide Falls Short

llm-server-docs is a documentation-only repository. There is no code to run, no CLI to install, and no automated provisioning script. A reader must follow each section manually, adapt the commands to their hardware, and resolve any version conflicts or distribution differences themselves.

The guide targets Debian specifically. The README notes that most Linux distros should follow a similar process, but the exact package names, service management commands, and driver installation steps are written for Debian. Users on Ubuntu, Fedora, or other distributions will need to translate some steps.

For a team that wants to provision multiple identical servers or reproduce the setup reliably, a documentation guide is the wrong tool. An Ansible playbook, a Terraform configuration, or a purpose-built distribution like Umbrel would provide repeatable, automated deployment that this guide does not. The guide also has no built-in testing: there is no way to verify that a configuration is correct short of running the full stack and checking whether it works.

LM Studio is a common alternative for engineers who want a local LLM setup with minimal manual configuration. It provides a graphical interface on macOS and Windows, bundles model downloading and inference in a single application, and does not require Docker, GPU power configuration, or a reverse proxy. The trade-off is that LM Studio runs on a desktop machine rather than a dedicated headless server, and it does not expose the same collection of ancillary services that llm-server-docs assembles.

Maintenance Status and License

The repository is not archived. The last push was on 2026-06-30. The project has no GitHub releases; the documentation lives in README.md and the docs/ directory. The MIT license allows free use, modification, and distribution with attribution. The repository has a docs/ directory alongside the README, which the table of contents links into for the detailed per-component setup steps.

The guide covers a Troubleshooting section for SearXNG, Docker, SSH, Nvidia Drivers, Ollama, vLLM, and Open WebUI. It also covers a Monitoring section. The Notes section is divided into Software and Hardware subsections, with hardware notes specific to the dual RTX 3090 build used as the reference configuration.

Editorial conclusion

llm-server-docs is the right resource for an engineer who wants to run a full-featured LLM server on dedicated Debian hardware and does not mind assembling ten separate open-source components. It is not the right starting point for someone on Windows or macOS who wants a graphical interface out of the box, or for a team that needs a reproducible deployment they can maintain across machines without reading through the docs folder. Before following any section, verify that the commands in that section match your specific GPU vendor and driver version, since the guide was built around a dual RTX 3090 system and AMD GPU power limiting steps are explicitly skipped.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. varunvasudeva1/llm-server-docs on GitHub