textgen

An open source desktop app for running large language models locally, with text, vision, tool calling, and an OpenAI and Anthropic compatible API. The README leads with portable builds that claim setup in about a minute.

The README claims getting started takes about a minute. Portable builds exist for Linux, Windows, and macOS with CUDA, Vulkan, ROCm, and CPU-only options, and all dependencies are included. The builds are compatible with GGUF models from llama.cpp. For people who need more, a one-click installer per OS runs a start script, asks for the GPU vendor, and opens http://127.0.0.1:7860 in the browser once installed. A full installation for extra backends needs about 10GB of disk space and downloads PyTorch.

The chat side covers several modes. Instruct mode is for instruction following, while chat-instruct and chat modes handle talking to custom characters, with prompts formatted automatically using Jinja2 templates. Vision support attaches images to messages for visual understanding. File attachments accept text files, PDFs, and docx documents to discuss their contents. Messages can be edited, versions navigated, and conversations branched at any point, and a notebook tab offers free-form text generation outside chat turns.

Multiple backends are listed, including llama.cpp, Transformers, ExLlamaV3, and TensorRT-LLM, and you can switch between them and models without restarting. An OpenAI and Anthropic compatible API exposes chat, completions, and messages endpoints with tool-calling support, positioned as a local drop-in replacement. Models can call custom functions during chat, including web search, page fetching, and math, with each tool as a single Python file, and MCP servers are supported. A training tab fine-tunes LoRAs on multi-turn chat or raw text, and an image generation tab works with diffusers models like Z-Image-Turbo.

The privacy claims are blunt: 100 percent offline and private, with zero telemetry, no external resources, and no remote update requests. The interface supports dark and light themes, syntax highlighting for code blocks, and LaTeX rendering for math. Built-in and community extensions include text to speech, voice input, and translation, with the full list in the extensions directory.

The whole README reads like a privacy-first counterpoint to hosted LLM services, with local runs and an API that mimics the services many teams already use.

• Source 1 · Project repository
• Source 2 · README