LocalGPT: Rust AI Assistant with Persistent Memory and 3D World Builder

What LocalGPT ships in its two Rust binaries

LocalGPT is a Rust workspace with two published crates. The first is localgpt: an AI assistant for local use, running as an interactive terminal session, a single-question command, or a background daemon with an HTTP API. The second is localgpt-gen: a standalone world-building binary that takes natural language prompts and produces 3D scenes using the Bevy game engine, then exports them as glTF/GLB, HTML, or screenshots.

Both binaries install from crates.io without a source checkout:

cargo install localgpt-gen
cargo install localgpt

Neither binary requires Node.js, Docker, or Python. The README describes this as a deliberate design constraint: single binary, all dependencies compiled in. The workspace version is 0.3.6. Rust edition 2024 is used, and the Cargo.toml workspace lists 16 internal crates covering the core, CLI, server, sandbox, mobile FFI, generation, markdown, world types, world export, Bevy integration, audio, sync, world agent, relay, and bridge components. The project also provides a docker-compose.yml for teams who prefer a containerized setup with a pre-configured daemon: it mounts the workspace directory and exposes port 31327, storing embeddings in a named volume for the FastEmbed local embedding model cache. Alternatively, the project can be cloned and built from source with cargo run.

localgpt-gen: AI-driven 3D world creation with Bevy

localgpt-gen takes a natural language prompt and generates a 3D scene. It supports parametric primitive shapes including box, sphere, cylinder, capsule, plane, torus, pyramid, tetrahedron, icosahedron, and wedge; PBR materials with color, metalness, roughness, emissive, alpha, and double-sided properties; point, spot, and directional lights; and named behaviors including orbit, spin, bob, look_at, pulse, path_follow, and bounce. Audio can be attached as ambient (wind, rain, forest, ocean, cave) or spatial emitters.

Worlds can be saved as reusable skills and loaded later:

localgpt-gen "Create a desert scene with pyramids and a UFO hovering above"
localgpt-gen --scene ./world.glb
localgpt-gen --verbose

Headless mode runs generation without a display window, useful for batch jobs or CI pipelines:

localgpt-gen headless --prompt "Build a cozy cabin in a snowy forest"

localgpt-gen also exposes an MCP server so that Claude Desktop, VS Code, Cursor, Zed, and other MCP-compatible editors can call it as a tool:

localgpt-gen mcp-server

Adding it to Claude Desktop requires a short entry in ~/Library/Application Support/Claude/claude_desktop_config.json on macOS:

{
  "mcpServers": {
    "localgpt-gen": {
      "command": "localgpt-gen",
      "args": ["mcp-server"]
    }
  }
}

The memory system in localgpt-gen learns palette preferences, lighting preferences, and entity templates across sessions and applies them automatically in future generations.

Installing and configuring the AI assistant

After cargo install localgpt, three commands cover the main modes:

localgpt chat
localgpt ask "What is the meaning of life?"
localgpt daemon start

The daemon mode starts an HTTP server on port 31327 (as set in the docker-compose.yml, which binds it to 127.0.0.1:31327 by default) and exposes a web UI at GET /, a chat endpoint at POST /api/chat, a streaming SSE variant at POST /api/chat/stream, and memory search at GET /api/memory/search?q=query. A full CLI command reference is available:

localgpt chat                   # Interactive chat
localgpt ask "question"         # Single question
localgpt daemon start           # Start daemon
localgpt memory search "query"  # Search memory
localgpt config show            # Show config
localgpt paths                  # Show resolved paths

Configuration is stored in a config.toml file at the XDG config directory. To connect to a local LM Studio instance with no API key:

[agent]
default_model = "openai/qwen/qwen3.5-35b-a3b"

[providers.openai]
api_key = "lm-studio"
base_url = "http://127.0.0.1:1234/v1"

For cloud providers, the Anthropic example in the README uses:

[agent]
default_model = "claude-cli/opus"

[providers.anthropic]
api_key = "${ANTHROPIC_API_KEY}"

The README lists LM Studio, Ollama, Anthropic, OpenAI, xAI, GLM, Vertex AI, and CLI providers as supported. The config.toml reference is at website/docs/configuration.md in the repository. Run localgpt paths to see the resolved XDG directories for config, data, state, and cache on the current machine.

Persistent memory system and autonomous heartbeat

The AI assistant stores long-term context in markdown files under a workspace directory:

<workspace>/
├── MEMORY.md     # Long-term knowledge (auto-loaded each session)
├── HEARTBEAT.md  # Autonomous task queue
├── SOUL.md       # Personality and behavioral guidance
└── knowledge/    # Structured knowledge bank

MEMORY.md is loaded automatically at the start of every session. HEARTBEAT.md holds autonomous tasks the assistant works through in the background when the daemon is running. SOUL.md provides persistent personality and behavioral guidance that is injected into every conversation.

The knowledge directory is indexed with SQLite FTS5 for full-text keyword search and sqlite-vec for semantic search using local embeddings. Both search types are accessible through the CLI:

localgpt memory search "query"

This architecture means memory survives restarts and is human-readable: any text editor can open MEMORY.md and inspect or edit what the assistant remembers. No proprietary format or database export is required. The Cargo.toml workspace dependencies show rusqlite 0.40 with the bundled, functions, vtab, and load_extension features, plus sqlite-vec 0.1.7-alpha.10 for the vector store. These are compiled into the binary, so no separate SQLite installation is needed on the host machine. The autonomous heartbeat (HEARTBEAT.md) lets a user delegate recurring tasks to the assistant; when the daemon is running, it checks and advances the task queue in the background without requiring an active chat session.

Security model and kernel-level sandboxing

LocalGPT implements what the README calls defense-in-depth security. The combination of OS-level sandboxing, a protected policy file, and prompt injection defenses is designed to limit what the agent can do outside its configured workspace, even if a user feeds it a prompt designed to escape those constraints. On Linux, Landlock and seccomp provide kernel-enforced sandboxing at the filesystem and syscall level. On macOS, Seatbelt is used. The Docker compose configuration drops all Linux capabilities and sets no-new-privileges, runs the container as a non-root user, and mounts /tmp as noexec.

Prompt injection defenses include marker stripping, pattern detection, and content boundary enforcement. The POLICY.md file holds standing instructions for the agent. It is sanitized before injection into the prompt and is explicitly protected from agent self-modification: the agent cannot overwrite its own policy file.

Signed policy files are an additional layer. The README documents this in website/docs/policy.md and website/docs/sandbox.md. This combination is more structured than most local AI assistant tools, which typically rely on prompt-level instructions alone without OS-layer enforcement.

Limitations and comparison with Ollama

LocalGPT depends on an external LLM provider for all inference. It does not bundle or download a model. A user must have LM Studio, Ollama, or a cloud API account configured before the assistant produces any output. This is an explicit design choice: LocalGPT is an application layer above the model server, not a model server itself.

The world builder (localgpt-gen) has no built-in path to convert an existing 3D asset into a prompt-modifiable world. The --scene flag loads an existing glb file but the README does not document editing or mixing existing external assets with generated geometry.

Ollama is the most common comparison point. Ollama is a local model server that runs LLMs via an OpenAI-compatible API on Linux, macOS, and Windows. Its focus is serving models; it does not include a persistent memory system, autonomous task scheduling, policy enforcement, or the world builder. LocalGPT can use Ollama as a provider, adding the assistant features on top of Ollama's model serving. Teams who only need an API endpoint for local models do not need LocalGPT; the assistant features are the differentiator.

Editorial conclusion

LocalGPT is worth evaluating if you need a local AI assistant that retains context across sessions without sending data to the cloud, or if you want to generate and export 3D scenes from natural language. The world builder is an unusual capability with no direct equivalent in most AI assistant tools. The security model is thoughtful: kernel-enforced sandboxing on both Linux and macOS, a policy file the agent cannot modify, and signed policy rules. Teams who only need a local model server and no assistant layer should look at Ollama instead; LocalGPT treats Ollama as a provider, not a replacement. The project was last updated on 2026-09-27 and is at version 0.3.6.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. localgpt-app/localgpt on GitHub
4. Project website
5. README