llama.vscode: Local FIM Completion and an Agent Inside VS Code

What llama.vscode actually replaces

The extension targets a narrow workflow: text and code completion where the model runs on your machine rather than a vendor endpoint. The README describes it as "Local LLM-assisted text completion, chat with AI and agentic coding extension for VS Code," and the feature list is built around that: auto-suggest on input, Tab to accept a suggestion, Shift + Tab to accept only the first line, Ctrl/Cmd + Right Arrow to accept the next word, and Ctrl + L to toggle a suggestion manually.

The audience is developers who already have llama.cpp installed, or who are willing to install it, and who have hardware to spare. The README's recommended settings are expressed purely in VRAM tiers, from more than 64GB down to less than 8GB, which tells you the project is not pretending to be hardware-neutral. If your machine cannot hold a code model in memory, the extension still installs and still talks to a server, but the README's CPU-only section carries an explicit warning that quality will be significantly lower.

That framing matters for adoption decisions. This is not a drop-in replacement for a hosted assistant. It is a front end for a server you operate.

The FIM constraint and why it shapes model choice

The README states plainly that the plugin requires FIM-compatible models and links to a Hugging Face collection for them. FIM stands for fill-in-the-middle: the model is trained to complete text between a prefix and a suffix rather than only continuing forward from a prompt. That is the mechanism behind auto-suggest on input, where the editor sends the code before the cursor and the code after it, and the model returns what belongs between them.

This constraint is the first thing to check before installing anything. A chat-tuned model that has never seen FIM training data will not behave the way the completion path expects, regardless of how good it is at conversation. The README points to the ggml-org collection as the source of compatible models, and the predefined model list in the extension includes local and external entries such as DeepSeek V4 Flash 0731, Qwen3.7 Flash, GPT-6 Astra and Claude Fable 5.1. Those external entries sit alongside the local ones, so the extension is not strictly local-only in what it can be pointed at, but the completion quality story in the README is built around local FIM models.

One consequence worth noting: the extension's own model management (add, remove, export, import, and a search-and-download path to Hugging Face) exists because picking and placing the right FIM model is part of the setup burden, not something the extension can abstract away.

How the extension talks to llama.cpp

The architecture visible in the README is a client-server split. llama.vscode is the VS Code client; llama.cpp runs as a server that the extension connects to. The README's manual instructions use llama serve with named model presets, and the CPU-only examples pin the port explicitly with --port 8012. That port number is the one concrete piece of wiring the README exposes, and it is the value to check first when completion silently does nothing.

The server side carries the tuning flags. The CPU-only examples show -ub 512 -b 512 --ctx-size 0 --cache-reuse 256 for the 1.5B model and -ub 1024 -b 1024 --ctx-size 0 --cache-reuse 256 for the 0.5B model. The --cache-reuse flag connects to a feature the README calls out directly: support for very large contexts even on low-end hardware via smart context reuse, with a link to the llama.cpp pull request that introduced it. So the extension's context handling is not implemented entirely in TypeScript; part of the capability lives in the server's KV cache reuse behaviour.

Context assembly on the client side is described as a ring context with chunks from open and edited files and yanked text, and the scope of context around the cursor is configurable. The README does not give the config key names for those settings, so the exact keys are something you would have to read from the extension's contributed settings in the marketplace listing or the source.

Getting a working setup: the commands the README gives

Installation is two layers. First the extension from the VS Code marketplace, also available on Open VSX. Second, llama.cpp. The README says this is now automatic on starting llama-vscode, using the official llama.cpp shell script or brew on Mac and Linux or Winget on Windows, and it keeps the manual instructions for reference.

Manual installation is one command per platform: brew install llama.cpp on macOS, winget install llama.cpp on Windows. For other systems the README points to the latest binaries or a source build and says to add the bin folder to the path. Linux users get the binaries route rather than an automated install.

Once llama.cpp is present, the README's flow is: open the llama-vscode menu from the status bar or with Ctrl+Shift+M, choose "Install/Upgrade llama.cpp" if you want the automatic path, then choose "Select/start env..." to pick an environment. The env concept groups models, and selecting or deselecting an env selects or deselects every model inside it. The README also lists predefined envs for different use cases such as completion only, chat plus completion, and chat plus agent.

Server startup for the recommended path is a single preset flag. More than 64GB VRAM: llama serve --fim-qwen-30b-default. More than 16GB: llama serve --fim-qwen-7b-default. Less than 16GB: llama serve --fim-qwen-3b-default. Less than 8GB: llama serve --fim-qwen-1.5b-default. Models pulled with -hf land in ~/Library/Caches/llama.cpp/ on macOS, ~/.cache/llama.cpp on Linux, and LOCALAPPDATA on Windows.

The agent view and its tooling

Llama Agent is a separate surface inside the extension, opened with Ctrl+Shift+A or from the menu. The README says it works with local models and names gpt-oss 20B as the best choice for now, with the caveat that it could work with external models such as those from OpenRouter. It supports MCP tools from MCP Servers installed and started in VS Code, and ships nine internal tools.

Two of those internal tools are described in enough detail to be useful. custom_tool returns the content of a file or a web page. custom_eval_tool lets you write your own tool in JavaScript, described as a function with an input and a return value string. There is also an attach-selection-to-context action and a configurable maximum loop count for the agent.

Two other features sit outside the editor entirely. A Telegram bot provides access to the llama-vscode agents from a phone. Deep links in the form vscode://ggml-org.llama-vscode?view=agent&prompt=Hello open VS Code if it is not already running, switch to the agent view, and prefill the prompt. The README does not document authentication or access control for the Telegram path, which is the kind of thing to verify before pointing it at a repository you care about.

The agent also has a Kimi K3 entry covering dynamically loaded tools, described as tools on demand, when used from moonshot.ai. That is an external-provider feature rather than a local-model one.

Where this approach breaks down

The clearest limitation is stated by the project itself: CPU-only configurations come with the note that quality will be significantly lower. The recommended presets are all sized by VRAM, and the smallest, --fim-qwen-1.5b-default, is listed for machines with less than 8GB. Below that you are in the CPU-only examples, which use 1.5B and 0.5B Qwen2.5-Coder quantisations. Those are small models. Completion on a 0.5B model is a different product from completion on a 30B one, and the README does not blur that distinction.

A second constraint is the FIM requirement. If your preferred local model is not FIM-compatible, it is not a candidate for the completion path no matter how strong it is at chat. The README directs you to a specific Hugging Face collection rather than saying any code model will do.

A third is operational. The extension depends on an external server process that you start, configure and keep running. The README describes automatic install of llama.cpp, but the server settings, port and model selection remain your responsibility, and the README's own CPU examples show how much flag tuning is involved. There is no claim in the material that the extension manages server lifecycle beyond the install step.

Finally, the documentation is uneven. Feature bullets are dense, but several features (ring context, context scope, the Telegram bot) are named without configuration details in the README, which defers to the wiki. Expect to read the wiki and the extension's settings UI rather than the README alone.

How it compares to llama.vim and to hosted assistants

The README names two reference points. The first is llama.vim, the Vim/Neovim plugin listed under Other IDEs. The README says the initial implementation of llama.vscode was done by Ivaylo Gardev using llama.vim as a reference. The two share a lineage and, by implication, a server-side dependency on llama.cpp, so the difference is the editor integration rather than the model stack. If you work across both editors, that shared backend is the reason the setup cost is paid once.

The second reference point is implicit: hosted completion services. The difference is not model quality, it is where inference happens and who pays for it. A hosted assistant sends your code to a remote endpoint and returns completions without asking you to size a model against your VRAM. llama.vscode inverts that. Your code stays on your machine, and in exchange you supply the GPU, choose a FIM model, run llama serve, and accept the quality ceiling that your hardware imposes. The README's VRAM tiers are effectively a statement that the ceiling moves with your card.

The agent view complicates the comparison slightly, because the README says it could work with external models from providers like OpenRouter. At that point you are back to sending context off-machine for the agent path while keeping completion local, which is a mixed posture worth being deliberate about.

Licence, maintenance and what to verify first

The extension is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the standard permissive arrangement and it is compatible with shipping the extension inside a corporate VS Code install. Nothing in the material suggests a separate commercial tier or a licence key. This is a description of the licence text, not legal advice; if you are redistributing the extension or bundling it, have your own counsel read the MIT terms.

Maintenance cadence is visible from the release history: v0.0.63, v0.0.64 and v0.0.65 landed across roughly two and a half weeks in August and September 2026, and the repository is not archived. The version numbers are still in the 0.0.x range, which is a reasonable signal that interfaces and settings may still move between releases. Pin a known-good version if you deploy this across a team rather than letting the marketplace update silently.

One cost the README does not quantify is disk. Models downloaded via -hf are cached in platform-specific directories, and the recommended presets go up to a 30B model for the 64GB VRAM tier. Those are large downloads, and the extension's Hugging Face search-and-download feature makes it easy to accumulate several. Budget the cache directory before you start pulling presets.

What to verify first, concretely: that llama serve starts with the preset matching your VRAM, that the server is reachable on the port the extension expects (the CPU examples use 8012), and that the model you selected is FIM-compatible per the ggml-org collection. If completion works on a single file with a small model, the wiring is correct and quality is the only remaining variable.

Editorial conclusion

Adopt llama.vscode if you already run llama.cpp or are willing to, and you want completion, chat and an agent view driven by models on your own hardware. Skip it if you have no GPU budget for a FIM model, or if you expect the extension to manage model lifecycle for you. Before committing, run llama serve with the flag matching your VRAM tier, confirm the server answers on port 8012, and check that your chosen model appears in the ggml-org FIM collection, because the README states the plugin requires FIM-compatible models.

Official sources

1. ggml-org/llama.vscode on GitHub
2. Issues
3. License: MIT
4. README
5. Releases