forge: a reliability layer for self-hosted LLM tool-calling

The failure forge targets: a small model that cannot reliably emit a tool call

The problem is not that an 8B local model cannot call tools. It is that it calls them inconsistently: a truncated JSON argument, a tool name spelled slightly wrong, a turn where the model answers in prose instead of invoking anything. The README frames forge as "a reliability layer for self-hosted LLM tool-calling" and states that the guardrails (rescue parsing, retry nudges, response validation) apply even with zero required steps. That last clause matters. You do not have to describe a workflow to get the reliability behaviour; you can hand forge a tool set and let the model choose order and frequency.

The audience follows from that. Someone running a quantised model behind llama-server on their own GPU, who wants tool-calling that does not fall over on the third turn, is the target reader. The README also names a second audience explicitly: people already inside a coding harness such as opencode, aider or Cline, who can put the proxy in front of their existing client instead of rewriting anything. Forge is domain-agnostic by design, so the tool schemas are yours, not the project's.

One loop, three entry points, and where forge stops

Forge is deliberately not an orchestrator. The README lists multi-agent graphs, DAG planners and cross-agent coordination as out of scope, and that constraint shapes everything else. There is no scheduler that decides which agent runs next. There is one agentic loop, and forge makes the tool calls inside it reliable.

The three entry points sit at different depths. The proxy server is a drop-in that speaks both the OpenAI chat-completions API and the Anthropic Messages API at /v1/messages, sitting between any client and a local model server. The client believes it is talking to a smarter model. WorkflowRunner is the library path: you define tools, pick a backend, and forge manages the full lifecycle including system prompts, tool execution, context compaction and guardrails. Guardrails middleware is the shallowest integration, described in the README as composable, and demonstrated in examples/foreign_loop.py, where you keep your own orchestration loop and forge only validates responses, rescues malformed tool calls and enforces required steps.

Workflow structure is opt-in through three named parameters: required_steps, prerequisites and terminal_tool. The quick-start example sets required_steps to an empty list and terminal_tool to "get_weather", which tells the loop that the weather lookup ends the run. SlotWorker is a separate piece for multi-agent setups: it gives priority-queued access to a shared inference slot with auto-preemption, so specialist workflows can share one GPU slot rather than each holding a model resident.

Getting it running: two installs with different ownership

The distribution split is unusual and worth reading carefully. The standalone proxy bundles forge, a private Python runtime and the Anthropic SDK, so the host needs neither Python nor pip. On Linux and macOS the README gives `curl -fsSL https://raw.githubusercontent.com/antoinezambelli/forge/main/install.sh | sh`, and on Windows PowerShell `irm https://raw.githubusercontent.com/antoinezambelli/forge/main/install.ps1 | iex`. After a refreshed terminal, `forge-proxy init` creates a profile and `forge-proxy check` validates it. The README is explicit that the standalone installer does not install a backend executable, a model, a GPU stack, a service, credentials or client configuration. You supply all of that.

The Python package is a separate install: `pip install forge-guardrails` for core, or `pip install "forge-guardrails[anthropic]"` to add the Anthropic client. It requires Python 3.12+ and a running LLM backend. Critically, the Python package intentionally does not install a global forge-proxy command. You run the proxy implementation as `python -m forge.proxy`, and the README states the standalone installer is the sole owner of the forge-proxy command and its update and uninstall lifecycle. If you install both, you have two proxy paths with one lifecycle manager, and that is by design rather than an accident.

Backend setup is a choice of three. llama-server is recommended, and the README notes the top 10 eval configs all run on it: `llama-server -m path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf --jinja -ngl 999 --port 8080`. The --jinja flag is doing real work there, since it enables the chat template the tool-calling format depends on. Ollama is the easier alternative, described as slightly weaker on harder workloads, via `ollama pull ministral-3:8b-instruct-2512-q4_K_M`. Anthropic needs no local GPU at all; you install the extra and export ANTHROPIC_API_KEY.

Inside the quick start: pydantic schemas, a context budget, and a terminal tool

The quick-start code is short enough to read as a specification of the API surface. Tools are declared as a dictionary of ToolDef objects, each pairing a ToolSpec (name, description, and a pydantic model for parameters) with a plain Python callable. The example's GetWeatherParams is a single city field with a description, which is the schema the model sees.

Context handling is configured separately and is not optional in the example: `ContextManager(strategy=TieredCompact(keep_recent=2), budget_tokens=8192)`. TieredCompact with keep_recent=2 means older turns are compacted while the two most recent are retained, and budget_tokens=8192 caps the window. That pairing is the part most likely to need tuning, because a tool set with large schemas competes with conversation history for the same budget. The runner is then constructed as `WorkflowRunner(client=client, context_manager=ctx)` and invoked with `await runner.run(workflow, "What's the weather in Paris?")`.

The client in the example is LlamafileClient with a gguf_path, mode="native" and recommended_sampling=True. Sampling defaults are part of the reliability story here: a model that samples badly will produce malformed calls that the rescue parser then has to fix, and there is a limit to how much parsing can recover. The README lists supported backends as generic OpenAI-compatible endpoints, Ollama, llama-server, Llamafile, vLLM and Anthropic.

The eval numbers, and why you should not treat them as your numbers

The README claims forge takes an 8B local model from single digits to 84% across forge's 26-scenario v0.7.0 eval suite, and lifts Sonnet 4.6 from 85% to 98% on the same workload. Two caveats are stated in the same paragraph and deserve equal weight. The Anthropic numbers were measured in v0.6.0 and were not re-run in v0.7.0 because the cost is non-trivial. And the suite is forge's own, with 26 scenarios chosen by the project.

That does not make the numbers meaningless. A jump from single digits to 84% on any fixed suite is large enough that the underlying mechanism, not the scenario selection, is probably doing most of the work. But the shape of your tool schemas is what the rescue parser and retry nudges operate on, and a suite built around forge's own examples will flatter forge's own parsing heuristics. Treat 84% as a ceiling reached under favourable conditions, and treat your own tool set as the only eval that matters. The README's own framing supports this: it recommends llama-server partly because the top 10 eval configs run on it, which is a statement about the project's test matrix, not about your hardware.

Where forge is the wrong tool, and what to use instead

The README's own exclusion list is the first limitation: no multi-agent graphs, no DAG planners, no cross-agent coordination. If your problem is deciding which of five specialist agents handles a request, forge does not decide that. SlotWorker addresses the adjacent concern of sharing one GPU slot between specialist workflows, but sharing a slot is not the same as planning across agents.

The second limitation is that forge is not a coding harness and does not try to be. For coding specifically, the README points at proxy mode as the way to lift an existing harness rather than replace it. That is an honest positioning: opencode, aider and Cline already contain the file-editing, diff and repository-context machinery that a coding agent needs, and forge adds guardrails underneath them instead of competing. If you want a coding agent, install one of those and put forge-proxy in front of it. If you want a general agent framework with planning, tool registries and multi-agent handoff, LangGraph is the obvious comparison point, and the difference is structural rather than a matter of features. LangGraph models the workflow as a graph you author, with nodes and edges you control explicitly. Forge does the opposite: it keeps the loop open and intervenes when a call fails, with required_steps and prerequisites as optional constraints rather than a mandatory graph. Choosing between them is choosing whether you want to specify the control flow or recover from its absence.

The third limitation is environmental. The proxy bundles a Python runtime but installs no model, no GPU stack and no backend, so a machine without a working llama-server or Ollama instance gets nothing from it. And the Python package's refusal to install a global forge-proxy command means anyone mixing the two installs has to remember which one owns updates.

Maintenance, release cadence and the MIT licence

The release history in the supplied material shows three versions in the two weeks before the last push: v0.9.5 on 2026-08-30, v0.9.4 on 2026-08-26 and v0.9.3 on 2026-08-22. The subjects are narrow and specific: equivalent dual-auth compatibility, Anthropic reasoning capture, and a proxy command ownership hotfix. That cadence suggests active maintenance of the proxy surface in particular, and the v0.9.3 hotfix around command ownership is consistent with the two-install split described above being a live source of friction rather than a settled design.

For upgrade cost, the material does not describe a deprecation policy or a compatibility guarantee across minor versions, so pinning is the prudent reading rather than a documented recommendation. The standalone installer supports exact-version installation according to the README's pointer to docs/PROXY_INSTALLATION.md, which also covers profiles, updates, recovery and uninstall. Licence is MIT, which permits commercial use and modification with the usual requirement to retain the copyright and permission notice; that is a statement about the licence text, not legal advice, and anyone embedding forge in a distributed product should read the LICENSE file rather than this paragraph. The dependency surface is small: Python 3.12+, an optional Anthropic extra, and whatever backend you run separately. There is no homepage listed for the project, so the repository and its docs directory are the documentation.

Editorial conclusion

Adopt forge if you already run llama-server, Ollama, Llamafile, vLLM or an OpenAI-compatible endpoint and your failure mode is malformed or missing tool calls inside one loop. Do not adopt it if you need a DAG planner, multi-agent coordination or a coding harness, all of which the README places out of scope. Before committing, verify two things yourself: that your backend accepts the --jinja flag llama-server needs for tool-calling templates, and whether the 84% figure, measured on forge's 26-scenario v0.7.0 eval suite, transfers to your own tool schemas. Run `forge-proxy init` and `forge-proxy check` first if you only intend to use the proxy.

Official sources

1. antoinezambelli/forge on GitHub
2. Issues
3. License: MIT
4. README
5. Releases