Shepherd Model Gateway: an engine-agnostic router for self-hosted LLM fleets

The problem SMG addresses: many engines, many APIs, one client contract

If you run vLLM on one node and SGLang on another, and you also call OpenAI for overflow, your application code has to know about three request shapes, three error formats, three token counting conventions and three sets of streaming semantics. SMG's answer is a single endpoint. The README describes it as an "Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments" that centralizes worker lifecycle management and balances traffic across self-hosted engines and cloud providers behind one unified endpoint. The intended user is a platform or inference team operating several engines, not an individual calling one model. The API surface listed in the README is broad: OpenAI Chat Completions, Completions, Embeddings, Rerank and Classify; Responses and Conversations for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize and detokenize; and MCP tool execution. That breadth is the point. A client that speaks OpenAI Chat Completions can be pointed at SMG and reach a TensorRT-LLM worker without a code change, assuming the model and parser support line up.

Cache-aware routing and the radix tree over KV state

The routing layer is where SMG claims its main advantage. The README states that cache-aware routing "tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX", with load modeling that accounts for queued token work and KV pressure. The mechanism, as described, is prefix reuse: if a request shares a prefix with work a worker has already processed, sending it to that worker avoids recomputing the prefix. This is the same class of idea as prefix-aware routing in other gateways, but SMG's claim is that it tracks state across five different engines rather than one. The README lists ten routing policies: cache_aware, least_load, power_of_two, consistent_hashing, prefix_hash, bucket, round_robin, random, manual and passthrough. That spread matters because cache_aware is not always right. With short prompts and no shared prefixes, a radix tree lookup adds bookkeeping for no reuse. With highly skewed traffic, least_load or power_of_two may distribute better. The documentation points to a routing page under concepts/routing/load-balancing, and that page is where the actual selection criteria would need to be, since the README only names the policies.

The gRPC pipeline and prefill/decode disaggregation

SMG describes an "industry-first gRPC pipeline" with native streaming gRPC to engines, prefill/decode and encode disaggregation, and DP-aware routing. The architecture diagram in the README shows clients flowing through a gateway layer and a router layer to gRPC workers, HTTP workers and external APIs. Two details are worth separating. First, the transport: HTTP workers are reached over HTTP, and gRPC workers over a streaming gRPC path. Second, the disaggregation: prefill and decode can be separate worker roles, and there is a third encode stage for vision. The README's Kubernetes discovery section mentions "per-role prefill/decode/encode selectors", which confirms that roles are a first-class concept in service discovery rather than a naming convention. DP-aware routing means data-parallel engine replicas are addressed individually. If your engines are not configured for disaggregated serving, most of this layer is inert and you are using SMG as a load balancer with a larger dependency footprint. The README does not state which engine versions support each disaggregation mode, so that has to be confirmed against the engine documentation, not SMG's.

Getting it running: install, launch flags, and what the default port tells you

The README gives four install paths. Docker is `docker pull lightseekorg/smg:latest`. Kubernetes is `helm install smg oci://ghcr.io/smg-project/charts/smg`. Python is `pip install smg`. Rust is `cargo install smg`, and the README notes it needs protoc, which follows from the gRPC pipeline. Launch is a single binary. A single worker: `smg launch --worker-urls http://localhost:8000`. Multiple workers with cache-aware routing: `smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware`. High availability uses `--enable-mesh`, `--mesh-advertise-host` and `--mesh-peer-urls`. The client-facing example is a curl to `http://localhost:30000/v1/chat/completions` with a model name and a messages array, so the default listen port is 30000 and the OpenAI path is `/v1/chat/completions`. Note what is not in the quick start: no config file example, no API-key setup, no model-to-worker mapping. The README's feature table points to a service-discovery page for Kubernetes pod watchers with label selectors, and to a tokenization-and-parsing page for the 21 tool-call parsers and 16 reasoning parsers. Those pages carry the configuration surface that the quick start omits.

Where SMG is the wrong tool

The clearest case against SMG is a single engine. If one vLLM or SGLang process serves your traffic, SMG adds a network hop, a second process to monitor, and a routing decision that has exactly one correct answer. The cache-aware machinery has nothing to choose between. A second case is a team that cannot operate a stateful control plane. High availability is described as mesh networking with SWIM gossip and CRDT-replicated state, and the launch flags include advertise host and peer URLs. That is a cluster with its own failure modes, and the README does not describe what happens to in-flight requests during a mesh partition. A third case is API coverage assumptions. The README lists Responses, Conversations, Realtime over WebSocket and WebRTC, and MCP tool execution, but it does not publish a matrix showing which of those work against which backend. Anthropic Messages against a vLLM worker is not the same problem as Anthropic Messages against Anthropic. Treat the coverage list as a statement of gateway endpoints, not a guarantee per backend. Finally, WebAssembly plugins and MCP execution with approval policies widen what runs inside your gateway process. If your threat model does not accept third-party logic in the request path, that surface is a liability you are taking on for features you may not use.

How SMG differs from a plain reverse proxy and from LiteLLM

A generic reverse proxy such as nginx or Envoy can spread requests across backends and terminate TLS. It cannot track which worker holds which KV-cache prefix, because it has no visibility into the engine's cache state. That is the specific gap SMG fills: the radix-tree tracking described in the README is engine-aware bookkeeping that a layer-7 proxy has no access to. The closer comparison is LiteLLM, which also presents one OpenAI-compatible endpoint across providers. The difference in approach is where the intelligence sits. LiteLLM's primary job is provider translation and key management in Python, with routing policies layered on top. SMG is a Rust binary whose stated focus is the routing decision itself, with cache-aware selection, queued-token load modeling, prefill/decode disaggregation and DP-aware addressing as core mechanisms rather than add-ons. The trade-off is ecosystem reach. LiteLLM's provider list is longer and its Python surface is easier to extend in-process. SMG's extension point is WebAssembly plugins, which is a harder boundary to write against but a stronger one to isolate. If your problem is provider translation, LiteLLM is the smaller commitment. If your problem is GPU utilization across self-hosted engines with shared prefixes, that is the case SMG is built for.

Maintenance cost, release cadence and the Apache-2.0 terms

The release history shows v1.10.1 on 2026-08-27, v1.10.0 two days earlier, and v1.9.0 on 2026-07-30. That is a minor release roughly monthly with patch releases between, and it implies you should expect to track versions rather than pin once and forget. The repository was last pushed 2026-09-10 and is not archived. Upgrade cost concentrates in the areas that touch your configuration: routing policy behaviour, parser detection for tool calls and reasoning, and the mesh protocol, since CRDT-replicated state between peers assumes compatible versions on both ends. The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. Apache-2.0 also requires that you preserve copyright and licence notices and state significant changes if you redistribute. That is the general shape of the licence and not legal advice; if you are embedding SMG in a product, read the LICENSE file and get your own review. The README does not describe a commercial support offering, an SLA, or a long-term support branch, so plan for community-paced maintenance.

Editorial conclusion

Adopt SMG if you already run more than one self-hosted inference engine and want a single OpenAI-compatible endpoint with cache-aware routing, priority admission and multi-tenant keys in front of them. Do not adopt it if a single vLLM or SGLang instance serves your traffic, or if you cannot run a stateful control plane with mesh gossip and a WebAssembly plugin surface. Verify first that the Kubernetes service discovery path or the static --worker-urls list matches your deployment shape, that the routing policy you intend to use (cache_aware, least_load, power_of_two) is exercised against your actual engine versions, and that the Responses and Conversations APIs you depend on are covered for your backend, since the README lists API coverage without a per-backend compatibility matrix.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. smg-project/smg on GitHub