# Gateway API Inference Extension: routing LLM traffic through ext-proc

> The Kubernetes SIG project turns any gateway that already speaks Envoy external processing into an inference gateway, moving request scheduling for self-hosted generative models into a pluggable Endpoint Picker. It is worth a look for platform teams running model servers on Kubernetes, but the routing packages that carried most of the weight have just moved out to llm-d.

**kubernetes-sigs/gateway-api-inference-extension** — Gateway API Inference Extension

- Repository: https://github.com/kubernetes-sigs/gateway-api-inference-extension
- Website: https://gateway-api-inference-extension.sigs.k8s.io/
- Stars: 793 · Forks: 321
- Language: Go
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/kubernetes-sigs-gateway-api-inference-extension

## Turning an existing Envoy gateway into an inference gateway

Gateway API Inference Extension is a Kubernetes SIG project written in Go under the Apache-2.0 license. Its premise is narrow and practical: if a gateway can already speak Envoy's external processing protocol and already implements Gateway API, then it can be repurposed as an inference gateway without being rewritten. The README names Envoy Gateway, kgateway and the GKE Gateway as examples of gateways this applies to. The audience is platform teams who host generative models in-cluster and want to expose their OpenAI-compatible chat completion endpoints to other workloads, either inside the cluster or outside it.

The value proposition is scheduling rather than proxying. A generic gateway balances requests across backends, which is the wrong objective when backends are GPU-backed model servers with uneven cache state and different adapter sets. The project lists improvements to tail latency and throughput for completion requests, and it does so through a request scheduling algorithm the README describes as kv-cache and request cost aware. The goal is to avoid evictions and queueing as load grows.

The project also places itself inside a wider stack. It can sit under a higher level AI Gateway such as LiteLLM, Solo AI Gateway or Apigee, so that self-hosted models and model-as-a-service providers are reachable through one entry point. That layering is what lets a platform team share a pool of foundation model servers across many client model names without multiplying accelerators.

## How the Endpoint Picker intercepts a request through ext-proc

The data path runs through the Envoy external processing filter. A component called the Endpoint Picker, abbreviated EPP, communicates with the gateway over that protocol and acts as the router. It intercepts each incoming inference request and routes it to a model server replica it considers optimal. In other words, the gateway keeps doing what a gateway does, and the scheduling logic sits one hop behind it in a protocol-defined extension point rather than inside the proxy binary.

That design has an obvious attraction. Scheduling code changes far more often than proxy code, and the external processing protocol lets it change independently. It also means the project can serve multiple gateway implementations at once, which is why the same extension can sit behind Envoy Gateway, kgateway or GKE Gateway without maintaining three integrations.

The plugin architecture is also what made an external collaboration possible. The README describes how the pluggable architecture of the inference gateway was used to enable the llm-d Router, and how the project worked with that team on a disaggregated serving solution in which vLLM and the inference gateway are customised together. The stated direction is to generalise the disaggregated serving plugins and set a standard for disaggregated serving across any model server that follows the protocol described in the project proposals, in particular the model server protocol proposal under the proposals directory.

## Scheduling on metrics the model server reports back

Scheduling decisions are not hard-coded. The README defines an Inference Scheduler as an extendable component that decides which endpoint is best in terms of cost or performance for a given inference request, using data called Metrics and Capabilities. That data is provided by model serving platforms and covers performance, availability and capability signals. Two named examples are prefix cache status and the availability of LoRA adapters.

This is the part that differs most from a plain load balancer. Prefix cache state tells the scheduler whether a given replica has already processed a shared prefix, so a request can be steered to the replica that can reuse cached work instead of paying to recompute it. LoRA availability tells it which replicas can serve a request at all when the request names an adapter. Both signals come from the model server rather than from the gateway, which is why the project treats model server integration as a first-class concern rather than an afterthought.

The repository layout backs this up. There is an api directory and an apix directory for the declarative APIs, a client-go directory, a cmd directory for entrypoints, an internal directory, a pkg directory, and a conformance directory that the README identifies as the primary home for conformance test development. The Go module is sigs.k8s.io/gateway-api-inference-extension and it requires go 1.26.0, along with the Envoy control plane, controller-runtime, the Kubernetes API machinery at v0.37.1 and gRPC, which is what the ext-proc conversation rides on.

## Declarative routing for adapters, traffic splits and model upgrades

Beyond request placement, the project adds Kubernetes-native declarative APIs. The README states that these route client model names to use-case specific LoRA adapters, control incremental rollout of new adapter versions, support A/B traffic splitting, and enable safe blue-green upgrades of base models and model servers. The mechanism matters more than the feature list: the mapping from a client-facing model name to a concrete adapter and a concrete backend becomes an object in the cluster rather than a routing table inside a proxy, so it can be reviewed, audited and changed the way other Kubernetes resources are.

There is an important caveat about which APIs remain here. The Endpoint Picker, the InferenceObjective and InferenceModelRewrite APIs, and the Body Based Router packages have all moved to other repositories. The Endpoint Picker and its associated APIs now live in llm-d/llm-d-router, and the Body Based Router packages live in llm-d/llm-d-inference-payload-processor. The README states plainly that no new code will be accepted in those packages here and that they will be archived soon.

What stays is narrower and still useful. This repository continues to host the lightweight Endpoint Picker, referred to as LWEPP, and the InferencePool API, and it remains the primary location for conformance test development. A reader planning a new deployment should treat the repository as the API and conformance anchor, and treat llm-d as the home of the routing implementation and the payload processing work.

## Building the lightweight Endpoint Picker image

The README does not publish a step-by-step install guide, so the honest place to look for the mechanics is the build system. The Makefile at the repository root is where the image names, the container tool and the build targets are defined, and it is the same file a contributor or a platform engineer would read first. The following lines are taken verbatim from it and describe how the lwepp image is named and where it is pushed.

```bash
STAGING_IMAGE_REGISTRY ?= us-central1-docker.pkg.dev/k8s-staging-images
IMAGE_REGISTRY ?= $(STAGING_IMAGE_REGISTRY)/gateway-api-inference-extension
LWEPP_IMAGE_NAME := lwepp
```

The build itself is driven by a container tool, defaulting to Docker, and by the buildx command rather than a plain build. The Makefile notes that the target commands are only tested with Docker, which is scaffolded by default, and that a different tool such as podman may require replacing the variable.

```bash
CONTAINER_TOOL ?= docker
DOCKER_BUILDX_CMD ?= docker buildx
IMAGE_BUILD_CMD ?= $(DOCKER_BUILDX_CMD) build
```

Two other details are worth knowing before a first build. The base image defaults to a distroless static image, so the container has no shell or package manager, which is a deliberate supply chain choice but does mean debugging inside a running pod is harder. And ENVTEST_K8S_VERSION is pinned to 1.31.0 for the envtest binaries, so local controller tests run against an older Kubernetes than the v0.37.1 libraries the module depends on.

## Where this approach stops being the right tool

Several limits are visible in the project's own documentation. First, the dependency on the external processing protocol is a hard requirement, not a preference. A gateway without an ext-proc filter cannot become an inference gateway through this project at all, and the set of qualifying gateways is whatever Envoy Gateway, kgateway, GKE Gateway and similar implementations happen to provide.

Second, the scheduling signals must come from model servers that follow the model server protocol described in the proposals. The project's own direction is to set a standard for disaggregated serving across any protocol-adherent model server, which concedes that not every server speaks it yet. A scheduler that cannot read prefix cache or adapter state from the backend has much less to work with, and the README does not publish measured figures for how routing decisions affect tail latency, so any performance expectation has to be validated in your own cluster.

Third, and most consequential, the project is deliberately shrinking. The Endpoint Picker, the rewrite APIs and the Body Based Router are leaving, and the repository that remains is a reference implementation plus API and conformance work. That is a reasonable choice for a SIG project that wants one home for the standards, but it means an adopter who wants to write a custom scheduler or a custom payload processor should start in llm-d rather than here.

## Choosing between the reference EPP and a production router

A realistic alternative exists, and it is inside the same ecosystem. The llm-d Router is the project where the full Endpoint Picker and the InferenceObjective and InferenceModelRewrite APIs now live, and llm-d itself is the project behind llm-d-inference-payload-processor for the Body Based Router. The difference in approach is one of depth and ownership. The llm-d Router is the actively evolving implementation with the surrounding integrations, including the vLLM partnership described in the README and the disaggregated serving work. The lightweight EPP in this repository is described as a reference implementation provided for conformance test purposes.

That framing is the right way to read the split. If your goal is conformance, a reference scheduler that matches the API is enough, and the conformance suite here is where the behaviour is pinned down. If your goal is production traffic with custom scheduling, you are assembling across two repositories, and the pieces must be kept on compatible versions. The repository version is the other half of that decision: recent releases include v1.6.0 on 2026-08-17, v1.6.1 on 2026-09-11 and v1.6.2 on 2026-09-17, and the last push to the repository was on 2026-10-19. Nothing in the release notes indicates a stable release cadence, so pin a tag rather than tracking the branch.

## Conclusion

Adopt this project if you already run model servers on Kubernetes and your gateway supports Envoy external processing and Gateway API, since the InferencePool API, the lightweight Endpoint Picker and the conformance suite are where the remaining value sits. Do not adopt it as the home for your own Endpoint Picker or body based router logic, because those packages no longer accept new code here and are scheduled for archiving. Before committing, verify that your gateway implementation exposes an ext-proc filter, confirm which release tag your build pins since the project published v1.6.2 on 2026-09-17, and check the llm-d repositories that now own the Endpoint Picker and the model rewrite APIs your deployment depends on.

## FAQ

### What is Gateway API Inference Extension used for?

It upgrades a gateway that supports Envoy external processing and Gateway API into an inference gateway for self-hosted generative models on Kubernetes, routing requests to model server replicas through a pluggable Endpoint Picker.

### Does Gateway API Inference Extension require Envoy Gateway, or can it work with other gateways?

The README names Envoy Gateway, kgateway and the GKE Gateway as examples, and states the extension applies to any ext-proc capable proxy or gateway. The requirement is support for the external processing protocol and Gateway API, not a specific proxy vendor.

### Where do the Endpoint Picker and the Body Based Router live now?

The Endpoint Picker and the InferenceObjective and InferenceModelRewrite APIs moved to llm-d/llm-d-router, and the Body Based Router packages moved to llm-d/llm-d-inference-payload-processor. This repository keeps the lightweight EPP, the InferencePool API and conformance tests, and no longer accepts new code in the moved packages.

### What signals does the inference scheduler use to pick an endpoint?

It uses Metrics and Capabilities reported by model serving platforms, which cover performance, availability and capabilities such as prefix cache status and LoRA adapter availability, according to the README.

## Sources

- [kubernetes-sigs/gateway-api-inference-extension on GitHub](https://github.com/kubernetes-sigs/gateway-api-inference-extension)
- [License: Apache-2.0](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/LICENSE)
- [Project website](https://gateway-api-inference-extension.sigs.k8s.io/)
- [README](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/README.md)
- [Releases](https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/kubernetes-sigs-gateway-api-inference-extension