AI Gateway Labs: APIM Policies as Teaching Material for Model, Tool and Agent Traffic

What the repository actually is, and what it is not

The name invites a wrong assumption. AI-Gateway is not a gateway you install. The README describes it as labs that explore AI models, MCP servers and agents with the AI Gateway powered by Azure API Management and Microsoft Foundry, and the repository layout supports that reading: each entry under labs/ is a Jupyter notebook paired with Bicep infrastructure templates and APIM policies that you deploy into your own Azure subscription. The gateway itself is Azure API Management. This repository is the curriculum around it.

That framing answers the audience question quickly. The material targets engineers who have already decided to put APIM in front of model endpoints and now need to know which policy fragment does what. The labs are grouped into three tracks: models (load balancing, token rate limiting, semantic caching, model routing, a FinOps framework), tools (Model Context Protocol, MCP client authorization, function calling, realtime audio with MCP), and agents (Foundry Agent Service, the OpenAI Agents SDK, Gemini with MCP tools, A2A-enabled agents). Someone evaluating whether to use APIM at all will find the notebooks assume that decision rather than argue it.

The primary language is Jupyter Notebook, which is a deliberate choice with consequences. Notebooks let a reader execute one cell at a time and see intermediate responses, which suits policy debugging. They also do not diff well, do not run in CI without extra tooling, and tempt copy-paste into production. Treat the notebooks as documentation you can execute, not as source you vendor.

The mechanism: Bicep provisions the gateway, policies shape the traffic

The data flow described in the README is conventional for an API gateway. A client calls an APIM endpoint. APIM applies policy, which may enforce OAuth 2.0 or managed identity authentication, apply content safety filtering, consult a semantic cache, check a token quota, or select a backend from a pool. The request then reaches the model endpoint or tool backend, and logging and token metrics are emitted for observability.

The interesting part is where the policy lives. Each lab carries its own Bicep templates and APIM policies, so the gateway configuration is versioned alongside the notebook that explains it. That is a better arrangement than a tutorial that tells you to click through the Azure portal, because it means the policy XML is inspectable before you deploy anything. If you want to know how token rate limiting is expressed, the answer is in the lab's policy files, not in prose.

Model Context Protocol support appears in the tools track, with labs covering MCP generally, an MCP client authorization flow, and a combination of the realtime voice API with MCP tools. The README states that MCP gives plug and play tools with OAuth credential management. Credential handling is the part worth studying closely, since it is the piece most teams get wrong when they wire tools into an agent.

The agent track leans on external orchestration frameworks: Foundry Agent Service, the OpenAI Agents SDK, Google Gemini models, and A2A-enabled agents where an MCP agent acts as an A2A server. The gateway's role in these labs is to sit between the orchestrator and the model or tool, which is the same position it occupies in the model labs. Nothing in the supplied material suggests the agent labs change the gateway's architecture; they change what is being routed.

Getting a lab running: uv, a Codespace, and one notebook

The README's prerequisites are explicit: Python 3.12 or later, uv as the package manager, VS Code with the Jupyter extension, an Azure subscription with Contributor and RBAC Administrator roles, and an authenticated Azure CLI. The RBAC Administrator requirement is not incidental. Several labs create role assignments as part of their Bicep deployment, and a subscription where you cannot grant roles will fail partway through provisioning.

The documented setup sequence is short. Clone the repository, change into it, then run uv sync followed by uv pip install -r pyproject.toml, open VS Code with code ., and select the .venv interpreter created by uv sync as the Jupyter kernel. The README notes that you can skip local setup entirely with GitHub Codespaces, which is the faster path if you only want to read a notebook's output before committing to a subscription.

One rough edge in the documented commands: uv sync already installs the project's dependencies from pyproject.toml, so the follow-up uv pip install -r pyproject.toml looks redundant. It is harmless, but a reader following the Quick Start literally will run two installs where one should do. If you hit a resolution problem, check whether the second command is the cause before debugging your environment further.

Beyond the labs, the tools/ folder holds smaller notebooks for tracing, streaming, a rate limit tester, an OAuth client, and a mock server that the README describes as an OpenAI API mock for local development and testing. The mock server is the most useful of these for anyone who wants to exercise a policy without spending on model calls.

The Copilot Agent Skills, and where the material thins out

The repository ships six Copilot Agent Skills intended to help you author new labs: lab-creator for scaffolding notebooks with Bicep and policies, apim-bicep and apim-terraform for generating infrastructure, apim-policies for APIM XML, apim-kql for KQL queries that the README says control models, tools and agents, and mcp-builder for MCP servers. The README's example section for creating a new lab is cut off in the supplied text, so the exact invocation is not something I can state.

That truncation matters less than a broader observation about the repository's shape. There are no releases listed. For a sample repository that is normal, but it means there is no changelog to consult and no version pin to depend on. If you build internal labs by forking this one, you inherit whatever the main branch looked like on the day you forked, and you own the divergence from then on. The last push recorded is 2026-09-10, so the material is being maintained, but maintenance here means the samples keep working, not that an API surface stays stable.

The licensing is MIT, which is permissive and allows reuse inside commercial work. A caveat that is not legal advice: the notebooks deploy Azure resources and reference Microsoft Foundry and Azure API Management services, so the MIT grant covers the repository contents, not the terms under which you consume those Azure services. Check the service terms separately.

Where a notebook lab is the wrong tool

The clearest limitation is that this is not a deployable artifact. There is no package to add to a requirements file, no container image, no Helm chart. A team looking for an LLM proxy they can run next to their application will not find it here. What they will find is a set of policy examples they must transplant into their own APIM instance and then maintain.

The second limitation is the subscription requirement. Every lab is designed to be deployed to a live Azure subscription with Contributor and RBAC Administrator roles. There is no documented offline mode for the labs themselves, and the mock server in tools/ mocks the OpenAI API rather than APIM, so it does not remove the need for a gateway instance if you want to test gateway behaviour.

The third is the notebook format itself. Because each lab is a notebook with step-by-step instructions, the state of your APIM instance between cells is real and persistent. Re-running a provisioning cell after a partial failure can leave resources in an inconsistent state, and the notebooks do not appear to be idempotent by design. If you are the sort of team that wants a lab reproducible from a single command in CI, you will end up extracting the Bicep and policies out of the notebooks and running them yourself. That is a reasonable thing to do, but it is work the repository does not do for you.

How this differs from running LiteLLM as your model proxy

The obvious alternative for someone who wants a gateway in front of multiple models is LiteLLM, a self-hosted Python proxy that normalises provider APIs behind an OpenAI-compatible surface and offers its own routing, budget and key-management features. The difference in approach is architectural rather than feature-by-feature.

LiteLLM is a process you run. You install it, point it at provider keys, and it becomes the thing your application talks to. AI-Gateway is not a process; it is a set of instructions for configuring Azure API Management, which is a managed service you pay for and operate through Azure tooling. Choosing between them is largely a question of where you want the gateway to live. If your workloads already sit in Azure and your organisation has APIM, the policy-based approach keeps authentication, logging and quota enforcement inside a system your platform team already governs. If you want a single binary you can run in a container anywhere, with no Azure dependency, LiteLLM is the closer fit and the AI-Gateway labs will not help you.

A second alternative worth naming is writing the policy yourself against the APIM documentation without using this repository. That is viable if you only need one capability, say token rate limiting, and you are comfortable reading the APIM reference directly. The labs earn their place when you want several capabilities and want to see how they compose, or when you want to hand a working example to an engineer who has not used APIM policies before.

Maintenance cost and what to check before you commit

The upgrade cost here is not a dependency bump. It is the cost of keeping your APIM configuration aligned with a moving repository while your own policies diverge. Because there are no releases, you cannot diff version to version; you compare commits. If you fork the repository and adapt the labs, budget for reading the upstream changes to the Bicep templates and policy XML rather than expecting a migration guide.

The MIT licence removes most friction on the code itself. The practical constraint is elsewhere: the labs assume Azure API Management, and APIM capabilities vary by tier. The README links to the APIM genai-gateway-capabilities documentation, and that page is where you should confirm that the policy a given lab relies on is available in the tier you run. Semantic caching and token rate limiting in particular are the kind of features that tend to be tier-sensitive.

A sensible first move is to pick one lab that maps to a problem you already have, open its Bicep template and policy files before opening the notebook, and read them end to end. If the policy XML is legible to you, the rest of the repository will be. If it is not, the labs will teach you APIM policy syntax at the same time as they teach you AI gateway patterns, which is a longer project than the Quick Start implies.

Editorial conclusion

Adopt it if you already operate Azure API Management and need a working reference for token rate limiting, backend pool load balancing or MCP credential handling before writing your own policy XML. Do not adopt it if you want a runtime component: nothing here ships as a package you import, and the notebooks assume Contributor plus RBAC Administrator on a real subscription. Verify first that your APIM tier supports the specific policy a lab exercises, and open the lab's Bicep template before running the notebook, because that template is where the actual gateway configuration lives.

Official sources

1. Azure-Samples/AI-Gateway on GitHub
2. Issues
3. License: MIT
4. Project website
5. README