AI System Design: a map of the LLM stack for engineers

What the guide is, and what the repository contains

AI System Design is described as a complete guide to learn AI System Design step by step, from LLM inference, GPUs, KV Cache and caching through to RAG, vector databases, AI agents, MCP, multi-agent systems, voice AI, guardrails, evaluation, observability and cost optimization, together with a step-by-step framework for handling an AI system design interview. Everything is presented in one place, explained in simple words, with detailed write-ups for each deep dive.

The repository itself is minimal. The top level has a .gitattributes file, the LICENSE, the README and an assets folder, and the primary language is Markdown. So the deliverable is the document, and the assets are images referenced from it.

The audience list is long and specific, and it reads as a career map rather than a marketing line. The guide is aimed at people who want to become an AI Engineer, a Gen AI Engineer, an LLM Engineer, an Agentic AI Engineer, an AI Agent Engineer, a Forward Deployed Engineer, an AI Solutions Architect, an AI Platform Engineer, an Applied AI Engineer, a Machine Learning Engineer, an MLOps Engineer, an LLMOps Engineer, or a Backend Engineer building AI products.

The document is prepared and maintained by the founder of Outcome School, Amit Shekhar, who teaches the AI and Machine Learning program there. The project is Apache-2.0 licensed, the default branch is main, and the last push was on 2026-10-19 with no GitHub releases.

There is no code to run and nothing to install, and the guide says so itself in the sense that it describes itself as a guide to learn from rather than a system to deploy. That makes the trade explicit from the start: you get a map of the territory and you still have to walk it.

The inference layer, where most AI design conversations start

The first block of the table of contents is the hardware and serving layer, and its ordering is a design argument in itself. It opens with an LLM recap and the big picture, then an inference server section with a subsection on choosing an inference engine, and then a hardware section covering GPU, TPU and LPU.

The metrics section is where the guide becomes concrete about what you actually need to measure. There are separate subsections for time to first token, tokens per second, throughput and cost per token. Treating those four as distinct quantities rather than one speed number is the point, because they trade against each other in ways that affect user experience and bill separately.

Then there is a section on prefill and decode as the two phases of LLM inference, with subsections on chunked prefill and prefill-decode disaggregation. That is the most operationally specific content in the list, because prefill is compute-bound and decode is memory-bound, and a design that treats them as one phase is making a scheduling decision by accident.

The KV cache appears again later under caching, alongside KV cache compression, prompt cache, semantic cache and embedding cache. So the guide treats caching in two distinct places, the mechanism that lives inside the inference server and the application-level caches that live in front of it, which is a reasonable way to organise the subject and a hint at the level of detail behind each heading.

The overall shape here is the one an interviewer probes. Given a latency target and a traffic profile, what do you run on, and what do you measure?

Scaling, estimation and load balancing

The scaling section follows the serving section, and it is organised around the two axes and the two mechanisms that matter. There is vertical scaling and horizontal scaling, then tensor parallelism and pipeline parallelism, and a subsection on how to choose between them.

After that comes auto scaling for AI and a section on back-of-the-envelope estimation for AI, with three subsections that tell you what a candidate is expected to be able to do: token estimation, GPU estimation and cost estimation. Estimation is a separate skill from scaling, and putting it in the guide as its own section is a reasonable editorial choice, since a design interview often starts with a back-of-the-envelope question and a candidate who cannot estimate is stuck.

Load balancing for LLM servers has its own section with three approaches. Least outstanding tokens treats the queue rather than the request as the unit of balance. Prefix-aware routing sends requests to the server that already holds the relevant prefix, which saves prefill work. Sticky sessions keep a conversation on one server so its cached prefix stays valid.

Those three are ordered in a way that suggests a preference, and they are also the three places where the KV cache leaks into the architecture, since a balanced server has a smaller cache hit rate than a sticky one.

The guide also has a section on LLM routing, meaning model routing, which is a different layer again: choosing which model answers a given request rather than which server serves it. A design that treats model choice and server placement as one decision is missing a degree of freedom.

RAG, and the many ways it can be wrong

The retrieval section is the longest in the table of contents, and its subsections read like a catalogue of RAG variants rather than a single approach. There is a vector database section with a subsection on vector index types, and then the RAG section itself with subsections on document parsing and ingestion, chunking strategies, hybrid search, query transformation with HyDE, reranking, ColBERT and late interaction, agentic RAG, GraphRAG and vectorless RAG.

Listing agentic RAG, GraphRAG and vectorless RAG side by side is the useful part. Those are three different answers to the same question about where the retrieval logic lives and whether it needs a vector index at all, and a guide that names them in one place is telling you the choice is a design decision rather than a default.

The subsection ordering is also a sequence. Parsing and ingestion, then chunking, then search, then reranking, then the architectural variants. That is roughly the order of the problems, since a poor chunking decision cannot be rescued by a good reranker.

Reranking and ColBERT get their own entries rather than being folded into retrieval, which suggests they are treated as the most effective addition to a basic pipeline. A vector database section with a subsection on index types follows later in the list, immediately before RAG, and that ordering suggests the guide treats the index as a component you pick rather than a foundation you build on.

The context window management section sits next, with subsections on context rot and lost in the middle and RoPE decay, context engineering, truncation, summarization, sliding window, compaction and hierarchical memory. That is a rare set of headings, and the lost-in-the-middle framing in particular is the kind of thing that changes how you write a prompt.

Agents, tool calling and where MCP fits

The agent section starts with the five core parts, then covers how an AI agent works end to end, the types of AI agents, computer use and browser agents, the common failure modes, and a subsection contrasting AI orchestration with AI agents. There is then a subsection on loop engineering and another on graph engineering, which is the clearest statement in the guide that agent reliability is a control flow problem.

The failure modes subsection is the one to read first in that list, because a guide that enumerates failure modes is telling you the design work is about preventing specific behaviours rather than about making the model smarter.

Tool calling gets its own section, and the Model Context Protocol gets a section with three subsections: how MCP works, why MCP matters for AI system design, and when to use MCP. There is also a subsection on when MCP is overkill.

That last one is unusual and worth noticing. A section on when a protocol is not needed is a signal of engineering maturity rather than enthusiasm, and it is paired with a separate section on agent skills, which are the lighter alternative to a full protocol server.

Around the agent sections there are the operational topics that a deployed agent needs: structured output, memory for AI agents, and, earlier in the list, the AI gateway, rate limiting in AI with subsections on tokens per minute, requests per minute, concurrent requests and cost-based rate limiting, async processing for long AI tasks, message queues in AI systems, and streaming responses with server-sent events and WebSockets.

Putting the gateway and the rate limiter before the agents in the ordering is a deliberate choice. It is the ordering of a service that has to survive its own traffic, which is a different list from the ordering of a tutorial.

Where the guide is useful, and where it is not

The strength of this resource is coverage and ordering. The table of contents is a map of a full stack, from the inference server up to the agent loop, and it is arranged so that each layer appears where a reader meets it in practice. The subsystem choices are named rather than glossed over, with vertical against horizontal, tensor against pipeline, three load balancing strategies, and three named RAG variants, and the section on when MCP is overkill suggests the author has had to argue that point in a review.

The weakness is that it is a reading resource with no code, no benchmarks and no numbers to check. A repository containing a README and an assets folder cannot be run, and you cannot verify a claim in it against an experiment. For a design interview that is a feature, since you are expected to reason rather than to quote. For production work it is a starting point for your own numbers.

The maintenance model is the other thing to weigh. The guide carries a note saying that AI System Design is moving very fast, that it will continue to grow as more blogs are written on new topics, and that you should bookmark it and come back for a refresher. That is honest, and it also means there is no versioned copy to cite, since the repository publishes no releases and the document is one file that keeps changing.

The alternative worth comparing is learning the layer from its own source. Each tool in the stack, the inference engine, the vector database, the agent framework, the protocol, has its own documentation and its own design documents, and for a specific implementation question those are better than any survey. The guide earns its place in the gap between them, where the question is which layer you need and what goes in it, and in interviews where you have to hold the whole shape in your head at once.

Editorial conclusion

Adopt this guide as a reading list if you are preparing for AI system design interviews or joining a team that is building on LLM inference and retrieval, since the table of contents is unusually complete on the parts people skip, and the honest framing that the field moves fast makes it a refresher rather than a manual. Keep away if you need runnable code, a library or benchmarks you can cite, because the repository holds a README and an assets folder and nothing else. Before you rely on any specific section, check the maintenance note, which says the guide will keep growing as new blogs are written, and note the last push was on 2026-10-19 with no tagged releases to pin an older version to.

Frequently asked questions

Official sources

1. amitshekhariitbhu/ai-system-design on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README