Mooncake: A KVCache-Centric Disaggregated Architecture for LLM Serving

What Mooncake Solves and Who It Is For

Mooncake addresses a specific bottleneck in large-scale LLM serving: the memory and bandwidth cost of KV caches. In standard serving, each request's KV cache is tied to a single GPU, which limits batch size and creates inefficiencies when prefill and decode phases have different resource profiles. Mooncake decouples these phases by making the KV cache a first-class, distributed resource. The project is the serving platform for Kimi, Moonshot AI's LLM service, and the README claims that under real workloads this architecture lets Kimi handle 75% more requests while still meeting SLOs. That number comes from the project itself; you should treat it as a vendor-reported figure, not an independent benchmark. The intended audience is engineers running large-scale inference clusters, particularly those already using or planning disaggregated prefill/decode. If you operate a single node or a small cluster, the complexity of a distributed KV cache pool is likely overkill.

The KVCache-Centric Architecture and Transfer Engine

The core idea is to treat the KV cache as a shared, distributed store rather than a per-GPU private resource. Mooncake comprises a store component and a transfer engine. The store manages KV cache blocks across a pool of nodes, enabling cross-instance sharing and reuse. The transfer engine handles moving data between nodes, with support for RDMA, which is key for low-latency, high-bandwidth transfer. The README highlights integrations where this matters: vLLM's Mooncake Store connector, SGLang's hierarchical KV caching backend, and TensorRT-LLM's cache transmission utilities. The mechanism is visible in the integration points: a connector in the inference engine sends or receives KV cache data via the transfer engine, while the store acts as a central coordinator. This is not a simple sharding scheme; it is a disaggregated design where the cache is a separate tier from compute. The architecture is described in a FAST25 paper, which is a good source for the detailed data flow, but the README alone gives enough to understand the high-level pattern.

Getting It Running: Installation and Configuration

The README does not provide a step-by-step quickstart, but it does point to several concrete artifacts. The transfer engine is available on PyPI as mooncake-transfer-engine, with variants for different hardware: mooncake-transfer-engine-cuda13, non-cuda, npu, musa, efa, and rocm. That suggests installation is as simple as pip installing the appropriate package for your accelerator. There is also a Docker image at kvcacheai/mooncake. The documentation lives at https://kvcache-ai.github.io/Mooncake/, which is the authoritative source for configuration. The README mentions integration with vLLM via a MooncakeStoreConnector and a MooncakeTransferEngineConnector, and with SGLang via a hierarchical KV caching backend. To run Mooncake, you would likely start the store service, then configure your inference engine to use the connector, specifying the store's address and the transfer engine's RDMA settings. The exact config keys are not in the README, so you must consult the docs. The existence of multiple PyPI packages indicates that hardware support is a first-class concern, not an afterthought.

Integrations and Ecosystem Reach

Mooncake's value is amplified by its integrations. The README lists vLLM, SGLang, TensorRT-LLM, and even PyTorch Ecosystem membership. vLLM officially features Mooncake Store, and SGLang supports it as a hierarchical caching backend. These integrations mean you do not have to build your own connector; you can use existing hooks. For example, vLLM has a Mooncake connector for PD-disaggregated setups, and SGLang uses Mooncake for EPD (Encode-Prefill-Decode) disaggregation. The transfer engine is also used for weight transfer in distributed RL, as seen in SGLang's RDMA-based P2P update, which the README claims achieves 7x faster weight updates for a 1T-parameter model. That claim is from a blog post, not a peer-reviewed paper, but it shows the engine's versatility. The breadth of integrations is a strong signal that the project is not a silo; it is designed to plug into existing serving stacks. However, each integration is a separate code path, and you must verify which version of Mooncake works with your engine version.

Limitations and Failure Modes

Mooncake is not a drop-in solution. It requires a disaggregated serving setup, which is a significant architectural change. If your inference engine does not support a Mooncake connector, you cannot use it without custom development. The README does not mention any fallback for non-RDMA networks; the transfer engine's performance depends on RDMA, so clusters without RDMA (like InfiniBand or RoCE) will likely see degraded performance. The project also introduces new operational components: a store and a transfer engine, both of which can fail or become bottlenecks. The README does not discuss failure handling or recovery, so you should assume that a store outage could impact all connected inference nodes. Another limitation is the hardware-specific PyPI packages: if you use a niche accelerator, you may find that the non-CUDA variants have less testing or slower updates. The README also notes a 'high-performance version' of the P2P store is open-sourced separately as checkpoint-engine, which suggests the main Mooncake store may have performance trade-offs for certain workloads.

Alternatives and How They Differ

The main alternative is to keep KV caches local to each GPU, as in standard vLLM or SGLang deployments without Mooncake. That approach is simpler but limits cross-request reuse and makes disaggregated prefill/decode harder. Another alternative is FlexKV, a distributed KV store from Tencent and NVIDIA that, according to the README, now supports distributed KV cache reuse with the Mooncake Transfer Engine. FlexKV is a different store implementation but can use Mooncake's transfer engine, so it is not a pure competitor but a complementary option. If you are looking for a non-RDMA approach, you might consider a simpler cache pool like Redis-based caching, but that lacks the zero-copy RDMA transfer that Mooncake provides. The key difference is Mooncake's focus on disaggregation and RDMA; alternatives that do not use RDMA will have higher latency and lower bandwidth for cache transfer. The README also mentions RBG + SGLang HiCache + Mooncake as a cloud-native solution, which is a more elastic deployment model, but it is an integration, not a standalone alternative.

Maintenance and Upgrade Cost

The project is actively maintained, with the latest release v0.3.13 pushed on 2026-08-26, and a history of point releases. The README shows a steady stream of integrations and updates, indicating a fast-moving project. That is a double-edged sword: you get new features, but you also face frequent changes that can break your setup. The PyPI packages are versioned, so you must pin versions to ensure compatibility with your inference engine. The license is Apache-2.0, which is permissive and allows commercial use, but you should review the full license text for any patent or attribution clauses. The project also has a Slack channel for community support, which is useful but not a substitute for documentation. The maintenance cost is non-trivial: you need to track upstream changes, test new releases, and manage the store and transfer engine as part of your infrastructure. If you adopt Mooncake, budget for ongoing operational work, not just initial integration.

Editorial conclusion

Adopt Mooncake if you run large-scale LLM serving with disaggregated prefill/decode or need high-throughput KV cache sharing across nodes, especially with RDMA-capable hardware. Skip it if your workload is single-node or you lack the infrastructure to manage a distributed cache pool. Before adopting, verify the specific integration path for your inference engine (vLLM, SGLang, TensorRT-LLM) and confirm the transfer engine supports your accelerator type (CUDA, NPU, MUSA, ROCm, EFA). Also check the license implications under Apache-2.0 and the operational overhead of running a dedicated store. Mooncake is proven in production at Moonshot AI, but its complexity is justified only when disaggregation is already on your roadmap.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes