TokenSpeed: A C++ Control Plane and Python Execution Plane for Agentic LLM Inference

What TokenSpeed Solves and Who It Is For

The engine's core components are a modeling layer, a scheduler, kernels, and an entrypoint. The modeling layer uses a local-SPMD design with a static compiler that generates collective communication from module-boundary placement annotations, so users do not hand-write parallelism logic. The scheduler is a C++ control plane with a Python execution plane, where request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine. The type system enforces safe KV resource reuse at compile time, which is a strong guarantee compared to runtime checks. Kernels are pluggable, with a portable public API and a centralized registry, including an MLA implementation for Blackwell. The entrypoint is an SMG-integrated AsyncLLM for low-overhead CPU-side request handling. This architecture is visible in the README, but the actual code layout is not described in detail.

How the Control Plane and Execution Plane Split Works

The key mechanism is the separation of control and execution. The control plane is implemented in C++ as a finite-state machine, which manages request lifecycles and KV cache state. The type system is used to enforce safe resource management at compile time, meaning that invalid state transitions or double frees are caught during compilation rather than at runtime. This is a significant departure from engines like vLLM, which handle scheduling in Python. The execution plane is in Python, allowing fast iteration for researchers and engineers. The README states that this lowers cognitive load while providing strong correctness guarantees in the core scheduling system. The finite-state machine approach means that the scheduler is deterministic and testable, but it also implies that any change to the scheduling logic requires recompiling the C++ code, which could slow experimentation.

Kernels as a Modular Subsystem

TokenSpeed treats kernels as a first-class modular subsystem, separate from the core engine. The public API is portable, and there is a centralized registry with a selection model. The README mentions an extensible plugin mechanism for heterogeneous accelerators. This is notable because most inference engines bake kernels into the core, making it hard to swap in custom implementations. The registry approach allows users to select different kernels per model or hardware. The README claims one of the fastest MLA implementations on Blackwell for agentic workloads, but no benchmark numbers are given in the README itself. The PyTorch blog link mentions 580 TPS on Qwen3.5-397B-A17B, but that is an external claim, not verified here. The modular kernel system is a genuine differentiator, but its practical benefit depends on the quality of the built-in kernels and the ease of writing new ones.

Getting TokenSpeed Running: Commands and Configuration

The README does not provide installation commands or a quick start code snippet. It links to a Getting Started guide and a Launching a Server guide, but the actual commands are not in the material. The documentation index is at https://lightseek.org/tokenspeed/, with guides for getting started, launching a server, model recipes, server parameters, compatible parameters, and parallelism. The server parameters page likely includes flags for model path, tensor parallelism, and other settings, but those are not reproduced in the README. The repository is in Python, so installation likely involves pip or building from source, but that is unconfirmed. The release v0.1.0 is the only release, dated 2026-07-24, so the project is at an early stage. To get it running, a user must follow the external docs, which are not part of the README. This is a limitation for evaluation: the core value proposition cannot be tested without visiting external pages.

A Real Limitation: Young Project and Sparse Documentation

TokenSpeed is at version 0.1.0 with a single release. The README is high-level and marketing-oriented, with links to blogs and news rather than technical documentation. There is no installation guide in the README, no API reference, and no example code. The claims of performance and usability are not backed by reproducible benchmarks in the repository. The architecture is complex: a C++ control plane and a Python execution plane require a build system and cross-language integration, which is a significant maintenance burden. For a production engine, the lack of a stable API and the need to compile C++ code could be a barrier. The project may be the wrong tool for teams that need a quick, well-documented deployment or that do not have C++ build expertise. The finite-state machine scheduler is a strong design, but it is only as good as its implementation, and there is no evidence of extensive testing or community adoption yet.

Alternative: vLLM and Its Different Approach

The main alternative is vLLM, which is a popular open source inference engine. vLLM uses a Python-based scheduler with continuous batching and PagedAttention for KV cache management. The key difference is that vLLM does not separate the control plane from the execution plane; both are in Python, with kernels in C++/CUDA. This makes vLLM easier to modify and debug, but it lacks the compile-time safety guarantees of TokenSpeed. vLLM is mature, with extensive documentation, a large community, and support for many models and hardware. TokenSpeed's approach is more ambitious: it moves the scheduler to C++ and uses the type system to enforce safety. This could yield better performance and correctness, but it comes at the cost of complexity and a less mature ecosystem. For teams that prioritize ease of adoption and broad support, vLLM is the safer choice. For teams that need the specific performance characteristics of TokenSpeed and are willing to invest in the learning curve, TokenSpeed may be worth evaluating.

Maintenance and Upgrade Cost, and License

TokenSpeed is licensed under MIT, which is permissive and allows commercial use, modification, and redistribution with attribution. This is a positive for adoption, as it imposes minimal legal restrictions. The maintenance cost is potentially high due to the dual-language architecture. The C++ control plane must be compiled and maintained separately from the Python execution plane, and any changes to the finite-state machine require C++ expertise. The project has a single release, so there is no history of upgrades to assess. The README mentions a blog and talks, but no contribution guidelines or development roadmap. The last push was in July 2026, but the repository is not archived, so it is actively maintained at least in terms of commits. The reliance on external blogs for technical details suggests that the core documentation is not yet comprehensive. Before adopting, users should verify that the project is actively maintained and that the C++ build is reproducible on their target systems.

Editorial conclusion

Adopt TokenSpeed if you run agentic workloads on NVIDIA Blackwell GPUs and need low-latency inference with compile-time safety for KV cache management. Do not adopt it if you need mature documentation, broad hardware support, or a stable API, since the project is at v0.1.0 with no installation instructions in the README. Before adopting, verify the actual performance claims on your hardware, check the docs for a working install command, and confirm that the C++ control plane builds on your system. TokenSpeed is a promising design, but it is not yet a drop-in replacement for vLLM.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes