TileRT: a pinned-binary runtime for millisecond-scale token generation

The latency problem TileRT is aimed at, and who feels it

Most serving stacks optimize tokens per second across a batch. TileRT inverts that. The README states its goal is to push latency limits without reducing model size or quality, targeting millisecond-level time per output token for models with hundreds of billions of parameters. The named use cases are high-frequency trading, interactive AI, real-time decision-making, long-running agents, and AI-assisted coding. What those have in common is that one request's wall-clock time matters more than how many requests share a GPU. If your workload is offline summarization or a queue of batch jobs, the design premise does not apply to you, and the hardware demands below will look absurd. The project is explicitly positioned against throughput-optimized systems, which is a useful signal: this is a tool for a narrow latency band, not a general replacement for a serving framework. The release history supports the narrow framing. The first public release targeted a single model, DeepSeek-V3.2-Exp, and later releases added GLM-5 and GLM-5.1. Support is model-by-model, not universal.

Tile-level decomposition and how the runtime overlaps work

The mechanism described in the README is a tile-level runtime engine driven by a compiler. LLM operators are decomposed into fine-grained tile-level tasks, and the runtime reschedules computation, I/O, and communication across multiple devices with heavy overlap. The stated purpose is to minimize idle time and improve hardware utilization. That matters on a multi-GPU node because the failure mode of low-latency decoding is not raw arithmetic throughput, it is bubbles: a device waiting on a collective, a copy in flight while tensor cores sit idle. Reordering at tile granularity gives the scheduler more freedom to fill those gaps than a kernel-level or graph-level approach would. The README also says the underlying compiler techniques will be shared with the community as they land in TileLang and TileScale, which tells you the runtime is the productized surface of a compiler research effort rather than a standalone serving layer. For PD disaggregation, the split is specific: vLLM handles prefill and TileRT handles decode, exposed behind an OpenAI-compatible endpoint, with GLM-5/5.1 and DeepSeek-V3.2 named as supported. That is a pragmatic division, since prefill is compute-heavy and more tolerant of batching, while decode is the latency-sensitive phase the project is built around.

Getting it running: the environment table is the real documentation

TileRT v0.1.5 ships as a pre-built binary wheel, and the README is blunt that it is linked against the exact ABI of the listed versions. Other combinations of Python, CUDA, or PyTorch are described as untested and not guaranteed to work. The stated hard requirements are 8x NVIDIA B200, an NVIDIA driver supporting the CUDA 13.2 runtime, Linux x86_64 with glibc >= 2.28 (manylinux_2_28), Python 3.12, torch==2.11.0+cu130, transformers==4.46.3, and tokenizers==0.20.3. The README calls these hard requirements rather than lower bounds, which is an unusual and honest framing. The recommended path is the prebuilt image, mirrored to two registries: docker pull ghcr.io/tile-ai/tilert:cu132-latest or docker pull tileai/tilert:cu132-latest. You then launch a container with all 8 B200 GPUs attached. The supplied README is truncated mid-sentence at that launch step, so the exact docker run flags, the generation invocation, and the MTP configuration are not visible in the material available here. Treat any command beyond the two pulls as something to read from the repository itself, not from this article. The practical consequence is that installation is not a pip install into an existing environment; it is a container launch on a specific node class.

Multi-token prediction and what the acceptance-length numbers actually say

MTP arrived in v0.1.2-alpha.1, with the release note claiming decoding rates up to 590 tokens/s at mtp=3 under synthetic workloads. The caveat is in the phrase synthetic workloads, and it is worth taking seriously. The v0.1.5 chart caption is more informative: it compares TileRT without MTP, with MTP at an average acceptance length of 3.2, and the peak under best-case acceptance of 4.0, on GLM-5.1-FP8 over 8x B200 with output length 1K and input length 1K to 192K. The gap between the average and best-case bars is the whole story of speculative decoding. Acceptance length is a property of the draft model and the text being generated, not a knob you set once, so the average figure is the one that describes production and the peak is the one that describes a favorable prompt. If you evaluate TileRT on the peak bar you will be disappointed by real traffic. The mtp=3 setting appears in the release note but the README section covering MTP generation is outside the visible material, so the exact configuration key is not something this article can state. Plan to read it from the repository.

Where the pinned-wheel approach breaks down

The limitation is structural, not incidental. A wheel compiled against one ABI means the project cannot follow you onto a different GPU generation, a different CUDA major version, or a different Python minor version without a new build. The README's own warning about untested combinations is the honest version of this: the maintainers are not claiming broad compatibility, they are claiming one supported configuration. That has three consequences. First, upgrading PyTorch for an unrelated reason can break TileRT, because torch==2.11.0+cu130 is pinned. Second, a driver that does not provide the CUDA 13.2 runtime is a blocker you cannot work around in userspace. Third, the 8x B200 requirement excludes essentially every other accelerator, including the previous generation, so this is not a tool you evaluate on a spare A100. There is also a scope limitation: model support is enumerated per release. If your model is not DeepSeek-V3.2 or GLM-5/5.1, the material here gives no indication that it will run, and no adapter or conversion path is described. The correct conclusion for an unsupported model is that TileRT is the wrong tool, not that it needs configuration.

How this differs from a general-purpose serving stack

The natural comparison is vLLM used on its own. vLLM schedules continuous batches across requests, which raises aggregate throughput and is the reason it is the default choice for most deployments; individual request latency is a byproduct of queueing and batch composition rather than the objective. TileRT's README describes the opposite priority, responsiveness over high-throughput batch processing, and its runtime reschedules tile-level tasks across devices instead of batching whole requests. The two are not mutually exclusive in the v0.1.5 design: the PD disaggregation release explicitly pairs vLLM prefill with TileRT decode behind an OpenAI-compatible endpoint. That is the most interesting architectural fact in the release history, because it concedes that vLLM's prefill path is good enough and confines TileRT to the phase where its tile-level scheduling pays off. If you are choosing between them, the question is not which is faster but which phase dominates your latency budget. Long prompts with short outputs push work into prefill, where the disaggregated setup hands the work back to vLLM anyway.

Maintenance cost, licence, and what to check before committing

TileRT is MIT licensed, which permits commercial use and modification, but the licence does not give you the ability to rebuild the wheel; that depends on whether the compiler toolchain behind it is published, and the README says only that compiler techniques will be shared gradually as they land in TileLang and TileScale. Until then you are consuming binaries on the maintainers' release cadence, which the release list shows is roughly every one to two months, with a post-release (v0.1.5.post2) following v0.1.5 within days. Each upgrade means re-validating the pinned stack, because a new wheel may pin new versions of torch or transformers. Budget for that as recurring work, not a one-time setup. There is also a hardware cost that the README does not discuss: 8x B200 is an expensive node, and if your traffic does not need millisecond TPOT, the same hardware running a batching server will serve far more users. The honest framing is that TileRT converts hardware into latency, and that trade only makes sense when latency is the product. This is a description of the licence terms as stated in the repository metadata, not legal advice; if you plan to redistribute or embed the runtime, read the MIT text and the wheel's bundled third-party notices yourself.

Editorial conclusion

Adopt TileRT only if you already run 8x NVIDIA B200 on Linux x86_64 and can reproduce the pinned stack (Python 3.12, torch==2.11.0+cu130, transformers==4.46.3, tokenizers==0.20.3), because the wheel is linked against that exact ABI and other combinations are documented as untested. Do not adopt it for throughput-oriented batch serving, for non-B200 hardware, or for a host where you cannot control the Python and CUDA versions. Verify first that the model you need is actually supported (the release notes name DeepSeek-V3.2 and GLM-5/5.1), that your driver provides the CUDA 13.2 runtime, and that glibc is at least 2.28; then start from the published container image rather than installing the wheel on a general-purpose host.

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. tile-ai/TileRT on GitHub