tt-metal: Tenstorrent's Two-Layer Stack for Wormhole and Blackhole AI Chips

The Problem: No Software, No AI Accelerator

Tenstorrent sells AI accelerators, the Wormhole and Blackhole boards, but hardware without a usable software stack is just a paperweight. tt-metal is that stack. It solves the problem of actually getting neural networks to run on these chips, from high-level model definitions down to hand-written kernels. The target audience is twofold: ML engineers who want to deploy models like Llama or Mixtral without writing low-level code, and systems programmers who need to squeeze performance out of the hardware by writing custom kernels. The README makes this split explicit, presenting TT-NN as the Python and C++ operator library and TT-Metalium as the low-level programming model. If you are not using Tenstorrent hardware, this project has nothing for you. It is a hardware-specific software repository, not a portable framework.

Two Layers, One Repository: TT-NN and TT-Metalium

The repository is organized around two distinct but related layers. TT-NN is the higher-level interface, a library of neural network operators that you call from Python or C++. It is what you use when you want to build or port a model without thinking about the underlying hardware. TT-Metalium sits below it, exposing a kernel programming model for direct control of the Tensix cores. The README describes TT-Metalium as a 'low-level programming model, enabling kernel development for Tenstorrent hardware.' The two layers share the same repository, which is convenient for development but also means that changes to the low-level API can ripple up into the operator library. The project also includes a models directory with demos for specific architectures, which serves as both a validation suite and a starting point for users who want to adapt a model.

What the Hardware Demands: Parallelism and Data Formats

Tenstorrent's chips are many-core designs, and the software must expose that parallelism. The README's performance tables show tensor parallel (TP) and data parallel (DP) factors, with Llama 3.3 70B running at TP=32 on a Galaxy system. That means the model is split across 32 devices, and the software must handle the communication and synchronization. TT-Metalium's tech reports cover the matrix engine and data formats, which are the hardware-specific details that matter for performance. The data formats report, updated September 2024, explains how different numeric formats are stored and computed, which is critical for both accuracy and throughput. The matrix engine report, from the same month, describes the hardware unit that accelerates matrix multiplication. If you are writing kernels, you need to understand both. The documentation is there, but it is dense and assumes you are comfortable with the hardware architecture.

Getting Started: Installation and First Steps

The README points to an INSTALLING.md file for setup, which is the first thing you should read. There is no single command that gets you running; the installation is likely to involve building the project from source, given the rapid release cadence and the need to match hardware revisions. The repository links to simple kernel examples in the TT-Metalium documentation, which is the recommended starting point for newcomers. You will need a Tenstorrent device, either Wormhole or Blackhole, because the software talks directly to the hardware. The README also mentions a Developer Hub and a Discord server, which are practical resources for troubleshooting. Do not expect a pip install to just work. You are committing to a build process, and you should budget time for it.

Performance Numbers: What the README Claims

The README includes performance tables for several models. These are not independent benchmarks; they are metrics collected from the project's own model demos. The README is careful to note that results may vary when using other runtimes such as vLLM. For Llama 3.3 70B on a Galaxy system with batch 32, it reports a time to first token of 53 milliseconds and a throughput of 2268.8 tokens per second. Qwen 2.5 7B on an n300 with batch 32 shows 109 ms TTFT and 707.2 tokens per second. These numbers are useful as a sanity check, but you should treat them as vendor-reported. If you are planning a deployment, you need to reproduce these on your own hardware and workload. The README also defines the metrics clearly, distinguishing TTFT from inter-token latency and total throughput, which is helpful for comparing across models.

Limitations: Young Stack, Rapid Changes, Hardware Lock-In

The most obvious limitation is that this software only works on Tenstorrent hardware. There is no CPU fallback, no GPU port, no emulation layer that would let you develop without the chip. That is a deliberate design choice, but it means your entire development environment is tied to the hardware. The README also notes that Blackhole software optimization is under active development, which is a polite way of saying that Blackhole support is not as mature as Wormhole. The release cadence is aggressive, with multiple dev releases per week, which suggests the API is still shifting. If you build a model against one release, a later release may break it. The README's model tables reference specific TT-Metalium releases for each model, so you are expected to pin versions. That is a maintenance burden you must accept.

Alternatives: CUDA and ROCm, But Not for This Hardware

The natural alternatives to tt-metal are the software stacks for other accelerators: CUDA for NVIDIA GPUs and ROCm for AMD GPUs. Those stacks are more mature, have larger ecosystems, and support a wider range of models out of the box. But they do not run on Tenstorrent hardware. If you are comparing tt-metal to CUDA, you are not really comparing like for like, because the hardware is different. The only meaningful alternative within the Tenstorrent ecosystem is the vLLM Tenstorrent plugin, which the README mentions. That plugin uses tt-metal under the hood but provides a higher-level serving interface. So the choice is not tt-metal versus something else; it is tt-metal as the foundation, with optional layers like vLLM on top. If you are evaluating whether to buy Tenstorrent hardware, the real question is whether tt-metal's maturity is acceptable for your workload.

Maintenance and Licensing: Apache-2.0 and a Fast-Moving Target

The project is licensed under Apache-2.0, which is permissive and allows commercial use, modification, and distribution, with the usual conditions about retaining copyright notices. That is a favorable license for adoption. The maintenance picture is more complex. The repository is actively developed, with multiple releases per week, and the default branch is main. That means you are either tracking a moving target or pinning to a specific release and managing your own patches. The README shows that models are tied to specific releases, so upgrading the stack may require updating your model code. The project also participates in a bounty program for issues, which suggests an active community but also that the project relies on external contributions for some fixes. Before adopting, check the issue tracker and the MODEL_UPDATES.md file to see how quickly breaking changes are communicated.

Editorial conclusion

Adopt tt-metal if you own or plan to buy Tenstorrent Wormhole or Blackhole hardware and need to run modern LLMs like Llama 3.3 70B or Qwen 2.5 at scale. It is the only supported software path for that hardware, so if you are not committed to Tenstorrent silicon, it is the wrong tool; you would be better served by CUDA or ROCm stacks. Before adopting, verify that your specific model is in the Model Matrix, confirm your hardware revision (Wormhole vs. Blackhole) matches the release you plan to use, and check the latest release notes for known issues, as the project is under active development with frequent version bumps.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes