Tessera is a whole model stack written to one goal, and its tests check properties rather than outputs

One goal, end to end, and no framework underneath

The framing is a single sentence: distil a large teacher into a small student, then serve that student efficiently. Keeping that goal end to end is why the repository touches custom GPU kernels, sharded training, an inference engine, quantization and a serving front end, and the readme is explicit that none of it is a toy.

What makes that possible to read is a fallback design that is stated in the second paragraph. It runs and is unit-tested on a laptop, on a CPU or on the Apple graphics runtime. The accelerated kernels are written for one vendor's hardware, and on anything else the model transparently falls back to an unfused reference path. The kernels are then checked against that reference whenever a graphics card is available.

That single decision is what turns this from an impressive-looking repository into something you can actually run. Most from-scratch kernel projects require the hardware you do not have in order to execute a single line. Here the fallback is not a degraded mode, it is the reference implementation the fast path is measured against, which means the same tests cover both.

The architecture is one teacher, one student, one engine. The teacher is described as being several times larger than the student, training runs through a fully sharded data parallel setup at the third stage, and the student comes out quantized in one of three formats into an engine built around three techniques that are all listed in the same diagram.

The tests are the argument, and they check invariants

The readme makes a point of this: the repository is small enough to test thoroughly, and most tests check a property rather than a fixed output. That is the correct instinct for a numerical systems project, where a golden output tells you the number did not change but not that the number is right.

Six properties are listed. Incremental decoding with the cache produces the same result as a full forward pass, which is the invariant everything in the serving layer depends on. Each accelerated kernel matches its unfused reference within floating point tolerance. The independently written reference implementation matches the main one to a stated tolerance. The sharded optimizer matches the single-process one step for step, both in one process and across two ranks of the collective backend, which is the check that distinguishes a real sharding implementation from one that only works when you are not actually sharded.

Two of the remaining three are about failure rather than correctness, and they are the ones that matter in production. Self-speculation reproduces greedy decoding exactly, which is the property that makes speculative decoding safe to enable. And the engine drains every request under tight memory and preemption without leaking cache blocks.

That last one is the check you want most and most projects do not have. A memory leak in a paged allocator under preemption does not show up in a functional test; it shows up as a server that slowly runs out of memory over a week. Writing that test is a statement about what the author was worried about.

Sharding written from scratch, and verified against the thing it replaces

The training side is a decoder transformer with the standard modern choices: a root-mean-square normalization, rotary position embeddings, grouped-query attention, and a gated feed-forward. Nothing exotic, which is correct; this project is about the machinery around the model, not about the model.

The distillation losses are where the interesting choices are, and there are three of them. A temperature-scaled divergence handles the soft distribution, which is the part of distillation that actually transfers knowledge. An optional hard cross-entropy term against the true labels is included, which is a good instinct: it lets you weight how much of the signal is the teacher's distribution versus the data. And hidden-state matching, which aligns intermediate representations rather than only output distributions.

The fully sharded training implementation is the part that would make most readers stop. It is written from scratch rather than imported, using flat-parameter sharding with a sharded optimizer, and the reason is testability: because it is written here, it can be compared step for step against the single-process optimizer, in one process and across two ranks. A sharding implementation inherited from a library can only be tested for plausibility.

Checkpoints are atomic and sharded with resume-from-latest. Atomic is the operative word: a checkpoint write that is interrupted halfway is a training run you lose, and the alternative is a checkpoint directory that silently contains a mixture of two states.

Kernels in two languages, always checked against something

The kernel directory is split in two, and the split is informative. One half is written in a higher-level kernel language with an autotuner; the other is raw compiled code for the low-level memory work, with profiling annotations and notes about how to look at them.

The attention kernel implements online softmax, which is the trick that lets a flash-attention style kernel avoid materialising the full score matrix, plus causal masking, grouped-query attention support, and autotuned tile sizes. Alongside it are a fused normalization kernel, a fused gated feed-forward matrix multiply, and an eight-bit weight-only matrix multiply that dequantizes inside the loop rather than materializing a full-precision copy first.

That last one is the detail worth pausing on. The naive way to do low-bit inference is to dequantize the weights into a temporary full-precision buffer and then call a normal matrix multiply. Doing the dequantization inside the loop avoids the temporary entirely, which is the difference between a kernel that fits in memory and one that does not.

The readme is equally clear about what is missing: there is no fused backward kernel for attention, and no fused paged-attention kernel for the decode path. Both are marked in the code rather than mentioned in passing, which is the honest way to do it. An inference engine that only accelerates the prefill pass is still a useful thing, but it is not the thing the readme's own list of serving features implies.

Three serving techniques that all interact badly if you get them wrong

The engine has three parts, and they are listed as a chain in the diagram because each one exists to serve the others.

The first is a block-paged key-value cache with a reference-counted allocator that enables prefix sharing. Paging is what lets a request's cache occupy arbitrary blocks instead of requiring a contiguous allocation, which is what stops one long request from fragmenting everything else. The reference counting is what makes prefix sharing safe, because a shared block cannot be freed while anything still points at it.

The second is a continuous-batching scheduler that recomposes the batch every step, with admission control and preemption under memory pressure. Recomposing every step is the difference between a system that can accept a request mid-generation and one that can only add work between requests. Preemption means a request can be paused and its blocks reclaimed under pressure, which is what makes the tight-memory test above pass.

The third is speculative decoding with the standard accept-and-reject scheme: draft several tokens cheaply, verify them against the full model, keep the prefix that matches. The self-drafting benchmark reports acceptance in the high nineties, which is unusually high and is a direct consequence of the verification test being exact.

Quantization is separate and comes in three flavors: eight-bit weight-only, a hardware-aware method, and an eight-bit floating point format. The first is the practical one for a laptop, the last needs hardware this project explicitly says it has not yet written a kernel for.

A Rust gateway and a second implementation in another framework

The serving front end is written in Rust with an async runtime and a web framework, and it calls into the Python engine through the standard foreign-function interface rather than over a network. That choice matters: an in-process call has microsecond overhead where an HTTP call has milliseconds, and at the edge of a decode loop the difference is the difference between the gateway being free and being visible in the profile.

The gateway's stated responsibilities are HTTP handling and admission back-pressure. The second one is the interesting half, and it is a correct division of labour: the Python engine decides what fits in memory, and the Rust layer refuses work before it queues rather than after it starts.

The second implementation is more unusual. There is a forward pass reimplemented in a completely different framework, compiled through its own compiler stack, and its only stated purpose is to act as an independent parity check against the main implementation. A framework with a different compiler and different numerics will disagree with your implementation in ways your implementation will never disagree with itself, which makes it a genuinely useful test rather than a redundant one.

Around that sit the interpretability helpers: activation hooks, a logit lens, and detection of a specific class of attention head that does induction. The last one is a real finding from the mechanistic interpretability literature, and having a detector for it in the repository means you can check whether a trained student learned the mechanism rather than only the behaviour.

The benchmarks are on a laptop, and the readme says so first

Every number in the benchmark table comes from an Apple laptop running the unfused reference path, and the readme says that in the sentence before the table rather than in a footnote. It calls the numbers a floor rather than what the kernels do on real hardware, and tells you to run the marked test and the benchmark scripts on a graphics card for numbers that matter.

That framing is what you want from a benchmark table. The figures are a forward pass in the hundred-thousand-tokens-per-second range on a tiny model with a short sequence, a measured attention throughput, a decode rate for six concurrent requests, and a speculative acceptance rate near certainty. None of those are impressive in isolation and all of them are honest.

The install path is short, and the whole sequence fits in one block:

git clone https://github.com/zengxiao-he/tessera && cd tessera
python -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu   # or a CUDA build
pip install -e ".[dev]"

pytest -m "not gpu"          # CPU tests; the kernel tests skip without a GPU
tessera info                 # list presets and parameter counts
python examples/serve.py     # continuous batching + speculative decoding
python examples/train_distill.py --steps 30
python examples/interp_demo.py

Four example scripts follow: one serving, one distilling for a handful of steps, one interpretability, and a command that lists the available presets with their parameter counts, which is the fastest way to see what is in the box. There is also a gateway test path: run the Rust test suite, run the server, and send one request to a generate endpoint with a prompt and a token budget.

The layout section is worth a glance too, because a repository this size benefits from a map, and the directory structure is exactly the dependency structure with the gateway and the reference implementation as siblings rather than as layers:

tessera/            model, kernels, quant, serve, distill, interp, data
  kernels/triton/   Triton kernels      kernels/cuda/  raw CUDA C++
  serve/            paged KV, scheduler, speculative, engine
  distill/          KD losses, FSDP, checkpoint, trainer
tessera-rs/         Rust tokio/axum gateway + PyO3 bindings
jax_ref/            JAX/XLA reference
tests/  examples/  benchmarks/  docs/

Editorial conclusion

What this project is actually for is reading. Almost every piece of a production inference stack exists here in a form small enough to hold in your head, and every component has a reference implementation next to it that it is checked against, which is the discipline that makes such a project trustworthy rather than merely impressive. Two honest limits, both stated in the readme. The benchmark numbers come from a laptop running the unfused reference path, so they are a floor and not a claim about the kernels. And the backward pass, the fused decode kernel and the tensor-core path for the newer hardware are listed as not done. Read it as a working reference implementation, which is a more useful thing than a half-finished product would be.

Frequently asked questions

Official sources

1. Issues
2. README
3. zengxiao-he/tessera on GitHub