# openTPU: an AI-designed accelerator where RTL, simulator and compiler share one repo

> FeSens/openTPU puts a SystemVerilog accelerator, its instruction set, a bit-exact simulator, a Triton-style kernel language and the host software that drives a real PCIe card in a single monorepo. The measured decode rates are real but modest, and every claim rests on FPGA hardware you have to own.

**FeSens/openTPU** — An open-source AI accelerator, developed by AI: RTL, ISA, simulator, compiler and profiler in one repo. Runs Qwen3, LFM2.5 and Qwen3.5 on a Kintex-7 PCIe card.

- Repository: https://github.com/FeSens/openTPU
- Stars: 826 · Forks: 66
- Language: Verilog
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/fesens-opentpu

## RTL, ISA, simulator, compiler and host software in one monorepo

The repository layout is the argument. Directories named rtl/, sim/, asic/, boards/, targets/, tools/, docs/ and opentpu/ sit next to a Makefile and a pyproject.toml, which means the hardware design, the instruction set, a bit-exact simulator, a kernel language with its compiler and the host software all live under one version history.

The README calls openTPU an open-source AI accelerator developed by AI, and it frames the project as a study of how far agents can go at hardware design. openTPU brings the lessons of the earlier auto-arch-tournament repository to the problem of building a chip that can run the inference stack the agents built. For a reader, the practical value is that the layers line up: the simulator and the card can be compared directly because they implement the same ISA.

The Python side is packaged as opentpu, version 0.1.0, requiring Python 3.10 or newer with numpy and textual as dependencies. It also exposes a console script set for the host tools:

```toml
[project.scripts]
otpu-chat = "opentpu.host.chat:main"
otpu-smi = "opentpu.host.smi:main"
otpu-selftest = "opentpu.host.selftest:main"
otpu-setup = "opentpu.host.pcie_setup:main"
```

An optional extra called minicpmv adds pillow and torchvision for MiniCPM-V image handling, and pytest is configured to look in tests/ with the repository root on the path.

## The simulator and the card agree token for token

Correctness is the part the README is strictest about. The design runs ten modern models with their real weights on an Inspur YPCB-00338 card, a Xilinx Kintex-7 xc7k480t with two DDR3 channels, and the card produces the same tokens as the simulator, bit for bit. The claim is repeated per configuration: every configuration matches the simulator token for token, per position and with the resident decode program.

That equivalence is what makes the simulator useful rather than decorative. A simulator that approximates the hardware forces you to test on the card for every change, and a card is scarce. A bit-exact simulator lets most of the work happen in software, with the card reserved for confirmation.

The measurement method is documented rather than asserted. tools/qual/perf.py measures decode as 64 greedy tokens after a 512-token prompt, with the host's argmax in the loop rather than a streamed one, so device counts count only cycles where the accelerator runs and wall counts add the host. Prefill is the 512-token prompt, executed on the device, and DRAM traffic comes from the card's own counters while it runs. The README also notes that the Gemma 4 E2B base checkpoint matches Hugging Face's greedy tokens on three prompts.

## Decode on this card is bound by DRAM bandwidth

The performance table is the most useful thing in the README, and it tells a consistent story: every configuration sits near the bandwidth ceiling. LFM2.5-230M in int8 decodes at 59.0 tok/s on the device and 52.3 tok/s wall, with prefill at 295.6 tok/s and DRAM traffic of 14.5 GB/s, which is 85 percent of peak. Qwen3-0.6B in int8 decodes at 21.6 tok/s with 14.4 GB/s and 84 percent of peak. Moving to a 4-bit head lifts LFM2.5-230M to 85.8 tok/s and Qwen3-0.6B to 31.3 tok/s, mostly because the weights move less.

The larger models show the ceiling clearly. Phi-4-mini at 3.8B parameters decodes at 3.99 tok/s in int8 and 6.56 tok/s with a 4-bit head, and SmolLM3-3B reaches 5.00 tok/s in int8. In every one of those rows the DRAM figure sits at 16.0 GB/s and 94 percent of peak. The README states the cause plainly: decode is bound by DRAM.

The hardware behind those figures is modest. The production image runs at 133.33 MHz, uses LiteDRAM controllers calibrated by a small CPU inside the memory core, and has a four-column systolic matrix unit feeding the stream engine. DDR3-1066 on both channels gives a 17.1 GB/s peak, and the host is a machine called opentpu built around an Intel Core i7-4790. These are FPGA numbers on a consumer board, not a datacentre part.

## Agents pick the architecture, then the tournament measures it

The unusual part of openTPU is not the accelerator, it is how the accelerator was designed. The Makefile exposes an architecture tournament, and the comment block at the top of the file documents the interface:

```bash
#   make tourney COMP=otpu_coll N=1 K=1 [AGENT=claude|codex] [EVAL=yosys|vivado-remote]
#   make tourney-report [COMP=otpu_coll]
```

A run picks a component to compare, a number of candidate designs, how many rounds, which agent does the design work (claude or codex) and how each candidate is evaluated. The evaluation choice is the interesting one, because it is a real alternative in approach: yosys evaluates candidates with an open-source synthesis flow, while vivado-remote evaluates them with Vivado on a separate build host. Two further targets exist. The fmax tournament sweeps the whole-design frequency across named component families such as otpu_native_dram, otpu_tmem, otpu_mxu, otpu_seq, otpu_dma, otpu_xunit, otpu_vpu, otpu_quant, otpu_coll, otpu_actram and otpu_fp, while the units tournament runs each unit on its own without a full build.

The design philosophy visible in the Makefile comments is that the units on the measured worst path families come first, and that a champion is kept per component. The default is that lint and the test gates run remotely, with EXEC set to remote, and the local machine can be used instead. This is a hardware development loop where machine-written candidates compete on measured constraints.

## Host tools for the card: chat, telemetry, PCIe setup and memory calibration

Everything on the host side is a console script, and the set shows what the workflow actually requires. otpu-setup points at opentpu.host.pcie_setup, otpu-i2c at opentpu.host.i2c, otpu-memcal at opentpu.host.memcal and otpu-ddr4 at opentpu.host.ddr4, so enumerating the PCIe device, talking to it over I2C and calibrating memory are separate, explicit steps rather than something that happens automatically.

For interactive use there is otpu-chat and otpu-serve, which serves an API, plus otpu-smi for the status view of the card and otpu-lens for a lens tool. The README's demonstration image shows otpu-chat running LFM2.5-230M on the card while otpu-smi shows the card's utilization and DRAM bandwidth. Two more scripts handle process state and diagnostics: otpu-lock holds the device, and otpu-diag collects diagnostics.

The order of those steps is the practical part. A fresh board needs PCIe enumeration and setup, then memory calibration, then a self test, before a chat session or a served model makes sense. The README shows a calibration that runs when the image starts, with the core's CPU calibrating both DDR3 channels in 12 seconds and no host involvement, and it contrasts that with the previous image built with the Xilinx MIG, a two-column matrix unit and a 120.755 MHz clock. Decode stayed within 2.3 percent of that older image in every configuration, while prefill became 1.3x to 2.0x faster.

## Mixture-of-experts models run by streaming experts from host storage

The card has 4 GiB, which is not enough for a large mixture-of-experts model, so openTPU handles that case by streaming. The card routes each token and computes every expert, keeping the experts in per-layer slots in its own DRAM, and the host only copies missing experts from a pool file into those slots at the link's rate. The details live in docs/offload.md, section 10.

The measured results show how far this goes. LFM2.5-8B-A1B, with 8.5B parameters and 1.7B active, runs at 10.6 tok/s over 160 tokens, with 98.5 percent of expert uses hitting the resident slots and 5.2 MB streamed per token. Qwen3.5-35B-A3B, 34.7B parameters with 3.0B active, runs at 3.95 tok/s and matches Hugging Face's 16 greedy tokens, but 62 percent of expert uses miss the slots and 153 MB is streamed per token at 1.41 GB/s over PCIe.

That contrast is the real lesson of the offload path. When the active experts mostly stay resident, the design is fast and the PCIe link is quiet. When they do not, the host link becomes the bottleneck and the token rate falls accordingly. Whether a given model behaves well is a property of its routing pattern, not of the accelerator.

## What the Kintex-7 form factor cannot do

Several limits are visible in the project's own numbers. The board holds 4 GiB of memory, and the README states plainly that Gemma 4 E2B in int8 does not fit, which is why its per-layer embedding tables are kept on the card only in the 4-bit configurations. The Qwen3.5-4B int8 image is over 4 GiB, so that combination is out of reach as well.

The performance ceiling follows from the same hardware. At 133.33 MHz on DDR3-1066 with a 17.1 GB/s peak, every measured configuration lands between 82 and 94 percent of DRAM peak, so the matrix unit is rarely the limiting factor. A Phi-4-mini decode at 3.99 tok/s is a fair picture of what this class of board delivers for a mid-sized model.

There are project-level constraints too. The repository publishes no GitHub releases, so there is nothing to pin a deployment against beyond a commit, and the Python package is version 0.1.0. The licence is Apache-2.0, which permits use and redistribution including in commercial work, provided the notice and licence terms are carried along, and asic/ and boards/ directories point at the flows and board definitions that a serious port would start from.

One more limitation deserves attention for anyone planning to reproduce this. The numbers depend on the specific card, and the README separates them by date and build, from the first three models measured on 2026-09-29 with the production image deploy_champ_e698dcd7, through 2026-09-30 for LFM2-2.6B, SmolLM3-3B and Phi-4-mini, to 2026-10-01 for Qwen3.5 and Gemma 4 with build B, deploy_fused133c_79c5707a. Reading a row without checking which build produced it will mislead you.

## Conclusion

openTPU suits engineers who want to read an accelerator end to end, from a matmul in Python to the wires, and who already have a Kintex-7 PCIe card to test on. It does not suit teams looking for a production inference stack, since decode speed on the card is bounded by DRAM bandwidth, the design targets FPGA rather than an ASIC flow, and the repository publishes no releases to pin against. Before adopting, check what the Makefile tourney targets produce on your own machine and read docs/board.md section 5, where the earlier images and their numbers are kept.

## FAQ

### What hardware does openTPU need to run on?

The README describes an Inspur YPCB-00338 card with a Xilinx Kintex-7 xc7k480t and two DDR3 channels, sitting in a host machine named opentpu built around an Intel Core i7-4790. Host tools such as otpu-setup, otpu-i2c and otpu-memcal handle enumeration, I2C and memory calibration.

### How accurate is the openTPU simulator compared with the hardware?

The README states that every configuration matches the simulator token for token, per position and with the resident decode program, and that the card produces the same tokens as the simulator bit for bit. It also notes that the Gemma 4 E2B base checkpoint matches Hugging Face greedy tokens on three prompts.

### Which models does openTPU run?

The results table covers ten models with real weights, including LFM2.5-230M, Qwen3-0.6B, Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, Gemma 4 E2B, Gemma 4 E4B, LFM2-2.6B, SmolLM3-3B and Phi-4-mini at 3.8B, in int8 and in 4-bit weight variants.

### How is the hardware architecture of openTPU chosen?

Through an architecture tournament driven by the Makefile, where make tourney takes a component, a number of candidates and an agent (claude or codex), and an evaluation backend (yosys or vivado-remote). A separate tourney-fmax target sweeps whole-design frequency across the unit families, and tourney-report summarises a component.

### What licence is openTPU released under?

Apache-2.0, as stated in the LICENSE file and in the pyproject.toml project metadata. That is a statement about the licence terms rather than legal advice.

## Sources

- [FeSens/openTPU on GitHub](https://github.com/FeSens/openTPU)
- [Issues](https://github.com/FeSens/openTPU/issues)
- [License: Apache-2.0](https://github.com/FeSens/openTPU/blob/main/LICENSE)
- [README](https://github.com/FeSens/openTPU/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/fesens-opentpu