APEX: an FPGA LLM tile where the KV cache is a hardware codec
An inference chip design that runs a real LLM (Qwen2.5-0.5B) on FPGA — one transformer decoder layer in RTL, every silicon value bit-exact against a golden model. 0.56 tok/s measured, a 140× climb, full evidence trail.
At a glance
- What is it?
- APEX is a verification-first transformer decoder layer in RTL, brought up on FPGA with Qwen2.5-0.5B at a measured 0.56 tok/s. The bet is architectural: compress the KV cache inside the datapath instead of treating it as a software problem.
- Who is it for?
- Adopt APEX if you are building an edge inference accelerator and want a decoder layer whose every block has a bit-exact NumPy reference, an FPGA bring-up on Qwen2.5-0.5B, and a KV codec already inside the datapath. Do not adopt it if you need a deployable inference stack: the charter excludes DRAM controller, PCIe and NoC, so there is no path from this tile to a working system without you writing the rest.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 28 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The two costs APEX picks a fight with
The README names two dominant costs at the edge: moving weights, and remembering context. The first is bandwidth. The second is the KV cache, which grows with every token and gets re-read on every new token, so a long conversation turns an inference engine into a memory-traffic engine. Most accelerators handle the KV cache in software: quantize it on the host, store it, hope. APEX's stated bet is different. It puts the KV codec inside the datapath, so keys and values are compressed the moment they are produced and decompressed the moment they are consumed, with an importance unit that watches which parts of the context matter and spends bits there. The intended audience is hardware engineers building edge inference accelerators, and researchers who want a decoder layer they can inspect block by block rather than a vendor IP block. The scope is stated plainly: one tile, no DRAM controller, no PCIe, no NoC, out of scope by charter.
One decode step, seven GEMMs, one systolic array
The architecture diagram in the README shows the data flow for a single new token. An activation vector x enters through a seam, passes RMSNorm, and hits the MXE, an INT8 systolic array that computes W_Q·x, W_K·x and W_V·x. RoPE is applied to Q and K, then the KVQ engine compresses K and V before they reach on-tile SRAM. The cache is stored as INT4 plus an outlier lane. On the read side, the MXE computes Q·K̂ᵀ, an online softmax runs in the ASU, and the MXE computes P·V̂ with dequantization happening in flight. A TIP importance unit feeds back into that path to adjust precision adaptively. The output projection W_O·attn is added to the residual to produce y, and the FFN branch repeats the pattern: RMSNorm, W_gate and W_up, SwiGLU, W_down, residual. The design point worth noting is that the MXE is time-multiplexed across all seven matrix jobs rather than replicated. That saves area and creates a scheduling problem, which is what the layer walker exists to solve: the README describes a sequencer choreographing the array, with weight streaming feeding it.
Bit-exact means bit-exact, and the repo enforces it
The methodological claim is the strongest thing here. Every RTL block in the tree is verified against an executable NumPy reference, and the README is explicit that this is not close and not within tolerance: bit-identical. Testbenches are mutation-tested, and evidence is machine-generated under what the README calls an anti-fabrication rule. That combination is unusual in open hardware. Mutation testing in particular changes the question from did the tests pass to would the tests have caught a wrong implementation, which is the question that matters when you are about to tape out or hand a block to someone else. The README also states that every number in the repository is labeled as measured or projected, and points to STATUS.md, a results index, and a master table under docs/design. If you are evaluating this project, that labeling discipline is the first thing to check, because it is the difference between a research artifact and a demo.
What was actually run, and on what
The README reports Qwen2.5-0.5B running end-to-end through the verified pipeline on real FPGA hardware, at a measured 0.56 tok/s. The same document describes that figure as a 140x climb, which is a claim about progress within the project rather than a comparison to any other platform. Treat it as a bring-up milestone, not a throughput result: half a token per second is not a serving number. Qwen2.5-7B tokens have run through the software-verified golden pipeline, but the README is careful to say not through silicon. The architecture is sized for 7B-class models, with head_dim = 128 present in RTL, and the paper architecture that wraps the tile into a full chip is specified in docs/spec/APEX7B_SPEC.md with per-number provenance. That split between what ran on the board and what ran in simulation is stated clearly, and it is the right way to present it.
Getting it running: what the README actually gives you
The reproduction path is a section called Reproduce everything, backed by a results index under docs/results and a master table at docs/design/MASTER_TABLE.md. The README does not reproduce the individual commands in the excerpt available here, so treat the repository's own scripts as the source of truth rather than anything paraphrased. What you can verify before running anything is the layout: docs/spec for the block diagram and the 7B spec, docs/design for the master table, docs/results for the evidence index, and STATUS.md at the root for the measured-versus-projected split. The primary language is Python, which is consistent with the golden-model-first method: the NumPy reference is the executable specification, and the RTL is checked against it. Expect the FPGA flow to depend on vendor tooling that is not part of this repository.
The charter is the limitation
APEX is one tile, and the README says so without hedging: no DRAM controller, no PCIe, no NoC, out of scope by charter. That is the honest boundary and also the reason this is not a drop-in accelerator. Weights have to arrive from somewhere, the host has to talk to the tile somehow, and the tile has to talk to memory. None of that is here. A second constraint is the model class. The demonstrated end-to-end path is a 0.5B model; 7B exists in RTL at head_dim = 128 and in the golden pipeline, but the README does not claim silicon results for it. If your target is a 7B deployment, you are reading a specification, not a measurement. Third, the KV compression scheme trades accuracy for memory traffic by construction. The README describes an importance unit spending bits where they count, which implies a policy that can misjudge. Whether that policy holds up across context lengths is something the results index should answer, and you should read it before assuming it does.
How this differs from quantizing the KV cache on a GPU
The obvious alternative is the standard one: run the model on a GPU or NPU and quantize the KV cache in software, using a library that stores compressed keys and values and decompresses them in a kernel. The difference is where the codec sits. In the software approach, compression and decompression are memory operations around the compute, and the datapath sees full-precision or already-quantized tensors. In APEX, the KVQ engine sits between the projection output and SRAM, and the dequantization happens on the read side of the attention products, inside the same pipeline that computes Q·K̂ᵀ and P·V̂. The importance unit also lives in that loop, adjusting precision per token rather than per model. Whether that placement buys anything depends on whether KV traffic, not GEMM throughput, is your bottleneck. For a 0.5B model on a modern GPU, it almost certainly is not. For a long-context edge workload where the cache dominates, the argument in the README is worth taking seriously.
Maintenance, licence, and who this is for
The licence is Apache-2.0, which permits commercial use, modification and redistribution with the usual attribution and notice requirements, and includes a patent grant. That is a permissive choice for a hardware design and removes the licensing question from your adoption decision. It does not tell you anything about whether the design is correct, and it does not give legal advice; if you plan to ship silicon derived from this RTL, have counsel read the notice files. On maintenance: the repository is not archived, the last push recorded is 2026-08-18, and no releases were retrieved, so there is no versioned artifact to pin against. You would be tracking main. The dependency surface is Python plus an FPGA vendor toolchain, which means the maintenance burden is mostly in the toolchain, not the code. The layer walker and the MXE scheduling are the parts most likely to need changes if you alter the model shape, since a time-multiplexed array means every change to the seven matrix jobs touches the sequencer. Adopt this if you want a verified decoder layer and a KV codec you can study and extend. Look elsewhere if you need a complete inference system, a supported release, or throughput numbers you can put in a product plan.
Editorial conclusion
Adopt APEX if you are building an edge inference accelerator and want a decoder layer whose every block has a bit-exact NumPy reference, an FPGA bring-up on Qwen2.5-0.5B, and a KV codec already inside the datapath. Do not adopt it if you need a deployable inference stack: the charter excludes DRAM controller, PCIe and NoC, so there is no path from this tile to a working system without you writing the rest. Before committing, run the golden-model tests against the RTL yourself, read STATUS.md to separate the measured 0.56 tok/s figure from the projected ones, and check whether the KVQ importance unit's bit allocation matches your context lengths.
Community notes