EdgeRock: a reasoning model that keeps a constant-size attention history

The claim is a fixed history budget, not a bigger context

EdgeRock brings strong reasoning to a fixed history budget. EdgeRock-4B, the first open source release, supports text and image reasoning, and its constant-size history leaves more GPU memory available for sustained reasoning and concurrent serving. More model sizes are in preparation.

Three numbers carry the argument, and the README states them with the comparisons attached. Strong mathematical reasoning is 82.9 percent mean AIME pass@1, versus 18.8 percent for the strongest evaluated bounded-state baseline. O(1) history memory is a fixed 72 MiB attention KV per sequence, 99.1 percent less than Qwen3.5-4B. And higher serving throughput, 6.42 times Qwen3.5-4B's output throughput at each model's best measured batch.

The framing is careful in a way that is worth copying. This is not a claim about an unbounded context. It is a claim about a class of models, described as bounded-state, where the attention state does not grow with the conversation.

That is the interesting idea, because the memory a transformer needs for a long conversation is dominated by the KV cache, and a fixed budget removes that variable. The trade is that the model has to summarise rather than remember, and the README is careful in the benchmark section too, noting that the AIME figure is a mean pass@1 across 2024 to 2026, the KV comparison is at 256K context, and the serving throughput uses 32K input and 32K output on one Hopper 80GB GPU at each model's best measured batch, counting output tokens only, with elapsed time including prefill.

Each of those footnotes narrows the comparison, and reading them is how you know how much weight the headline numbers carry.

The quick start clones, installs a vendored transformers, and downloads weights

The quick start is a shell sequence, and it shows that this repository ships a modified transformers rather than using a released one.

git clone https://github.com/rockmatrix-ai/EdgeRock.git
cd EdgeRock
python -m venv .venv
source .venv/bin/activate
python -m pip install ./wheels/transformers-5.7.0.dev0-py3-none-any.whl -e .

The wheel installed is a development build of transformers, 5.7.0.dev0, and the requirements file confirms it is pinned that way with a relative path to the same file in the wheels directory. That is a strong signal about how the model integrates: the architecture is not yet in a released transformers, so the repository carries the fork that knows how to load it.

The weights then come from Hugging Face, with a narrow include list so you do not pull the evaluation data along with the checkpoints:

hf download rockmatrix-ai/EdgeRock --local-dir checkpoints/EdgeRock-4B \
  --include '*.safetensors' '*.json' '*.jinja' 'tokenizer*' 'vocab*' 'merges*' \
  --exclude 'evaluation_data/*'

And finally the config and the custom modelling files are copied next to the weights, and the demo is run:

cp config.json configuration_edgerock.py modeling_edgerock.py checkpoints/EdgeRock-4B/
python run.py demo --model checkpoints/EdgeRock-4B \
  --prompt 'Solve 17 × 23, explaining your reasoning.' --max-new-tokens 2048

The model directory is a Hugging Face repository, with a ModelScope mirror listed alongside it and a collection of quantized releases linked separately. There is also a technical report PDF in the docs directory, and the repository has a Chinese README alongside the English one.

Seven quantised builds, each with its own caveats

The model directory is the part of the README that most rewards reading, because each format comes with a statement about what has and has not been validated.

The original model is BF16 for text and images. There is an NF4 build, an INT8 build and an FP8 build, each with an included compact loader. The AWQ and GPTQ builds are 4-bit with an included packed INT4 GPU runtime, and the NVFP4 build is W4A16 with BF16 reference loading.

Then the caveat that matters: the compact packages validate text inference on Hopper, and you should see each card for its conversion checks. Original-model benchmark scores do not establish quantized-package accuracy or throughput.

That sentence is the whole point of reading this section. The 82.9 percent AIME figure is an original-model number, and applying it to a 4-bit AWQ build is not something the project claims to have measured.

The other runtime families have the same shape of qualification. The MLX packages come in 8-bit, 6-bit, 4-bit and mixed 4-bit variants, and the README says they use the included EdgeRock MLX runtime, that functional checks were run on Linux CUDA, and that Apple Metal has not been tested. The GGUF build has 7 profiles with a custom EdgeRock backend, and the README is explicit that GGUF requires the custom EdgeRock RMA backend, that stock llama.cpp and Ollama are not supported, and that the bundled native binaries target Linux x86-64 Hopper. The ONNX packages cover a CUDA 4-bit, a CPU 4-bit and a CUDA 8-bit build, and they require their supplied stateful runner, with mobile ONNX Runtime untested.

The mobile previews for Google AI Edge Gallery are INT8 and INT4 LiteRT-LM CPU builds, and the README says full-model long-memory checks and Android device validation are in progress and that image and audio inputs are not included.

The runtime package is the artefact in this repository

The package manifest describes the project as an inference and evaluation runtime for EdgeRock-4B, named edgerock-runtime at version 0.2.0, requiring Python 3.10 or newer.

The base dependencies are torch 2.6 or newer, transformers pinned to the 5.7.0.dev0 development build, safetensors, numpy and accelerate. The CUDA extra adds the pieces needed for the fused attention path, with fla-core, tilelang, causal-conv1d and flash-attn.

The evaluation extra is a much longer list, and it tells you what the benchmark harness needs: pyarrow and duckdb for result handling, scipy and sympy for mathematics, pillow and nltk for text work, math-verify and latex2sympy2-extended for checking answers, antlr4-python3-runtime for parsing, sentencepiece and absl-py, z3-solver for symbolic reasoning, emoji, syllapy and langdetect for language handling, and immutabledict.

The entry points are the interesting part. There is a console script edgerock mapped to run:main, and a vllm plugin entry point, edgerock mapped to vllm_edgerock:register, registered under the vllm.general_plugins group. So the runtime can be served through vLLM rather than only through the bundled run.py.

The package layout follows from that. There are four py-modules, run, configuration_edgerock, modeling_edgerock and vllm_edgerock, with the runtime and sglang_models directories included as packages. The repository top level shows the same split, with a runtime/ directory, an sglang_models/ directory, a vllm_edgerock.py at the root, a run.py entry point, a Dockerfile and a USAGE.md for the inference guide.

The container targets Hopper, and the requirements pin a lot

The Dockerfile in the repository shows what the deployment target is.

The base image is the NVIDIA PyTorch container at 26.03 for Python 3, the working directory is /opt/edgerock-runtime, and the requirements and the wheels directory are copied in first. The pip install is run with no cache directory and no build isolation, with CAUSAL_CONV1D_FORCE_BUILD set to true so that the causal convolution extension compiles from source rather than pulling a wheel, and the package itself is installed with no dependencies so the pinned requirements file stays authoritative.

The entry point runs Python in isolated mode with the run script:

ENTRYPOINT ["python", "-I", "/opt/edgerock-runtime/run.py"]

The requirements file is a long list of exact pins, and it includes the relative path to the vendored transformers wheel, which is the same file the quick start installs by hand. So the container and the manual install end up with the same transformers build, which is the point of pinning it that way.

The rest of the pins are all exact versions, including tokenizers, accelerate, pyarrow, duckdb, numpy, scipy, sympy, pillow, nltk, math-verify, antlr4, sentencepiece, absl-py, regex, packaging, huggingface-hub, fla-core, tilelang, apache-tvm-ffi, torch-c-dlpack-ext, z3-solver, causal-conv1d, emoji, syllapy, langdetect and immutabledict.

Exact pins for a research runtime are the right call, because a numerical library moving under you will change your benchmark numbers, and the whole point of publishing a score is that it is reproducible.

Editorial conclusion

Adopt EdgeRock if you need a small reasoning model with a bounded memory footprint, on a Hopper GPU, and you can work from a published checkpoint rather than training one. Do not adopt it on a non-Hopper machine or without reading the per-package caveats, because the quantized builds validate text inference on Hopper only, the MLX packages were tested on Linux CUDA rather than Apple Metal, and the GGUF path needs a custom backend rather than stock llama.cpp. Before you plan around the numbers, read the footnote under the hero figure, since the AIME mean, the KV comparison and the throughput comparison each use a specific measurement setup.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. rockmatrix-ai/EdgeRock on GitHub