eLLM: A CPU Inference Framework That Bets on DDR Instead of HBM
eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs
At a glance
- What is it?
- eLLM is a Rust LLM inference server for CPU-only machines that trades memory capacity for bandwidth, keeping a static computation graph and a non-paged KV cache so long prompts are prefilled in one pass. It targets agent workloads, and it only makes sense on hardware most teams do not have.
- Who is it for?
- Adopt eLLM only if you already own an AVX-512 FP16 Xeon or EPYC machine with 128 GB or more of DDR and your workload is a long-running agent session where time to first token and context continuity matter more than raw concurrency. Do not adopt it if you need multi-model serving, a stable API surface, or you are running on a laptop, an ARM box, or a GPU cluster you have already paid for.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 6 days ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The bandwidth gap eLLM is trying to close
GPU inference is limited by HBM capacity and by how much of that bandwidth a single decode step can consume. CPU servers have the opposite profile: DDR is cheap and plentiful, but per-core bandwidth is far lower. eLLM's README frames the whole design as a response to that asymmetry, describing a strategy of trading storage for computation. The claim is that on long-horizon work, the CPU's larger memory wins.
The target user is narrow. The README lists Open Claw computer-use agents, code copilots working across large repositories, RAG that retrieves during execution rather than up front, and deep research loops running for hours or days. What those share is a session that stays alive and keeps growing its context. That is the workload eLLM is built around, not single-shot chat.
The hardware requirement is the first filter. The README asks for AVX-512 FP16 support and 128 GB of memory or more, on Linux x86-64, with Xeon Gen4+ or AMD EPYC recommended. If you are reading this on a workstation without AVX-512 FP16, the project is not for you regardless of how good the design sounds.
Static graph, non-paged KV, and why the sequence dimension is oversized
Four mechanisms are described in the README, and they depend on each other.
The first is what the project calls an elastic static computation graph. eLLM builds one graph and accesses tensors in what the README terms a dimension-first layout, so that elements at the same logical coordinates map to stable memory locations. The consequence is that the same graph serves different input lengths without being rebuilt. That matters because rebuilding a graph per request is exactly the repeated work a long session cannot afford.
The second is a static-shape, non-paged KV cache. Instead of paged block management, eLLM preallocates one fixed-shape tensor and locates KV entries by tensor coordinates, reading contiguously along the sequence dimension. The README's stated rationale is reduced metadata maintenance, address mapping, and dynamic allocation, plus fewer TLB and cache misses. This is a real trade-off, not a free win: a paged cache can grow and shrink with load, while a fixed tensor is committed at startup. Sizing it wrong means either wasted memory or a hard ceiling.
The third mechanism is the oversized sequence dimension. The README says eLLM reserves a large enough sequence dimension to build an effectively unbounded KV tensor, which is what allows full single-pass prefill over an entire long prompt with no chunking and no repeated parameter loading.
The fourth is the session cache. KV state persists across turns, so a new turn prefills only the new input. The README describes this as state continuity at the mechanism level. For an agent that appends a tool result every few seconds, that is the difference between a growing bill and a flat one.
Getting it running: the actual commands and flags
The README gives a concrete path. You need rustup, Python 3, and curl. The toolchain is pinned to nightly by rust-toolchain.toml in the repository, so you do not choose a channel yourself.
Download Qwen3-Coder-30B-A3B-Instruct from Hugging Face and place it at models/Qwen3-Coder-30B-A3B-Instruct inside the cloned repository. Then:
git clone https://github.com/lucienhuangfu/eLLM.git cd eLLM cargo build --release --bin main ./target/release/main --model-path models/Qwen3-Coder-30B-A3B-Instruct --chunk-size 50000 --sequence-length 50000 --batch-size 1
Three flags carry the design. --sequence-length sets the reserved sequence dimension, which is the size of the KV tensor you are committing to at startup. --chunk-size is paired with it in the example at 50000. --batch-size 1 reflects the README's position that decode should run with a smaller batch so each request gets a larger share of memory bandwidth.
The README warns that first startup may take longer while weights and the computation graph are initialized. That is consistent with a static-graph design: the cost is paid once, up front, rather than per request. There is no published figure for how long that initialization takes, so treat it as an unknown you will measure on your own machine.
On the API side, the README states eLLM is vLLM API compatible, which is the practical reason a team could try it without rewriting a client. The README does not enumerate which endpoints are implemented, so that compatibility claim needs checking against your own client before you plan around it.
Where the design breaks down
The static-shape KV cache is the most consequential limitation. Because the tensor is preallocated rather than paged, concurrency is bounded by what you reserved, not by what the machine could theoretically hold. The README acknowledges this indirectly, stating that single-instance concurrency is lower than GPU solutions. It argues real QPS is still higher because end-to-end latency is smaller. That argument may hold for long sessions, but it does not hold for a burst of short unrelated requests, where a paged cache would simply serve more of them.
Model support is the second constraint. The supported list is Qwen3 series, with Qwen3.8 marked as in development. There is no mention of Llama, Mistral, Gemma, or any vision or embedding model. If your stack is not Qwen3, the project is not currently an option, and the README gives no indication of a conversion path.
The project is also young and moving. v0.0.1 was open-sourced in December 2025, v0.0.2 in April 2026, and v0.0.3 in September 2026. The README says code is pushed to the main branch monthly. Monthly pushes on main, with no long-term support branch described, means you should expect to track main or pin a tag and carry patches yourself.
Finally, the README's performance claims are stated without a published benchmark harness. It asserts roughly two orders of magnitude improvement in prefill over existing CPU frameworks and says inference speed can exceed GPUs. There is no linked methodology, dataset, or reproduction script in the material provided. Treat those numbers as the authors' claims, not as measurements you can rely on until you reproduce them.
How it differs from llama.cpp and from vLLM on GPUs
The obvious CPU comparison is llama.cpp. Both run quantized or half-precision transformer inference on CPU, both are written in systems languages, and both can serve an OpenAI-style HTTP API. The difference is in the memory model. llama.cpp's KV cache is allocated per sequence and managed dynamically, which makes it flexible across many concurrent short requests. eLLM goes the other way: one fixed-shape tensor, coordinates instead of block tables, and a session that persists across turns. That makes eLLM better suited to a single long-lived agent session and worse suited to a general-purpose multi-tenant endpoint.
The GPU comparison is vLLM. vLLM's paged attention exists precisely to pack many sequences into limited HBM. eLLM inverts the premise: it assumes HBM is the scarce resource and DDR is not. The README's argument is that for long-horizon work, the cost of repeatedly loading parameters dominates, and loading them once into a large DDR-backed graph beats cycling them through HBM. Whether that holds depends entirely on your prompt lengths and session durations.
A third option worth naming is simply not running inference yourself. If your agent traffic is modest, a hosted API removes the 128 GB hardware requirement entirely. eLLM's case rests on owning the hardware already.
Licence and the cost of keeping it running
eLLM is AGPL-3.0. That is a copyleft licence with a network clause: if you modify eLLM and expose it to users over a network, the licence's terms attach to your modified version. For an internal-only deployment the practical effect is smaller than for a product built on top of it. This is not legal advice, and if you plan to ship eLLM inside a commercial service, the licence question is worth raising with counsel before you build on it, not after.
Maintenance cost is real but bounded. The dependency surface is small: Rust, a pinned nightly toolchain, Python 3 and curl for the chat client. There is no Kubernetes operator, no scheduler, and no separate cache tier described. Upgrades mean rebuilding the binary and restarting the service, which also means paying the graph initialization cost again and dropping any live session KV state. The README does not describe a warm-reload or state-migration path, so plan for a restart window.
The hardware cost is the larger line item. A Xeon Gen4+ or EPYC machine with 128 GB of DDR is not a spare laptop. The README's cost argument is that you reuse existing CPU machines and avoid water cooling and high-power supplies, which is fair if those machines are already sitting in your data center. If you have to buy them, the comparison against a GPU node changes.
What to check before you commit a machine to it
Start by confirming AVX-512 FP16 specifically. AVX-512 alone is not enough, and the flag is absent on many recent consumer and some server parts. Run lscpu and look for avx512_fp16 in the flags line. If it is missing, stop there.
Next, size the KV tensor against your real workload. --sequence-length 50000 in the README example is a starting point, not a recommendation. Multiply the per-token KV footprint of Qwen3-Coder-30B-A3B-Instruct by the context length you actually need, add the weight footprint, and check it against available DDR. Because the cache is static, oversizing wastes memory you paid for and undersizing produces a hard failure at a context length you cannot predict from the outside.
Then verify the output alignment claim. The v0.0.3 release notes state that inference results are fully aligned with SGLang CPU. That is a strong claim and an easy one to test: run the same prompts through both and diff the outputs. If your application is sensitive to token-level differences, do this before anything else.
Finally, check the vLLM API surface against your client. The README says eLLM is vLLM API compatible but does not list endpoints. Point your existing client at the server and see what breaks. That single test will tell you more about adoption cost than any of the performance claims.
Editorial conclusion
Adopt eLLM only if you already own an AVX-512 FP16 Xeon or EPYC machine with 128 GB or more of DDR and your workload is a long-running agent session where time to first token and context continuity matter more than raw concurrency. Do not adopt it if you need multi-model serving, a stable API surface, or you are running on a laptop, an ARM box, or a GPU cluster you have already paid for. Before committing, verify three things yourself: that your CPU actually exposes AVX-512 FP16 (not just AVX-512), that the Qwen3-Coder-30B-A3B-Instruct weights fit alongside the KV tensor you size with --sequence-length, and that v0.0.3's SGLang-aligned outputs hold for your own prompts, since the README claims alignment but publishes no comparison harness.
Community notes