Krasis: a hybrid LLM runtime that keeps MoE experts in CPU RAM and runs the rest on your GPU

The problem Krasis targets: MoE weights that do not fit in VRAM

A 397B-parameter mixture-of-experts checkpoint does not fit on a 32 GB card, and the usual answer is to rent a 96 GB card or accept CPU-only inference. Krasis takes a third route. The README describes it as an LLM runtime for running large MoE models on NVIDIA consumer GPUs, built around fast GPU prompt processing, GPU-executed decode, and HCS expert residency management so models much larger than VRAM can still run locally. The benchmark table makes the intent concrete: an RTX 5090 with 32 GB is listed running Qwen3.5-397B-A17B at 10.04 tok/s decode, and a 123.6B Nemotron model at 41.87 tok/s. The audience is someone with one consumer or prosumer NVIDIA card and enough system RAM to hold the cold portion of the expert set. It is not a general-purpose runtime for dense models, and the README does not present it as one.

HCS: hot and cold experts, and why that is the whole design

The mechanism the README names is HCS, which it describes as managing hot/cold expert residency between VRAM and CPU RAM. Prefill runs on the GPU in full, decode is GPU-executed, and the expert weights are the part that gets split. The README lists the machinery around that split: measured startup calibration, prompt-conditioned reload, a dynamic recency tail, per-stage budgets, soft-tier reload caps, and safe eviction and reload paths. Read together, those items describe a system that measures what it can hold rather than assuming a fixed split, and that reloads experts based on what the current prompt is likely to touch. The recency tail is the interesting part: experts used recently stay resident, so a prompt that keeps hitting the same expert group degrades less than one that wanders. That also means throughput depends on the prompt, which is why a single tok/s number for a model is less informative than the shape of the workload.

The runtime path moved out of Python, and the README says so plainly

The README states that the current runtime is no longer the early Python-hot-path prototype, and that serving uses Rust/CUDA orchestration, CUDA kernels, cached quantized weights, and measured VRAM budgeting. Python is kept for launcher, setup, and model loading. This matters if you are evaluating an older write-up or an older release: v0.1.64 is described as the previous stable release, and the current line is presented as a major change from it, with hot-path work moved into Rust/CUDA for serving, decode orchestration, timing, HCS operations, and benchmark-critical paths. Two surfaces were removed rather than ported: Session messenger integration and other prototype-era surfaces, and AWQ and Polar4 are deprecated for new production runs. The README points current production at HQQ attention plus k6v6, k4v4, or BF16 KV. If you have scripts pinned to AWQ or Polar4, they are on a path the project has stopped maintaining.

Getting it running: installer, prerelease script, and the caches it builds

Native Windows has a downloaded installer, KrasisSetup-1.0.16-win64.exe, which the README says installs for the current user, bundles its own Python runtime, and adds Krasis to the Start Menu. Linux and WSL2 use a shell script: curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- prerelease. Note the version mismatch in the README: the install section points at v1.0.16 while the release feed shows v1.0.21-rc.11, so the install instructions and the release line are not in step. Maintenance commands are krasis update and krasis prerelease. First run is slower because Krasis builds optimized local caches, and later runs reuse them; those artifacts live under ~/.krasis. The README warns that disk usage must cover the source model plus the cache artifacts, and that system RAM should be sized for the selected quantized cache and the HCS backing store. Larger models need substantial RAM even when VRAM is limited. That is the real hardware bill: a big card is not enough, and a fast NVMe drive is not optional.

KV cache modes and HQQ attention: the quality and memory dial

Krasis exposes compact KV cache modes, k6v6 which the README calls the quality-oriented launcher default, and k4v4 for tighter VRAM budgets. Attention can run in HQQ4, HQQ6, HQQ8, or an auto mixed profile, with cache build and rebuild support. The benchmark rows pair these differently for a reason: DeepSeek-V4-Flash-0731 is listed as INT4/HQQ8/BF16 cache at 30.08 tok/s decode on a 96 GB card, while Step-3.7-Flash at roughly two thirds the parameters uses INT4/HQQ4/k4v4 and reaches 55.40 tok/s. Those are not comparable measurements of the models; they are different memory and precision settings on different checkpoints. The README is explicit that you should read the per-family quality and tool-use limitations rather than assuming every quantized runtime is equally faithful. Treat k4v4 as a budget setting you validate on your own prompts, not as a free win.

Where Krasis is the wrong tool

The README's tradeoffs section rules out several cases directly. Krasis targets NVIDIA GPUs with CUDA, Ampere and newer; there is no AMD or Apple silicon path described. Input models should be BF16 safetensors, so a GGUF-only workflow is not the intended input even though gguf-model-support is a repository topic. The design is aimed at MoE checkpoints: the residency manager exists because experts are sparse, and a dense model of comparable size has no cold expert pool to park in RAM, so the mechanism has less to work with. First-run cache building means a fresh deployment is slow before it is fast, which is awkward for autoscaling or ephemeral containers. And the release feed is a run of release candidates, rc.9, rc.10, rc.11 within about two days, with the README's install instructions still pointing at v1.0.16. If you need a frozen runtime for a production service, that cadence is a reason to wait for a stable tag.

How this differs from llama.cpp

llama.cpp is the obvious comparison, and the repository topics themselves list llama-cpp-alternative. The difference is where each puts the boundary. llama.cpp's well-known partial offload splits layers across GPU and CPU, so the split is per layer and the same split applies to every token. Krasis splits inside the MoE layer, keeping attention and decode on the GPU while HCS moves individual experts between VRAM and CPU RAM based on measured budgets and recent use. For a sparse MoE model that is a different bet: layer offload pays a fixed per-token penalty across the whole model, while expert residency pays only when the router selects an expert that is currently cold. That is also the risk. If your prompts route broadly across experts, the reload path runs more often and the advantage narrows. Krasis is also narrower in scope: CUDA and BF16 safetensors, against llama.cpp's much wider backend and format coverage. Pick Krasis for a specific large MoE model on one NVIDIA card; pick llama.cpp when portability or format breadth matters more than peak speed on that class of model.

Maintenance cost and the licence question

The repository metadata reports the licence as NOASSERTION, which means GitHub could not map the licence file to a known SPDX identifier. That is not a statement about what the licence permits; it means you have to open the licence file and read it before you depend on Krasis commercially or redistribute a bundled build. The README does not discuss licensing terms, so there is nothing further to verify from the supplied material. On upgrade cost, the project ships GitHub-release based installation plus krasis update and krasis prerelease, and the README mentions full wheel packaging for Python 3.10 through 3.13 with vendored CUDA sidecars. That packaging is a real convenience, but the rc-heavy release feed and the README's stale v1.0.16 install pointer mean an upgrade can move you between cache formats. Since Krasis builds and reuses caches under ~/.krasis, and the README notes cache build and rebuild support for HQQ attention, plan for a rebuild after a version jump rather than assuming the old cache is reused.

Editorial conclusion

Adopt Krasis if you have an NVIDIA Ampere-or-newer card, a large amount of system RAM, and a specific MoE checkpoint you want to run locally without renting a 96 GB card. Do not adopt it if your target model is a dense transformer, if you are on AMD or Apple silicon (the README targets CUDA only), or if you need a stable API surface: the release history is a run of release candidates. Before committing, verify three things against your own hardware: the first-run cache build time and the disk headroom it needs under ~/.krasis, peak system RAM during HCS eviction and reload, and the per-family quality and tool-use limitations the README tells you to read instead of assuming every quantized runtime is equally faithful.

Official sources

1. brontoguana/krasis on GitHub
2. Issues
3. README
4. Releases