KTransformers: CPU-GPU Expert Offload for Large MoE Models

The VRAM Wall That KTransformers Is Built Around

Mixture-of-experts models break the usual assumption that a model's weights must sit in GPU memory. A model can carry hundreds of billions of parameters while activating only a fraction per token, because the router sends each token to a small number of experts. The weights still exist. They still have to be stored somewhere, and in a conventional serving stack that somewhere is VRAM. KTransformers takes the opposite position: use host DRAM alongside the GPU, with CPU expert offload. The README's framing is CPU-GPU heterogeneous computing, and the project's own history describes the payoff in memory terms rather than throughput terms, for example the August 2024 note about decreasing DeepseekV2's required VRAM from 21G to 11G.

The audience follows from that. This is for someone with a workstation or a single server that has a large amount of system RAM and one or a few consumer GPUs, who wants to run a model that would otherwise need a multi-GPU node. It is also for people who want to fine-tune such a model, since the project ships an SFT path through LLaMA-Factory. It is not aimed at a team that already has an eight-GPU node and just wants maximum tokens per second, because in that setting the offload machinery is overhead rather than a solution.

How the CPU-GPU Split Actually Works

The README says both user-facing capabilities, Inference and SFT, come from the kt-kernel source tree, and the updates list CPU-GPU expert scheduling alongside native BF16 and FP8 per-channel precision support, plus CPU expert offload tutorials.

The precision story is more specific than a single quantized format. The project added native BF16 and FP8 per-channel precision in January 2026, and introduced native block-FP8 LoRA fine-tuning in August 2026 (PR #2141).

The project also lists a three-layer GPU-CPU-Disk prefix cache, which extends the same tiering idea from weights to the key-value cache, so that long-context reuse does not require recomputation.

Instruction Sets Decide Which Kernel Path You Get

The hardware matrix is the part most likely to trip up a first attempt, because the CPU side is not a generic fallback. The release history shows a progression of CPU backends: AMX-Int8 and AMX-BF16 support in April 2025, an AVX2-only CPU backend for KT-Kernel inference in March 2026, and LoRA fine-tuning on compatible AVX512 x86 CPUs including AMD servers without requiring AMX in v0.7.0. Those are three different capability tiers, and the fine-tuning path in particular was AMX-gated before v0.7.0.

What this means in practice is that two machines with the same core count and the same RAM can produce very different results depending on which of these instruction sets the CPU exposes. A server without AMX is not excluded, but it will land on a different kernel path with different throughput characteristics. The project does not publish a single performance number that covers all of them. The README's older entries mention figures like up to 3 to 28x speedup for Deepseek-R1 and V3 on a single 24GB GPU with 382G DRAM, but those are tied to a specific hardware and model configuration and should be read as a description of that setup, not as a general claim.

GPU vendor coverage is similarly broad and similarly uneven in maturity. The updates list Intel Arc GPU support in May 2025, ROCm on AMD GPU in March 2025, and Ascend NPU support in October 2025, with a DeepSeek-V4-Flash tutorial for a single Ascend NPU with CPU expert offload added in August 2026. Each of those is a separate path through the code, and the existence of a tutorial is the main signal that a path has been exercised end to end.

Getting It Running: The Entry Points the Repo Names

The repository splits its documentation into two user-facing capabilities, inference and SFT, and the README points at kt-kernel/README.md for the first and doc/en/SFT/KTransformers-Fine-Tuning_Cookbook.md for the second. That split is worth respecting when you install, because the two paths have different dependency sets and, as the v0.7.0 note shows, different CPU requirements.

For inference, the starting point is the kt-kernel README, with model-specific tutorials under doc/en/kt-kernel/. The updates list tutorials for GLM-5.3-flash, MiniMax-M3, GLM-5.2, Kimi-K2-Thinking, and others, each at a path like doc/en/kt-kernel/GLM-5.2-Tutorial.md. For a first run, pick the tutorial that matches your model rather than adapting a different one, since the loaders and precision settings differ per architecture. Precision selection is exposed through the native precision path documented at doc/en/kt-kernel/Native-Precision-Tutorial.md, and the expert placement is configured through the mechanism described in doc/en/kt-kernel/experts-sched-Tutorial.md.

For fine-tuning, the cookbook covers hardware checks, installation, BF16, FP8 and INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting. The integration is with LLaMA-Factory, and the project also documents RL-DPO fine-tuning at doc/en/SFT/DPO_tutorial.md. A separate quick start lives at doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md. The project also publishes a cloud path combining KTransformers, AutoDL and LLaMA-Factory as a PDF under doc/zh/, which addresses rental hardware rather than owned hardware.

setup.py describes the top-level package as lightweight: pip install ktransformers installs kt-kernel at the matching version, ktransformers[sft] adds transformers-kt and accelerate-kt, and ktransformers[sglang] adds sglang-kt. Model-specific steps live in the per-capability documents and tutorials.

Where the Offload Design Costs You

The central trade-off is bandwidth. Expert weights live in host memory, so every token that routes to a CPU-resident expert requires those weights to cross into the compute path. The cost does not disappear. A workload with high expert diversity per token, where the router spreads activations across many experts, will move more data than a workload that concentrates on a few. The project's own emphasis on prefix caching and on expert scheduling suggests the authors treat this as the main bottleneck to manage rather than a solved problem.

The second cost is operational. This is described in its own README as a research project, and the release cadence supports that reading: v0.7.0 in August 2026, v0.7.0.post4 and v0.7.1 in September, and v0.7.2 on 2026-10-15, with model support updates landing between them. Day0 support entries for new models appear frequently, which is a benefit if you want the newest architecture and a liability if you want a version that will still be patched in two years.

The third cost is the hardware matrix itself. Because the CPU backend determines which kernel path you get, and because the fine-tuning path only recently relaxed its AMX requirement, a configuration that works on one machine may need a different code path on another. That is a real constraint for anyone planning to standardize on a single image across a heterogeneous fleet.

Finally, the project is not the right tool when the model already fits. If your weights fit in VRAM with room for the KV cache, the offload layer adds complexity and a CPU-side dependency for no memory benefit. The same applies if your priority is a stable HTTP serving interface with predictable latency under load; the project does integrate into SGLang, per the October 2025 roadmap entry.

Compared With a Conventional Quantized Serving Stack

The obvious alternative for someone facing a VRAM limit is aggressive quantization in a standard serving stack: run the model at a low bit width so that everything still fits on the GPU, and avoid host memory entirely. The difference in approach is where the compression happens. Quantization reduces the size of every weight so the whole model fits in VRAM, which typically costs accuracy uniformly across layers and can degrade more at very low bit widths. KTransformers instead keeps the expert weights at a chosen precision in host memory and moves them on demand, trading PCIe bandwidth for the ability to use a larger effective memory pool without pushing quantization as far.

The two are not mutually exclusive, and the project's own history shows it: the updates mention support for unsloth 1.58 and 2.51 bit weights and IQ1_S/FP8 hybrid weights, which is quantization applied inside the offload framework. So the real comparison is not quantize versus offload, it is whether the extra memory tier is worth the scheduling complexity for your workload.

A second alternative is simply to use fewer, larger GPUs. That is the conventional answer and it works, but it changes the hardware budget rather than the software. KTransformers exists precisely for the case where that budget is fixed. A third option is to run a smaller dense model that fits comfortably. That avoids the problem entirely and is the right answer more often than people admit, particularly when the task does not require the specific capabilities of a large MoE model.

Maintenance, Versioning and the Apache-2.0 Terms

KTransformers is licensed under Apache-2.0, which permits commercial use, modification and redistribution provided the license and notices are preserved. That is a permissive arrangement with no copyleft obligation on your own code. It does not, however, come with any warranty, and the license text is standard in disclaiming liability. Nothing here constitutes legal advice; if you are shipping a product that embeds the framework, have counsel review the NOTICE requirements and any third-party kernel licenses bundled with it, since the repository depends on external linear backends and GPU kernel code.

The maintenance cost is the more practical concern. The project tracks new model releases closely, which means the codebase moves. Upgrading between minor versions can change which CPU instruction sets are required, as v0.7.0 did for LoRA fine-tuning on AVX512 CPUs. If you pin a version, pin the model tutorial that goes with it.

For a research group or an individual engineer, that cadence is the point: you get access to new architectures early. For a production team, it means budgeting engineering time for version tracking, and it means testing each upgrade against your specific CPU and GPU combination rather than assuming the previous configuration still applies.

Editorial conclusion

Adopt KTransformers if you have a fixed hardware budget, a model whose experts dominate its parameter count, and the appetite to track a fast-moving research repository rather than a frozen serving stack. Do not adopt it if you need a stable API surface, long-term support guarantees, or a deployment that a platform team can operate without reading kernel-level documentation. Before committing, verify three things against your own hardware: that your CPU exposes the instruction set the chosen kernel path requires (AMX, AVX512, or AVX2), that the specific model you intend to serve has a tutorial in the doc tree, and that the kt-kernel README's supported configuration matches your GPU count and memory layout.

Frequently asked questions

Official sources

1. kvcache-ai/ktransformers on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases