KTransformers: CPU-GPU Expert Offload for Large MoE Models
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
At a glance
- What is it?
- KTransformers is a research framework that splits large mixture-of-experts models between GPU and CPU, using optimized kernels to keep expert weights in system memory. It targets engineers who want to run or fine-tune models that do not fit in VRAM, and it assumes you are willing to work inside a moving research codebase.
- Who is it for?
- Adopt KTransformers if you have a fixed hardware budget, a model whose experts dominate its parameter count, and the appetite to track a fast-moving research repository rather than a frozen serving stack. Do not adopt it if you need a stable API surface, long-term support guarantees, or a deployment that a platform team can operate without reading kernel-level documentation.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The VRAM Wall That KTransformers Is Built Around
Mixture-of-experts models break the usual assumption that a model's weights must sit in GPU memory. A model can carry hundreds of billions of parameters while activating only a fraction per token, because the router sends each token to a small number of experts. The weights still exist. They still have to be stored somewhere, and in a conventional serving stack that somewhere is VRAM. KTransformers takes the opposite position: keep the expert weights in host memory and the DRAM that most servers already have in quantity, and keep the attention layers and the dense parts on the GPU. The README's framing is CPU-GPU heterogeneous computing, and the project's own history describes the payoff in memory terms rather than throughput terms, for example the August 2024 note about decreasing DeepseekV2's required VRAM from 21G to 11G.
The audience follows from that. This is for someone with a workstation or a single server that has a large amount of system RAM and one or a few consumer GPUs, who wants to run a model that would otherwise need a multi-GPU node. It is also for people who want to fine-tune such a model, since the project ships an SFT path through LLaMA-Factory. It is not aimed at a team that already has an eight-GPU node and just wants maximum tokens per second, because in that setting the offload machinery is overhead rather than a solution.
How the CPU-GPU Split Actually Works
The mechanism is expert offload with a scheduling layer on top. The repository separates the work into kt-kernel, which holds the CPU-optimized kernel operations, and the surrounding Python framework that loads models, routes tokens, and moves data. The README describes kt-kernel as CPU-optimized kernel operations for heterogeneous LLM inference, and the updates list a CPU-GPU expert scheduling tutorial alongside native BF16 and FP8 per-channel precision support. That combination is the core of the design: expert weights are held at a reduced precision in host memory, and the scheduler decides which experts are needed for the current batch so that only those are pulled across the PCIe bus.
The precision story is more specific than a single quantized format. The project added native BF16 and FP8 per-channel precision in January 2026, and later added block-FP8 LoRA fine-tuning that loads FP8 routed-expert weights directly from the checkpoint without materializing a complete BF16 copy. That last detail matters for anyone who has tried to fine-tune a large MoE model on limited hardware: the usual failure is not compute, it is the memory spike from dequantizing the whole model into a training dtype before the first step. Loading FP8 experts in place avoids that spike.
The data flow, as far as the documentation describes it, runs in this order. The model is loaded with the attention and dense layers placed on the GPU and the routed experts placed in host memory. For each forward pass, the router produces expert assignments, the scheduler maps those assignments onto available CPU and GPU compute, the CPU kernel path executes the experts it owns, and the results are combined. The project also lists a three-layer GPU-CPU-Disk prefix cache, which extends the same tiering idea from weights to the key-value cache, so that long-context reuse does not require recomputation.
Instruction Sets Decide Which Kernel Path You Get
The hardware matrix is the part most likely to trip up a first attempt, because the CPU side is not a generic fallback. The release history shows a progression of CPU backends: AMX-Int8 and AMX-BF16 support in April 2025, an AVX2-only CPU backend for KT-Kernel inference in March 2026, and LoRA fine-tuning on compatible AVX512 x86 CPUs including AMD servers without requiring AMX in v0.7.0. Those are three different capability tiers, and the fine-tuning path in particular was AMX-gated before v0.7.0.
What this means in practice is that two machines with the same core count and the same RAM can produce very different results depending on which of these instruction sets the CPU exposes. A server without AMX is not excluded, but it will land on a different kernel path with different throughput characteristics. The project does not publish a single performance number that covers all of them. The README's older entries mention figures like up to 3 to 28x speedup for Deepseek-R1 and V3 on a single 24GB GPU with 382G DRAM, but those are tied to a specific hardware and model configuration and should be read as a description of that setup, not as a general claim.
GPU vendor coverage is similarly broad and similarly uneven in maturity. The updates list Intel Arc GPU support in May 2025, ROCm on AMD GPU in March 2025, and Ascend NPU support in October 2025, with a DeepSeek-V4-Flash tutorial for a single Ascend NPU with CPU expert offload added in August 2026. Each of those is a separate path through the code, and the existence of a tutorial is the main signal that a path has been exercised end to end.
Getting It Running: The Entry Points the Repo Names
The repository splits its documentation into two user-facing capabilities, inference and SFT, and the README points at kt-kernel/README.md for the first and doc/en/SFT/KTransformers-Fine-Tuning_Cookbook.md for the second. That split is worth respecting when you install, because the two paths have different dependency sets and, as the v0.7.0 note shows, different CPU requirements.
For inference, the starting point is the kt-kernel README, with model-specific tutorials under doc/en/kt-kernel/. The updates list tutorials for GLM-5.3-flash, MiniMax-M3, GLM-5.2, Kimi-K2-Thinking, and others, each at a path like doc/en/kt-kernel/GLM-5.2-Tutorial.md. For a first run, pick the tutorial that matches your model rather than adapting a different one, since the loaders and precision settings differ per architecture. Precision selection is exposed through the native precision path documented at doc/en/kt-kernel/Native-Precision-Tutorial.md, and the expert placement is configured through the mechanism described in doc/en/kt-kernel/experts-sched-Tutorial.md.
For fine-tuning, the cookbook covers hardware checks, installation, BF16, FP8 and INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting. The integration is with LLaMA-Factory, and the project also documents RL-DPO fine-tuning at doc/en/SFT/DPO_tutorial.md. A separate quick start lives at doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md. The project also publishes a cloud path combining KTransformers, AutoDL and LLaMA-Factory as a PDF under doc/zh/, which is the only document in the supplied material that addresses rental hardware rather than owned hardware.
What the README does not give is a single pip install line or a top-level configuration file format. Installation instructions live in the per-capability documents, and the exact commands depend on which kernel path and which model you selected. Treat the model tutorial as the install guide.
Where the Offload Design Costs You
The central trade-off is bandwidth. Expert weights live in host memory, so every token that routes to a CPU-resident expert requires those weights to cross into the compute path. The scheduler's job is to make that crossing worth it, by batching and by keeping frequently used experts closer to the GPU, but the cost does not disappear. A workload with high expert diversity per token, where the router spreads activations across many experts, will move more data than a workload that concentrates on a few. The project's own emphasis on prefix caching and on expert scheduling suggests the authors treat this as the main bottleneck to manage rather than a solved problem.
The second cost is operational. This is described in its own README as a research project, and the release cadence supports that reading: v0.6.3 in June 2026, v0.6.4 in July, v0.7.0 in August, with model support updates landing between them. Day0 support entries for new models appear frequently, which is a benefit if you want the newest architecture and a liability if you want a version that will still be patched in two years. There is no long-term support branch described in the material.
The third cost is the hardware matrix itself. Because the CPU backend determines which kernel path you get, and because the fine-tuning path only recently relaxed its AMX requirement, a configuration that works on one machine may need a different code path on another. That is a real constraint for anyone planning to standardize on a single image across a heterogeneous fleet.
Finally, the project is not the right tool when the model already fits. If your weights fit in VRAM with room for the KV cache, the offload layer adds complexity and a CPU-side dependency for no memory benefit. The same applies if your priority is a stable HTTP serving interface with predictable latency under load; the project does integrate into SGLang, per the October 2025 roadmap entry, but the supplied material does not describe the maturity of that integration.
Compared With a Conventional Quantized Serving Stack
The obvious alternative for someone facing a VRAM limit is aggressive quantization in a standard serving stack: run the model at a low bit width so that everything still fits on the GPU, and avoid host memory entirely. The difference in approach is where the compression happens. Quantization reduces the size of every weight so the whole model fits in VRAM, which typically costs accuracy uniformly across layers and can degrade more at very low bit widths. KTransformers instead keeps the expert weights at a chosen precision in host memory and moves them on demand, trading PCIe bandwidth for the ability to use a larger effective memory pool without pushing quantization as far.
The two are not mutually exclusive, and the project's own history shows it: the updates mention support for unsloth 1.58 and 2.51 bit weights and IQ1_S/FP8 hybrid weights, which is quantization applied inside the offload framework. So the real comparison is not quantize versus offload, it is whether the extra memory tier is worth the scheduling complexity for your workload.
A second alternative is simply to use fewer, larger GPUs. That is the conventional answer and it works, but it changes the hardware budget rather than the software. KTransformers exists precisely for the case where that budget is fixed. A third option is to run a smaller dense model that fits comfortably. That avoids the problem entirely and is the right answer more often than people admit, particularly when the task does not require the specific capabilities of a large MoE model.
Maintenance, Versioning and the Apache-2.0 Terms
KTransformers is licensed under Apache-2.0, which permits commercial use, modification and redistribution provided the license and notices are preserved. That is a permissive arrangement with no copyleft obligation on your own code. It does not, however, come with any warranty, and the license text is standard in disclaiming liability. Nothing here constitutes legal advice; if you are shipping a product that embeds the framework, have counsel review the NOTICE requirements and any third-party kernel licenses bundled with it, since the repository depends on external linear backends and GPU kernel code whose terms are not enumerated in the supplied material.
The maintenance cost is the more practical concern. The project tracks new model releases closely, which means the codebase moves. Upgrading between minor versions can change which CPU instruction sets are required, as v0.7.0 did for LoRA fine-tuning on AVX512 CPUs. If you pin a version, pin the model tutorial that goes with it, because a tutorial written for v0.6.4 may reference loaders or precision flags that changed in v0.7.0. There is no described deprecation policy or compatibility guarantee across releases.
For a research group or an individual engineer, that cadence is the point: you get access to new architectures early. For a production team, it means budgeting engineering time for version tracking, and it means testing each upgrade against your specific CPU and GPU combination rather than assuming the previous configuration still applies.
Editorial conclusion
Adopt KTransformers if you have a fixed hardware budget, a model whose experts dominate its parameter count, and the appetite to track a fast-moving research repository rather than a frozen serving stack. Do not adopt it if you need a stable API surface, long-term support guarantees, or a deployment that a platform team can operate without reading kernel-level documentation. Before committing, verify three things against your own hardware: that your CPU exposes the instruction set the chosen kernel path requires (AMX, AVX512, or AVX2), that the specific model you intend to serve has a tutorial in the doc tree, and that the kt-kernel README's supported configuration matches your GPU count and memory layout.
Community notes