KTransformers v0.6.2 Is the First Open-Source Inference Stack That Treats the CPU as a First-Class Citizen for MoE

KTransformers v0.6.2 from Tsinghua MADSYS runs 671B-param DeepSeek V3 on one 24GB GPU plus RAM. SOSP'25 paper, AMX-optimized kernels, integrates into SGLang. Most teams are still overpaying 8x for the same workload.

Hey guys, Mr. Technology here. KTransformers is an open-source CPU-GPU heterogeneous inference engine from Tsinghua's MADSYS lab that solves a problem nobody else in open source has solved: running a 671B-parameter Mixture-of-Experts model without renting 8x H100s.

What You Need to Know - What it is: Open-source CPU/GPU heterogeneous inference and fine-tuning framework for MoE LLMs (kt-kernel CPU kernels + SGLang backend integration) - License: Apache 2.0 - Languages: Python, C++, CUDA, with ISA-specific assembly for AMX / AVX-512 / AVX2 - GitHub: kvcache-ai/ktransformers — 17.4K stars, 1,339 forks, active as of last week - Current version: v0.6.2 (May 3, 2026); MiniMax-M3 Day-0 June 21, GLM-5.2 Day-0 June 17, DeepSeek-V4-Flash May 2.

Why MoE Inference Has Been a Tax Until Now

DeepSeek V3 is 671B parameters with 37B activated per token. Kimi-K2-Thinking, GLM-5, GLM-5.2, Qwen3-Next, DeepSeek-V4-Flash — all sparse mixtures with hundreds of billions of total weights and a small fraction active per forward pass.

The dense-inference engines we know — vLLM, SGLang, llama.cpp — were built for dense models and bolted MoE on. They assume you either have VRAM for the whole model or you don't run it. That assumption is wrong for MoE. The 634B of "cold" experts in DeepSeek V3 don't need to be hot in VRAM. They need to be cheap, abundant, and reachable when routing picks them.

KTransformers treats that asymmetry as a first-class design constraint. Hot experts stay on GPU, cold experts live in CPU memory, NUMA-pinned and AMX-quantized. CUDA Graphs capture the hybrid execution so host-device sync overhead disappears. Expert deferral overlaps CPU compute with GPU attention. Detail in the SOSP'25 paper.

The Three Tricks That Make It Win

1. AMX-specialized CPU kernels. Hand-written AMX (Intel Advanced Matrix Extensions) kernels hit 21.3 TFLOPS sustained on a single Xeon socket — 3.9x faster than PyTorch native. That is the gap between "MoE inference is slow" and "MoE inference is free" for Sapphire Rapids and Granite Rapids owners.

2. NUMA-aware tensor parallelism. KTransformers pins expert weights to the owning NUMA node, yielding up to 63% decoding throughput improvement on dual-socket hardware versus the usual "treat all memory as one pool" approach.

3. Expert deferral. Because of residual connections, Transformers tolerate small delays to intermediate computations. KTransformers defers some experts to the next layer so CPU compute overlaps GPU attention: up to 1.45x decoding throughput, accuracy variation below 0.5%.

The LMSYS integration blog breaks down the GPU kernel-launch overhead drop from 20%+ to nearly zero via CUDA Graph capture. Benchmarks on a single 24GB GPU plus 382GB DRAM show 3-28x speedup over a naive DeepSeek-R1/V3 deployment. On 8x L20 + Xeon Gold 6454S they hit 227.85 tokens/s total / 87.58 tokens/s output at 8-way concurrency on DeepSeek-R1-0528 FP8.

How It Compares to vLLM and SGLang

vLLM is the right call for dense Llama and Qwen. SGLang is the default agent-serving stack. Both treat MoE as "dense-with-sparse-routing" — neither treats the CPU as a first-class device for cold experts.

KTransformers does not compete with either. It is a CPU-side kernel library and expert-scheduling policy that plugs into SGLang via the sglang-kt fork, so you keep your existing serving stack and the MoE part gets 10x cheaper to run. SGLang teams add a one-line --kt-method flag. vLLM shops use the standalone kt CLI.

The Day-0 support cadence — MiniMax-M3 June 21, GLM-5.2 June 17, DeepSeek-V4-Flash May 2 — tells you who the maintainers are shipping for. Not dense-model benchmarks. The MoE era that 2026 has actually become.

What It Looks Like to Actually Run It

The kt-kernel PyPI wheel is the low-level CPU kernel library. The kt CLI and the SGLang integration (sglang-kt) are how you serve traffic. Canonical run for Qwen3-30B-A3B on 2x Xeon Gold 6454S + a single RTX 4090:

bash

pip install kt-kernel sglang-kt
python -m sglang.launch_server \
    --host 0.0.0.0 --port 30000 \
    --model /mnt/data/models/Qwen3-30B-A3B \
    --kt-weight-path /mnt/data/models/Qwen3-30B-A3B \
    --kt-cpuinfer 64 \
    --kt-threadpool-count 2 \
    --kt-num-gpu-experts 32 \
    --kt-method BF16 \
    --attention-backend flashinfer \
    --mem-fraction-static 0.80 \
    --chunked-prefill-size 16384

One GPU, 64 physical CPU cores, 128GB+ of system RAM, and you get production-grade serving of a 30B MoE. --kt-num-gpu-experts 32 decides which experts stay hot on GPU. --kt-method picks the CPU backend (AMXINT8, AMXINT4, FP8, BF16, or LLAMAFILE for AVX2-only systems). For fine-tuning, the project ships an LLaMA-Factory integration for DPO/RLHF on the same topology.

The Take

If you are serving dense Llama or Qwen, keep your vLLM. If you are serving DeepSeek V3, Kimi-K2, GLM-5, MiniMax-M3, or Qwen3-MoE and you have at least one decent CPU socket, KTransformers is the most important open-source drop of 2026 for your stack. CPU memory is 10-20x cheaper than H100 VRAM per GB, AMX turns that cheap memory into a usable inference substrate, and the SOSP'25 paper plus the SGLang integration plus Day-0 support for every major MoE release this year means the project is not vanishing in six months.

If you have a Sapphire Rapids or Granite Rapids box with a single 24GB GPU sitting idle and a few hundred GB of RAM, you can run DeepSeek V3 at interactive speeds today. There is no longer a technical excuse to keep renting 8x H100s for it.

Repo: github.com/kvcache-ai/ktransformers — Apache 2.0, 17.4K stars, 1,339 forks, v0.6.2 (May 3, 2026). MiniMax-M3 Day-0 June 21, 2026. SOSP'25 paper. SGLang integration. Fine-tuning via LLaMA-Factory.