LMDeploy Is the C++ Inference Engine You Skipped Past — And It's Been Eating vLLM's Lunch for Two Years

What it is

LMDeploy is an inference and serving stack from Shanghai AI Lab (the InternLM / MMRazor / MMDeploy teams). It ships two engines: TurboMind, a C++/CUDA engine forked from NVIDIA's FasterTransformer, and PyTorchEngine, a pure-Python engine for fast iteration. Both share the same CLI (lmdeploy serve api_server) and Python SDK — you swap engines with a flag, not a rewrite.

The problem it solves: vLLM is decent general-purpose plumbing, but on the InternLM / Qwen / DeepSeek axis — particularly with 4-bit quantization or MoE architectures — it's been measurably slower than TurboMind for about two years. The numbers are in their release notes, not in a blog post.

Why it's interesting

Three things keep drawing me back:

1. Real, reproducible wins over vLLM — InternLM2-20B with GQA hit 16+ RPS, ~1.8× vLLM. MXFP4 GPT-OSS on H800: 1.5× vLLM (Sept 2025). The gap holds on Qwen and DeepSeek variants. 2. DeepSeek-ready before the hype cycle — V3/R1 support in January 2025; FlashMLA, DeepGemm, DeepEP, MicroBatch, EPLB by April 2025; prefill/decode disaggregation via Mooncake + DLSlime by June 2025. Running a DeepSeek MoE in production today? LMDeploy is one of the lowest-friction paths. 3. Quantization composes cleanly — W4A16 AWQ, online int8/int4 KV cache quant, and prefix caching stack simultaneously. The 4-bit AWQ path runs ~2.4× FP16 throughput.

How it works under the hood

The C++ engine is the interesting half:

• Blocked KV cache — tokens packed into fixed-size GPU blocks; the scheduler swaps whole blocks in and out of GPU memory. Lower fragmentation than vLLM-style paging on long contexts, simpler bookkeeping.
• Persistent batching + dynamic split&fuse — long prefill requests are sliced and interleaved with decode steps of running requests. No single prompt stalls the batch. DistServe and Mooncake formalized this idea later; TurboMind shipped it years earlier.
• Tensor parallelism with custom all-reduce kernels and GQA-optimized paths (the GQA kernels drove the InternLM2-20B 1.8× win).
• W4A16 AWQ with dequant fused into the GEMM — the 2.4× over FP16 comes from avoiding full FP16 materialization per matmul, not magic.
• Online KV cache quant — cache lives in int8/int4, so a 70B model at 32k context fits in reasonable memory. Stacks with prefix caching so you don't re-pay for system prompts.
• Prefill/decode disaggregation (mid-2025) — prefill and decode run on separate instance pools via Mooncake + DLSlime without a custom serving layer.

The PyTorch engine is the pragmatic fallback: pure Python, CUDA graphs, easier to adapt when a new model drops before TurboMind has a kernel. Ship fast with it; switch to TurboMind for production throughput.

When to reach for it

• Reach for LMDeploy when serving DeepSeek V3/R1, Qwen, InternLM, or other MoE-heavy Chinese models; when running 4-bit AWQ in production; when doing VLMs with prefix caching; when deploying on Huawei Ascend; or when you want prefill/decode disaggregation without building it yourself.
• Stay on vLLM if model coverage is #1 — it supports new architectures days faster and the ecosystem is wider. On a stock Llama-3 70B FP16 deployment on H100s, the perf gap is small enough that ecosystem wins.
• SGLang wins for structured-output agentic workloads via RadixAttention and its DSL. Don't fight that.
• TensorRT-LLM still wins on raw kernel perf for single-model hyperscale deployments, but engine compilation takes a week. Only worth it for one-model-at-a-time scenarios.

Gotchas

• Best perf gains are on InternLM / Qwen / DeepSeek. On Llama-3 expect 5–15%, not 1.8×. Set expectations correctly.
• The C++ engine has a heavier dep tree than vLLM. Use the official Docker images unless you enjoy debugging CUDA toolchains.
• Quantization + prefix-cache combos can be finicky on very long contexts — read the kv_quant docs before enabling everything at once.
• pip install lmdeploy works again as of v0.12.3 (April 2026). Don't trust guides older than mid-2026.

Take

If you're running a vanilla Llama-3 70B FP16 on H100s, keep vLLM. If you're running DeepSeek V3/R1, Qwen, InternLM, or any AWQ-quantized model on commodity NVIDIA boxes — and you're not on LMDeploy — you're leaving 30–80% of your GPU budget on the table. The C++ engine, the quantization story, and the DeepSeek optimizations make it the most underrated serving stack in open-source right now. English-language discourse largely ignored it because it came from the Chinese open-source ecosystem. That window is closing.