How I Cut LLM Latency in Half with Speculative Decoding

Last month I was staring at a 12-second response time for a code completion feature. Unacceptable. I tried batching, caching, quantization — all helped marginally. Then I implemented speculative decoding and the same request dropped to 5.8 seconds. The improvement was immediate and dramatic. Here's what I learned implementing it.

What Speculative Decoding Actually Is

Standard LLM inference is *sequential* — each token depends on all previous tokens. You generate token 1, then token 2, then token 3. The GPU sits idle while the model thinks.

Speculative decoding inverts this. You use a small "draft" model to generate a sequence of candidate tokens quickly — usually 4 to 8 tokens. Then you verify all of them in a *single forward pass* of the large model. If the draft model was right, you accepted multiple tokens for the price of one verification pass. If it was wrong, you discard the bad speculation and continue.

The result: the large model runs at full utilization, not waiting around for token-by-token generation. Throughput goes up dramatically, especially on longer responses.

The Setup

I'm using vLLM with a Llama 3 8B as the target model and a tiny 80M parameter model as the draft. Here's the config that worked:

from vllm import LLM, SamplingParams

Target: the big model that does verification

target_model = LLM(

model="meta-llama/Llama-3-8B-Instruct",

gpu_memory_utilization=0.85,

tensor_parallel_size=1,

max_model_len=4096,

)

Draft: tiny model for fast speculation

draft_model = LLM(

model="microsoft/Phi-3-mini-4k-instruct",

gpu_memory_utilization=0.15, # Small footprint

tensor_parallel_size=1,

)

Speculative decoding params

sampling_params = SamplingParams(

temperature=0.7,

top_p=0.9,

max_tokens=512,

speculative_model=draft_model,

num_speculative_tokens=5, # Draft generates 5 tokens ahead

)

The key parameter is `num_speculative_tokens`. Too few and you lose the benefit. Too many and the rejection rate kills your speedup. In practice, 4-8 works well for most tasks. I settled on 5 as a default and tune it per-use-case.

Measuring the Win

I ran a simple benchmark: 200 requests with varied prompt lengths and completion targets.

The 31% rejection rate at 8 tokens looks bad but the math still works out — you're verifying 8 tokens per forward pass and only discarding 2-3 on average. Latency nearly halved. Throughput more than doubled.

The Gotchas Nobody Warns You About

**1. Draft model quality matters more than size.** I tried a 250M draft and got worse results than my 80M. The draft needs high prediction accuracy on *your specific task distribution*. A generic 250M model outperformed a specialized 80M on code tasks because it had better next-token intuition for code patterns. Test your draft on real queries.

**2. Memory overhead is real.** The draft model takes VRAM too. With an 8B target at 85% GPU memory, I had about 12% left for the draft. The 80M Phi-3-mini fit comfortably. A larger draft won't fit.

**3. Batch size interacts poorly with speculative decoding.** When you batch multiple requests, each request needs its own speculation sequence. This breaks the efficiency gains at high batch sizes. If you're doing heavy batching, test whether speculative decoding helps at all at your batch size.

**4. Temperature must be low (< 1.0).** Speculative decoding relies on the draft and target models agreeing on high-probability tokens. At temperature 1.0 or higher, the probability distribution flattens and the models disagree more often, which tanks the acceptance rate.

**5. It's not always faster.** For very short responses (under 50 tokens), the overhead of speculation exceeds the benefit. The draft generates 5 tokens, the target verifies them, but if you only needed 30 tokens total, you wasted work. Enable speculative decoding selectively for longer completions.

When to Use It

Speculative decoding is the right call when:

• Response length is unpredictable and often > 100 tokens
• You're latency-sensitive (chat, autocomplete, real-time)
• You're GPU-bound, not memory-bandwidth-bound
• You have VRAM headroom for a small draft model

Skip it when:

• Your responses are consistently short
• You're throughput-bound (high batch sizes, offline batch processing)
• You're already memory-constrained

The Bottom Line

I was skeptical — speculative decoding sounded like a research trick that wouldn't hold up in production. It did. The latency improvement was immediate and the implementation was straightforward with vLLM's built-in support.

If you're running inference on vLLM and haven't tried it, start with a 5-token speculation window and a small draft model. Benchmark on your actual workload. The numbers usually speak for themselves.

— *Mr. TECHNOLOGY*

*P.S. If you're on an older GPU without enough VRAM for a second model, look into "self-speculative decoding" — the target model drafts from earlier layers while later layers verify. Slower to implement but uses zero extra memory.*