Imagine you're writing a document with a very fast but occasionally inaccurate assistant, and a very accurate but slow editor. Without speculative decoding: You wait for the slow editor to write each word. It's accurate but takes forever. With speculative decoding: The fast assistant quickly drafts 5-10 words. The slow editor reviews all of them at once (in parallel), accepting the correct ones and fixing any mistakes. You get the editor's accuracy with the assistant's speed. Speculative decoding does the same for AI. A small, fast model (draft model) generates several tokens quickly. The large, accurate model (target model) verifies them all at once. If the draft was right, you've generated multiple tokens in the time it takes to generate one. If the draft was wrong, the target model corrects it. This technique can achieve 2-3x speedup while maintaining the exact same output quality as the large model.
Imagine you're writing a document with a very fast but occasionally inaccurate assistant, and a very accurate but slow editor. Without speculative decoding: You wait for the slow editor to write each word. It's accurate but takes forever. With speculative decoding: The fast assistant quickly drafts 5-10 words. The slow editor reviews all of them at once (in parallel), accepting the correct ones and fixing any mistakes. You get the editor's accuracy with the assistant's speed. Speculative decoding does the same for AI. A small, fast model (draft model) generates several tokens quickly. The large, accurate model (target model) verifies them all at once. If the draft was right, you've generated multiple tokens in the time it takes to generate one. If the draft was wrong, the target model corrects it. This technique can achieve 2-3x speedup while maintaining the exact same output quality as the large model.
Speculative decoding exploits the fact that verifying N tokens takes roughly the same time as generating 1 token in autoregressive decoding. By having a draft model propose multiple tokens, the target model can verify them in parallel. The Process: Step 1: Draft Phase Small draft model generates K candidate tokens (typically K=5-10) Draft model is fast (small, optimized) Draft tokens may contain errors Step 2: Verification Phase Large target model processes all K draft tokens in parallel Target model computes probabilities for each position Compare draft tokens with target model's predictions Step 3: Acceptance/Rejection For each draft token, accept if it matches target model's distribution Reject at first mismatch, regenerate from that point Repeat the process Mathematical Insight: Standard decoding: Generate 1 token per forward pass Speculative decoding: Generate up to K tokens per forward pass (if draft is correct) Expected speedup: 2-3x in practice (depends on draft quality) Draft Model Selection: Same Architecture, Smaller Size: Llama-70B target + Llama-7B draft Different Architecture: Any fast model that approximates target Medusa: Multiple prediction heads on target model (no separate draft) Eagle: Learned draft model specific to target Key Factors for Success: Draft Quality: Higher draft accuracy = more tokens accepted = better speedup Draft model should approximate target model's distribution Typical acceptance rate: 60-80% Draft Length (K): Too short: Limited speedup Too long: More rejections, wasted computation Optimal K: 5-10 tokens (task-dependent) Hardware Utilization: Verification phase must be memory-bound (not compute-bound) Works best on high-memory-bandwidth hardware (H100, A100) Variants: Standard Speculative Decoding: Separate draft and target models Most common implementation Flexible but requires two models in memory Medusa: Multiple prediction heads on target model No separate draft model needed Lower memory overhead SpecInfer: Uses ensemble of draft models Higher acceptance rate More complex implementation EAGLE: Learned draft model with context features Higher acceptance rate than standard draft State-of-the-art performance
# Speculative decoding with Hugging Face Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load target model (large, accurate)
target_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-70b-hf",
torch_dtype=torch.float16,
device_map="auto"
)
# Load draft model (small, fast)
draft_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
torch_dtype=torch.float16,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-70b-hf")
# Speculative decoding function
def speculative_decode(
prompt: str,
target_model,
draft_model,
tokenizer,
max_new_tokens: int = 100,
K: int = 5 # Number of tokens to draft
):
"""Generate text using speculative decoding."""
inputs = tokenizer(prompt, return_tensors="pt").to(target_model.device)
generated_ids = inputs.input_ids
for _ in range(max_new_tokens // K):
# Step 1: Draft phase - generate K tokens with draft model
draft_outputs = draft_model.generate(
generated_ids,
max_new_tokens=K,
do_sample=False, # Greedy for deterministic drafting
return_dict_in_generate=True,
output_scores=True
)
draft_tokens = draft_outputs.sequences[0, -K:]
# Step 2: Verification phase - verify all K tokens with target model
# Target model processes all K tokens in parallel
with torch.no_grad():
target_outputs = target_model(
torch.cat([generated_ids, draft_tokens.unsqueeze(0)], dim=1)
)
target_logits = target_outputs.logits[0, -K-1:-1] # Logits for K positions
target_probs = torch.softmax(target_logits, dim=-1)
# Step 3: Acceptance/rejection
accepted_tokens = []
for i in range(K):
draft_token = draft_tokens[i]
target_token = target_probs[i].argmax()
# Accept if draft matches target's most likely token
if draft_token == target_token:
accepted_tokens.append(draft_token)
else:
# Reject and use target's token
accepted_tokens.append(target_token)
break # Stop at first mismatch
# Append accepted tokens
generated_ids = torch.cat([
generated_ids,
torch.tensor(accepted_tokens, device=generated_ids.device).unsqueeze(0)
], dim=1)
# Stop if we generated enough tokens
if len(accepted_tokens) < K:
break
return tokenizer.decode(generated_ids[0], skip_special_tokens=True)
# Usage
prompt = "Once upon a time in a land far away,"
result = speculative_decode(prompt, target_model, draft_model, tokenizer)
print(result)
# Achieves 2-3x speedup compared to standard decoding
# while producing identical output qualitySpeculative decoding is critical for reducing inference costs at scale: Why It Matters: Cost Reduction: 2-3x speedup = 2-3x cost reduction for inference User Experience: Faster responses improve user satisfaction Scalability: Serve more users with same infrastructure Competitive Advantage: Lower costs enable better pricing Enterprise Applications: High-Traffic APIs: Reduce costs for millions of requests Real-Time Applications: Meet strict latency requirements Long-Form Generation: Speed up document generation, code completion Interactive Systems: Chatbots, code assistants, voice assistants Cost Example: Standard decoding (Llama-70B): $0.06 per 1M tokens Speculative decoding (Llama-70B + 7B draft): $0.02 per 1M tokens Savings: 67% cost reduction with identical quality Implementation Considerations: Draft Model Selection: Choose draft that approximates target well Memory Requirements: Need to load both models (or Medusa heads) Tuning: Optimize K (draft length) for your specific use case Monitoring: Track acceptance rates to measure effectiveness Popular Implementations: vLLM: Built-in speculative decoding support TensorRT-LLM: NVIDIA's optimized implementation llama.cpp: Speculative decoding for CPU inference Medusa: Open-source implementation
A chef and sous-chef preparing a meal. The sous-chef (draft model) quickly preps ingredients based on the recipe. The head chef (target model) reviews everything at once, accepting correct prep and fixing mistakes. The meal is prepared much faster because the head chef doesn't have to do everything from scratch — they just verify and correct.
Imagine you're writing a document with a very fast but occasionally inaccurate assistant, and a very accurate but slow editor. Without speculative decoding: You wait for the slow editor to write each word. It's accurate but takes forever. With speculative decoding: The fast assistant quickly drafts 5-10 words. The slow editor reviews all of them at once (in parallel), accepting the correct ones and fixing any mistakes. You get the editor's accuracy with the assistant's speed. Speculative decoding does the same for AI. A small, fast model (draft model) generates several tokens quickly. The large, accurate model (target model) verifies them all at once. If the draft was right, you've generated multiple tokens in the time it takes to generate one. If the draft was wrong, the target model corrects it. This technique can achieve 2-3x speedup while maintaining the exact same output quality as the large model.
Speculative decoding exploits the fact that verifying N tokens takes roughly the same time as generating 1 token in autoregressive decoding. By having a draft model propose multiple tokens, the target model can verify them in parallel. The Process: Step 1: Draft Phase Small draft model generates K candidate tokens (typically K=5-10) Draft model is fast (small, optimized) Draft tokens may contain errors Step 2: Verification Phase Large target model processes all K draft tokens in parallel Target model computes probabilities for each position Compare draft tokens with target model's predictions Step 3: Acceptance/Rejection For each draft token, accept if it matches target model's distribution Reject at first mismatch, regenerate from that point Repeat the process Mathematical Insight: Standard decoding: Generate 1 token per forward pass Speculative decoding: Generate up to K tokens per forward pass (if draft is correct) Expected speedup: 2-3x in practice (depends on draft quality) Draft Model Selection: Same Architecture, Smaller Size: Llama-70B target + Llama-7B draft Different Architecture: Any fast model that approximates target Medusa: Multiple prediction heads on target model (no separate draft) Eagle: Learned draft model specific to target Key Factors for Success: Draft Quality: Higher draft accuracy = more tokens accepted = better speedup Draft model should approximate target model's distribution Typical acceptance rate: 60-80% Draft Length (K): Too short: Limited speedup Too long: More rejections, wasted computation Optimal K: 5-10 tokens (task-dependent) Hardware Utilization: Verification phase must be memory-bound (not compute-bound) Works best on high-memory-bandwidth hardware (H100, A100) Variants: Standard Speculative Decoding: Separate draft and target models Most common implementation Flexible but requires two models in memory Medusa: Multiple prediction heads on target model No separate draft model needed Lower memory overhead SpecInfer: Uses ensemble of draft models Higher acceptance rate More complex implementation EAGLE: Learned draft model with context features Higher acceptance rate than standard draft State-of-the-art performance
Speculative decoding is critical for reducing inference costs at scale: Why It Matters: Cost Reduction: 2-3x speedup = 2-3x cost reduction for inference User Experience: Faster responses improve user satisfaction Scalability: Serve more users with same infrastructure Competitive Advantage: Lower costs enable better pricing Enterprise Applications: High-Traffic APIs: Reduce costs for millions of requests Real-Time Applications: Meet strict latency requirements Long-Form Generation: Speed up document generation, code completion Interactive Systems: Chatbots, code assistants, voice assistants Cost Example: Standard decoding (Llama-70B): $0.06 per 1M tokens Speculative decoding (Llama-70B + 7B draft): $0.02 per 1M tokens Savings: 67% cost reduction with identical quality Implementation Considerations: Draft Model Selection: Choose draft that approximates target well Memory Requirements: Need to load both models (or Medusa heads) Tuning: Optimize K (draft length) for your specific use case Monitoring: Track acceptance rates to measure effectiveness Popular Implementations: vLLM: Built-in speculative decoding support TensorRT-LLM: NVIDIA's optimized implementation llama.cpp: Speculative decoding for CPU inference Medusa: Open-source implementation