Sampling

Imagine you're at an ice cream shop with 100 flavors. The shop ranks them by popularity: Greedy sampling: Always pick the #1 most popular flavor (vanilla). Predictable but boring. Random sampling: Pick any flavor randomly. Creative but might get "pistachio garlic" (nonsense). Top-k sampling: Pick randomly from the top 5 most popular flavors. Balanced variety. Top-p (nucleus) sampling: Pick from flavors that together make up 90% of popularity. Dynamic selection. Sampling is how the AI picks the next word. Different strategies give different balances of predictability and creativity.

The Simple Version

Detailed Explanation

After a language model processes input, it outputs a probability distribution over its entire vocabulary (e.g., 50,000 tokens). Sampling selects which token to generate next. Sampling Strategies: Greedy (Argmax): Always select the token with highest probability Deterministic: same input → same output Can produce repetitive, bland text Random (Multinomial): Sample from the full probability distribution Highly diverse but can produce incoherent text Controlled by temperature parameter Top-k: Restrict sampling to the k most likely tokens k=1 is greedy; k=vocab_size is random Fixed cutoff regardless of probability distribution shape Top-p (Nucleus): Select smallest set of tokens whose cumulative probability exceeds p Dynamic cutoff: adapts to distribution shape p=0.9 typically works well Typical Sampling: Select tokens with probability close to the "typical" information content Avoids both very high and very low probability tokens Parameters: Temperature: Scales the distribution (0=greedy, 1=raw, >1=flatter) Top-k: Number of tokens to consider Top-p: Cumulative probability threshold Repetition Penalty: Discourages repeating tokens Frequency/Presence Penalty: Further controls repetition

Code Example

# Comparing sampling strategies
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")

# Greedy (deterministic)
greedy = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print("Greedy:", tokenizer.decode(greedy[0]))

# Random with temperature
random = model.generate(**inputs, max_new_tokens=20, do_sample=True, temperature=1.0)
print("Random:", tokenizer.decode(random[0]))

# Top-k sampling
topk = model.generate(**inputs, max_new_tokens=20, do_sample=True, top_k=50)
print("Top-k=50:", tokenizer.decode(topk[0]))

# Top-p (nucleus) sampling
topp = model.generate(**inputs, max_new_tokens=20, do_sample=True, top_p=0.9)
print("Top-p=0.9:", tokenizer.decode(topp[0]))

Key Characteristics

Controls Diversity: Determines how varied outputs are
Task-Dependent: Different tasks need different sampling strategies
Non-Deterministic: Most strategies produce varied outputs
Configurable: Multiple parameters allow fine-grained control

Why It Matters

Sampling strategy directly impacts AI output quality and user experience: Use Cases by Strategy: Greedy (temp=0): Code generation, data extraction, factual Q&A Balanced (temp=0.7, top-p=0.9): General conversation, summarization Creative (temp=1.0, top-p=0.95): Creative writing, brainstorming, marketing Best Practices: Start Conservative: Begin with temperature=0.7 for most applications Test Extensively: Evaluate outputs across multiple runs Document Choices: Record which parameters work for each use case Monitor Quality: Track coherence at different settings

Real-World Analogy

A jazz musician choosing the next note. Greedy sampling always plays the most expected note (safe but predictable). Random sampling plays any note (creative but possibly chaotic). Top-p sampling plays from a curated set of musically appropriate notes (balanced creativity and coherence).

Common Misconceptions

Myth: Higher temperature always means better, more creative outputs.
Reality: Higher temperature increases diversity but also increases incoherence risk. There's an optimal range for each task.
Myth: Sampling is random noise.
Reality: Sampling is guided by the model's probability distribution. It's controlled randomness, not chaos.
Myth: Greedy sampling is always best for factual tasks.
Reality: While greedy is deterministic, it can produce repetitive text. Top-p with low temperature often works better.

Sampling

The Simple Version

Detailed Explanation

Code Example

Key Characteristics

Why It Matters

Real-World Analogy

Common Misconceptions

Related Terms

Related Articles

Sources & Further Reading

Sampling: Direct Definition

Technical Definition

Why It Matters