Methods for measuring model quality, reliability, and real-world performance.
Imagine you're comparing cars. You don't just look at them and guess which is faster. You take them to a racetrack, measure their 0-60 times, top speed, fuel efficiency, and handling. These standardized tests let you objectively compare different cars. Benchmarking does the same for AI. We use standardized tests (like MMLU for knowledge, HumanEval for coding, GSM8K for math) to measure how well different models perform. This lets us objectively compare GPT-4 vs. Claude vs. Llama and track improvements over time. Without benchmarking, we'd have no way to know if a new model is actually better or just marketed as better.
Instead of taking one single practice test to see if you're ready for the final, you take 5 different practice tests, each covering a different part of the material. This gives you a much more accurate idea of what you actually know.
When a student accidentally gets a copy of the final exam before taking it. The AI isn't actually smarter; it just memorized the exact questions and answers from the test it's supposed to be taking, making its score completely fake.
Imagine a calculator. If you type "2 + 2" and press equals, you always get "4." Every single time. No exceptions. That's deterministic behavior — same input, same output, always. Now imagine a magic 8-ball. You ask it a question, shake it, and it gives you an answer. But if you ask the same question again, you might get a different answer. That's non-deterministic — same input, different outputs. Most AI language models are non-deterministic by default (they can give different answers to the same question). But you can make them deterministic by setting the temperature parameter to 0, which forces the model to always pick the most likely next word.
The gap between how well a student does on the homework (training data) versus the actual final exam (real-world data). If they memorized the homework answers, their generalization error is huge.
Imagine a student taking a test who doesn't know the answer. Instead of writing "I don't know," they confidently write a detailed, plausible-sounding essay full of made-up facts, fake historical dates, and invented quotes, hoping the teacher will be impressed by the writing style and not notice the content is completely false. That's an AI hallucination. The model isn't "lying" intentionally; it's just doing what it was trained to do: predict the next most likely word. Sometimes, the most likely-sounding sequence of words is a complete fabrication.
Imagine rolling a die. Even if you roll it the exact same way twice, you might get different numbers. That's non-deterministic — the outcome varies even with identical starting conditions. Most AI language models work this way by default. Ask the same question twice, and you might get slightly different answers. This variability is actually a feature, not a bug — it allows the AI to be creative, explore different solutions, and avoid getting stuck in repetitive patterns. You can control the degree of non-determinism using the temperature parameter. Higher temperature = more variability. Lower temperature = more consistency.
Imagine you're playing a word-guessing game. Your friend says, "The sky is..." If you guess "blue," you're not very surprised. The perplexity is low. If your friend says, "The sky is... refrigerator," you are highly surprised. The perplexity is very high. For an AI, perplexity measures exactly this: how shocked the model is by the actual next word in a sentence, based on what it predicted. A good language model should be "surprised" by bad grammar or nonsense, and "unsurprised" by coherent, natural text.