Wiki

LLM Benchmarks Decoded: What MMLU, HumanEval, and GSM8K Actually Measure for Developers

A developer-focused breakdown of the most-cited LLM benchmarks — MMLU, HumanEval, GSM8K, MATH, Chatbot Arena — including what they test, their hidden biases, and how to use scores to pick a model for real-world deployment.

Wiki Updated 30 July 2026 6 min read Lena Walsh

Journalists Protest against rising violence during march in Mexi | by Knight Foundation | openverse | by-sa

You are staring at a leaderboard: Model A scores 90.1% on MMLU, Model B 89.2%, Model C 88.4%. Which one do you deploy? The numbers look precise, but they hide a deeper problem — each benchmark measures a narrow skill that may not match your use case. MMLU tests multiple-choice knowledge, not reasoning. HumanEval checks isolated Python functions, not real-world codebases. GSM8K rewards correct arithmetic answers, not explanation quality. This guide unpacks the five most cited benchmarks, exposes their blind spots, and gives you a practical filter for turning leaderboard scores into deployment decisions.

What a Benchmark Actually Tests — and What It Skips

A benchmark is a fixed-input, automated-scoring task. For LLMs, three categories dominate:

Knowledge and reasoning: MMLU, ARC, HellaSwag, GPQA
Code generation: HumanEval, MBPP, SWE-bench
Math and logic: GSM8K, MATH, MATH500

Each benchmark has a narrow evaluation scope. MMLU uses 14,079 multiple-choice questions from 57 academic subjects — but it never checks whether the model can explain why an answer is correct, or handle ambiguous phrasing. HumanEval requires the model to complete a Python function that passes all unit tests, yet it tests only 164 problems, all in English, and ignores code readability, security, or error handling. GSM8K gives a grade-school math problem and expects an exact numeric answer — models that achieve 90%+ often rely on pattern matching rather than genuine arithmetic reasoning.

Why Benchmarks Are Misleading

Benchmark scores are reproducible (if the evaluation setup is disclosed), but their practical value decays for three reasons:

Data contamination. Many models have seen benchmark examples during training. A high MMLU score can reflect memorization, not reasoning. For example, some early GPT-4 evaluations on GSM8K were later rerun with different few-shot prompts and showed a 6–8 percentage point drop.

Narrow scope. MMLU does not test instruction following, safety, or long-context retrieval. HumanEval does not measure code maintainability. GSM8K ignores explanation quality.

Saturation. Older benchmarks like SQuAD, GLUE, and SuperGLUE are now near-perfect for strong models, offering little differentiation. Newer variants (MMLU-Pro, GPQA) raise the ceiling but still cover only a few domains.

Vendors routinely cherry-pick benchmarks that flatter their model. A model card may list MMLU and GSM8K but omit HumanEval or safety tests. Independent sources like the Open LLM Leaderboard (Hugging Face) and Chatbot Arena (crowdsourced preference) provide a check, but each has its own bias: Arena scores depend on voter demographics (often favoring models with more polished style), and the Leaderboard uses static evaluation sets that may not match the original benchmark protocol.

Core Benchmarks in Detail

Benchmark	What it tests	Format	Typical pass criterion	Known limitations
MMLU	Multi-subject knowledge (57 domains)	14,079 multiple-choice questions	Accuracy	Potential contamination; measures fact recall more than reasoning depth
HumanEval	Python function synthesis	164 programming problems	pass@k (all unit tests pass)	Tiny dataset; English-only; not representative of multi-file repos
GSM8K	Grade-school math word problems	8,500 problems	Exact match on answer	Simple arithmetic; many models top out above 90%, reducing differentiation
MATH	Competition-level math (AMC 12, AIME)	5,000 problems	Accuracy	Harder than GSM8K but still only math; no explanation scoring
Chatbot Arena	Overall helpfulness and user preference	Crowdsourced pairwise votes (vs. ChatGPT and others)	Elo score	No task granularity; voting may favor politeness over correctness; demographic bias

Additional benchmarks worth knowing: MMLU-Pro (harder subset with longer options), GPQA (graduate-level science questions), SWE-bench (real GitHub issues with pull requests), and the KILT collection for knowledge-intensive tasks. For retrieval-augmented generation (RAG) pipelines, Natural Questions, TriviaQA, and the newer RAG benchmark sets are more relevant than MMLU.

Benchmark Contamination: The Hidden Variable

Contamination happens when benchmark data leaks into the training set. A model that has seen the exact GSM8K problems during pre-training will score artificially high. Researchers have found that even simple n-gram overlap checks catch contamination in many open models. For example, the Llama 2 technical report showed that certain benchmarks (like TruthfulQA) had high overlap with pre-training data. To mitigate:

Always check the release date of the benchmark against the model’s training cutoff. A model trained after a benchmark release is suspect.
Prefer benchmarks that use held-out or rotated subsets, such as the newer MMLU-Redux or the LM Evaluation Harness’s decontaminated splits.
Use the Open LLM Leaderboard’s reproduction runs, which keep evaluation code and prompts transparent.

A Practical Decision Framework for Developers

Rather than trusting a single number, combine multiple signals in this order:

Identify your primary task. Code assistant → prioritize HumanEval, MBPP, and SWE-bench over MMLU. Factual QA → MMLU, GPQA, Natural Questions. Math tutoring → GSM8K and MATH, but also test explanation quality manually.

Check the evaluation date. A score from January 2025 is likely stale by May 2025, especially for rapidly updated APIs. Some vendors (e.g., OpenAI, Anthropic) release updated scores with new system prompts.

Demand independent reproduction. A vendor-published 90% MMLU score without a detailed setup (few-shot examples, temperature, split) is unverified. Use the Open LLM Leaderboard or your own run of the lm-evaluation-harness.

Run your own micro-benchmark. Create 20–50 representative prompts that match your actual pipeline. Keep the same system message, temperature, and max tokens across models. The small sample often reveals failures that broad benchmarks miss.

Factor in cost and latency. A model that is 1% better on MMLU may be 5× slower or 10× more expensive per token. Multiply accuracy by throughput and price to get a practical score.

Sources and Caveats

MMLU: Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding. https://arxiv.org/abs/2009.03300
HumanEval: Chen et al. (2021). Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374
GSM8K: Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. https://arxiv.org/abs/2110.14168
Chatbot Arena: Zheng et al. (2024). Judging LLM-as-a-Judge with Chatbot Arena. https://arxiv.org/abs/2403.04510
Open LLM Leaderboard: Hugging Face. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

Caveats: This guide reflects benchmarks as of March 2025. New benchmarks and scoring paradigms appear regularly. Scores from commercial model card pages may use different evaluation prompts, temperature settings, or validation splits — always prefer peer-reviewed or independently replicated results. The decision framework above does not replace a dedicated evaluation pipeline; it is a starting point for informed comparison. For deeper guidance, see ReviewArticle’s “How RAG Benchmarks Differ from LLM Benchmarks” and “Comparing Open vs. Closed Models: A Developer Checklist.”