MMLU: The Massive Multitask Language Understanding Benchmark

What MMLU Measures

MMLU (Massive Multitask Language Understanding) is a benchmark covering 57 academic subjects across STEM, social science, humanities, and professional fields. Questions range from elementary mathematics to medical diagnosis, legal reasoning, and macroeconomics, each presented as a 4-choice multiple-choice question. A model must demonstrate genuine knowledge across this diversity to score well. It is the most widely cited benchmark for measuring the broad knowledge encoded in a Foundation Model's weights, distinct from skills like Chain of Thought reasoning or RAG-augmented factual recall, which are evaluated separately.

Developed by UC Berkeley researchers Dan Hendrycks, Collin Burns, and colleagues, and published in 2020, MMLU was designed to measure whether LLMs had acquired human-level academic knowledge across the breadth of subjects that a well-educated person might know. At the time of release, it was one of the first benchmarks ambitious enough to include professional-level knowledge domains alongside standard academic subjects. It became the de facto standard for measuring 'knowledge breadth' precisely because it's so broad, so well-validated, and because it arrived at exactly the right moment, when LLMs were first showing signs of real academic competence.

How MMLU Is Constructed

MMLU's 14,079 questions were sourced from freely available practice exams, textbooks, and standardized test preparation materials. The 57 subject categories span everything from abstract algebra and anatomy to world religions and US foreign policy. Each question has exactly four answer choices with exactly one correct answer. Questions were collected and filtered for clarity, correctness, and discriminatory power, ambiguous or poorly-worded questions were removed.

The subjects are organized into four broad categories: STEM (mathematics, physics, chemistry, biology, computer science, engineering), Social Sciences (psychology, economics, sociology, political science), Humanities (philosophy, history, law, literature), and Other Professional Fields (medicine, finance, accounting). This breadth is intentional: no model should be able to score well by being deeply expert in only a few areas.

A critical property of MMLU's design is that it tests knowledge that a model must have internalized during pretraining, not knowledge it can retrieve in-context or reason toward from first principles. This makes it a pure test of what a model 'knows' in its weights, independent of Chain of Thought prompting or tool use. MMLU without Chain of Thought versus with Chain of Thought produces different scores, which is why papers must specify the evaluation protocol.

What the Scores Actually Mean

Human expert performance on MMLU is estimated at around 89%. Random chance gives 25%, the floor on a 4-choice benchmark. The useful range is therefore 25% to 100%, with 89% representing the human expert ceiling. Models have progressed dramatically across this range: GPT-3 scored 43.9%, GPT-3.5 Turbo scored 70%, GPT-4 scored 86.4%, and current frontier models like Gemini 2.5 Pro, GPT-5, and Claude Opus 4 reach 89-93%.

Scores above 70% indicate broad academic knowledge. Above 85% suggests approaching or matching human expert performance on average. Above 89% means exceeding the estimated human expert baseline, which frontier models now routinely achieve. However, averages hide critical variation: a model scoring 90% overall may perform at 97% in mathematics and 83% in clinical medicine, domain-specific evaluation is often more informative than the aggregate score, especially when selecting a model for a specific professional application.

Reading Subject-Level Scores

The most actionable information in MMLU is not the aggregate score but the subject-level breakdown. Models with identical overall scores can differ dramatically by domain: a model might lead on medical genetics but trail on formal logic; another might be exceptional at history but weaker on electrical engineering. These patterns reflect training data composition, fine-tuning emphasis, and architectural biases.

For practitioners, subject selection matters. If you're building a legal AI, the MMLU professional law and jurisprudence subjects are the relevant signals. For medical applications, check anatomy, clinical knowledge, medical genetics, and professional medicine separately. For financial applications, high school macroeconomics, econometrics, and professional accounting are the meaningful categories. Never choose a model based solely on aggregate MMLU without checking the subjects that match your application domain.

Datasets of subject-level MMLU scores are publicly available. Papers from AI labs routinely report the full per-subject breakdown alongside the aggregate. When subject-level data isn't provided by a lab, it's sometimes possible to find independent evaluations on platforms like the Open LLM Leaderboard (Hugging Face) or HELM (Stanford).

MMLU's Limitations and the Saturation Problem

As models have approached and exceeded the 'human expert' baseline, MMLU's discriminating power at the frontier has declined sharply. In mid-2025, top models score within 2-3 percentage points of each other, a gap within the margin of evaluation variance due to sampling, prompting choices, and hardware differences. This is the benchmark saturation problem: MMLU can still filter out weak models, but it can no longer meaningfully rank the best ones against each other.

Benchmark contamination is a significant concern. MMLU questions circulate widely on the internet, in textbooks, practice test sites, and study forums that were indexed in training data. Models trained on internet text have likely encountered many MMLU questions before evaluation, which may inflate apparent performance. Unlike GPQA, whose questions are deliberately novel, MMLU's sourcing from existing test materials makes contamination control difficult.

A structural limitation: MMLU doesn't evaluate the capabilities that matter most in production. It doesn't test Agent tool-use, Hallucination rates, instruction-following quality, long-context reasoning, or the kind of multi-turn clarification that characterizes real user interactions. A model scoring 93% on MMLU may still produce incorrect outputs in real-world professional workflows, the benchmark is a necessary filter, not a sufficient signal.

MMLU vs. MMLU-Pro: Which to Use

MMLU-Pro is the harder successor benchmark designed specifically to address MMLU's saturation problem. It extends questions to 10 answer choices (from 4), selects harder questions that require multi-step reasoning, and filters out questions answerable by simple pattern matching. MMLU-Pro scores are substantially lower than MMLU scores for the same models, typically 10-15 percentage points, which restores discriminating power at the frontier.

When to use MMLU: checking that a model has broad academic knowledge, evaluating mid-tier models where MMLU is not yet saturated, or comparing against historical baselines established with MMLU. When to use MMLU-Pro: comparing frontier models where MMLU scores are compressed, evaluating models for professional applications requiring genuine reasoning, or building evaluations where you want differentiation across top models. For work published after 2024, MMLU-Pro is increasingly the expected standard.

Current Top Scores

As of mid-2025 on standard MMLU: GPT-5 reaches approximately 92.0%, o3 achieves 91.8%, Gemini 2.5 Pro approximately 91.5%, and Claude Opus 4 approximately 90.4%. The differences between these models are small, well within the margin of statistical variance, confirming that MMLU no longer meaningfully differentiates frontier models from each other. A mid-tier model like Llama 3.1 70B (79.3%) is meaningfully separated from frontier models; frontier models are not meaningfully separated from each other.

On MMLU-Pro, the field spreads out more usefully. Frontier reasoning models score in the 70-80% range; standard instruct-tuned frontier models score 65-75%; capable mid-tier models score 50-65%. This spread makes MMLU-Pro the more useful benchmark for 2025 evaluations. If you're reading a model report and it only cites MMLU, look for MMLU-Pro scores before drawing conclusions about relative capability.