HumanEval: OpenAI's Python Coding Benchmark Explained

What HumanEval Tests

HumanEval is a coding benchmark from OpenAI consisting of 164 hand-written Python programming problems. Each problem provides a function signature and docstring describing what the function should do; the model must complete the implementation. Correctness is verified by running the accompanying unit tests against the generated code, there's no ambiguity, no human judgment involved. Either the code passes the tests or it doesn't.

Problems range from simple string manipulation and list operations to more involved algorithmic tasks: sorting, searching, recursion, basic number theory. The benchmark specifically targets function-level completion, given a specification, write a working function, which is a clear, measurable, and practically relevant capability. It does not test multi-file reasoning, debugging, test writing, or the kind of open-ended software engineering that characterizes real development work.

How HumanEval Was Built

HumanEval was introduced by OpenAI's Mark Chen and colleagues in the 2021 paper 'Evaluating Large Language Models Trained on Code,' released alongside the Codex model. The 164 problems were written by hand by OpenAI researchers rather than scraped from existing sources, this was intentional, to avoid contaminating training data for models trained on GitHub. Each problem was validated to confirm that the unit tests actually covered meaningful correctness criteria and that the problems had unique correct solutions.

The benchmark's hand-crafted, non-contaminated origin gave it credibility that scraped benchmarks lacked. In 2021, it was genuinely hard: Codex (12B parameters) achieved only 28.8% pass@1. The progression from 28.8% in 2021 to 90%+ by 2024 charts the dramatic improvement in code generation over just three years, one of the clearest trajectories in LLM capability research.

Pass@k: The Evaluation Metric

HumanEval uses the pass@k metric: what fraction of problems does the model solve correctly if given k independently sampled attempts? Pass@1 (one attempt per problem) measures whether the model's first-try solution is correct. Pass@10 (ten attempts) measures whether at least one of ten sampled solutions passes, a much more lenient criterion. Pass@100 is even more lenient. All three are reported in the literature; comparing them across papers without checking which k is being used produces misleading conclusions.

Pass@1 is the most practically relevant metric because most production applications need the first response to be correct. Generating 10 solutions and selecting the best one is computationally expensive and requires a separate selection mechanism (either a discriminator or running tests, which may not be available in production). Pass@10 is useful for research purposes and for understanding the model's solution variance, a high pass@10 but low pass@1 suggests the model has high variance and could benefit from techniques like self-consistency or best-of-N sampling.

There's also a subtlety in how pass@k is computed: naively sampling k solutions and checking if any pass produces a biased estimate. The statistically correct approach uses an unbiased estimator from the Chen et al. paper, sampling n > k solutions and computing the expected pass@k. This detail matters for comparing numbers across papers that may use different computation methods.

Current Leaderboard

As of mid-2025: o3 achieves 99%+ pass@1 on HumanEval, o4-mini reaches 97.9%, GPT-5 approximately 95%, Claude Sonnet 4.6 around 92%, and GPT-4o at 90.2%. The absolute scores are now so high, nearly every frontier model exceeds 90%, that HumanEval has essentially lost discriminating power among top models. A model scoring 97% vs one scoring 93% on HumanEval is not meaningfully different in practical coding performance.

The benchmark remains useful for two purposes: establishing a floor (any serious coding model should exceed 80%), and evaluating mid-tier and small models where HumanEval scores haven't yet saturated. For models like Llama 3.1 8B (72.6%) or Mistral 7B (~40%), HumanEval meaningfully differentiates quality. For the top 10 frontier models, the scores are too compressed to be informative.

MBPP and LiveCodeBench: The Complementary Benchmarks

MBPP (Mostly Basic Python Problems) is a complementary coding benchmark of 374 crowd-sourced Python problems. It's somewhat broader than HumanEval and also saturated at the frontier. The combination of HumanEval and MBPP, often reported together as 'HumanEval + MBPP', provides slightly better coverage than either alone, but both face the same fundamental saturation issue among frontier models.

LiveCodeBench is the living replacement: it continuously adds fresh problems from competitive programming platforms, Codeforces, LeetCode, AtCoder, with a rolling window to prevent contamination. Because the problems are genuinely new and haven't circulated online, contamination is minimal and scores reflect real coding ability. Frontier model scores on LiveCodeBench are substantially lower than HumanEval, typically 60-80% for frontier models, dropping to 40-60% for the harder problems, making it the more discriminating benchmark in 2025.

What HumanEval Cannot Tell You

HumanEval measures isolated function generation from a specification, a very specific sub-task of software engineering. It doesn't test: understanding large multi-file codebases, debugging existing code with subtle bugs, writing test suites, explaining code, refactoring for readability or performance, or the iterative back-and-forth of real development where requirements evolve. A model scoring 95% on HumanEval may still struggle significantly with realistic engineering tasks.

The gap is not hypothetical. When researchers compare HumanEval scores with SWE-Bench performance, where models must resolve actual GitHub issues, the correlation is imperfect. A model that excels at generating clean solutions to well-specified small problems doesn't automatically excel at navigating ambiguous requirements, understanding legacy code, or producing patches that pass a complex existing test suite. HumanEval is a necessary qualification bar, not a sufficient predictor of engineering capability.

Which Coding Benchmark to Use

For evaluating code generation from specifications: HumanEval + MBPP as a floor check, LiveCodeBench for meaningful differentiation. For real-world software engineering capability: SWE-Bench Verified. For code editing in existing repos: the Aider benchmark. For competitive programming capability: LiveCodeBench or Codeforces evaluation sets. No single benchmark captures all of coding ability, a complete coding evaluation uses at least two of these in combination.

For practitioners building AI coding tools, SWE-Bench Verified is the most predictive available benchmark for agentic software engineering, and LiveCodeBench is the most predictive for algorithmic coding assistance. HumanEval scores should be consulted as a baseline sanity check but should not be the primary differentiator when comparing frontier models. If a model report only cites HumanEval, look for additional coding benchmarks before drawing conclusions.