SWE-Bench
A benchmark that measures an AI agent's ability to resolve real GitHub issues from open-source repositories, a rigorous test of practical software engineering capability.
SWE-Bench (Software Engineering Benchmark) was introduced by Princeton NLP in 2024. It consists of 2,294 real GitHub issues from popular Python open-source repositories like Django, scikit-learn, and Flask. Each issue comes with a codebase snapshot, the issue description, and a test suite that validates the correct fix. An AI agent must understand the issue, locate the relevant code, implement a fix, and pass the tests.
SWE-Bench is widely considered the most practically meaningful benchmark for coding AI because it tests the full software engineering workflow, not just generating code from scratch, but understanding existing codebases, reproducing bugs, and writing targeted fixes. Performance on SWE-Bench correlates more closely with real-world engineering utility than benchmarks like HumanEval.
The SWE-Bench Verified subset (500 human-validated problems) is used most often for direct model comparisons. Scores have improved dramatically: early agents scored under 5%, while frontier agents in 2025 exceed 50% on the verified set. Aider, Devin, and Claude Code are among the tools evaluated on SWE-Bench.
Read next
Devin: The Autonomous AI Software Engineer
Devin by Cognition is the most ambitious AI coding agent, an autonomous software engineer that can browse the web, use tools, and complete multi-hour engineering tasks independently.
Aider: AI Pair Programming in Your Terminal
Aider is an open-source AI coding assistant that runs in your terminal, edits your local files, and commits changes to git. The power tool for developers who live in the command line.
Cursor: The AI-First Code Editor Review
A deep-dive into Cursor, the VS Code fork that put AI at the center of the editing experience. Who it's for, what makes it different, and whether it's worth the switch.