SWE-bench and Aider leaderboards for coding models
SWE-bench and Aider are both useful for coding model comparison because they test patching, editing, and multi-language problem solving in different ways.
“Coding ability” sounds like one skill, so one leaderboard should settle which model has most of it. In practice a model can sit near the top of SWE-bench and mid-table on Aider — because the two boards measure different verbs: navigating a real repository to resolve an issue, and obeying an exact edit instruction in a file you never see execute.
Source
SWE-bench Leaderboards — By: SWE-bench team · Type: Benchmark family (viewed Sep 2026)
Aider LLM Leaderboards — By: Paul Gauthier (Aider) · Type: Benchmark (last updated Nov 20 2025)
I keep these two bookmarked as a pair because I don’t want chat scores when picking a coding model — I want evidence it can edit files and finish real software tasks. Each board answers a different half of that question, and together they shortlist models faster than either alone.
Two boards, two verbs, one cost column
SWE-bench tests the agent loop. Its core set takes 2,294 real GitHub issues from 12 Python repositories; the model gets a repo checkout and must produce a patch that passes the project’s hidden tests. The human-filtered Verified subset (500 instances) is the default comparison, now standardised so every model runs in the same minimal harness (mini-SWE-agent, bash-only — a deliberately spartan 100-line agent, so scores reflect the model rather than fancy tooling). The family keeps growing: Multilingual (300 tasks, 42 repos, 9 languages), Multimodal (issues described with screenshots), and the newest pair — CodeClash, which evaluates models as goal-oriented developers, and ProgramBench, which tests writing meaningful software from scratch rather than patching.
The current Verified table is a cost lesson. Claude 4.5 Opus (high) leads at 76.8%, averaging $0.75 per instance. Third-equal is MiniMax M2.5 at 75.8% — for $0.07 per instance, a tenth of the price. Near the bottom of the value curve sits gpt-oss-120b at 26.0% ($0.06): same budget as MiniMax, wildly different outcome. Once a board publishes cost alongside score, “which model is best” quietly becomes “which model is best per dollar at the score I need.”
Aider tests the edit loop. Its polyglot benchmark runs 225 hard Exercism exercises across C++, Go, Java, JavaScript, Python, and Rust, and grades whether the model can follow precise instructions and produce correct edits without a human intervening — the pair-programming experience, not the autonomous-agent one. The top of the current table: gpt-5 (high) 88.0% for $29.08 to run the whole benchmark, o3-pro (high) 84.9% for $146.32. The value pick is again instructive: DeepSeek-V3.2-Exp Reasoner hit 74.2% — fifth — for $1.30, roughly 1% of o3-pro’s bill, while gpt-4.5-preview burned $183.18 to score 44.9%.
Reading them together is the point. The skills differ — repo navigation, multi-file reasoning, and running tests on SWE-bench; exact diff discipline and instruction fidelity on Aider — so a model that ranks well on both is genuinely rounded, and one that doesn’t is telling you where its weakness lives. One caution the boards themselves don’t advertise: Aider’s table is frozen (last updated November 2025) while SWE-bench carries February 2026 entries, so cross-board comparisons silently mix eras. And both are still someone else’s repositories — your codebase’s patterns, test quality, and tooling remain the final exam.
My take
What comparing the two reveals is that the benchmark harness is the skill being measured. mini-SWE-agent’s 100 lines deliberately strip away scaffolding, which is why its scores predict autonomous agents; Aider’s structured diff formats and architect/editor modes measure a conversational discipline. When a lab quotes a coding score, the harness is doing half the work — the same weights can gain or lose double-digit points between the two philosophies. The cost columns are the quiet revolution here: both boards now make price a first-class ranking axis, which kills the “best model” question for good and replaces it with the only one a budget ever asked — what does a resolved instance cost?
Where this is useful
- Shortlisting coding models before spending a weekend on local trials — cross-check both boards, not one.
- Budgeting agent runs: use the per-instance dollar figures to estimate what a hundred automated fixes will cost before committing.
- Judging vendor claims: a lab quoting a coding score without naming the harness and setting is quoting marketing.
- Less useful for greenfield work — patching existing repos (SWE-bench) and guided edits (Aider) both differ from building from scratch; that’s what ProgramBench is starting to measure.
Apply this: next model choice, pick two candidates that clear your score bar on both boards, then run three real tasks from your own backlog against each — issue fix, small feature, refactor — and compare cost and correctness on your code before deciding.
What I want to remember
Remember: The best coding benchmark is still not my repo. Leaderboards shortlist; local tasks decide. And always read the cost column — rank without price is half an answer.