Inference
Carlos E. Jimenez et al.

Software Engineering Benchmark Verified

A curated, human-verified subset of SWE-bench that tests models on resolving real GitHub issues from popular open-source Python repositories like Django, Flask, and scikit-learn.

ModelsModels with a result.75
Top resultBest result on this test.96.0%Claude Opus 5
Top-3 spreadPoints from first to third.1.0 pts
YearYear of release.2024

Result and price

020406080100$0.1$0.3$1$3$10$30$100RESULTOUTPUT PRICE PER 1M TOKENS · LOG SCALE

Results

75 results
#ModelResultThe score from the source.
01Claude Opus 596.0%
02Claude Mythos 595.5%
03Claude Fable 595.0%
04Claude Opus 4.888.6%
05Claude Opus 4.7 (Adaptive)87.6%
06Ornith-1.5-397B86.0%
07Claude Sonnet 585.2%
08GPT-5.3 Codex85.0%
09Ornith-1.0-397B82.4%
10Claude Opus 4.580.9%
11Claude Opus 4.680.8%
12DeepSeek V4 Pro 081380.6%
13MiniMax M380.5%
14Qwen3.7 Max80.4%
15Kimi K2.680.2%
15Inkling-Small80.2%
17GPT-5.280.0%
18Claude Sonnet 4.679.6%
19DeepSeek V4 Flash 073179.0%
19Ornith-1.5-35B-A3B79.0%
21Qwen3.6 Plus78.8%
22dots3-note Preview78.4%
22BTL-478.4%
24MiMo-V2-Pro78.0%
25GLM-577.8%
26Apodex 1.177.7%
26Qwen3.7 Plus77.7%
28Inkling77.6%
28Mistral Medium 3.5 128B77.6%
30Muse Spark77.4%
31Qwen3.6-27B77.2%
31Claude Sonnet 4.577.2%
33Kimi K2.5 (Reasoning)76.8%
33Kimi K2.576.8%
35Grok 4.2076.7%
36Qwen3.5 397B76.2%
37Muse Glimmer 30B76.0%
38Ornith-1.0-35B75.6%
39MiMo-V2-Omni74.8%
40Laguna M.174.6%
41Claude 4.1 Opus74.5%
42Hy3 Preview74.4%
43GLM-4.773.8%
44MAI-Thinking-173.5%
45Qwen3.6-35B-A3B73.4%
45MiMo-V2-Flash73.4%
47Claude Haiku 4.573.3%
48Claude 4 Sonnet72.7%
49MAI-Code-1.1-Flash72.6%
50Qwen3.5-27B72.4%
51Qwen3.5-122B-A10B72.0%
52Nemotron 3 Ultra71.9%
53Laguna XS 2.170.9%
54Grok Code Fast 170.8%
55Solar Pro 470.6%
55Ornith-1.5-9B70.6%
57Solar Open 270.4%
58Laguna XS.269.9%
59Ornith-1.0-9B69.4%
60Qwen3.5-35B-A3B69.2%
61K-EXAONE 2.068.2%
61LongCat-Flash-Lite-Sparse68.2%
63Gemini 2.5 Pro63.8%
64Ternary Bonsai 2 27B60.8%
65Granite 4.2 30B57.0%
66GPT-4.154.6%
67ZAYA1-74B-Preview53.2%
68Nemotron 3.5 Lightning 30B A3B NVFP452.8%
69o3-mini49.3%
70LLaDA2.2-flash49.3%
71Claude 3.5 Sonnet49.0%
72Granite 4.2 8B47.7%
73MiniCPM5-2B46.4%
74DeepSeek V342.0%
75GPT-4.1 mini23.6%