Inference
Vals AI

SWE-bench

Solving production software engineering tasks

ModelsModels with a result.84
Top resultBest result on this test.97.0%Claude Opus 5
Top-3 spreadPoints from first to third.1.4 pts
UpdatedDate the source changed the results.1 Sept 2026

Result and cost

020406080100$0.003$0.01$0.03$0.1$0.3$1$3$10RESULTUS DOLLARS PER TASK · LOG SCALE

Results

84 results
#ModelResultThe score from the source.CostUS dollars to run one task.TimeTime to run one task.
01Claude Opus 597.0%$1.2910 min
02GPT-5.6 Sol96.2%$1.153 min
03Grok 4.695.6%$0.78510 min
04GPT-5.6 Terra95.4%$0.4013 min
04GLM-5.395.4%$0.33814 min
06Claude Fable 595.0%$2.056 min
07GPT-5.6 Luna93.0%$0.0433 min
08GLM-5.3-Flash92.0%$0.01864 min
09DeepSeek V4 Flash 042388.8%<$0.013 min
10Claude Opus 4.888.6%$1.929 min
11DeepSeek V4 Pro 042386.9%$0.1034 min
12Muse Spark 1.286.6%$0.5548 min
12Grok 4.586.6%$0.543 min
14Qwen3.8-27B86.0%$1.0810 min
15Claude Opus 4.8 Claude CodeClaude Code85.8%$0.6722 min
16Qwen3.8 Max85.6%$1.1342 min
17GLM-5.282.8%$0.7111 min
18GPT-5.582.6%$1.367 min
19Inkling-Small82.2%$0.1344 min
20Muse Spark 1.182.0%$0.3523 min
20Claude Opus 4.782.0%$2.427 min
22Gemini 3.7 Flash80.8%$1.446 min
23Gemini 3.8 Flash80.0%$2.1911 min
24Claude Sonnet 579.6%$1.4916 min
24Gemini 3.6 Flash79.6%$1.1913 min
24Composer 2.5Cursor CLI79.6%—0 s
27Gemini 3.5 Flash78.8%$0.9544 min
27Gemini 3.1 Pro Preview78.8%$0.7835 min
29GPT-5.478.2%$0.8015 min
29Claude Opus 4.6 (Adaptive)78.2%$1.226 min
29Kimi K2.7 Code78.2%$0.26515 min
32GPT-5.3 Codex78.0%$0.4554 min
33Inkling77.6%$0.5867 min
34Claude Sonnet 4.677.4%$1.39 min
35Gemini 3 Pro76.4%$0.7727 min
35GLM-5.176.4%$0.4579 min
35Claude Opus 4.5 Thinking76.4%$1.055 min
35GPT-5.5 CodexCodex76.4%$0.6522 min
39GPT-5.5 FactoryFactory76.2%$1.137 min
40GPT-5.275.8%$1.4511 min
41Gemini 3 Flash Preview75.0%$0.285 min
41MiniMax M375.0%$0.41612 min
41Gemini 3.5 Flash-Lite75.0%$0.3097 min
44MiniMax M2.174.8%$0.4396 min
45Muse Spark74.4%$0.4448 min
46MiniMax M2.574.2%$0.4577 min
47MiMo-V2.5-Pro74.0%$0.0238 min
48MiniMax M2.773.8%$0.46615 min
49Qwen3.6 Plus73.4%$0.8697 min
50GPT-5.4 mini73.0%$0.5115 min
51Qwen 3.6 Max (preview)72.8%$1.5718 min
52GPT-5.2-Codex72.4%$0.3123 min
53Grok 4.20 (Reasoning)72.2%$0.5162 min
54Grok 4.371.4%$0.2232 min
54GLM-5 Thinking71.4%$0.4499 min
56Qwen3.5 Plus Thinking71.2%$0.9648 min
57MiMo-V2.571.0%$0.014 min
58Claude Sonnet 4.5 Thinking70.0%$0.8076 min
58Kimi K2.5 Thinking70.0%$0.1686 min
58Qwen3.6-27B70.0%$0.91417 min
61GPT-5.169.8%$0.7138 min
61GPT-5.4 nano69.8%$0.0984 min
63GLM-4.769.4%$0.2046 min
64GPT-569.0%$0.274 min
65Qwen3.7 Max68.8%$3.2316 min
66DeepSeek V3.2 (Thinking)67.6%$0.6739 min
67Claude Haiku 4.5 Thinking66.6%$0.3664 min
68Mistral Medium 3.566.4%$419 min
69Ling 3.0 Flash65.2%$0.0323 min
70Qwen3.5 Flash64.4%$0.2636 min
71Gemini 3.1 Flash Lite Preview62.8%$0.1052 min
71Devstral62.8%$0.7474 min
73GPT-5 mini60.8%$0.053 min
74Kimi K2 Thinking60.2%$0.36627 min
75Grok 457.8%$1.4210 min
76Laguna M.157.6%$0.5215 min
77Laguna XS.255.2%$0.175 min
78Gemini 2.5 Pro54.4%$0.3523 min
79Nemotron Lightning 3p5 30B A3B52.8%$0.03215 min
80Grok 4 Fast (Reasoning)45.4%$0.0512 min
81Grok 4.1 Fast (Reasoning)41.4%$0.0282 min
81Mistral Large41.4%$0.3223 min
83GPT-OSS 120B33.6%$0.1724 min
84Command A7.8%$0.214 min