Carlos E. Jimenez et al.

Software Engineering Benchmark Verified

SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup.

In index
Coding
Best in coding
Source ↗

Models114

Top result96.0%Claude Opus 5

Top-3 spread1.0 pts

Year2024

Result and price

62 of 114 results

Results

114 results

#ModelResult

01Claude Opus 596.0%

02Claude Mythos 595.5%

03Claude Fable 595.0%

04Ember-192.2%

05Claude Opus 4.888.6%

06Claude Opus 4.7 (Adaptive)87.6%

07Ornith-1.5-397B86.0%

08Claude Sonnet 585.2%

09GPT-5.3 Codex85.0%

10Ornith-1.0-397B82.4%

11Inkling 2 Small82.2%

12Beam 1.0 501B-A23B81.6%

13Claude Opus 4.580.9%

14Claude Opus 4.680.8%

15Beam 1.0 501B-A23B NVFP480.7%

16DeepSeek V4 Pro 081380.6%

17MiniMax M380.5%

18Qwen3.7 Max80.4%

19Beam 1.0 501B-A23B FP880.2%

19Kimi K2.680.2%

19Inkling-Small80.2%

22GPT-5.280.0%

23Claude Sonnet 4.679.6%

24DeepSeek V4 Flash 073179.0%

24Ornith-1.5-35B-A3B79.0%

26Qwen3.6 Plus78.8%

27dots3-note Preview78.4%

27BTL-478.4%

29MiMo-V2-Pro78.0%

30GLM-577.8%

31Apodex 1.177.7%

31Qwen3.7 Plus77.7%

33Inkling77.6%

33Mistral Medium 3.5 128B77.6%

35Muse Spark77.4%

36Qwen3.6-27B77.2%

36Claude Sonnet 4.577.2%

38Kimi K2.5 (Reasoning)76.8%

38Kimi K2.576.8%

40Grok 4.2076.7%

41Ripple-65B Preview76.5%

42Qwen3.5 397B76.2%

43Muse Glimmer 30B76.0%

44Ornith-1.0-35B75.6%

45MiMo-V2-Omni74.8%

46Laguna M.174.6%

47Claude 4.1 Opus74.5%

48Hy3 Preview74.4%

48Step 3.5 Flash74.4%

50Flint74.2%

51GLM-4.773.8%

52MAI-Thinking-173.5%

53Qwen3.6-35B-A3B73.4%

53MiMo-V2-Flash73.4%

55Claude Haiku 4.573.3%

56Jais-3-MoVA-36B-A4B72.8%

57Claude 4 Sonnet72.7%

58MAI-Code-1.1-Flash72.6%

59Claude 4 Opus72.5%

60Qwen3.5-27B72.4%

60Jais-3-32B72.4%

62Qwen3.5-122B-A10B72.0%

63Nemotron 3 Ultra71.9%

64Nemotron 3.5 Super VL 120B A12B71.1%

64Nemotron 3.5 Super VL 120B A12B NVFP471.1%

66Laguna XS 2.170.9%

67Grok Code Fast 170.8%

68Solar Pro 470.6%

68Ornith-1.5-9B70.6%

70Solar Open 270.4%

71Jais-3-375B-A23B70.2%

71Jais-3-7B70.2%

73Laguna XS.269.9%

74Qwen3-Coder 480B-A35B69.6%

75Ornith-1.0-9B69.4%

76Qwen3.5-35B-A3B69.2%

77K-EXAONE 2.068.2%

77LongCat-Flash-Lite-Sparse68.2%

79DeepSeek V3.166.0%

80Kimi K265.8%

81Gemini 2.5 Pro63.8%

82Spark-X2.6-4B62.8%

83GPT-OSS 120B62.4%

84Claude 3.7 Sonnet62.3%

85Ternary Bonsai 2 27B60.8%

86GPT-OSS 20B60.7%

87Nemotron 3 Super 100B60.5%

88GLM-4.7-Flash59.2%

89DeepSeek-R1-052857.6%

90Granite 4.2 30B57.0%

91MiniMax M1 80k56.0%

92GPT-4.154.6%

93ZAYA1-74B-Preview53.2%

94Nemotron 3.5 Lightning 30B A3B NVFP452.8%

95LFM2.5-24B-A2B51.0%

96K-Exaone49.4%

97o3-mini49.3%

97LLaDA2.2-flash49.3%

99DeepSeek-R149.2%

100Claude 3.5 Sonnet49.0%

101Granite 4.2 8B47.7%

102Mellum2.1-12B-A2.5B-Thinking47.0%

103Devstral Small46.8%

104MiniCPM5-2B46.4%

105Sarvam 105B45.0%

106LFM3-1.7B43.4%

107DeepSeek V342.0%

108Claude 3.5 Haiku40.6%

109Sarvam 30B34.0%

110Gemini 2.5 Flash-Lite31.6%

111GPT-4.1 mini23.6%

112Gemini 2.0 Flash21.4%

113Gemini 2.0 Flash-Lite12.5%

114Gemini 1.5 Flash9.6%