NL2Repo

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

In index
Coding
Best in coding

Models36

Top result68.2%DeepSeek V4.1 Flash 1017

Top-3 spread6.7 pts

Year—

Result and price

25 of 36 results

Results

36 results

#ModelResult

01DeepSeek V4.1 Flash 101768.2%

02ERNIE 5.5 Flash Preview64.8%

03DeepSeek V4 Pro 081361.5%

04MiniMax M3.1 Flash61.2%

05LongCat-2.560.5%

06Ornith-1.5-397B59.5%

07Hy4 preview58.9%

08GLM-5.358.0%

09GLM-5.3-Flash56.3%

10Qwen3.8 Max55.9%

11DeepSeek V4 Flash 073154.2%

12dots3-note Preview49.8%

13Qwen3.8-Omni-Flash48.9%

13GLM-5.248.9%

15Ornith-1.0-397B48.2%

16Qwen3.8-Flash-Next48.1%

17Qwen3.7 Max47.2%

18Seed 2.1 Pro47.0%

19Ornith-1.5-35B-A3B46.2%

20Seed 2.1 Turbo43.7%

21Claude Opus 4.543.2%

22Qwen 3.6 Max (preview)42.9%

23GLM-5.142.7%

24Qwen3.8-27B42.3%

25MiniMax M342.1%

26Qwen3.7 Plus41.1%

27MiniMax M2.739.8%

28Qwen3.6 Plus37.9%

29Qwen3.6-27B36.2%

30GLM-535.9%

31Ornith-1.0-35B34.6%

32Ornith-1.5-9B32.4%

33Qwen3.5 397B32.2%

34Kimi K2.532.0%

35Qwen3.6-35B-A3B29.4%

36Ornith-1.0-9B27.2%