Inference
Epoch AI

OTIS Mock AIME 2024-2025

Evaluated by Epoch AI on their own harness.

ModelsModels with a result.168
Top resultBest result on this test.100.0%GPT-6 Astra
Top-3 spreadPoints from first to third.0.0 pts
YearYear of release.N/A

Result and price

020406080100$0.1$0.3$1$3$10$30$100$300RESULTOUTPUT PRICE PER 1M TOKENS · LOG SCALE

Results

168 results
#ModelResultThe score from the source.
01GPT-6 Astramax effort100.0%
01Claude Fable 5.1max effort100.0%
01GPT-5.5 Proxhigh effort100.0%
01GPT-5.6 Solmax effort100.0%
01GPT-5.5xhigh effort100.0%
01Qwen3.8 Max (0902)xhigh effort100.0%
07Claude Fable 5max effort99.7% ±0.3
07GPT-5.6 Terramax effort99.7% ±0.3
09Qwen3.8 Maxxhigh effort99.4% ±0.4
10Muse Spark 1.3xhigh effort99.2% ±0.6
10Grok 4.6xhigh effort99.2% ±0.5
12Claude Opus 5max effort98.9% ±1.1
12Gemini 3.8 Flashhigh effort98.9% ±0.9
14Claude Opus 4.8max effort98.3% ±1.4
14GPT-5.6 Lunamax effort98.3% ±1.2
16Grok 4.5high effort97.8% ±1.3
17Gemini 3.7 Flashhigh effort97.2% ±1.9
18DeepSeek V4 Pro 0423max effort96.7% ±2.0
19GPT-5.2xhigh effort96.1% ±2.7
20Gemini 3.5 Flashhigh effort95.6% ±2.7
20Gemini 3.1 Prohigh effort95.6% ±3.1
20Qwen3.7 Max95.6% ±3.1
20Kimi K2.7 Code95.6% ±3.1
20Gemini 3 Flashhigh effort95.6% ±3.1
25GPT-5.4xhigh effort95.3% ±3.2
26Gemini 3.6 Flashhigh effort94.2% ±3.1
27GLM-5.3-Flashmax effort93.9% ±2.2
28GLM-5.193.3% ±3.8
28Qwen3.6 Plus93.3% ±3.8
28Grok 4.3 Betahigh effort93.3% ±3.0
31Grok 4.2092.2% ±3.3
32Kimi K2.592.2% ±3.9
33Gemini 3 Pro91.4% ±3.7
33GPT-5high effort91.4% ±3.7
35GLM-5.3max effort91.1% ±3.5
35Claude Opus 4.6max effort91.1% ±4.3
35Qwen 3.6 Max (preview)91.1% ±4.3
38Inkling-Smallxhigh effort90.0% ±4.0
39Muse Spark88.9% ±4.7
40Inklingxhigh effort88.9% ±3.9
40GPT-5.4 minixhigh effort88.9% ±4.7
40GPT-OSS 120Bhigh effort88.9% ±4.5
43GPT-5.1high effort88.6% ±4.0
44GPT-5.4 nanohigh effort87.8% ±3.9
45Claude Opus 4.7max effort86.7% ±5.1
45GPT-5 minihigh effort86.7% ±4.3
45Qwen 3.5 Plus (hosted 397B-A17B)86.7% ±5.1
45Qwen3-235B-A22B (Jul 2025)86.7% ±5.1
49GLM-5.2max effort86.4% ±4.2
50Gemini 2.5 Pro (Jun 2025)84.7% ±4.8
51Qwen 3.5 Flash (hosted 35B-A3B)84.4% ±5.5
51Qwen3.6 Flash84.4% ±5.5
53Grok 484.0% ±5.0
54o3high effort83.9% ±4.4
55GLM-4.783.3% ±4.8
56Kimi K2 Thinking83.1% ±4.8
57Gemma 4 26B A4Bminimal effort82.2% ±5.8
57Qwen3.5 397B A17Bnone effort82.2% ±5.8
59Claude Opus 4.581.7% ±5.0
59o4-mini (high)high effort81.7% ±4.7
61GPT-5 nanohigh effort81.1% ±4.6
62Claude Sonnet 5max effort80.0% ±6.0
62Qwen3.7 Plusnone effort80.0% ±6.0
62GLM-580.0% ±6.0
62Gemini 3.1 Flash-Litehigh effort80.0% ±6.0
66Grok 3 Minihigh effort77.8% ±6.3
66Qwen3.7 Flashnone effort77.8% ±6.3
68o3-minihigh effort76.9% ±5.2
69Claude Sonnet 4.574.4% ±6.3
70Gemma 4 31Bminimal effort73.3% ±6.7
70o1high effort73.3% ±6.7
70Qwen3 Max73.3% ±5.9
73Gemini 2.5 Flash71.9% ±5.5
74Claude Sonnet 4.6max effort71.1% ±6.8
74Gemini 3.5 Flash-Litehigh effort71.1% ±6.8
76Qwen3.5-35B-A3Bnone effort70.0% ±6.0
77Qwen3.6-35B-A3Bnone effort68.9% ±7.0
78DeepSeek V3.268.4% ±7.5
79GPT-5.5 Instant68.1% ±6.2
80Qwen3.6-27Bnone effort66.7% ±7.1
81DeepSeek-R1 (May 2025)66.4% ±4.9
82Qwen3-30B-A3B (Jul 2025)66.3% ±6.0
83Claude Opus 4.164.4% ±7.2
84Qwen3.5 9Bnone effort61.7% ±6.0
85Claude Sonnet 461.1% ±7.0
86Claude Opus 460.0% ±7.4
87QwQ 32B59.2% ±6.3
88Gemini 2.0 Flash Thinking57.8% ±7.4
89Qwen3.5 4Bnone effort55.8% ±6.2
90Grok 3 [Beta]55.6% ±7.5
90DeepSeek R1 Distill Qwen 32B55.6% ±6.2
92DeepSeek-R153.3% ±7.5
93R1 Distill Llama 70B51.4% ±6.1
94Claude Haiku 4.551.3% ±7.1
95GPT-OSS 20Bhigh effort50.8% ±6.7
96DeepSeek R1 Distill Qwen 14B50.6% ±6.0
97Claude 3.7 Sonnet50.0% ±7.5
98O1 Minihigh effort46.9% ±6.5
99GPT-4.1 mini44.7% ±6.1
100GPT-4.138.3% ±6.3
101DeepSeek-V3 (Mar 2025)37.8% ±6.2
101GPT-4.537.8% ±7.3
103Mistral Medium 332.2% ±6.0
104Gemini 2.0 Flash31.1% ±6.3
104o1-preview31.1% ±7.0
106Mistral Small 3.230.3% ±5.8
107Magistral Small 1.030.0% ±5.9
108GPT-4.1 nano28.9% ±6.0
109Magistral Small 1.228.1% ±5.6
110MiniMax M3none effort26.7% ±6.7
111Qwen3 14Bnone effort25.8% ±4.8
112Qwen3 30B A3Bnone effort25.6% ±5.2
113Qwen3 32Bnone effort23.1% ±5.1
114Gemma 3 27B22.5% ±5.3
115Qwen3 8Bnone effort22.2% ±4.5
116DeepSeek R1 Distill Qwen 1.5B21.4% ±5.3
117Llama 4 Maverick20.6% ±4.9
118Qwen Plus 072817.8% ±4.4
119Gemma 3 12B16.7% ±4.2
120Qwen2.5 Max16.1% ±4.1
121DeepSeek V315.8% ±4.3
122Gemini 1.5 Pro14.9% ±5.2
123Phi-413.8% ±3.7
124Grok 211.5% ±3.4
125Gemini 1.5 Flash10.1% ±4.1
126Llama 3.1 405B9.7% ±3.2
127Claude 3.5 Sonnet (October 2024)8.5% ±2.6
128Mistral Large 28.1% ±2.5
129Qwen2.5-72B8.1% ±2.6
129Qwen3 1.7Bnone effort8.1% ±2.8
131Llama 4 Scout7.8% ±3.0
132Gemma 3 4B7.5% ±2.8
133Qwen2.5 32B7.4% ±2.1
134GPT-4o mini6.9% ±2.7
135GPT-4 Turbo (Apr 2024)6.7% ±2.4
136Claude 3.5 Sonnet6.5% ±2.2
137GPT-4o6.3% ±2.3
138Qwen Turbo6.1% ±2.2
139Mistral Small 36.0% ±2.7
140Llama 3.3 70B5.1% ±2.4
141Mistral Small 3.14.9% ±1.5
142Claude 3 Opus4.7% ±2.0
143Gemini 1.5 Flash 8B4.6% ±2.0
144Tulu 3 (Tülu 3) 70B4.4% ±1.8
145Claude 3.5 Haiku4.3% ±1.6
145Llama 3 70B4.3% ±2.2
147Llama 3.1-70B3.6% ±1.7
148Llama 3.2 90B2.6% ±0.9
149Claude 22.5% ±1.5
149Claude 3 Sonnet2.5% ±2.0
149Hermes 2 Theta Llama-3 70B2.5% ±1.2
149Qwen2.5 7B2.5% ±0.9
153GPT-3.5 Turbo2.2% ±2.2
154Mistral Large1.9% ±0.9
154Claude 2.11.9% ±1.0
154Llama 3-8B1.9% ±1.0
157Claude 3 Haiku1.8% ±0.8
158Llama 3.1-8B1.7% ±0.9
159Gemma 2 27B1.4% ±0.9
160Gemini 1.0 Pro1.1% ±0.6
160Gemma 3 1B1.1% ±0.5
160GPT-4 (Jun 2023)1.1% ±0.5
163DeepSeek LLM 67B0.8% ±0.5
164Gemma 2 9B0.6% ±0.4
164GPT-4 (Mar 2023)0.6% ±0.4
164Llama 3.2 1B0.6% ±0.4
167Mistral 7B0.3% ±0.3
168Llama 2-70B0.0%