Inference
Vals AI

MedQA

Evaluating language model bias in medical questions.

ModelsModels with a result.91
Top resultBest result on this test.96.5%o1
Top-3 spreadPoints from first to third.0.1 pts
UpdatedDate the source changed the results.16 Apr 2026

Result and cost

020406080100$0.00003$0.0001$0.0003$0.001$0.003$0.01$0.03$0.1$0.3RESULTUS DOLLARS PER TASK · LOG SCALE

Results

92 results
#ModelResultThe score from the source.CostUS dollars to run one task.TimeTime to run one task.
01o196.5%$0.05311 s
02GPT-5.196.4%<$0.0112 s
03Gemini 3.1 Pro Preview96.4%$0.01839 s
04GPT-596.3%$0.02135 s
05GPT-5.496.1%$0.04423 s
06o396.1%<$0.018.5 s
06GPT-5 mini96.1%<$0.0116 s
08Gemini 3 Pro96.0%$0.02922 s
09o4-mini (high)96.0%<$0.016.6 s
10Claude Opus 4.5 Thinking95.9%$0.14832 s
11Gemini 3 Flash Preview95.8%$0.01326 s
12Claude Opus 4.6 (Adaptive)95.4%$0.02730 s
13Qwen3.5 Plus Thinking95.2%<$0.0152 s
14o3-mini94.8%<$0.018.4 s
15Claude Sonnet 4.5 Thinking94.7%$0.03136 s
16Grok 4.20 (Reasoning)94.5%<$0.0112 s
17Kimi K2.5 Thinking94.4%<$0.0142 s
18GLM-5 Thinking94.3%<$0.0181 s
19GPT-5.294.1%$0.02219 s
20DeepSeek V3.2 (Thinking)93.9%<$0.0154 s
21GLM-4.793.7%<$0.0166 s
22Claude Opus 4.1 Thinking93.6%$0.08824 s
23GPT-5 nano93.3%<$0.0120 s
24Claude Opus 4.593.2%$0.04310 s
25Gemini 2.5 Pro Exp 03.2593.1%<$0.0111 s
26o1-preview93.0%$0.10916 s
27Claude Opus 492.9%$0.03812 s
28Claude Sonnet 4 Thinking92.7%$0.02527 s
29Kimi K2 Thinking92.6%<$0.014 min
30Claude Opus 4.192.5%$0.04214 s
31MiniMax M2.592.5%<$0.0134 s
32Grok 492.5%$0.02664 s
33Grok 292.3%<$0.014.1 s
34GLM-4.692.2%<$0.0129 s
35Grok 4.1 Fast (Reasoning)92.1%<$0.0118 s
36Grok 4 Fast (Reasoning)92.1%<$0.015.3 s
37Claude Sonnet 4.692.1%$0.03535 s
38Gemini 2.5 Flash (Preview)91.4%<$0.018.5 s
39GPT-OSS 120B91.4%<$0.0112 s
40GPT-4.191.2%<$0.013.1 s
41Gemini 2.5 Flash Preview Thinking91.2%<$0.0118 s
42MiniMax M2.191.2%<$0.0136 s
43Gemini 2.5 Flash Preview 04.17 Thinking91.0%<$0.018.9 s
44DeepSeek-R190.8%<$0.0142 s
45Qwen3 235B A22B90.6%<$0.0126 s
46Claude Sonnet 490.3%<$0.018.7 s
47Claude 3.7 Sonnet Thinking90.2%$0.01716 s
47O1 Mini90.2%<$0.015.9 s
49Grok 3 Mini Fast High Reasoning90.1%<$0.017.1 s
50GLM-4.590.0%<$0.0160 s
51Magistral Medium89.5%<$0.0117 s
52DeepSeek V3.289.5%<$0.0113 s
53Gemini 2.5 Flash Lite Preview Thinking88.9%<$0.0115 s
54Grok 3 Mini Fast Low Reasoning88.7%<$0.014.9 s
55Meta Llama 3.1 405B Instruct Turbo88.2%<$0.018.7 s
56GPT-4o88.2%<$0.013.4 s
57Qwen3 Maxqwen3-max-preview87.4%<$0.0120 s
58Qwen3 Maxqwen3-max87.4%<$0.010.0 s
59Gemini 2.5 Flash Preview 04.1786.7%<$0.012.3 s
60Meta Llama 3.1 70B Instruct Turbo84.8%<$0.014.8 s
61GPT-4.1 mini84.6%<$0.011.9 s
62Kimi K2 Instruct84.0%<$0.0111 s
63Grok 3 [Beta]83.8%<$0.017.5 s
64Claude 3.5 Sonnet83.2%<$0.015.9 s
65GPT-OSS 20B82.9%<$0.0125 s
66Magistral Small82.4%<$0.018.0 s
67GPT-4 Turbo82.0%$0.0128.4 s
68Gemini 2.0 Flash 00181.5%<$0.011.9 s
69DeepSeek V381.5%<$0.0113 s
70Command A80.5%<$0.012.9 s
71Gemini 2.5 Flash Lite Preview80.3%<$0.015.1 s
72Claude Haiku 4.5 Thinking79.6%$0.01424 s
73Mistral Large79.2%<$0.0117 s
74Mistral Medium78.2%<$0.017.3 s
75Qwen2.5 72B Instruct Turbo77.4%<$0.015.6 s
76Gemini 1.5 Pro 00276.5%<$0.015.9 s
77Grok 4.1 Fast Non Reasoning76.0%<$0.016.6 s
78Grok 4 Fast Non Reasoning75.4%<$0.012.4 s
79GPT-4o mini72.4%<$0.012.3 s
80GPT-4.1 nano68.2%<$0.011.4 s
81Jamba 1.5 Large68.1%<$0.016.0 s
82Mistral Small63.0%<$0.013.6 s
83Meta Llama 3.1 8B Instruct Turbo62.6%<$0.012.4 s
84Mixtral 8x22B Instruct v0.162.1%<$0.018.9 s
85GPT-3.5 Turbo58.5%<$0.011.6 s
86Jamba 1.5 Mini55.2%<$0.011.1 s
87Mixtral 8x7B V0.153.2%<$0.013.5 s
88Jamba Mini 1.652.5%<$0.012.2 s
89Llama 4 Scout 17B 16E Instruct50.9%<$0.014.8 s
90Jamba Large 1.650.7%<$0.017.3 s
91Llama4 Maverick Instruct Basic43.3%<$0.017.8 s
92Command R Plus2.7%<$0.016.1 s