Inference
Vals AI

MGSM

A multilingual benchmark for mathematical questions.

ModelsModels with a result.70
Top resultBest result on this test.N/A
Top-3 spreadPoints from first to third.N/A
UpdatedDate the source changed the results.9 Jan 2026

Result and cost

020406080100$0.00003$0.0001$0.0003$0.001$0.003$0.01$0.03$0.1RESULTUS DOLLARS PER TASK · LOG SCALE

Results

71 results
#ModelResultThe score from the source.CostUS dollars to run one task.TimeTime to run one task.
—Claude Opus 4.5 Thinking95.2%$0.06312 s
—Claude Opus 4.594.8%$0.0276 s
—Claude Opus 4.1 Thinking94.4%$0.05514 s
—Claude Sonnet 4.5 Thinking94.3%$0.01110 s
—Claude Opus 4.194.2%$0.0278 s
—GPT-5.294.0%$0.01111 s
—Gemini 3 Pro93.9%$0.01813 s
—Claude Opus 493.8%$0.02710 s
—o4-mini (high)93.4%<$0.018 s
—Gemini 3 Flash Preview93.3%<$0.0114 s
—Claude Sonnet 493.0%<$0.016 s
—GPT-5.193.0%<$0.018 s
—Claude 3.7 Sonnet Thinking93.0%$0.02624 s
—GPT-592.8%$0.01420 s
—GPT-5 mini92.6%<$0.0113 s
—Claude 3.5 Sonnet92.6%<$0.014 s
—Qwen3 235B A22B92.5%<$0.0132 s
—Llama4 Maverick Instruct Basic92.4%<$0.013 s
—Claude 3.7 Sonnet92.4%<$0.015 s
—DeepSeek-R192.3%<$0.0110 s
—Qwen3 Max92.1%<$0.0121 s
—Claude Haiku 4.5 Thinking92.1%<$0.0110 s
—GPT-OSS 120B92.0%<$0.016 s
—DeepSeek V391.9%<$0.0116 s
—Qwen3 Max91.8%<$0.0111 s
—o391.7%<$0.017 s
—o3-mini91.3%<$0.0116 s
—Grok 3 [Beta]91.3%<$0.016 s
—Llama 3.3 70B Instruct Turbo91.1%<$0.013 s
—Kimi K2 Instruct90.9%<$0.0112 s
—Grok 490.9%$0.0262 min
—Claude Sonnet 4 Thinking90.9%$0.01514 s
—Grok 4 Fast (Reasoning)90.9%<$0.014 s
—DeepSeek V3.290.9%<$0.019 s
—Mistral Medium90.9%<$0.015 s
—GLM-4.590.8%<$0.0190 s
—GPT-4o90.5%<$0.017 s
—Grok 3 Mini Fast High Reasoning90.4%<$0.0110 s
—Grok 3 Mini Fast Low Reasoning90.4%<$0.017 s
—Kimi K2 Thinking90.1%<$0.0148 s
—Gemini 2.5 Flash (Preview)89.9%<$0.013 s
—Gemini 2.5 Flash Preview Thinking89.8%<$0.016 s
—GLM-4.689.7%<$0.0134 s
—Grok 4.1 Fast (Reasoning)89.5%<$0.0114 s
—Gemini 2.5 Flash Lite Preview89.5%<$0.011 s
—o189.3%$0.05711 s
—GPT-5 nano89.3%<$0.0123 s
—Gemini 1.5 Pro 00289.2%<$0.013 s
—GPT-OSS 20B89.0%<$0.016 s
—Gemini 2.0 Flash 00189.0%<$0.012 s
—Gemini 2.5 Flash Lite Preview Thinking88.4%<$0.015 s
—GLM-4.788.2%<$0.0163 s
—Grok 4 Fast Non Reasoning88.0%<$0.012 s
—Llama 4 Scout 17B 16E Instruct88.0%<$0.013 s
—MiniMax M2.187.9%<$0.0114 s
—GPT-4.1 mini87.8%<$0.013 s
—GPT-4.187.7%<$0.012 s
—Grok 4.1 Fast Non Reasoning87.6%<$0.014 s
—Gemini 1.5 Flash 00286.6%<$0.011 s
—Mistral Large86.3%<$0.018 s
—Magistral Small86.3%<$0.018 s
—GPT-4o mini86.2%<$0.014 s
—Grok 286.1%<$0.016 s
—DeepSeek V3.2 (Thinking)86.0%<$0.0128 s
—Command A85.7%<$0.018 s
—Claude 3.5 Haiku84.6%<$0.014 s
—Mistral Small84.1%<$0.014 s
—Magistral Medium74.6%<$0.0112 s
—Jamba Large 1.671.2%<$0.0110 s
—GPT-4.1 nano69.3%<$0.011 s
—Jamba Mini 1.641.7%<$0.014 s