Inference
EN
EuroEval

EuroEval Italian

How well a model handles Italian, as the unweighted mean of EuroEval's primary metric over its 8 Italian datasets (sentipolc16, multinerd_it, scala_it, squad_it, ilpost_sum, include_it, multiloko_it, winogrande_it). The primary metric differs by task — MCC for classification, knowledge and common-sense reasoning, micro-F1 without MISC for entity recognition, F1 for reading comprehension, chrF++ for summarization, METEOR for simplification — and is always the first of the two the board prints. Only models measured on every dataset are included, so each mean covers the same ground. Most rows are the validation split; some older open-weight rows are the test split, which EuroEval ranks in the same table.

ModelsModels with a result.166
Top resultBest result on this benchmark.78.5%GPT-6.1 Sol
Top-3 spreadPoints from first to third.1.7 pts
YearYear of release.—

Result and price

39 of 174 results

Results

174 results
#ModelResultThe score from the source.
01GPT-6.1 Sol78.5% ±0.3
02GPT-6 Astra77.8% ±0.3
03Gemini 3.7 Flash76.9% ±0.3
04Gemini 3.6 Flash75.1% ±0.3
05Gemini 3 Flash Preview Thinkingthinking effort72.6% ±0.3
06GLM-5.3-Flash72.2% ±0.3
07GPT-5.6 Sol70.1% ±0.4
08GPT-6 Sol69.9% ±0.4
09DeepSeek Flash No Thinkingno-thinking effort67.9% ±0.5
10GPT-5.6 Terra67.8% ±0.4
11Qwen3.6 27B FP867.7% ±0.5
12Gemma 4 31Bgemma-4-31B-it66.8% ±0.4
13Gemini 3.1 Flash Lite Preview66.7% ±0.4
14GPT-5.4 minihigh effort66.2% ±0.5
14Qwen3.8-27B66.2% ±0.5
16Grok 4.1 Fast (Reasoning)66.1% ±0.4
17DeepSeek Flash Lowlow effort66.0% ±0.4
18GPT-5.4 minimedium effort65.8% ±0.4
19Qwen3.5 35B A3B FP865.7% ±0.5
20Claude Sonnet 4.665.3% ±0.5
21Claude Sonnet 4.5 Thinkingthinking effort65.1% ±0.3
22GPT-5.4 Nonenone effort62.7% ±0.5
23GPT-5.4 minilow effort62.5% ±0.5
24GPT-6 Luna62.1% ±0.6
25Gemini 3.5 Flash-Lite61.6% ±0.5
26Claude Sonnet 4.5 No Thinkingno-thinking effort61.1% ±0.3
26Gemma 4 26B A4Bgemma-4-26B-A4B-it61.1% ±0.5
28GPT-5 mini60.1% ±0.3
28GPT-5.260.1% ±0.4
30GPT-5.6 Luna60.0% ±0.5
31Gemma 4 12B IT59.5% ±0.6
32GPT-OSS 120B58.4% ±0.6
33Mistral Small 3.1 24B Base58.2% ±0.5
33NVIDIA Nemotron 3 Nano 30B A3B BF1658.2% ±0.6
35Cogito V1 Preview Qwen 32B57.5% ±0.3
36Mistral Small 3.2 24B57.3% ±0.4
37GLM-4.7-Flash57.1% ±0.5
37GPT-4.157.1% ±0.4
37NVIDIA Nemotron 3 Nano 30B A3B FP857.1% ±0.6
40Qwen3.5 9B56.9% ±0.6
41Mistral Small 3.1 24B56.6% ±0.4
42GPT-5.4 Mini Nonenone effort56.4% ±0.5
43Ministral 3 14B Base56.2% ±0.7
44Nemotron Cascade 2 30B A3B56.1% ±0.5
45GPT-5.4 nanomedium effort55.7% ±0.5
46Qwen3 4B Thinking55.4% ±0.4
46Gemini 2.5 Flash Lite Thinkingthinking effort55.4% ±0.4
48GPT-5.4 nanohigh effort55.1% ±0.5
49GPT-5 nano53.8% ±0.5
49Gemma 3 27B Pt53.8% ±0.6
51Qwen3 32B No Thinkingno-thinking effort53.5% ±0.3
52Qwen3.5-4B53.3% ±0.7
53Claude Haiku 4.553.1% ±0.5
54Cogito V1 Preview Qwen 14B52.6% ±0.3
55GPT-5.4 nanolow effort52.5% ±0.5
56Olmo 3.1 32B Think52.3% ±0.8
57Qwen3.5 9B Base51.3% ±0.7
58GPT-OSS 20Bmedium effort51.0% ±0.3
59Gemma 4 E4B IT50.9% ±0.6
60Ministral 3 8B Base50.5% ±0.8
61Qwen3 14B No Thinkingno-thinking effort50.3% ±0.3
61Gemma 4 31Bgemma-4-31B50.3% ±0.9
63Ministral 3 8B (Reasoning)49.9% ±0.4
64Exaone 4.0 32B49.7% ±0.7
65Gemma 3 12B49.2% ±0.3
66Nesso 4B49.1% ±0.3
67Apertus V1.5 70B48.7% ±0.7
68EuroLLM 22B Instruct 251248.4% ±0.4
69Domyn Small V1.048.1% ±0.6
70Ministral 3 14B Instruct48.0% ±0.4
70Gemini 2.5 Flash Lite No Thinkingno-thinking effort48.0% ±0.4
72Ministral 3 14B (Reasoning)47.8% ±0.5
72GPT-OSS 20Blow effort47.8% ±0.3
74Qwen3 8B No Thinkingno-thinking effort47.7% ±0.3
75Bielik 11B V3.0 Instruct47.6%
76NVIDIA Nemotron 3 Nano 30B A3B Base BF1646.9% ±0.8
77Gemma 3 12B Pt46.8% ±0.5
78Qwen3.5 4B Base46.4% ±0.7
79Gemini 3 Flash Preview No Thinkingno-thinking effort45.6% ±0.5
80Olmo 3.1 32B Instruct45.0% ±0.4
80EuroLLM 9B Instruct 251245.0% ±0.7
82GPT-5.4 Nano Nonenone effort44.2% ±0.6
82Ministral 3 8B Instruct44.2% ±0.6
84EuroLLM 22B 251244.0% ±0.6
85Ministral 3 3B Instruct43.7% ±0.3
86Aya Expanse 8B43.6% ±0.7
87Luciole 23B Instruct 1.143.4% ±0.6
88GPT-OSS 20Bhigh effort43.3% ±0.5
89Qwen3 4B No Thinkingno-thinking effort43.1% ±0.4
90Gemma 4 E2B IT42.3% ±0.8
91Llama 3.1 Tulu 3 8B SFT41.8% ±0.4
92Olmo 3 32B41.5% ±0.5
92Gemma 4 12B41.5% ±0.9
94Gemma 4 26B A4Bgemma-4-26B-A4B41.0% ±0.9
95Llama 3.1 8B Instruct40.9% ±0.3
96Ministral 3 3B (Reasoning)40.7% ±0.4
97Bielik Minitron 7B V3.0 Instruct40.5%
98Cogito V1 Preview Llama 8B40.3% ±0.6
99Llama 3.1 8B38.5% ±0.6
100Qwen3.5 2B38.1% ±0.8
101EuroLLM 9B Instruct37.7% ±0.4
102AMALIA 9B DPO37.6% ±0.6
103Qwen3 1.7B No Thinkingno-thinking effort37.0% ±0.4
104AMALIA 9B SFT36.9% ±0.5
105Gemma 3 4B36.7% ±0.4
106DFM Mimir V1.536.6% ±0.6
107SmolLM3 3B Base36.4% ±0.5
108EuroLLM 9B36.2% ±0.4
109NVIDIA Nemotron 3 Nano 4B BF1635.9% ±0.5
110Gemma 3 4B Pt35.4% ±0.4
111Olmo 3 7B35.2% ±0.4
112NVIDIA Nemotron 3 Nano 4B FP834.9% ±0.8
113Luciole 8B Instruct 1.134.2% ±0.9
114DFM Mimir33.6% ±0.6
115LFM2.5 8B A1B Base33.3% ±0.7
115Apertus V1.1 4B33.3% ±0.9
117SmolLM3 3B No Thinkingno-thinking effort33.2% ±0.3
118Llama 3.2 3B Instruct32.5% ±0.5
119Occiglot 7B Eu531.8% ±0.4
120Qwen3.5 0.8B31.4% ±0.6
121Apertus V1.1 4B Instruct31.1% ±0.7
121Olmo 3 7B Instruct31.1% ±0.4
123Olmo 3 7B Instruct DPO29.5% ±0.6
124Luciole 23B Base29.0% ±1.1
124Cogito V1 Preview Llama 3B29.0% ±0.4
126Teuken 7B Instruct V0.628.5% ±0.6
127Bielik 1.5B V3.0 Instruct28.0% ±0.3
127Occiglot 7B Eu5 Instruct28.0% ±0.5
129Olmo 3 7B Instruct SFT27.0% ±0.7
130Gemma 4 E4B26.4% ±0.9
131Oellm 9B 128K Theta32m V325.9% ±0.8
132Lucie 7B Instruct V1.125.3% ±0.6
133Qwen3.5 0.8B Base24.4% ±0.8
133YuLan Mini Instruct24.4% ±0.5
135Luciole 8B Base23.6% ±1.0
136LFM2.5-8B-A1B20.8% ±1.1
137Gemma 3 1B IT20.7% ±0.4
138Qwen3 0.6B No Thinkingno-thinking effort20.6% ±0.5
139Salamandra 7B Instruct20.4% ±0.5
140Aya 23 8B20.1% ±0.8
141Lucie 7B Instruct Human Data19.6% ±0.9
142LFM2 1.2B19.5% ±0.3
143Gemma 3 1B Pt19.3% ±0.4
144Apertus V1.1 1.5B Instruct18.7% ±0.6
145Salamandra 7B18.6% ±0.5
146Apertus V1.1 1.5B18.2% ±0.9
147Luciole 1B Instruct 1.118.0% ±0.7
148Gemma 4 E2B16.6% ±0.9
149Pleias 1.2B Preview16.4% ±0.6
150LFM2 700M15.4% ±0.3
151Apertus V1.1 0.5B15.3% ±0.8
152LFM2 350M15.2% ±0.4
152Apertus V1.1 0.5B Instruct15.2% ±0.6
154Pleias 3B Preview15.0% ±0.7
155Llama 3.2 1B14.7% ±0.5
156Luciole 1B Base14.1% ±0.8
157Nesso 0.4B Agentic13.1% ±0.8
158Ling Mini 2.012.9% ±0.6
159EuroMoE 2.6B A0.6B Instruct Preview12.8% ±0.5
160Pleias 350M Preview12.4% ±0.5
161SmolLM2 360M12.2% ±0.5
162Falcon H1 Tiny Multilingual 100M Instruct11.8% ±0.2
163Llama 3.2 1B Instruct11.7% ±0.4
164Gemma 3 270M9.9% ±0.7
165Falcon H1 Tiny Multilingual 100M Base9.4% ±0.2
166SmolLM2 360M Instruct8.8% ±0.4
167SmolLM2 135M8.5% ±0.3
168Gemma 3 270M IT7.5% ±0.5
169SmolLM2 135M Instruct6.4% ±0.4
170Zagreus 0.4B Ita6.2% ±0.7
171Baguettotron No Thinkingno-thinking effort4.7% ±0.3
172Marco Mini Global Base3.8% ±0.6
173Monad2.0% ±0.3
174Teuken 7B Base V0.60.0%