Mirac Suzgun et al.

BIG-Bench Hard

A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.

Not in index
Reasoning
Source ↗

Models8

Results

8 results

#ModelResult

—Claude 3.5 Haiku86.6%

—Soofi S 30B-A3B78.8%

—MiniCPM5-1B71.9%

—Phi-3.5-mini69.0%

—Granite-4.0-1B60.4%

—Granite-4.0-H-1B59.7%

—Granite-4.0-H-350M33.3%

—Granite-4.0-350M33.1%