Claude 3.5 Haiku vs Command A

Change models
Command ACohere
IndexAll categories, out of 100.
24.1 ±3.4 best
23.9 ±9.2
GapIndex points below the top.
Top
−0.2 · no clear gap
CoverageIndex weight that has results.
100% best
65%
SpeedOutput tokens per second.
—
—
EloLMArena rating and rank.
1255
1331 best
ReleasedMonth of release.
2024-10
—
Input / 1MUS dollars per 1M input tokens.
—
—
Output / 1MUS dollars per 1M output tokens.
—
—
1K chats2K tokens in, 500 out, each.
—
—
1K documents20K tokens in, 1K out, each.
—
—
1K agent tasks200K tokens in, 20K out, each.
—
—
ContextMaximum tokens in one request.
200K
—
AgenticMulti-step tasks with tools.
26.6 best
25.3
CodingCode writing and repair.
28.0 best
12.7
ReasoningLogic problems and puzzles.
31.4
—
MultimodalTasks with images and text.
14.5
—
KnowledgeFacts and expert knowledge.
24.4
26.7 best
MultilingualTasks in many languages.
31.1
—
InstructionTasks with strict prompt rules.
32.6
—
MathMath problems.
16.2
23.0 best

Benchmarks

9 shared
Only Claude 3.5 Haiku: 24Only Command A: 6
Claude 3.5 Haiku
Command A
70.3%123 / 149
79.7% best93 / 149
TaxEval v2Finance
57.4%135 / 143
61.4% best123 / 143
41.9% best127 / 142
35.1%136 / 142
37.9%129 / 135
48.5% best122 / 135
MMLU ProKnowledge
64.1%129 / 135
69.2% best122 / 135
CorpFin v2Finance
50.8% best106 / 131
46.0%114 / 131
AIMEMath
3.3%93 / 95
13.3% best85 / 95
MGSMMultilingual
84.6%68 / 74
85.7% best66 / 74
64.2%56 / 59
76.2% best44 / 59