VulcanBench v3

An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.

ModelsModels with a result.14
Top resultBest result on this test.89.9%Grok 4.5
Top-3 spreadPoints from first to third.1.5 pts
YearYear of release.2026

Result and price

020406080100$0.1$0.3$1$3$10$30$100RESULTOUTPUT PRICE PER 1M TOKENS · LOG SCALE

Results

14 results
#ModelResultThe score from the source.
01Grok 4.589.9%
04Grok 4.687.0%
12GLM-5.378.3%
14Kimi K373.7%