BIG-Bench Hard
A challenging subset of BIG-Bench tasks where models previously failed to match average human performance
23 challenging tasks from the BIG-Bench suite where prior language models fell below average human rater performance. Tasks include date understanding, logical deduction, tracking shuffled objects, and more. Evaluated with 3-shot chain-of-thought prompting.
Shows open-weight models only. Commercial API models (GPT-4o, Claude, Gemini) are not submitted to the Open LLM Leaderboard — their scores come from provider-reported benchmarks.
| # | Model | Score |
|---|---|---|
| 1 | internlm2 5 7b chat | 57.0% |
| 2 | Qwen2.5 32B Instruct | 56.5% |
| 3 | Qwen2.5 32B | 54.0% |
| 4 | gemma 2 27b it | 49.3% |
| 5 | Phi 3 medium 128k instruct | 48.5% |
| 6 |
| Qwen2.5 14B Instruct |
| 48.4% |
| 7 | gemma 2 9b it | 42.1% |
| 8 | DeepSeek R1 Distill Qwen 14B | 40.7% |
| 9 | Qwen2.5 7B | 35.8% |
| 10 | Qwen2.5 7B Instruct | 34.9% |
| 11 | gemma 2 9b | 34.1% |
| 12 | Llama 3.1 8B Instruct | 29.4% |
| 13 | Qwen2.5 Coder 7B Instruct | 28.9% |
| 14 | phi 2 | 28.0% |
| 15 | Qwen2.5 3B Instruct | 25.8% |
| 16 | EXAONE 3.5 7.8B Instruct | 25.7% |
| 17 | Meta Llama 3 8B | 24.5% |
| 18 | Llama 3.2 3B Instruct | 24.1% |
| 19 | Mistral 7B Instruct v0.2 | 22.9% |
| 20 | gemma 7b | 21.1% |
| 21 | Hermes 3 Llama 3.2 3B | 20.2% |
| 22 | Qwen2.5 1.5B Instruct | 19.8% |
| 23 | gemma 2 2b jpn it | 18.5% |
| 24 | gemma 2 2b it | 18.0% |
| 25 | DeepSeek R1 Distill Qwen 32B | 17.1% |
| 26 | Llama 3.2 3B | 14.2% |
| 27 | Qwen2 1.5B Instruct | 13.7% |
| 28 | gemma 2 2b | 11.8% |
| 29 | deepseek llm 7b chat | 11.3% |
| 30 | deepseek llm 7b base | 9.8% |
| 31 | Llama 3.2 1B Instruct | 8.7% |
| 32 | Qwen2.5 0.5B Instruct | 8.4% |
| 33 | gemma 2b | 8.2% |
| 34 | DeepSeek R1 Distill Qwen 7B | 7.9% |
| 35 | gemma 1.1 2b it | 5.9% |
| 36 | DeepSeek R1 Distill Llama 8B | 5.3% |
| 37 | gemma 2b it | 5.2% |
| 38 | gpt j 6b | 4.9% |
| 39 | falcon 7b instruct | 4.8% |
| 40 | DeepSeek R1 Distill Qwen 1.5B | 4.7% |
| 41 | Llama 3.2 1B | 4.4% |
| 42 | TinyLlama 1.1B Chat v1.0 | 4.0% |
| 43 | gpt2 large | 3.3% |
| 44 | distilgpt2 | 2.8% |
| 45 | gpt2 | 2.7% |
| 46 | pythia 160m | 2.2% |