Published benchmark scores
Published benchmark scores for the models we route, GPQA Diamond, SWE-bench, HLE and more, each linked to its source. Reported by makers, not measured by us.
What each model’s maker published for the benchmarks below, one score per row, each linked to the page it comes from. We don’t run these tests or change the numbers. A maker’s own score is not an independent measurement: effort, tools and test harnesses differ between makers, so compare with care.
GPQA Diamond
Graduate-level biology, physics and chemistry questions written to resist web search; the hardest 198. GPQA Diamond
Humanity’s Last Exam
Expert-written questions at the frontier of human knowledge, across many subjects. Humanity’s Last Exam
ARC-AGI-2
Visual puzzles that test learning a new rule from a few examples. ARC-AGI-2
| Model | Score | Setting | Source |
|---|---|---|---|
| Claude Opus 5 | 90.42% | max effort, ARC Prize verified, semi-private | Maker, 2026-07-24 |
| Anthropic: Claude Fable 5.1 | 90% | max effort, ARC Prize verified, semi-private | Maker, 2026-09-01 |
| google/gemini-3.1-pro | 77.1% | ARC Prize Verified, thinking high | Maker, 2026-02-19 |
| Anthropic: Claude Opus 4.7 | 75.83% | max effort | Maker, 2026-04-16 |
| Anthropic: Claude Opus 4.8 | 72.1% | max effort | Maker, 2026-07-24 |
| Google: Gemini 3.5 Flash | 72.1% | — | Maker, 2026-05-19 |
| Anthropic: Claude Sonnet 4.6 | 60.42% | high effort, 120k thinking tokens | Maker, 2026-02-17 |
| Anthropic: Claude Sonnet 4.6 | 58.3% | max effort, Verified | Maker, 2026-02-17 |
MMLU-Pro
Multiple-choice questions across 14 subjects, with ten options each. MMLU-Pro
AIME 2025
The 2025 American Invitational Mathematics Examination problems. AIME 2025
| Model | Score | Setting | Source |
|---|---|---|---|
| openai/gpt-oss-120b | 97.9% | with tools, reasoning high | Maker, 2025-08-05 |
| Anthropic: Claude Sonnet 4.6 | 95.6% | no tools, max effort | Maker, 2026-02-17 |
| openai/gpt-oss-120b | 92.5% | no tools, reasoning high | Maker, 2025-08-05 |
| DeepSeek R1 Distill Qwen3 8B 0528 | 76.3% | — | Maker, 2026-11-03 |
| Kimi K2 Instruct | 49.5% | Avg@64 | Maker, 2026-11-03 |
AIME 2026
The 2026 American Invitational Mathematics Examination problems. AIME 2026
| Model | Score | Setting | Source |
|---|---|---|---|
| Z.ai: GLM 5.2 | 99.2% | — | Maker, 2026-11-03 |
| MoonshotAI: Kimi K2.6 | 96.4% | thinking | Maker, 2026-11-03 |
| Z.ai: GLM 5.1 | 95.3% | — | Maker, 2026-11-03 |
| Qwen: Qwen3.6 27B | 94.1% | full AIME 2026 I & II | Maker, 2026-11-03 |
| Qwen: Qwen3.6 35B A3B | 92.7% | full AIME 2026 I & II | Maker, 2026-11-03 |
| Google: Gemma 4 31B | 89.2% | no tools | Maker, 2026-07-30 |
| Google: Gemma 4 26B A4B | 88.3% | no tools | Maker, 2026-07-30 |
| google/gemma-4-E4B-it | 42.5% | no tools | Maker, 2026-07-30 |
SWE-bench Verified
Real GitHub issues in Python projects, fixed so the project’s tests pass; a human-checked set of 500. SWE-bench Verified
SWE-Bench Pro
Harder, longer software-engineering tasks across several languages. SWE-Bench Pro
LiveCodeBench
Competitive-programming problems collected after a model’s training cut-off. LiveCodeBench
Terminal-Bench 2.0
Tasks completed by working in a real command-line terminal. Terminal-Bench 2.0
| Model | Score | Setting | Source |
|---|---|---|---|
| Anthropic: Claude Opus 4.7 | 69.4% | Terminus-2 harness, thinking disabled | Maker, 2026-04-16 |
| Z.ai: GLM 5.1 | 69% | Claude Code harness | Maker, 2026-11-03 |
| google/gemini-3.1-pro | 68.5% | Terminus-2 harness, thinking high | Maker, 2026-02-19 |
| DeepSeek: DeepSeek V4 Pro 0423 | 67.9% | think max | Maker, 2026-11-03 |
| MoonshotAI: Kimi K2.6 | 66.7% | thinking; Terminus-2 | Maker, 2026-11-03 |
| Z.ai: GLM 5.1 | 63.5% | Terminus-2 | Maker, 2026-11-03 |
| Qwen: Qwen3.6 27B | 59.3% | Terminus-2, avg of 5 runs | Maker, 2026-11-03 |
| Anthropic: Claude Sonnet 4.6 | 59.1% | max effort, no thinking | Maker, 2026-02-17 |
| MiniMax: MiniMax M2.7 | 57% | — | Maker, 2026-11-03 |
| DeepSeek: DeepSeek V4 Flash 0423 | 56.9% | think max | Maker, 2026-11-03 |
| Qwen: Qwen3.6 35B A3B | 51.5% | Terminus-2, avg of 5 runs | Maker, 2026-11-03 |
| Qwen: Qwen3.5-122B-A10B | 49.4% | — | Maker, 2026-11-03 |
| Qwen: Qwen3.5-27B | 41.6% | — | Maker, 2026-11-03 |
| Qwen: Qwen3.5-35B-A3B | 40.5% | — | Maker, 2026-11-03 |
τ²-bench
Customer-service conversations where an agent uses tools under a company’s policy, scored per domain. τ²-bench
| Model | Score | Setting | Source |
|---|---|---|---|
| google/gemini-3.1-pro | 99.3% | telecom, thinking high | Maker, 2026-02-19 |
| Anthropic: Claude Sonnet 4.6 | 97.9% | telecom | Maker, 2026-02-17 |
| Anthropic: Claude Sonnet 4.6 | 91.7% | retail | Maker, 2026-02-17 |
| google/gemini-3.1-pro | 90.8% | retail, thinking high | Maker, 2026-02-19 |
| Kimi K2 Instruct | 70.6% | retail; Avg@4 | Maker, 2026-11-03 |
| Kimi K2 Instruct | 65.8% | telecom; Avg@4 | Maker, 2026-11-03 |
| Kimi K2 Instruct | 56.5% | airline; Avg@4 | Maker, 2026-11-03 |
BrowseComp
Hard-to-find facts that take many web searches to locate. BrowseComp
OSWorld-Verified
Tasks on a real computer desktop, through its screen, mouse and keyboard. OSWorld-Verified
| Model | Score | Setting | Source |
|---|---|---|---|
| Anthropic: Claude Fable 5 | 85% | max effort | Maker, 2026-06-09 |
| MoonshotAI: Kimi K3 | 84.8% | max | Maker, 2026-11-03 |
| Qwen: Qwen3.8 27B | 84.3% | — | Maker, 2026-11-03 |
| Anthropic: Claude Opus 4.8 | 83.4% | first-attempt, avg of 5 runs | Maker, 2026-05-28 |
| Google: Gemini 3.6 Flash | 83% | — | Maker, 2026-07-21 |
| Anthropic: Claude Sonnet 5 | 81.2% | first-attempt, avg of 5 runs | Maker, 2026-06-30 |
| Google: Gemini 3.5 Flash | 78.4% | — | Maker, 2026-05-19 |
| Anthropic: Claude Opus 4.7 | 78% | first-attempt, avg of 5 runs | Maker, 2026-04-16 |
| google/gemini-3.1-pro | 76.2% | — | Maker, 2026-05-19 |
| Google: Gemini 3.5 Flash Lite | 74% | — | Maker, 2026-07-21 |
| Qwen: Qwen3.7 Plus | 73.3% | — | Maker, 2026-11-03 |
| MoonshotAI: Kimi K2.6 | 73.1% | thinking | Maker, 2026-11-03 |
| Anthropic: Claude Sonnet 4.6 | 72.5% | first-attempt | Maker, 2026-02-17 |
| Qwen: Qwen3.6 27B | 63.9% | — | Maker, 2026-11-03 |
| Qwen: Qwen3.5-122B-A10B | 58% | — | Maker, 2026-11-03 |
| Qwen: Qwen3.5-27B | 56.2% | — | Maker, 2026-11-03 |
| Qwen: Qwen3.5-35B-A3B | 54.5% | — | Maker, 2026-11-03 |
| Google: Gemini 3.1 Flash Lite | 54.3% | — | Maker, 2026-07-21 |
| Qwen: Qwen3.5-9B | 41.8% | — | Maker, 2026-11-03 |
MMMU
College-level questions that need reading images, charts and diagrams. MMMU
Scores are percentages; higher is better. Only models we can route are listed. A model with no published score has no row, and a missing score is not a zero. A setting shows how the maker ran the test, such as with or without tools.
References
Every score on this page was read from one of these 39 pages, published by the model’s maker. Each benchmark’s own page is listed beside them.
Where the scores come from
- ai.google.dev/gemma/docs/core/model_card_4: Scores for Google: Gemma 4 26B A4B, Google: Gemma 4 31B, google/gemma-4-E4B-it; as of 2026-07-30
- anthropic.com/claude-fable-5-1-mythos-5-1-system-card: Scores for Anthropic: Claude Fable 5.1; as of 2026-09-01
- anthropic.com/claude-fable-5-mythos-5-system-card: Scores for Anthropic: Claude Fable 5; as of 2026-06-09
- anthropic.com/claude-opus-4-7-system-card: Scores for Anthropic: Claude Opus 4.7; as of 2026-04-16
- anthropic.com/claude-opus-4-8-system-card: Scores for Anthropic: Claude Opus 4.8; as of 2026-05-28
- anthropic.com/claude-opus-5-system-card: Scores for Anthropic: Claude Opus 4.8, Claude Opus 5; as of 2026-07-24
- anthropic.com/claude-sonnet-4-6-system-card: Scores for Anthropic: Claude Sonnet 4.6; as of 2026-02-17
- anthropic.com/claude-sonnet-5-system-card: Scores for Anthropic: Claude Sonnet 5; as of 2026-06-30
- cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf: Scores for openai/gpt-oss-120b; as of 2025-08-05
- deepmind.google/models/model-cards/gemini-3-1-flash-lite: Scores for Google: Gemini 3.1 Flash Lite; as of 2026-03-03
- deepmind.google/models/model-cards/gemini-3-1-pro: Scores for google/gemini-3.1-pro; as of 2026-02-19
- deepmind.google/models/model-cards/gemini-3-5-flash: Scores for google/gemini-3.1-pro, Google: Gemini 3.5 Flash; as of 2026-05-19
- deepmind.google/models/model-cards/gemini-3-5-flash-lite: Scores for Google: Gemini 3.1 Flash Lite, Google: Gemini 3.5 Flash Lite; as of 2026-07-21
- deepmind.google/models/model-cards/gemini-3-6-flash: Scores for Google: Gemini 3.6 Flash; as of 2026-07-21
- huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B: Scores for DeepSeek R1 Distill Qwen3 8B 0528; as of 2026-11-03
- huggingface.co/deepseek-ai/DeepSeek-V4-Flash: Scores for DeepSeek: DeepSeek V4 Flash 0423; as of 2026-11-03
- huggingface.co/deepseek-ai/DeepSeek-V4-Pro: Scores for DeepSeek: DeepSeek V4 Pro 0423; as of 2026-11-03
- huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813: Scores for DeepSeek: DeepSeek V4 Flash 0731, DeepSeek: DeepSeek V4 Pro 0813; as of 2026-11-03
- huggingface.co/meta-llama/Llama-3.3-70B-Instruct: Scores for meta-llama/Meta-Llama-3.3-70B-Instruct; as of 2024-12-06
- huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct: Scores for meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8, meta-llama/Meta-Llama-3.3-70B-Instruct; as of 2025-04-05
- huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct: Scores for meta-llama/Llama-4-Scout-17B-16E-Instruct; as of 2025-04-05
- huggingface.co/MiniMaxAI/MiniMax-M2.7: Scores for MiniMax: MiniMax M2.7; as of 2026-11-03
- huggingface.co/mistralai/Mistral-Medium-3.5-128B: Scores for Mistral: Mistral Medium 3.5; as of 2026-11-03
- huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506: Scores for mistralai/Mistral-Small-3.2-24B-Instruct-2506; as of 2026-11-03
- huggingface.co/moonshotai/Kimi-K2-Instruct: Scores for Kimi K2 Instruct; as of 2026-11-03
- huggingface.co/moonshotai/Kimi-K2.6: Scores for MoonshotAI: Kimi K2.6; as of 2026-11-03
- huggingface.co/moonshotai/Kimi-K3: Scores for MoonshotAI: Kimi K3; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.5-122B-A10B: Scores for Qwen: Qwen3.5-122B-A10B; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.5-27B: Scores for Qwen: Qwen3.5-27B; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.5-35B-A3B: Scores for Qwen: Qwen3.5-35B-A3B; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.5-9B: Scores for Qwen: Qwen3.5-9B; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.6-27B: Scores for Qwen: Qwen3.6 27B; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.6-35B-A3B: Scores for Qwen: Qwen3.6 35B A3B; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.8-2.4T-A95B: Scores for Qwen: Qwen3.7 Max; as of 2026-11-03
- huggingface.co/Qwen/Qwen3.8-27B: Scores for Qwen: Qwen3.6 27B, Qwen: Qwen3.7 Plus, Qwen: Qwen3.8 27B; as of 2026-11-03
- huggingface.co/zai-org/GLM-5.1: Scores for Z.ai: GLM 5.1; as of 2026-11-03
- huggingface.co/zai-org/GLM-5.2: Scores for Z.ai: GLM 5.2; as of 2026-11-03
- huggingface.co/zai-org/GLM-5.3: Scores for Z.ai: GLM 5.3; as of 2026-11-03
- media.x.ai/v1/website/4p5-5184fdf9.pdf: Scores for SpaceXAI: Grok 4.5; as of 2026-07-20