Nominal: Measuring Version Progress in Frontier Language Models
Abstract
We introduce Nominal, a benchmark that evaluates 32 frontier LLMs from 10 labs on a single, fully reproducible metric: the version number printed in the model's name. Unlike existing evaluations, Nominal is immune to test-set contamination, requires no inference compute, and yields zero run-to-run variance. We find that scores have risen steadily over the evaluation window, with GPT-6.1 Sol establishing a new state of the art at 6.1.
Over time
Up is good. Up and to the left is very good. Hover a dot to see one lab's whole climb.
Leaderboard
32 models. Ties are broken by nothing; they're just tied.
| # | Model | Score | Released | Lab |
|---|---|---|---|---|
| 1 | GPT-6.1 Sol | 6.1 | 2026-09-29 |  OpenAI |
| 2 | GPT-6 Astra | 6.0 | 2026-09-03 | OpenAI |
| 3 | GPT-5.6 Sol | 5.6 | 2026-07-09 | OpenAI |
| 4 | Opus 5.5 | 5.5 | 2026-09-22 |  Anthropic |
| 5 | GLM-5.3 | 5.3 | 2026-08-14 |  Z.AI |
| 6 | GLM-5.2 | 5.2 | 2026-06-16 | Z.AI |
| 7 | GLM-5.1 | 5.1 | 2026-04-07 | Z.AI |
| 8 | Fable 5.1 | 5.1 | 2026-09-01 | Anthropic |
| 9 | Fable 5 | 5.0 | 2026-06-09 | Anthropic |
| 10 | Grok 4.7 | 4.7 | 2026-09-21 |  xAI |
| 11 | Grok 4.6 | 4.6 | 2026-08-12 | xAI |
| 12 | Grok 4.5 | 4.5 | 2026-07-08 | xAI |
| 13 | DeepSeek V4.1 Flash | 4.1 | 2026-09-10 |  DeepSeek |
| 14 | DeepSeek V4 Flash | 4.0 | 2026-07-31 | DeepSeek |
| 15 | DeepSeek V4 Pro | 4.0 | 2026-08-13 | DeepSeek |
| 16 | Gemini 4 Argon | 4.0 | 2026-09-30 |  Google |
| 17 | Qwen 3.8 Max | 3.8 | 2026-08-02 |  Qwen |
| 18 | Gemini 3.8 Flash | 3.8 | 2026-09-02 | Google |
| 19 | Qwen 3.7 Plus | 3.7 | 2026-06-01 | Qwen |
| 20 | Gemini 3.7 Flash | 3.7 | 2026-08-13 | Google |
| 21 | Qwen 3.6 Plus | 3.6 | 2026-04-02 | Qwen |
| 22 | Gemini 3.6 Flash | 3.6 | 2026-07-21 | Google |
| 23 | Qwen 3.5 Plus | 3.5 | 2026-02-16 | Qwen |
| 24 | Kimi K3 | 3.0 | 2026-07-16 |  Moonshot |
| 25 | Kimi K2.6 | 2.6 | 2026-04-20 | Moonshot |
| 26 | MiMo V2.6 Pro | 2.6 | 2026-09-21 |  Xiaomi |
| 27 | Kimi K2.5 | 2.5 | 2026-01-27 | Moonshot |
| 28 | MiMo V2.5 Pro | 2.5 | 2026-04-23 | Xiaomi |
| 29 | MiMo V2 Pro | 2.0 | 2026-03-18 | Xiaomi |
| 30 | Muse Spark 1.3 | 1.3 | 2026-09-05 |  Meta |
| 31 | Muse Spark 1.2 | 1.2 | 2026-08-05 | Meta |
| 32 | Muse Spark 1.1 | 1.1 | 2026-07-09 | Meta |
Questions people have actually asked
- Isn't this meaningless?
- Pretty much. But it's reproducible, nobody can contaminate the test set, and the error bars are zero.
- Why does DeepSeek V4 Pro score the same as V4 Flash?
- "Pro" isn't a number. We checked.
- Can a lab game this?
- Yes, by renaming the model. That's the only strategy, and some labs are clearly using it.
- Why is Kimi so low?
- Kimi K3 is a 3. Take it up with Moonshot.
Methodology
- Read the model's name.
- Find the number in it. That's the score. GPT-6.1 Sol gets 6.1, Kimi K3 gets 3.0, and Sol, Astra, Pro, Max and Flash get nothing.
- Release dates and names come from Artificial Analysis. Only LLMs count, so no image or video models.