The results
The same scores as the leaderboard: a test that failed is left out of its run's score, not counted as zero. Time, tokens and cost cover each whole run, wildcards included.
Ranking
Quality v3 over measured non-wildcard pages. Different workloads: browse the outputs and coverage before comparing models.
Score against output tokens
Quality v3 against full-run tokens. Different workloads and hardware affect this comparison.
Output tokens include the model's thinking. Like the time, tokens and cost table, it counts every test the model took, wildcards included. The score axis is zoomed to the data.
Score against time
Quality v3 against full-run time. Different workloads and hardware affect this comparison.
Time is how long the model took to answer, added up over its tests; judging is not counted. Like the time, tokens and cost table, it counts every test the model took, wildcards included. The score axis is zoomed to the data.
Where each model is strong and weak
One row per model, one column per test and per scoring criterion.
| By test | By what the judges scored | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Aero Lab | Jet Odyssey | Blast Island | Launch Day | Alex Optics | Afterburner | Hangar Ops | Hangar Planner | Rotor Rampage | Brief delivered | Working behaviour | Look and feel | Engineering quality | |
| claude-fable-5-2effort: high | 88 | 84 | 85 | 77 | 90 | — | — | — | — | 93 | 100 | 70 | 65 | |
| claude-opus-5-5effort: high | 89 | 79 | 84 | 79 | 66 | 86 | 89 | — | 81 | 90 | 96 | 71 | 63 | |
| gpt-6-astraeffort: high | 83 | 82 | 80 | 69 | 85 | 81 | — | 89 | 79 | 92 | 99 | 68 | 50 | |
Time, tokens and cost
What each model used across all its tests, wildcards included.
| Model | Tests | Time | Output tokens | Speed | Cost | Subscription used |
|---|---|---|---|---|---|---|
| claude-fable-5-2effort: highAnthropic | 6 | 2h 35m | 906k547k of it thinking | 97.2 tok/s | $55.39API price | 7-day window +6 pts |
| claude-opus-5-5effort: highAnthropic | 8 | 3h 29m | 1.30M753k of it thinking | 103.0 tok/s | $36.60API price | 5-hour window not comparable7-day window +2 pts |
| gpt-6-astraeffort: highOpenAI | 9 | 3h 31m | 322k75k of it thinking | 25.4 tok/s | ≈$31.69API price, est. | — |
Read from the account's usage window in whole percentage points, so a short run can show 0 points. It counts everything the account did in that time, not only this run.
Runs in this video
Every run behind the numbers. Scores name their comparison basis.
claude-fable-5-2effort: high
Anthropic · 84 / 100 · Quality v3 · 5/5 pages · 2h 35m · $55.39
88
How this score was reached…
84
How this score was reached…
85
How this score was reached…
77
How this score was reached…
90
How this score was reached…
81
How this score was reached…
claude-opus-5-5effort: high
Anthropic · 82 / 100 · Quality v3 · 8/8 pages · 3h 29m · $36.60
89
Earlier evaluations
7.6/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.7/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
79
Earlier evaluations
7.0/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
6.8/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
84
Earlier evaluations
7.5/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
7.2/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
79
Earlier evaluations
6.7/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.7/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
66
Earlier evaluations
6.0/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
5.6/10 · engineering-1 · claude-code/claude-sonnet-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
86
Earlier evaluations
8.3/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
8.5/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
89
Earlier evaluations
8.6/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.4/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
81
Earlier evaluations
7.9/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
7.5/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
gpt-6-astraeffort: high
OpenAI · 81 / 100 · Quality v3 · 8/8 pages · 3h 31m · ≈$31.69
83
Earlier evaluations
6.0/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.8/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
82
Earlier evaluations
6.6/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.6/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
80
Earlier evaluations
6.7/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.0/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
69
Earlier evaluations
7.1/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
5.8/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
85
Earlier evaluations
7.5/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.8/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
81
Earlier evaluations
8.2/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.8/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
89
Earlier evaluations
7.3/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
9.0/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
79
Earlier evaluations
7.2/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.8/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.
How this score was reached…
85
Earlier evaluations
6.9/10 · engineering-1 · claude-code/claude-opus-5-5 (engineering-1 rubric)
7.4/10 · Quality v1 · claude-code/claude-opus-5-5 (quality-1 rubric)
Different rubrics; these values are historical, not a score change.