What a score means

Quality v1: the same standard for every page

A finished page receives four scores out of ten: working behaviour (30%), visual quality (30%), engineering quality (20%), and usability and accessibility (20%). Their weighted sum is its Quality v1 score. The original prompt, task-specific weights and model identity are omitted from this quality assessment. Simple pages can score well; complexity and novelty earn no automatic credit.

The scale uses fixed anchors: 0 means unusable in that dimension, 2 severely flawed, 5 adequate with material weaknesses, 8 polished with minor faults, and 10 exceptional within the available evidence. The judge reviews source, rendered images and recorded runtime observations. Missing observations remain unknown. A model judgment is an assessment, not proof that every feature, device or accessibility requirement has been tested.

A run’s Quality v1 headline averages its measured non-wildcard pages equally. Coverage shows how many received this standard; failures and missing evidence do not become invented quality scores. Wildcards and unrelated experiments have their own page scores. Prompt fulfilment and automated checks stay separate. Earlier evaluations remain available in page details, labelled with their original rubric.

Quality v1 was checked on saved app and canvas outputs, matched copies with scripts removed, and repeat judgments before migration. This small calibration checks basic separation and repeatability; it is not a statistical confidence estimate. Changing the judge or quality rubric requires fresh validation and explicit versioning.

Fair comparisons and benchmark editions

An official score is the equal-weight mean, out of ten, across all tests in a frozen core edition. Each edition identifies the exact prompts, assets, checks, scoring weights and generation settings by fingerprint. A changed prompt or protocol creates a new edition identity; earlier evidence remains in its own edition.

The default leaderboard browses all tested configurations. Its order describes the outputs in this catalogue, not a controlled universal ranking of model capability. Optional comparison filters group the same edition, execution and attempt policy, and actual scoring revision. Direct generation and distinct agent harnesses have separate tracks. One attempt and best-of-N attempts do not compete in the same track. A completed test may score zero; an interrupted generation or missing judge score is incomplete evidence.

Other numbers you will see

Run average

The mean across that run's non-wildcard tests, including extra tests. It can differ from the official core score.

Estimated comparison

The historical exploratory ranking estimates some changed or missing test scores. These carry ≈ and stay separate from measured official rankings.

Experiment score

Feedback on a particular prompt or imported page. Unrelated experiments never form a global model ranking. With no original brief, prompt adherence is unavailable.

Repaired page

A separate variant, with its own evaluation. It never replaces an original benchmark score.

Scoring and checks

Historical scores used task-specific weights across functionality, visuals, code, UX and creativity. They are labelled with their legacy rubric and must not be subtracted from Quality v1 scores. Quality v1 keeps the four fixed weights above, however a task prompt changes. The run page names the actual judge and mode, including heuristic evaluation. A changed judge or rubric gets a different scoring fingerprint; re-evaluation retains earlier evidence locally. New evaluations record judge temperature, output budget and render settings; older evaluations without those records remain in a separate historical scoring revision. Automated checks report observed behavior, not a guarantee that every feature works.

The exact active prompts, hidden checks and detailed judge instructions remain private. Prompt fingerprints help identify matching inputs; they do not independently prove fairness. Experiment prompts are private unless deliberately released.

Time, tokens and cost

The XY chart is available under Charts · XY on the leaderboard. In All results it uses the displayed output score and full-run resources. Equivalent core tests uses core-test resources instead. Both include recorded generation attempts and continuations. These are different workload views; use matching edition and execution filters for controlled comparisons. Score history shows unjoined observations across dates, avoiding false trends between different prompts. Missing or incomplete measurements are omitted. Run details show full-run usage, including extra tests. Tokens differ across tokenisers and are not a universal compute measure.

Local speed depends on the recorded hardware. Parameter counts, active parameters and quantisation are separate facts; undisclosed values stay unknown. “No API charge” does not mean zero hardware or electricity cost. Charged API cost, subscription usage and API-equivalent estimates are different measurements.

Edition catalogue

{
  "max_continuations": 2,
  "max_html_size_bytes": 5242880,
  "max_tokens": 262144,
  "recording_duration_seconds": 15,
  "sandbox_timeout_seconds": 30,
  "screenshot_height": 1080,
  "screenshot_width": 1920,
  "temperature": null
}

{
  "max_continuations": 2,
  "max_html_size_bytes": 5242880,
  "max_tokens": 262144,
  "recording_duration_seconds": 15,
  "sandbox_timeout_seconds": 30,
  "screenshot_height": 1080,
  "screenshot_width": 1920,
  "temperature": null
}

{
  "max_continuations": 2,
  "max_html_size_bytes": 5242880,
  "max_tokens": 262144,
  "recording_duration_seconds": 15,
  "sandbox_timeout_seconds": 30,
  "screenshot_height": 1080,
  "screenshot_width": 1920,
  "temperature": null
}

{
  "max_continuations": 2,
  "max_html_size_bytes": 5242880,
  "max_tokens": 262144,
  "recording_duration_seconds": 15,
  "sandbox_timeout_seconds": 30,
  "screenshot_height": 1080,
  "screenshot_width": 1920,
  "temperature": null
}

{
  "max_continuations": 2,
  "max_html_size_bytes": 5242880,
  "max_tokens": 262144,
  "recording_duration_seconds": 15,
  "sandbox_timeout_seconds": 30,
  "screenshot_height": 1080,
  "screenshot_width": 1920,
  "temperature": null
}