Freshness
Public sample refreshed monthly while private holdout stays sealed until replacement tasks exist.
Back to Benchmark Lab
Research
Browser Operations Suite
Browser-agent tasks for navigation, structured extraction, authenticated workflows, form filling, and UI-state verification under changing pages.
Current leader
Frontier reasoning model
Usable for constrained workflows with fallback routing.
Score71
Pass rate74
Recovery69
Task mix
What the suite measures
Navigation30%
Form completion22%
Extraction24%
Screenshot verification24%
Model classProviderScorePassRetryP95Reviewer note
Frontier reasoning modelFrontier API provider717410%5764msUsable for constrained workflows with fallback routing.
Fast mid-tier modelFast hosted API provider606215%4874msUsable for constrained workflows with fallback routing.
Open-weight local modelSelf-hosted/open-weight stack434620%6000msUse only for narrow routing, triage, or privacy-constrained baselines.
Small routing modelLow-cost routing endpoint312825%4732msUse only for narrow routing, triage, or privacy-constrained baselines.
Task trace evidence
Inspect representative benchmark runs
These trace packets show the task brief, expected evidence, model outcomes, cost units, latency, and reviewer notes behind the aggregate score.
Extraction / public / Medium
Pricing page extraction
Navigate a provider pricing page, extract input/output token prices, and return source-linked structured data.
Top run: Frontier reasoning modelOpen trace
Form completion / holdout / Hard
Multi-step demo form
Complete a multi-step demo request form, handle validation errors, and verify the confirmation state with screenshot evidence.
Top run: Frontier reasoning modelOpen trace
Navigation / public / Medium
Invoice portal download
Navigate a mock vendor portal; download the latest invoice; verify file state; and extract the invoice number.
Top run: Frontier reasoning modelOpen trace
Extraction / holdout / Hard
Competitor feature map
Visit three product pages; extract pricing tiers and feature gates; normalize them into a comparable table.
Top run: Frontier reasoning modelOpen trace
Scoring rubric
Task success
State verification
Recovery quality
Human handoff rate
Run provenance
Generated at: 2026-05-16T00:00:00+05:30
Dataset version: 0.1.0
Trace ingest paths: data/benchmark-trace-input.json, data/benchmark-trace-runs.csv
Run date: 2026-05-16
Synthetic v0.1 benchmark dataset for website scaffolding. Aggregate suite rows are generated from script constants; task traces are ingested from data/benchmark-trace-input.json and data/benchmark-trace-runs.csv when present. Replace both with real model/provider runs as benchmark harnesses come online.
Leaderboard control metadata
Controls attached to this benchmark run
These fields make the suite auditable: the public/private split, freshness policy, leakage policy, repeat-run rule, retirement trigger, and provenance fields are generated with the benchmark data instead of being described only in prose.
Split
6 public / 10 private holdout tasks
Public share
38%
Holdout share
63%
Repeat rule
Repeat any result within five points of a leaderboard boundary across at least three seeds.
Leakage policy
Do not use tasks sourced from public examples, vendor demos, or training-contaminated snippets without replacement variants.
Retirement rule
Retire a task when frontier and mid-tier models cluster near the ceiling or when source material becomes widely circulated.
Required provenance
traceId, createdAt, split, source, modelVersion, runSeed, reviewerNote, retirementStatus
Next data step
Replace this synthetic v0.1 run with real provider traces.
The page is wired to generated data already, including JSON task packets and CSV trace rows. The next engineering task is to point the importer at actual benchmark harness exports with model name, provider, settings, latency samples, retries, tool traces, and reviewer notes.
Read methodology