Extraction / public trace / Medium
Navigate a provider pricing page, extract input/output token prices, and return source-linked structured data.
Expected evidence
Scoring focus
Common failure mode
Browser agents extract stale snippets or fail to distinguish input, cached input, and output pricing.
Expected output
A source-linked table separating input, cached input, output, and batch pricing where available.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-public-2026-05
Run seed
2026051640
Prompt packet
pricing-page-extraction-public-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite browser-operations-suite --task pricing-page-extraction
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
public
Difficulty
Medium
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Extracted the right fields and preserved source URL; minor formatting cleanup needed.
Answer excerpt
Returned input, cached-input, and output prices with the source URL preserved for review.
Failure reason
Minor formatting cleanup needed.
Fast hosted API provider
Accepted with review
Reached page and extracted most prices but missed cached-input distinction.
Answer excerpt
Returned provider prices for input and output tokens.
Failure reason
Missed cached-input distinction.
Self-hosted/open-weight stack
Partial
Found pricing page but mixed plan names with API prices.
Answer excerpt
The page includes multiple API price plans.
Failure reason
Mixed plan names with API prices.
Low-cost routing endpoint
Rejected
Could classify the page type only.
Answer excerpt
Pricing page detected.
Failure reason
Could classify the page type only.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report