Extraction / holdout trace / Hard
Visit three product pages; extract pricing tiers and feature gates; normalize them into a comparable table.
Expected evidence
Scoring focus
Common failure mode
Weak browser agents copy marketing bullets without normalizing feature availability by tier.
Expected output
A normalized comparison table with source URLs; tier names; gated features; and extraction confidence.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-holdout-2026-05
Run seed
2026051730
Prompt packet
competitor-feature-map-holdout-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite browser-operations-suite --task competitor-feature-map
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
holdout
Difficulty
Hard
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted with review
Strong multi-page extraction with minor label cleanup.
Answer excerpt
Created a normalized tier table and preserved source URLs for all pages.
Failure reason
One confidence label needed review.
Fast hosted API provider
Partial
Incomplete but recoverable.
Answer excerpt
Collected feature bullets from two of three pages.
Failure reason
Missed one page and skipped normalization.
Self-hosted/open-weight stack
Rejected
Could not maintain state across pages.
Answer excerpt
Summarized the first page only.
Failure reason
Failed multi-page workflow.
Low-cost routing endpoint
Rejected
Classification only.
Answer excerpt
Competitive research task.
Failure reason
No execution.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report