Form completion / holdout trace / Hard
Complete a multi-step demo request form, handle validation errors, and verify the confirmation state with screenshot evidence.
Expected evidence
Scoring focus
Common failure mode
Agents claim success after submit click even when the form remains on a validation-error state.
Expected output
A confirmed demo-request submission with evidence that the final confirmation state is visible.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-holdout-2026-05
Run seed
2026051650
Prompt packet
multi-step-demo-form-holdout-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite browser-operations-suite --task multi-step-demo-form
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
holdout
Difficulty
Hard
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted with review
Recovered from one validation error and captured final state; slow but reliable.
Answer excerpt
Submitted the form after correcting one validation error and captured final confirmation state.
Failure reason
Slow but reliable.
Fast hosted API provider
Partial
Completed visible fields but missed a hidden required dropdown.
Answer excerpt
Filled the visible fields and clicked submit.
Failure reason
Missed a hidden required dropdown.
Self-hosted/open-weight stack
Rejected
Failed to recover after validation error.
Answer excerpt
The form could not be submitted due to validation errors.
Failure reason
Failed to recover after validation error.
Low-cost routing endpoint
Rejected
Not suitable for stateful browser execution.
Answer excerpt
This is a browser form task.
Failure reason
Not suitable for stateful browser execution.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report