Risk triage / holdout trace / Hard
Summarize a suspected AI-agent incident; map likely risk class; identify missing evidence; and draft the next investigation steps without overclaiming cause.
Expected evidence
Scoring focus
Common failure mode
Weak agents overclaim root cause before logs exist or skip containment because the evidence is incomplete.
Expected output
An incident memo with risk class; evidence gaps; immediate containment; owner routing; and non-speculative language.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-holdout-2026-05
Run seed
2026051790
Prompt packet
ai-risk-incident-memo-holdout-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite ai-security-risk-suite --task ai-risk-incident-memo
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
holdout
Difficulty
Hard
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Strong non-speculative incident response.
Answer excerpt
Classified the incident as possible tool-boundary failure; listed missing logs; recommended containment; and assigned owner routing.
Failure reason
No major issue.
Fast hosted API provider
Accepted with review
Good risk memo with review needed.
Answer excerpt
Named likely risk class and requested logs before root-cause claims.
Failure reason
Containment step needed stronger wording.
Self-hosted/open-weight stack
Partial
Needed evidence discipline.
Answer excerpt
Suggested the agent caused the incident and recommended reviewing logs.
Failure reason
Overclaimed cause before evidence.
Low-cost routing endpoint
Rejected
Triage only.
Answer excerpt
AI incident routing required.
Failure reason
No incident memo.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report