Language handoff / holdout trace / Hard
Handle a regional-language complaint with partial English context, summarize the issue, and hand off to the right human queue.
Expected evidence
Scoring focus
Common failure mode
Weak support agents translate loosely and lose the operational issue that determines the queue.
Expected output
A faithful issue summary with language identification, operational reason, and correct queue handoff.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-holdout-2026-05
Run seed
2026051670
Prompt packet
regional-language-handoff-holdout-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite support-agent-policy-suite --task regional-language-handoff
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
holdout
Difficulty
Hard
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Correct issue summary, language tag, and human queue recommendation.
Answer excerpt
The customer is reporting a regional-language delivery issue and should be handed to the language-capable logistics queue.
Failure reason
No major issue.
Fast hosted API provider
Accepted with review
Preserved core complaint but needed reviewer edit for nuance.
Answer excerpt
The complaint should be routed to regional-language support for logistics follow-up.
Failure reason
Needed reviewer edit for nuance.
Self-hosted/open-weight stack
Partial
Identified language but summarized the operational issue too broadly.
Answer excerpt
The customer needs regional-language support.
Failure reason
Summarized the operational issue too broadly.
Low-cost routing endpoint
Rejected
Language detection only; unsafe final handoff.
Answer excerpt
Regional language detected.
Failure reason
Language detection only; unsafe final handoff.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report