Support / holdout trace / Hard
Classify a mixed Hindi-English support ticket, apply refund policy, and decide whether to escalate based on payment and delivery evidence.
Expected evidence
Scoring focus
Common failure mode
Weak agents over-apologize and promise a refund without checking the delivery timestamp.
Expected output
A bilingual-safe response that applies refund policy, cites delivery and payment evidence, and escalates only the payment reconciliation issue.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-holdout-2026-05
Run seed
2026051610
Prompt packet
hindi-english-refund-escalation-holdout-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite indian-enterprise-workflow-suite --task hindi-english-refund-escalation
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
holdout
Difficulty
Hard
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Handled code-switching, cited policy, and escalated only the payment reconciliation issue.
Answer excerpt
I understand the concern. The delivery timestamp means the standard refund path does not apply, but the payment-status mismatch should be escalated.
Failure reason
No major issue.
Fast hosted API provider
Accepted with review
Good policy application; tone needed minor localization edits.
Answer excerpt
Refund policy does not allow automatic refund here, but the payment issue can be escalated.
Failure reason
Tone needed minor localization edits.
Self-hosted/open-weight stack
Partial
Understood complaint but missed the delivery evidence and escalated too broadly.
Answer excerpt
We should escalate this refund request for manual review.
Failure reason
Missed delivery evidence and escalated too broadly.
Low-cost routing endpoint
Rejected
Classified intent but could not produce a safe final answer.
Answer excerpt
Refund intent detected.
Failure reason
Could classify intent but could not produce a safe final answer.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report