Refund decisions / public trace / Medium
Respond to an angry customer whose refund request sits just outside the stated policy window while preserving tone and escalation boundaries.
Expected evidence
Scoring focus
Common failure mode
Models either deny too harshly or invent a refund exception not present in policy.
Expected output
An empathetic support response that does not invent a refund exception, cites the policy boundary, and offers the approved escalation path.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-public-2026-05
Run seed
2026051660
Prompt packet
refund-policy-boundary-public-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite support-agent-policy-suite --task refund-policy-boundary
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
public
Difficulty
Medium
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Firm policy answer, empathetic tone, and correct escalation path.
Answer excerpt
I understand why this is frustrating. The request is outside the refund window, but I can escalate the approved exception-review path.
Failure reason
No major issue.
Fast hosted API provider
Accepted
Strong answer with one minor wording edit.
Answer excerpt
The policy window has passed, but we can route this to the approved review queue.
Failure reason
One minor wording edit.
Self-hosted/open-weight stack
Partial
Tone was acceptable but policy boundary was vague.
Answer excerpt
We may not be able to refund this, but support can review the issue.
Failure reason
Policy boundary was too vague.
Low-cost routing endpoint
Rejected
Classified refund intent but could not safely resolve.
Answer excerpt
Refund intent detected.
Failure reason
Could not safely resolve the policy boundary case.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report