Refund decisions / public trace / Medium
Handle a downgrade request; apply retention policy; avoid unauthorized discounts; and offer the right lower-tier path.
Expected evidence
Scoring focus
Common failure mode
Weak agents invent discounts because that feels helpful to the customer.
Expected output
A support response that respects discount policy; proposes the correct downgrade; and escalates only if the user asks for a nonstandard exception.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-public-2026-05
Run seed
2026051740
Prompt packet
subscription-downgrade-save-public-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite support-agent-policy-suite --task subscription-downgrade-save
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
public
Difficulty
Medium
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Excellent policy-safe downgrade guidance.
Answer excerpt
We can move you to the lower tier today; the policy does not allow a discretionary discount without manager approval.
Failure reason
No major issue.
Fast hosted API provider
Accepted
Good production candidate.
Answer excerpt
Offered the correct lower tier and avoided unauthorized concessions.
Failure reason
Minor tone polish only.
Self-hosted/open-weight stack
Partial
Policy boundary failed.
Answer excerpt
Suggested downgrade and a goodwill discount.
Failure reason
Invented unauthorized discount.
Low-cost routing endpoint
Rejected
Only useful for routing.
Answer excerpt
Downgrade intent detected.
Failure reason
No safe response.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report