Finance / holdout trace / Hard
Compare invoice; credit note; and ledger export to decide whether the vendor credit has been applied correctly.
Expected evidence
Scoring focus
Common failure mode
Weak models confuse credit note issuance with ledger application and close the task too early.
Expected output
A reconciliation memo showing invoice amount; credit note amount; ledger treatment; and required finance action.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-holdout-2026-05
Run seed
2026051690
Prompt packet
gst-credit-note-reconciliation-holdout-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite indian-enterprise-workflow-suite --task gst-credit-note-reconciliation
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
holdout
Difficulty
Hard
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Strong reconciliation with correct distinction between issued and applied credit.
Answer excerpt
The credit note exists but the ledger has not applied it against the invoice; finance should keep the payable open and request posting confirmation.
Failure reason
No major issue.
Fast hosted API provider
Accepted with review
Correct conclusion with one evidence gap.
Answer excerpt
The credit note matches the invoice discrepancy; ledger posting still needs confirmation.
Failure reason
Reviewer added one missing ledger citation.
Self-hosted/open-weight stack
Partial
Needed human correction on ledger interpretation.
Answer excerpt
The vendor has issued a credit note so the mismatch appears resolved.
Failure reason
Incorrectly treated issuance as application.
Low-cost routing endpoint
Rejected
Routing only; not safe for finance action.
Answer excerpt
Finance discrepancy detected.
Failure reason
No reconciliation.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report