Finance / public trace / Medium
Given an invoice, purchase order, and short email thread, identify the GST mismatch and draft a vendor-facing explanation with cited evidence.
Expected evidence
Scoring focus
Common failure mode
Models often explain the mismatch correctly but cite the wrong source row or miss the exception email.
Expected output
A short vendor-facing explanation identifying the GST mismatch, citing the invoice tax line, purchase order rate, and vendor email exception.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-public-2026-05
Run seed
2026051600
Prompt packet
gst-invoice-discrepancy-public-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite indian-enterprise-workflow-suite --task gst-invoice-discrepancy
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
public
Difficulty
Medium
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Correct mismatch, cited all three evidence points, and proposed a clean vendor follow-up.
Answer excerpt
The invoice uses a GST rate that does not match the purchase order, and the exception email explains why the vendor needs to revise the tax line before payment.
Failure reason
Minor formatting cleanup only.
Fast hosted API provider
Accepted with review
Correct calculation and tone; reviewer had to repair one citation label.
Answer excerpt
The GST discrepancy appears to come from a rate mismatch between the invoice and the PO.
Failure reason
One citation label pointed to the right document but the wrong row.
Self-hosted/open-weight stack
Partial
Found a mismatch but confused CGST/SGST allocation and needed human correction.
Answer excerpt
The invoice has a tax mismatch and should be checked against the purchase order.
Failure reason
Confused CGST/SGST allocation and omitted the vendor exception email.
Low-cost routing endpoint
Rejected
Useful triage signal only; generated an unsupported tax explanation.
Answer excerpt
This looks like a finance escalation.
Failure reason
Could route the task but generated an unsupported tax explanation.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report