Legal / public trace / Medium
Review a vendor renewal clause; identify renewal deadline; surface termination notice risk; draft an internal note with evidence.
Expected evidence
Scoring focus
Common failure mode
Weak models spot the renewal clause but miss the notice window that changes the recommendation.
Expected output
A concise internal note with renewal date; notice deadline; commercial risk; and cited contract evidence.
Score breakdown
Trace provenance
Score calculation ledger
Score equals the sum of weighted rubric components. Component earned points are allocated from the aggregate run score.
Model version
frontier-reasoning-eval-public-2026-05
Run seed
2026051680
Prompt packet
vendor-contract-renewal-risk-public-packet-v0.1
Artifact bundle
Replay scaffold generated from the current seed trace. Replace with real harness exports when model runs are available.
Replay command
pnpm benchmarks:replay --suite indian-enterprise-workflow-suite --task vendor-contract-renewal-risk
This command is intentionally documented before the real harness exists so the artifact contract is visible.
Payload preview
Split
public
Difficulty
Medium
Evidence fields
3
Model runs
4
Screenshot
Pending real browser or app screenshot artifact.
Model run evidence
This is the inspection layer that keeps benchmark scores honest: each model class gets an outcome, cost proxy, latency, and reviewer note.
Frontier API provider
Accepted
Correctly linked renewal risk to the notice deadline and produced an actionable internal note.
Answer excerpt
The renewal auto-extends unless notice is sent before the deadline; the owner should confirm termination or renegotiation this week.
Failure reason
No major issue.
Fast hosted API provider
Accepted with review
Good deadline extraction; reviewer tightened the business-risk framing.
Answer excerpt
The contract is close to renewal and should be reviewed before the notice period expires.
Failure reason
Needed clearer owner handoff.
Self-hosted/open-weight stack
Partial
Found the right section but failed the decision-critical date.
Answer excerpt
The vendor agreement may renew soon and legal should check it.
Failure reason
Missed the exact notice deadline.
Low-cost routing endpoint
Rejected
Useful as a triage label but unsafe for contract advice.
Answer excerpt
Legal review needed.
Failure reason
Could route only; no reliable evidence extraction.
Aggregate scores are useful only when reviewers can inspect the task packet, expected evidence, and the exact failure mode. This page is the pattern for publishing public samples while keeping harder holdout tasks private.
Return to suite report