Batch 1

Batch scope and review

Model quality on 21 Workday Adaptive Planning questions, using each model’s first completed answer to each question.

Dataset and rubric

0.1.0-draft · 0.1.0-draft

Published

2026-09-28

Review method

Provisional blinded AI-assisted rubric review. Documentation-backed credit includes calibrated partial credit without treating undocumented alternatives as proven false. Focused cross-answer calibration covered archived versions, prior-period recurrence and import modes. No independent expert or tenant validation was performed.

Coverage note

All 21 models completed and received provisional assessments for all 21 questions. Every score uses the first completed answer per question. All 532 attempts, including 91 noncompletions, remain in completion and cost accounting.

Limitations

• Scores are provisional documentation-backed rubric results, not independently validated ERP accuracy or statistically established rankings.
• The first completed answer is selected per model and question, never the highest-scoring retry. Original, recovery and completion requests used different limits; this is not an equal-compute comparison.
• Latency is the median of selected completed-answer elapsed times. Costs cover all candidate API attempts, including failures; unknown charges keep total cost unknown. Assistant review and infrastructure costs are not measured.
• Unverified behavior receives no unsupported credit under the owner-approved rule; this does not prove an undocumented alternative is false.
• Original collection requested 8192 output tokens and a 180-second timeout. The first 61 attempts used socket-inactivity timeout enforcement; the remaining 359 used a wall-clock deadline. This is not a uniform-deadline cohort.
• Failure-selected recovery only: originally completed answers were not repeated. Expanded 32768-token and 600-second request limits, with paced requests; historical owner stops and noncompletion cutoffs remain in the attempt evidence.
• Supplemental model collection used 32768-token and 600-second limits. A narrow owner-approved retention exception applied only to these nonconfidential benchmark prompts; provider retention was possible, training was disallowed, and no tools or credentials were supplied.
• Missing-answer completion only; earlier completed answers are retained. Requests used 32768-token and 600-second limits, fixed original provider routes, ZDR and no-training restrictions. Three consecutive noncompletions paused dispatch for diagnosis; no model was silently excluded.
• The final 17 missing MiMo answers used owner-approved Novita after DeepInfra rate limits, with the same model and frozen questions. Requests retained ZDR, no training, no fallback, 32768-token and 600-second limits. FP8 was catalog metadata, not independently verified inference precision.
• Quality scores use the first completed answer per model/question; incomplete assessment coverage is unscored.