Decision benchmarks

Every benchmark the Decision index tracks, with each core benchmark's share of the composite and how many models it covers.

48 benchmarks tracked.

• DecisionBench primary accuracy (v1.0, 23,900 rows)
• DecisionBench 15-bin top-label ECE
• DecisionBench mean soft-label NLL
• Decision Index area: Arts & Human Taste
• Decision Index area: Knowledge & Reasoning
• Decision Index area: Language Understanding
• Decision Index area: Retrieval & Classification
• Decision Index area: Tools & Automation
• Amazon ESCI (Decision Index panel)
• ANLI (Decision Index panel)
• API-Bank (Decision Index panel)
• BANKING77 (Decision Index panel)
• BBH (Decision Index panel)
• BFCL (Decision Index panel)
• BPoMP (Decision Index panel)
• BRIGHT (Decision Index panel)
• CLINC150+OOS (Decision Index panel)
• ContractNLI (Decision Index panel)
• GPQA Diamond (Decision Index panel)
• GSM8K (Decision Index panel)
• HellaSwag (Decision Index panel)
• HLE (Decision Index panel)
• Home appliance simulator (Decision Index panel)
• Humicroedit (Decision Index panel)
• iSarcasmEval (Decision Index panel)
• MMLU-Pro (Decision Index panel)
• NLI4CT (Decision Index panel)
• ToolRet (Decision Index panel)
• When2Call (Decision Index panel)
• WinoGrande (Decision Index panel)
• Calibration-sample accuracy (Decision Index)
• Calibration Brier (Decision Index)
• Calibration ECE (Decision Index)
• Decision Index Full score (0.3.2)
• Decision Index public index (0.3.2)
• JevBench Calibration axis (v1.6.1)
• JevBench Capability score (v1.6.1)
• JevBench Composite Option A (v1.6.1)
• JevBench Cost axis (v1.6.1)
• JevBench Intelligence axis (v1.6.1)
• JevBench public accuracy (vendor self-run)
• JevBench public Brier (vendor self-run)
• JevBench public ECE (vendor self-run)
• JevBench Speed axis (v1.6.1)
• S1Bench (13-subset human-label suite, Nimble harness)
• TypeSafe workflow evals (mean of 4 workflows)
• ULTRABENCH in-house harness: chance-corrected accuracy
• ULTRABENCH in-house harness: Brier skill score