OpenDecisions and the Capability Score
Small decision benchmarks can hide whether a model generalizes, so OpenDecisions broadens validation coverage and scores both answer quality and probability calibration.
Read the paper ↗What it is
- OpenDecisions A validation set of decision questions built to cover varied task families, question styles, difficulty levels, and languages.
- Capability score One score that gives equal weight to getting answers right and assigning probabilities that reflect uncertainty.
Why it matters
A narrow benchmark can miss important decision behaviors, while accuracy alone ignores whether downstream systems can trust the model’s confidence. OpenDecisions broadens the test set; Capability keeps both answer quality and calibration in view.
Before getting into the scoring details, here is the short version: OpenDecisions is a validation set designed to look more like the mix of decisions that public decision benchmarks ask about. Capability then combines how often a system gets decisions right with how trustworthy its confidence is. That matters when another system will use the probabilities, not just the selected answer.
-
Coverage: The public decision sets discussed here were small, and larger multiple-choice benchmarks mostly tested general knowledge. OpenDecisions responds with questions spanning decision families, formats, difficulty tiers, and languages. The canonical concrete example is an option-order pair: the question stays the same while the answer options move, so you can test whether a system’s choice is sensitive to presentation. That is one diagnostic family, not a guarantee that all forms of option bias are covered. What else does a benchmark need besides breadth?
-
Trustworthy labels: Items come from code, LLM authoring checked by a panel, and real user requests checked by that panel. For example, an authored item is kept only when the panel agrees unanimously, or when two of three agree and their average probability on the author’s answer reaches the stated threshold. Paired examples are removed together if either fails verification. Think of this as checking both the answer key and the consistency of a deliberately altered pair; it does not make every label infallible. How do we reward a system for being right without rewarding unjustified confidence?
-
Accuracy plus calibration: Imagine two systems choosing the same correct option, but one assigns a probability that better reflects how likely it is to be right. An accuracy-only score treats them alike; a system feeding decisions to another process may not. Capability gives equal weight to chance-corrected accuracy and calibration. In plain terms, one part asks whether the answer beats chance, and the other asks whether the stated probabilities line up with outcomes. The score is still a summary, and averaging can hide which side drove a result. When is a single combined score useful, and when do you need to inspect its parts?
The set’s reported relationship to JevBench is encouraging, but I would not treat any one validation set as a universal proxy for every deployment. The useful move is pairing broader, checked coverage with a score that does not mistake confidence for correctness.
Read the deep dive
Why People Wanted It
A decision model is only useful if its evaluation set resembles the decisions it is meant to make—and if the set is large enough to compare systems meaningfully. The existing public decision sets cited here contained only 100 to 350 questions, while JevBench-Open contained 231. Larger multiple-choice benchmarks existed, but they primarily tested general knowledge rather than decisions based on supplied information. So the real question was not just whether a model can answer a benchmark, but whether a validation set can cover decision tasks and distinguish systems in a way that tracks the leaderboard.
The Core Move
OpenDecisions supplies a broader validation set, and Capability combines chance-corrected accuracy with calibration:
The score gives equal weight to how often a system chooses correctly and how well its probabilities reflect uncertainty.
Why Coverage Has to Resemble the Decisions
Now the deeper dataset-design detail: OpenDecisions is intended to reflect JevBench’s task composition, not merely to add more generic multiple-choice questions. It contains 2,352 questions across 11 families, including routing, multilingual decisions, fairness pairs, business use cases, counting and arithmetic traps, negation pairs, option-order and perturbation pairs, composite policy rules, and exact-probability calibration questions.
The set spans choice, yes/no, and score questions; four difficulty tiers; and 23 languages, with 78% of questions in English. Its mix of question types, tiers, families, and languages follows the published composition of the public JevBench v1.6 split. The variety matters because a single aggregate over one narrow question style could miss where a decision system succeeds or fails. What makes those varied questions trustworthy enough to score?
Why the Labels Need Checks
Here is the more technical construction detail: items come from code-written and labeled questions, LLM-authored questions checked by a three-model panel, and real user requests that are also panel-checked. For the LLM-authored items, a question is retained when all three panel models agree with the author’s answer, or when two agree and their average probability on that answer is at least 0.5. Real requests are retained only when all three models agree.
Some examples are paired—for instance, a reordered set of answer options or an altered input detail—and both members are removed if either fails panel verification. Candidate questions are also checked against training data and released JevBench questions for overlap. This construction is a set of checks, not a guarantee that every possible ambiguity disappears. Does the resulting set actually tell us more than the older public sets?
Why Capability Counts Calibration
Now the scoring machinery: the accuracy term, Intelligence , is chance-corrected accuracy, calculated separately by question type and difficulty tier. Tiers receive weights of 0.1, 0.2, 0.3, and 0.4 from easy through hard; question types receive equal weights. For yes/no questions, the answer is correct when the more likely side is correct.
Calibration summarizes probability quality with one score per type. It uses the top-answer expected calibration error (ECE) on hard choice questions, the probability of “yes” on yes/no questions, and, for score questions, both top-answer ECE and ranked probability score (RPS). The reported Capability score averages and , so a system cannot earn the full measure merely by selecting the right option while assigning poorly calibrated probabilities. Why not report accuracy alone? Because downstream systems may rely on those probabilities, not only the chosen answer.
Minimal Causal Code
This plain loop shows the score’s final aggregation, not the detailed per-type and per-tier calculations:
# I and C are already computed from the evaluation set.
capability = (intelligence + calibration) / 2
What You Gain, and What You Lose
OpenDecisions offers broad coverage, explicit label checks, and a score that treats answer quality and calibration as separate ingredients. In the reported comparison, 14 public decision models appear in nearly the same order as on JevBench, with a Spearman correlation of 0.90; four systems not used during construction were predicted from a fit on the other ten with an error of 3.0 points. Those are encouraging validation results, not a promise that the same relationship holds for every model or future benchmark.
Capability also compresses distinct behaviors into one number: its equal blend can conceal whether a change came from accuracy or calibration, and the calibration term uses different measures for different question types. The set is a validation set designed around JevBench’s composition, not a universal measure of every decision task. Separate modules cover the answer-letter probability readout and supervised adaptation recipe.
A decision benchmark should test varied decisions, and its score should care both whether the answer is right and whether the confidence deserves trust.