Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Jonathan Geuter, Andrew Cheng, David Alvarez-Melis

arXiv 2610.17205 23 pages Read the paper

The short version

The problem: Many applications need to choose among fixed options and provide a confidence estimate, such as routing a support request to the right team. Generating a written answer takes extra time, and a confidence score is useful only if it is dependable.
The idea: The paper uses a language model’s existing scores for answer letters such as A, B, and C to assign probabilities to the choices in one pass. It also gives a short recipe for training the model on examples of decisions, so its choices improve without building a specialized system from scratch.
What it got: Without extra training, Gemma 4 31B scores 79.6 on the paper’s decision test, close to Jev’s 80.1. With training, it reaches 84.4, surpassing Jev on that test.

Modules from this paper

One-Pass Probabilities from Answer-Letter Scores

Generating a decision can require a separate decoding step; label options with letters and read their next-token scores to get probabilities in one model pass.

decision-modelslanguage-modelsinferenceprobabilities

OpenDecisions and the Capability Score

Small decision benchmarks can hide whether a model generalizes, so OpenDecisions broadens validation coverage and scores both answer quality and probability calibration.

evaluationdecision-modelscalibrationbenchmarks

Supervised Adaptation for Stronger Decisions

When a pretrained model’s answer choices are not reliable enough, adapt its internal representations with LoRA and train a decision head initialized from its answer-letter weights using supervised decision examples.

supervised-fine-tuningLoRAdecision-modelssoft-labels