One-Pass Probabilities from Answer-Letter Scores
Generating a decision can require a separate decoding step; label options with letters and read their next-token scores to get probabilities in one model pass.
Read the paper ↗What it is
- Answer-letter readout Put answer choices in the prompt as A, B, C, and so on, then use the model’s scores for those letters as scores for the choices.
- One-pass probabilities For one question, turn the available letter scores into a probability distribution from the model’s final prompt position, without generating an answer first.
Why it matters
You can get both a preferred option and a distribution over the alternatives without decoding a textual response. The method is bounded by the supported letter choices, and each separate question still takes its own model pass.
Before getting into the mechanics, here is the short version: instead of asking a model to write its answer, we give each choice a letter and inspect how strongly the model favors those letters as its next token. That gives us a probability distribution over the choices directly, without generating an answer string first.
-
Avoid decoding: If I only need a choice, generating a word or explanation is extra work. The canonical example is a support-ticket routing question: the prompt lists billing, technical, and refund, and the model is told to answer A, B, or C. We can read the letter preferences rather than wait for it to emit one. This skips answer generation, not the model’s forward computation.
-
Keep uncertainty visible: A generated “A” hides how close the runner-up was. Reading the scores across A, B, and C gives a distribution, so the interface can distinguish a decisive preference from a close call. The concrete ticket example still maps each letter back to its described option; the scores are not meaningful option names by themselves.
So what are those scores, exactly? At the final prompt position, the model has a hidden state that its output layer uses to score possible next tokens. Since the prompt asks for one uppercase letter and each answer letter is a single token, we can select the scores for the letters corresponding to the listed options. Softmax then turns those selected scores into probabilities over the options. In words: it makes the available letter scores compete with one another and normalizes them into a distribution. The paper’s supplied text gives this operation as a softmax but does not provide a readable displayed equation here.
For the canonical three-choice ticket example, we keep only A, B, and C in that competition. We don’t let an unused D or an unrelated vocabulary token take probability mass. The paper describes support for up to 16 options, using answer-letter rows A through P. This is a deliberately constrained readout: it works because the interface controls the answer format.
So does one pass answer every question in a request? No: the request example has separate questions, and each question gets its own forward pass. The one-pass benefit is per question: all its option-letter scores are available together, with no generated answer sequence. That distinction matters if you are thinking about batching a multi-question request, which is a separate engineering concern.
The same pattern applies when the choices are not ordinary categories. A yes/no question becomes two options, and a score question becomes ordered levels; the model still compares the lettered options. The canonical ticket example makes this concrete: “Is the customer owed money back?” is represented by true and false options, each with its own letter. That establishes the mechanics, but it leaves the practical question: how much should you trust probabilities that come from the model’s raw letter preferences?
Read the deep dive
Why People Wanted It
A decision model has to choose among a finite set of options, but making it generate an answer first means asking it to continue writing text. Could we instead use the model’s own next-token preferences to score the choices directly? The useful shift is to make the choice correspond to a small set of answer letters, then inspect those letters’ scores before generating anything.
The Core Move
Label the options A, B, C, and so on, and turn the model’s scores for those letters at the final prompt position into probabilities over the options. The paper gives this procedure in prose and displays the probability conversion as a softmax, but the supplied text does not preserve a readable mathematical formula; the important operation is softmax over the valid option-letter logits.
For a question with options, only the first answer-letter scores are used. The options’ descriptions remain in the prompt, while each option is represented at readout time by its letter. The largest probability identifies the preferred answer, and the full distribution carries confidence information as well.
Why One Position Can Represent the Choices
Here is the part that gets technical: the model’s final hidden state is passed through the language-model output head, and the rows associated with letters A through P provide the candidate scores. Because each letter is a single token, the model can score these candidates from the same final-position state rather than generate an answer sequence token by token. The readout supports up to 16 options.
The prompt asks for exactly one uppercase answer letter and uses the model’s standard chat template with thinking disabled. The decision is therefore framed as “which answer letter would come next?” The letter scores are converted into a distribution over the question’s available options; the unused letter scores do not become extra choices.
Why Each Question Still Needs Its Own Pass
Now the exact constraint: the request format can contain multiple named questions, but each question is answered in its own forward pass. Within one question, the letter probabilities come from one pass; a request containing several questions does not collapse those separate questions into one shared pass.
The format covers ordinary choices, yes/no questions represented as two options, and score questions represented as ordered answer levels. The available letter slots are reused for these different question types. Thus the single-pass claim is about producing a distribution for one question without generating its answer—not about answering every question in a multi-question request at once.
Minimal Causal Code
This plain PyTorch sketch makes the readout visible. It assumes next_token_logits are the model’s logits at the final prompt position, and that letter_token_ids identify the single-token letters in order. The source describes extracting letter logits and applying softmax, but does not provide a code listing or an explicit formula in the supplied pages.
import torch
# One model call for this question; take logits at the final prompt position.
output = model(**prompt, use_cache=False)
next_token_logits = output.logits[:, -1, :]
# Gather only the letter scores for the K options actually present.
K = len(options)
letter_ids = letter_token_ids[:K]
option_logits = next_token_logits[:, letter_ids]
# Normalize those candidate scores into probabilities over the options.
probabilities = torch.softmax(option_logits, dim=-1)
choice_index = probabilities.argmax(dim=-1)
What You Gain, and What You Lose
You get option probabilities without sampling or decoding a textual answer, and the scores are available after one forward pass for that question. The approach also gives a distribution rather than only a generated label. Its constraints are real: the prompt must establish the letter-to-option mapping, each answer letter must be a single token, and the readout is limited to the supported letter slots. The model’s scores over letters are the basis of the probabilities, so this is not an independent decision mechanism that removes the language model’s preferences or calibration behavior.
Turn each option into a letter, and the model’s final-position letter scores become the choice distribution without generating an answer.