Typed Decisions from Question-Conditioned Audio

Why People Wanted It

Suppose you ask an audio model whether two clips contain the same speaker, which of several scenes is present, or how good a recording sounds. A text-generating model can answer in words, but then a bounded decision has to be recovered from that text. The real question is whether the model can use its audio-and-question understanding directly, while returning an output whose shape matches the answer space.

The audio encoder and adaptor still provide acoustic evidence, and the decoder still interprets the question and its answer specification. Jev changes the final readout: instead of generating answer tokens, it reads a hidden state computed before any target-answer tokens and maps that state to one of three typed outputs .

The Core Move

Use the final question-conditioned, pre-answer decoder state as the input to a decision head, then select the readout matching the requested answer type: $h^* = D_\theta(\text{audio}, q, c)_{t^*}$, with $r_\tau = W_\tau h^* + b_\tau$.

Here $q$ is the question, $c$ specifies the primitive and legal answers, and $\tau$ selects CHOICE, NOUL, or SCORE. The state is computed before target-answer tokens, so causal attention does not let it see the gold answer .

Quiz

Which readout returns a probability distribution over candidate descriptions supplied in the question?

SCORE

CHOICE

NOUL

$ check

Why the Shared State Is Question-Conditioned

Now the version for readers who want more of the machinery: the key constraint is that the readout state must summarize both the audio and the requested decision, without including the answer itself. The question and answer specification are serialized into the text prefix; adapted audio embeddings are inserted at audio positions, and the shared decoder processes them together .

The readout position $t^*$ is the final token of that prefix. Since it precedes target-answer tokens and the decoder uses causal attention, its representation can depend on the audio and prompt, but not on the supervised answer tokens . The head has three affine projections from this same state: $W_C \in \mathbb{R}^{64\times d}$, $W_N \in \mathbb{R}^{1\times d}$, and $W_S \in \mathbb{R}^{10\times d}$, with $d=2048$ in the described implementation . The requested primitive chooses the active output and objective; three projections do not mean three questions are answered simultaneously [0, 1].

Why Candidate Masking Preserves the Meaning of Choices

Here is the part that gets technical: the choice and score projections have fixed capacity, but an individual prompt may define fewer legal outcomes. For CHOICE with $N$ candidates, only the first $N$ logits are normalized: $p_i = \frac{\exp(z_i)}{\sum_{j=0}^{N-1}\exp(z_j)}$, for $0 \le i < N$. In words, each supplied candidate competes only with the other supplied candidates; unused output positions are excluded .

The candidate descriptions occur in the prompt in the same order as their output indices. Thus an index has no permanent class meaning: the prompt supplies what each position means for this question . The described implementation supports 2–64 candidates . Why is this useful? It lets a fixed projection serve different answer sets without treating spare positions as legal answers.

Why the Three Readouts Mean Different Things

Now the deeper distinction: the same kind of hidden state is turned into different mathematical objects because the answer types ask different questions. CHOICE produces probabilities over its valid candidates and can select the highest-probability option . NOUL applies a sigmoid to its single logit, $p(\text{true})=\sigma(z)$; that is a probability for the stated proposition, while any threshold used to turn it into a yes/no action is a downstream rule .

SCORE instead normalizes over the first $K$ ordered levels and takes their probability-weighted index, $s=\sum_{k=0}^{K-1} k p_k$. In words, it returns the expected level index, not merely the most likely level [1, 2]. The described implementation supports 2–10 levels . If application levels are equally spaced from $a$ to $b$, the index expectation can be mapped to application units; for unevenly spaced levels, take the expectation over the declared values instead . That scale conversion is separate from the native index output . Do these typed probabilities alone guarantee calibration? No: probability quality still has to be evaluated separately .

Minimal Causal Code

This loop makes the valid-answer boundary visible. logits below are the projected outputs for one question; the hidden-state computation is omitted so the normalization and score are easy to inspect.

import torch
import torch.nn.functional as F

# logits: one fixed-capacity output vector; valid_count comes from the prompt.

def choice(logits, valid_count):
    valid_logits = logits[:valid_count]  # exclude unused candidate slots
    probs = F.softmax(valid_logits, dim=-1)
    selected_index = probs.argmax(dim=-1)
    return probs, selected_index

def noul(logit):
    return torch.sigmoid(logit)  # probability the proposition is true

def score(logits, valid_count):
    valid_logits = logits[:valid_count]  # exclude unused rating levels
    probs = F.softmax(valid_logits, dim=-1)
    levels = torch.arange(valid_count, device=logits.device, dtype=probs.dtype)
    expected_level = (probs * levels).sum(dim=-1)
    return probs, expected_level

What You Gain, and What You Lose

You gain outputs shaped like the decision: candidate probabilities for a supplied choice set, a probability for a binary claim, or a distribution-derived expected rating. The answer need not be generated token by token, and unused candidate or level slots do not enter the normalization [0, 1, 2].

You give up the flexibility of an unconstrained textual response at this readout: the question must specify a supported primitive and its legal answer space. The fixed-capacity choice and score projections also impose the implementation’s stated limits, and requests with multiple questions are evaluated separately rather than in one shared question pass [0, 1]. Finally, a probability-shaped output is not proof of calibration .

Quiz

A prompt supplies four valid rating levels, but the score projection has additional output positions. Which logits should enter the score softmax?

The single highest logit before normalization

All projection logits, because the extra positions represent implicit ratings

Only the four valid-level logits

$ check