Joint Learning from Partially Labeled Audio Tasks

Why People Wanted It

An audio dataset rarely gives every recording every kind of answer. One clip may be labeled with a sound category, another with a yes/no judgment, and another with an ordered rating. Treating all of these as if they were the same target either discards useful supervision or asks the model to learn from labels that were never observed.

So the real question was not just how to predict different answers. It was how to put heterogeneous questions into one training stream while making sure each example teaches only what its annotation actually supports.

The Core Move

Represent each supervised item as an audio-conditioned question with an active answer primitive, valid criteria, and an observed target; apply only that primitive’s loss when its target is present.

For example, the same training system can compare a category distribution with a class target, a binary probability with a true/false label, or an ordered rating distribution with a rating annotation. Missing targets contribute no loss.

Quiz

A recording has a reliable Choice label but no Score annotation. What should its supervised update do?

Apply all primitive losses using the Choice label as their target.

Apply the Choice loss and omit Score loss for that recording.

Treat the missing Score as level zero.

Skip the recording entirely because it lacks a complete set of labels.

$ check

Why the Supervision Format Can Hold Different Tasks

Now the version for readers who want more of the machinery: an item can be described as audio, question and criteria, active primitive, and target, while its supervision is defined only for the observed target. The active primitive determines how the target is interpreted; the valid criteria specify the answer space for that question. This lets one stream contain categorical, binary, and ordinal examples without pretending they share a label geometry.

For Choice, let $C_i$ be the valid candidates, $z_{ik}$ their logits, and $t_{ik}$ the target mass. The probability is $p_{ik}=\operatorname{softmax}_{k\in C_i}(z_{ik})$, and the loss is $-\sum_{k\in C_i}t_{ik}\log p_{ik}$. In words: normalize only over valid answers, then penalize probability assigned away from the target distribution. A hard label is the special case where all target mass sits on one candidate. Unused output positions are excluded and receive no target mass.

For Noul, the binary proposition has probability $p_i=\sigma(z_i)$ and observed outcome $y_i\in[0,1]$. Its binary cross-entropy is $-y_i\log p_i-(1-y_i)\log(1-p_i)$. In words: reward probability on the observed side of the proposition and penalize probability on the other side; fractional annotations can represent a graded target. The label geometry is now true versus false, not a set of interchangeable categories. What should the loss do when the possible answers have an order?

Why an Ordered Rating Needs More Than Category Matching

Here is the part that gets technical: Score supervises a distribution over ordered levels and combines distribution matching, cumulative-distribution distance, and expected-value error. For levels $0$ through $K_i-1$, let $p_{ik}$ and $t_{ik}$ be predicted and target probabilities; their means are $\mu_i^p=\sum_k k p_{ik}$ and $\mu_i^t=\sum_k k t_{ik}$. In words: each distribution also implies a scalar rating by averaging its level indices.

The loss is $L_i^{score}=L_{CE}+0.25L_{CDF}+0.10L_{mean}$, where $L_{CE}$ matches the level distribution, $L_{CDF}$ penalizes differences between cumulative probabilities across ordered thresholds, and $L_{mean}$ applies SmoothL1 to the predicted and target means on the native rating scale. In words: match the annotation, notice how far probability has moved along the order, and keep the expected rating near its target. The coefficients are part of the stated objective; they do not turn the ordered levels into nominal categories. How does this per-example logic scale when annotations are absent or task frequencies differ?

Why Missing Labels and Sampling Change the Objective

Now the exact constraint: each loss reduction processes one observed question, and an example with no reliable target for a primitive contributes no loss for that primitive. If $t_i$ is the active primitive for example $i$, the decision objective is the expectation of its corresponding loss, $L_{decision}=\mathbb{E}_i[L_{t_i}(i)]$. In words: each sampled example contributes the objective selected by its own task, rather than an invented target for every possible task.

That makes the training stream’s sampling distribution consequential: the relative exposure of Choice, Noul, and Score is controlled by how often each appears in the stream. The overall SFT objective also includes the backbone’s aggregated load-balancing loss and router z-loss, with auxiliary coefficients; it does not include a next-token or multi-token prediction objective. Missing-label handling prevents unsupported gradients, but it does not by itself balance task exposure. How much of the final behavior comes from annotation coverage, and how much from the sampler?

Minimal Causal Code

A plain loop makes the key rule visible: choose the loss from the observed primitive, and skip examples without an observed target.

import torch
import torch.nn.functional as F

def observed_loss(logits, example):
    # Missing targets teach nothing for this example.
    if example["target"] is None:
        return None

    if example["primitive"] == "choice":
        valid = example["valid_mask"]
        target = example["target"]
        log_probs = F.log_softmax(logits[valid], dim=-1)
        return -(target[valid] * log_probs).sum()

    if example["primitive"] == "noul":
        target = torch.as_tensor(example["target"], dtype=logits.dtype)
        return F.binary_cross_entropy_with_logits(logits.squeeze(), target)

    if example["primitive"] == "score":
        target = example["target"]
        probs = F.softmax(logits, dim=-1)
        levels = torch.arange(logits.numel(), device=logits.device, dtype=logits.dtype)
        mean_pred = (probs * levels).sum()
        mean_target = (target * levels).sum()
        cdf_pred = probs.cumsum(dim=-1)
        cdf_target = target.cumsum(dim=-1)
        ce = -(target * probs.clamp_min(1e-8).log()).sum()
        cdf = (cdf_pred - cdf_target).abs().mean()
        mean = F.smooth_l1_loss(mean_pred, mean_target)
        return ce + 0.25 * cdf + 0.10 * mean

    raise ValueError("unknown primitive")

losses = []
for logits, example in batch:
    loss = observed_loss(logits, example)
    if loss is not None:
        losses.append(loss)

if losses:
    batch_loss = torch.stack(losses).mean()
    batch_loss.backward()

What You Gain, and What You Lose

The format lets one training stream use categorical, binary, and ordered annotations without converting them into a single misleading target type. Missing annotations are not treated as negatives or guessed labels, and each observed target gets a loss suited to its structure. Ordered ratings can express that nearby levels are less displaced than distant ones through the cumulative and mean terms.

The tradeoff is that task exposure still depends on sampling: a primitive shown less often receives less relative training exposure. The objective also depends on annotation quality and on choices such as the Score loss weights. A unified interface makes mixed supervision manageable; it does not make sparse labels complete, remove task imbalance, or guarantee that ratings from different sources mean the same thing.