Calibrated Null-Probability Decoding

Why People Wanted It

A typed extractor has two different questions to answer: does this candidate belong to any requested type, and, if so, which type fits best? A single threshold on the winning type’s probability confuses those questions. If a schema contains overlapping labels—a broad type and a more specific one, for example—the model can spread its evidence across both. Neither individual probability need be large even when the total evidence for a non-null label is.

There is a second problem: training can deliberately give null candidates less loss weight. That can make the model’s reported null odds too low, so its confidence no longer matches its decisions. The real question is how to recover a usable accept/reject confidence without changing which non-null type wins.

The Core Move

First threshold the probability of belonging to any non-null type; only after accepting a candidate, choose its most likely type. Correct the null logit for its training weight, then fit a temperature and decision threshold on calibration data.

For candidate $c$, let $p_c(m)$ be its probability for type $m$, including the null class $0$. The decision is

$1-p_c(0)\geq\tau\quad\Rightarrow\quad \hat m=\arg\max_{m\in\mathcal K}p_c(m).$

The threshold asks whether any queried type applies; the argmax selects the type.

Quiz

A candidate has evidence shared across a broad type and a fine-grained type, so neither type alone has a high probability. What should the extraction gate examine?

The sum of all non-null type probabilities, 1 minus the null probability.

The null logit alone, without converting it to a probability.

Only the probability of the single most likely type.

$ check

Why the Total Non-Null Probability Is the Right Gate

Here is the part that gets technical: the gate is about total non-null mass, not the confidence of the label that happens to win. Suppose a span plausibly matches both a broad and a fine-grained type. Its probability can be divided between those types, while their combined mass still supports extracting the span. Thresholding only the maximum type probability can reject it for that reason.

The rule uses $1-p_c(0)$ as the gate and then uses $\arg\max_{m\in\mathcal K}p_c(m)$ for the label. In words: test the probability of any queried type first, and let the most probable type break the tie only for candidates that pass. The same decision structure applies to relation candidates; overlapping spans are allowed.

This separates acceptance from classification, but what if the null probability itself has been distorted during training?

Why Down-Weighting Null Candidates Miscalibrates Their Score

Now the version for readers who want more of the machinery: reducing the loss weight of null candidates changes the balance the model learns between null and typed outcomes. The resulting null odds are reported too low, as can happen when negatives are down-sampled. This is a confidence problem, not a reason to reorder the non-null types.

If null candidates receive weight $\omega$, correct the null logit $z_0$ by adding $\log(1/\omega)$ before temperature scaling:

$z'_0=z_0+\log(1/\omega).$

This raises the null score to compensate for its reduced training-loss weight. Since the same adjustment is made to the null class and does not alter the relative odds among types, the ranking of candidate types is unchanged at a fixed temperature. The correction needs no additional data. But corrected scores still need a decision scale and a cutoff—how should those be chosen?

Why Temperature and Threshold Need Separate Calibration

Here is the deeper calibration detail: temperature changes the sharpness of the class probabilities, while the extraction threshold controls how much non-null probability is enough to emit a candidate. For entities, fit temperature by minimizing multi-class Brier score and choose the threshold to maximize F1 on calibration splits from the training sources. For relations, use temperature $T=1$ and fit the threshold on REBEL with gold entities. Event predictions reuse entity and relation calibration parameters.

The order matters: add the null correction before dividing logits by temperature. Refit the temperature and threshold after applying the correction. In the reported calibration comparison, average expected calibration error fell from 0.33 to 0.23 for entities, 0.25 to 0.09 for relations, and 0.51 to 0.39 for event arguments, with little change in F1. Those averages show the intended tradeoff: improve confidence estimates while keeping extraction accuracy similar, not guarantee identical predictions everywhere.

Minimal Causal Code

This deliberately exposes the order of operations. logits has null in column zero and one column per queried type.

import math
import torch


def decode(logits, null_weight, temperature, threshold):
    # Restore null score before temperature scaling.
    corrected = logits.clone()
    corrected[..., 0] += math.log(1.0 / null_weight)

    # Calibrate probability sharpness.
    probabilities = torch.softmax(corrected / temperature, dim=-1)

    # Decide whether any queried type applies.
    non_null_probability = 1.0 - probabilities[..., 0]
    accepted = non_null_probability >= threshold

    # Select a type only among candidates that passed the gate.
    predicted_type = probabilities[..., 1:].argmax(dim=-1) + 1
    return accepted, predicted_type

What You Gain, and What You Lose

You gain a decision rule that does not penalize a candidate merely because its non-null evidence is split across overlapping types. The null correction addresses the confidence bias introduced by down-weighting null candidates, and temperature plus threshold fitting makes the gate a calibrated operating point.

You do not get a universal threshold: the temperature and cutoff are fitted using calibration data, and the reported setup uses different fitting choices for entities and relations. The null correction preserves relative type odds at a fixed temperature, but refitting temperature and threshold can change confidence-based acceptance. Calibration improves confidence estimates with little change in average F1 in the reported results; it is not a promise that every dataset or candidate is unchanged.

Quiz

What does adding the null-logit prior correction accomplish at a fixed temperature?

It replaces the need to fit a decision threshold.

It raises the null score to compensate for its reduced training-loss weight, without changing relative type odds.

It raises every type score equally and changes which non-null type ranks first.

$ check