UltraBERT: Saturation and Masking Choices for Encoder Tasks

Why People Wanted It

More pretraining tokens can make an encoder better at predicting its training objective without making it better at the tasks you care about. That matters because a lower masked-token loss can look like progress even when downstream performance has stopped moving. So the real question was never simply how long to train: after downstream performance levels off, which pretraining choices still shape the model’s usefulness?

A second choice is how much text to hide during masked-language-model pretraining. A heavily masked example asks the model to reconstruct more missing content; a lightly masked one preserves more of the original context. The downstream preference turns out to depend on whether the task is classification or retrieval.

The Core Move

Treat downstream saturation as a point to compare masking choices, then select the masking ratio for the task rather than assuming one ratio serves every use. In the reported comparisons, classification favored lower masking, while retrieval favored higher masking.

Quiz

In the reported saturated encoder comparisons, how did masking-ratio preferences differ by downstream task?

Higher masking favored both classification and retrieval.

Lower masking favored both classification and retrieval.

The masking ratio had no observed relationship to either task.

Lower masking favored classification; higher masking favored retrieval.

$ check

Why Falling Loss Stops Predicting Downstream Gains

Here is the first deeper layer, for readers who want more of the measurement logic: the key distinction is between optimizing the pretraining objective and improving a fine-tuned task score. For 150M and 1.3B encoders trained with 30% masking, intermediate checkpoints were evaluated on GLUE, SQuAD 2.0, and BEIR while training continued. Downstream scores leveled off around 1–2 trillion tokens even as training and validation losses kept decreasing. BEIR showed no significant improvement beyond its first reported evaluation point, at 275B tokens.

That gap makes loss an incomplete stopping signal for these encoder tasks: it can keep improving after task performance has flattened. The result is specific to the tested encoder setups and evaluations, not a claim that every encoder or task saturates at the same budget. What should you compare once more tokens are no longer buying clear downstream gains?

Why More Masking Hurts Classification at Saturation

Now the version for readers who want the masking comparison in detail: masking removes visible context from the input, so raising the ratio makes reconstruction more demanding but also corrupts more of the context a representation can use. The comparison spans three encoder sizes and masking ratios up to 50%, with token budgets chosen to account for slower convergence at lower ratios. Across the reported GLUE and SQuAD 2.0 results, classification performance degraded consistently as the masking ratio increased.

This is the tradeoff that saturation helps expose. At smaller training budgets, a higher ratio can help a model converge faster; when the comparison trains to saturation, that speed advantage is less central, and the cost of additional corruption becomes more visible. The reported result does not say that masking is always harmful, only that these classification evaluations preferred less of it under the tested saturated training setup. If classification wants more context preserved, what happens when the downstream task is retrieval?

Why Retrieval Prefers a More Used Representation Space

Here is the more technical layer for the second task: retrieval reverses the observed preference. Across the tested model scales, 50% masking achieved the highest BEIR average nDCG@10, while classification favored lower ratios. The measurements also found that effective rank—the number of dimensions meaningfully used by mean-pooled representations—tended to increase with masking ratio.

That association offers a plausible representation-level clue: retrieval may benefit when vectors make use of more of their available dimensions. But the measurements do not establish that higher effective rank causes the retrieval improvement; the two could be related or independent. There is therefore no single best masking ratio for both task families in these comparisons. How do you keep that task-specific choice visible in the simplest possible training loop?

Minimal Causal Code

This loop makes the masking ratio an explicit choice. It is illustrative rather than a reproduction of the experiments; the selection happens when constructing the batch.

import torch

# Choose the ratio for the downstream task being prepared for.
mask_ratio = 0.15  # lower for classification; higher for retrieval

for tokens in loader:
    corrupted, targets = make_mlm_batch(tokens, mask_ratio=mask_ratio)
    logits = encoder(corrupted)
    loss = masked_token_loss(logits, targets)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

What You Gain, and What You Lose

The practical gain is a clearer allocation of pretraining effort: do not treat falling language-model loss as proof that downstream scores are still rising, and do not assume classification and retrieval want the same masking ratio. In the reported experiments, lower masking favored GLUE and SQuAD 2.0, while 50% masking led on BEIR.

The cost is task dependence. A ratio chosen for classification can give up retrieval performance, and the reverse choice can hurt classification. Saturation is also an empirical observation from particular model sizes, token budgets, and evaluations—not a universal token threshold. The design question is therefore not “Which ratio is best?” but “Which task’s tradeoff are you choosing?”