Preference Learning for Natural Speech

Why People Wanted It

A text-to-speech model may already know how to pause, group words, and vary its tone, yet fail to do those things consistently. A single reference recording or a fixed rule cannot capture what sounds natural in every context: a pause that helps a sentence land can make a phone number sound broken. So the real question was never just how to make speech intelligible. It was how to make the model choose its more natural delivery more often without losing the words it was meant to say.

The Core Move

Generate several takes of the same sentence, score each take for contextual naturalness and word accuracy, then increase the likelihood of the takes that score well relative to their peers.

For sentence $q$, sample takes $o_1,\ldots,o_G$ and assign each a reward $r_i$. GRPO turns each reward into a group-relative advantage:

$\hat A_i=\frac{r_i-\operatorname{mean}(r_1,\ldots,r_G)}{\operatorname{std}(r_1,\ldots,r_G)}$

A positive advantage says “make this take more likely”; a negative one says “make it less likely.”

Quiz

For a sentence, one take scores above its group's average reward. What does its positive GRPO advantage signal?

Train a critic to predict the take's reward.

Ignore the take because raw scores cannot guide updates.

Change the sentence so the group scores become equal.

Make that take more likely relative to the group's other takes.

$ check

Why Two Rewards Pull Against Different Failure Modes

Here is the part that gets technical: the reward combines a contextual naturalness score with a word-error check, because optimizing either one alone leaves a predictable gap. The judge receives both the audio and intended text; a speech-to-text system supplies the check against the words that were requested.

Naturalness is not a fixed acoustic target. The same pause can sound thoughtful in one sentence and misplaced in another, so the judge evaluates delivery in context. Word error rate (WER) checks whether words were substituted, omitted, or added. Naturalness alone could reward relaxed but unclear speech; WER alone could favor careful, flat delivery. Combining them makes the update answer two questions at once: did this sound appropriate here, and did it say the right thing?

The reward can be written schematically as $r_i=f(n_i,w_i)$, where $n_i$ is the naturalness score and $w_i$ is the word-error signal. In words, the combined reward uses the judge’s assessment and the transcription check; their relative weighting determines how those signals trade off. Why should the naturalness judge be trusted to understand context rather than just acoustic polish?

Why the Judge Needs Text as Well as Audio

Now the version for readers who want more of the machinery: the naturalness judge is a multimodal language model that listens to a take while reading the sentence it was supposed to deliver. This pairing matters because identical timing can be appropriate for one text and awkward for another.

The judge’s score is therefore not simply a measurement of pauses or speaking rate. It is a learned assessment of how the audio fits the text and its conversational context. A general-purpose judge may have a broad sense of natural speech, but “good” delivery still depends on the product’s voice and situation; agreement with human listeners is an important check on that judgment. What turns these scores into a learning signal without training a second model to predict them?

Why GRPO Uses the Other Takes as Its Reference

Here is the deeper optimization step: GRPO compares takes from the same sentence, rather than treating a raw reward as meaningful on its own. A sentence that is intrinsically difficult may receive lower scores across all its takes; a simple sentence may score highly even when its delivery barely varies. Within-group comparison makes the learning signal about which delivery worked better for that sentence.

For group rewards $r_1,\ldots,r_G$, each take’s normalized advantage is its distance from the group mean, scaled by the group’s reward spread. Every token in a take receives that same advantage, so a better-than-group take pushes its sampled sequence toward higher probability, while a worse-than-group take pushes it downward. Unlike PPO’s learned critic, the group itself supplies the reference. That removes the need for a separate critic model, but it means the method needs multiple takes per sentence and cannot learn a preference from a group whose scores do not distinguish its members. How much does that relative signal buy, and what does it cost?

Minimal Causal Code

This loop makes the group comparison visible. reward() stands for the combined naturalness and word-accuracy scoring; the update step is intentionally left schematic because a real policy-gradient implementation also tracks token log-probabilities.

for sentence in sentences:
    takes = [voice.sample(sentence) for _ in range(group_size)]
    scores = [reward(sentence, take) for take in takes]

    mean = sum(scores) / len(scores)
    variance = sum((score - mean) ** 2 for score in scores) / len(scores)
    spread = max(variance ** 0.5, 1e-8)
    advantages = [(score - mean) / spread for score in scores]

    # Positive advantage raises the sampled take's likelihood; negative lowers it.
    policy_gradient_update(voice, sentence, takes, advantages)

What You Gain, and What You Lose

Group-relative feedback makes the learning target comparative: it favors the stronger delivery among takes of the same sentence, while the combined reward asks for naturalness without abandoning the words. Using the group mean instead of a learned critic avoids maintaining that extra model.

The tradeoff is sampling and judgment. Several takes are needed to form a useful comparison, and if their scores are tied or noisy, the group offers little reliable direction. The judge can also encode a generic idea of naturalness that differs from a product’s intended voice; the word check does not resolve that mismatch. Separate guardrails address drift and human checkpoint selection, but they are not the learning rule described here.

Quiz

Why compare several takes of the same sentence rather than use each raw reward as the whole learning signal?

It removes the need to generate multiple takes.

It provides a sentence-specific reference and avoids needing a separate learned critic.

It guarantees the judge agrees with every human listener.

It makes word accuracy irrelevant to the reward.

$ check