Policy Learning over Hybrid Trajectories
Hybrid trajectories contain continuous latent steps and environment observations with no token likelihood, so HybridRL factors policy probability over sampled text positions and replays the recorded context to update them.
Read the paper ↗What it is
- Hybrid policy factorization A hybrid trajectory contains sampled text, continuous latent thoughts, and tool observations. Policy likelihood is attached to sampled text decisions, while the other parts provide context.
- Exact latent-context replay During the update, replay the recorded latent inputs and tool observations instead of generating replacements, then rescore the sampled text decisions.
- Hybrid policy update Use the recomputed text likelihoods to update the policy against rewards for the complete recorded trajectory.
Why it matters
Ordinary token-based reinforcement learning cannot assign a token likelihood to continuous latent thoughts or environment responses. This approach makes the policy update well-defined at sampled text positions without discarding the hybrid context that shaped them.
Before getting into the mechanics, here is the short version: HybridRL learns from the text decisions an agent actually samples, while keeping its latent thoughts and tool results in the replayed trajectory. That matters because a continuous latent vector is not a sampled token, but it can still change what the agent says next.
- Likelihood: Standard token-based policy updates expect a likelihood for each sampled action. A hybrid agent sometimes produces a continuous latent thought instead, so there is no token likelihood to score there.
- Context: A later text choice depends on what came before, including latent thoughts and tool results. If replay changes that history, it is scoring a different decision context.
- Update: The useful move is to score sampled text positions while replaying the complete recorded context that led to them.
Take a concrete example: an agent reasons in a latent state, asks a search tool a question, receives snippets, and then writes an answer. The answer tokens were sampled by the agent; the snippets came from the environment; the latent state was produced through the fixed HybridLink. Only the answer's sampled text positions need policy likelihoods, but those likelihoods need the latent state and snippets in their prefix. The analogy is a notebook: you can update the choices written on a page while keeping the earlier notes intact; unlike an ordinary notebook, the latent notes are embeddings rather than readable text.
So why not simply regenerate the whole trajectory under the updated model? Because the latent inputs might change, and then the text likelihood would be conditioned on a different history. HybridRL records latent thoughts, text, and tool observations during rollout; during optimization it replays those inputs and recomputes likelihoods for sampled text tokens. In other words, it keeps the old context fixed while asking how likely the current policy finds the recorded text decisions.
The policy factorization says the trajectory's policy likelihood comes from those sampled text positions. Latent continuations are deterministic given the fixed HybridLink and preceding history, and tool observations come from the environment. Put plainly, the policy's probability factors are for sampled text, but each factor sees the full preceding hybrid context. The supplied paper pages state this factorization, but the formula itself is not present in the provided text excerpt, so I am not writing a reconstructed equation.
What does the update do with those scores? It combines the replayed text-position probabilities with trajectory rewards and group-normalized advantages, then applies a standard RL objective. The important boundary is that the probability terms are text-position terms, whereas reward belongs to the complete recorded trajectory. What does this buy, and what does it ask you to preserve? It avoids inventing likelihoods for latent vectors, but requires faithful replay of the latent and tool context. Overall, I like the clean separation: score what was sampled, preserve what conditioned it.
Read the deep dive
Why People Wanted It
The problem is that a hybrid agent does not produce a sampled token at every point in its trajectory. Some positions are latent thoughts represented as continuous vectors; tool observations come from the environment. Standard token-level reinforcement learning needs likelihoods for sampled actions, so these parts do not have the ordinary token likelihood it expects. The real question is whether policy learning must score every part of the trajectory, or whether it can score just the decisions that were actually sampled while still giving them their full context.
The Core Move
Treat sampled text positions as the policy's stochastic decisions, and keep the recorded latent thoughts and tool observations in the trajectory as context. The paper states this factorization in Proposition 4.1, but the displayed formula is absent from the supplied page text, so no equation is reproduced here.
Why Only Sampled Text Positions Carry Policy Likelihood
Now the version for readers who want more of the machinery: the key distinction is between a sampled decision and a deterministic or externally supplied part of the trajectory. With fixed HybridLink, a latent continuation is computed from the preceding history; its conditional probability is one. A tool observation is generated by the environment, not sampled by the agent's policy. Deterministic mode transitions likewise contribute conditional mass one.
The chain rule for trajectory probabilities therefore leaves policy likelihood factors only at the sampled text positions, denoted by . In words, the likelihood of the hybrid trajectory is the product of the model's text-token probabilities, each conditioned on the complete preceding hybrid context. The factorization does not mean latent thoughts or tool results are irrelevant: they affect the context for later text decisions, even though they are not themselves policy-likelihood factors.
Why Replay Must Preserve the Recorded Latents
Here is the part that gets technical: the likelihood for a text decision depends on the prefix that preceded it, including the latent inputs used during rollout. During optimization, replay the trajectory in embedding space, leave those recorded latent inputs unchanged, and recompute the sampled text-token likelihoods with a differentiable forward pass. This is exact latent-context replay: the update evaluates text decisions against the same latent context that was present when the trajectory was collected.
If the update instead substituted newly generated latent inputs, the prefix would no longer be the recorded rollout context. The replayed likelihood would then refer to a changed trajectory history. Keeping the recorded latents fixed makes the context faithful while allowing the text likelihoods to be recomputed under the current policy.
Why the Update Is a Text-Position Update over a Full Trajectory
Now connect that replay to the policy update: the rollout records each trajectory's latent thoughts, text, and tool observations; rewards and group-normalized advantages are computed for the trajectories; replay obtains current-policy probabilities at sampled text positions; and the policy objective updates the model using those probabilities. The ratio terms are computed at those positions, not at continuous latent positions.
The optimization is therefore narrow in where it measures likelihood and broad in what it conditions on: text positions supply the sampled policy decisions, while the full recorded hybrid prefix supplies their context. What remains is how to update a policy without pretending that every part of a hybrid trajectory was a sampled token.
Minimal Causal Code
This sketch makes the replayed context and text-position scoring visible. It is schematic: it does not implement the paper's full RL objective or model interfaces.
# Each step contains a recorded input embedding and, when sampled by the
# policy, the corresponding text token. Tool observations are also replayed.
log_probs = []
history = []
for step in recorded_trajectory:
# Preserve the exact recorded latent input, text input, or observation.
hidden = model.forward_step(step.input_embedding, history)
if step.is_sampled_text:
token_log_probs = model.text_log_probs(hidden)
log_probs.append(token_log_probs[step.token_id])
history.append(hidden)
trajectory_log_prob = sum(log_probs)
What You Gain, and What You Lose
You gain a well-defined policy likelihood for a hybrid trajectory without assigning token probabilities to continuous latent vectors or environment observations. Exact replay also keeps the context for each rescored text decision aligned with the recorded rollout.
The tradeoff is that the update depends on recording and faithfully replaying the latent inputs and tool observations. The policy likelihood is not a token-by-token account of every trajectory component; it is a factorization over sampled text positions, conditioned on the hybrid context. The paper keeps HybridLink fixed during this procedure to preserve deterministic latent transitions.
Score the sampled text decisions, but replay the whole recorded hybrid context that made those decisions meaningful.