HybridLink: Bridging Latent Reasoning and Text

Why People Wanted It

A model can reason by generating text, or it can take recurrent steps through continuous hidden states. Text is inspectable and can carry tool calls, but each step must be decoded and embedded again. A latent step avoids that round trip, yet feeding a last-layer hidden state directly back as an input embedding creates a representation mismatch with the pretrained input space. So the real question is not simply how to add more internal steps: how can the model move between latent reasoning and ordinary text without making either mode an awkward guest in the other’s interface?

The Core Move

HybridLink is a lightweight residual adapter that maps a hidden state into a next-step input embedding, while special entry and exit anchors let generation switch between latent and text modes.

$\tilde e_{t+1}=h_t+W_2\,\sigma(W_1h_t)$

The residual keeps the hidden state’s semantics in the result; the learned projections adjust it toward the input-embedding space.

Why the Residual Link Bridges Two Spaces

Now the version for readers who want more of the machinery: the key constraint is that the adapter has to change the hidden state’s distribution without discarding what the state means. Let $h_t$ be the backbone’s last-layer state, and let $W_1$ and $W_2$ be linear projections with an activation $\sigma$ between them. The adapter’s output is $h_t+W_2\sigma(W_1h_t)$: the learned branch supplies a correction, while the skip connection carries the original state forward. This makes the module a residual alignment step, not a fresh projection that must reconstruct the full representation from scratch.

The resulting vector is used as the next input in latent mode. That gives the model a recurrent sequence of continuous representations without decoding each one into a token and embedding it again. The exact projection dimensions and activation are implementation choices; the defining structure is the residual correction. What does that buy—and where does the model decide to use it?

Why Anchors Make Mode Changes Part of Generation

Here is the part that gets technical: text mode follows the usual token-generation path until the model emits the special entry anchor <latent>. The next forward pass switches paths: instead of taking a vocabulary token through the embedding layer, it takes the preceding hidden state through HybridLink. In latent mode, this hidden-to-input update repeats at each step.

The exit decision is also tied to generation state: the latent span ends when the last-layer hidden state aligns with the exit anchor </latent>. The model then returns to text generation. Entry and exit are therefore not a fixed schedule imposed from outside; the mode can continue or switch as generation proceeds. This describes the interface and its stated alignment condition, not a particular numerical similarity threshold. How can a span stay latent yet still be inspected?

Why a Latent Step Can Still Be Read Out

Now for the interpretability detail: latent generation need not decode a token at every step, but a latent step can be inspected afterward by mapping its HybridLink output to the nearest token embedding. That nearest-embedding readout gives a readable pseudo-text view of the latent state; it is an inspection route, not a requirement that the recurrent computation itself pass through token decoding.

The distinction matters: the model can continue its internal computation in continuous space while a person uses the readout to examine what a span appears to express. The resulting pseudo-text is a view into the representation, not a claim that the latent trajectory was generated as ordinary text. What do you trade for combining these paths?

Minimal Causal Code

This loop isolates the latent update: each last-layer state is corrected by the adapter and used directly as the next input. A real model also needs the text-mode path and its entry/exit decisions.

import torch
import torch.nn as nn

class HybridLink(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.w1 = nn.Linear(d, d)
        self.w2 = nn.Linear(d, d)
        self.act = nn.GELU()

    def forward(self, h):
        return h + self.w2(self.act(self.w1(h)))

# `backbone_step` consumes one input-space vector and returns its last-layer state.
link = HybridLink(d=hidden_size)
state = first_hidden_state
for _ in range(latent_steps):
    next_input = link(state)          # visible hidden -> input-space update
    state = backbone_step(next_input) # next recurrent latent step

What You Gain, and What You Lose

You gain recurrent latent steps without token decoding and re-embedding at every step, plus explicit transitions back to ordinary text generation. The residual adapter is designed to preserve state semantics while aligning it toward the input space, and nearest-token-embedding readout offers a way to inspect latent spans.

You also add a learned interface and a mode-transition mechanism: the latent path depends on the adapter, and interpretation depends on a nearest-embedding view that is not itself the latent computation. Text remains useful for tool interaction and readable output; latent steps are not automatically text just because they can be read out. HybridLink addresses the within-generation bridge, while other parts of the system handle carrying hybrid context across turns and training reliable mode behavior.

Quiz

What is the role of HybridLink in latent mode?

It converts each latent step into a sampled vocabulary token before the next pass.

It replaces the backbone’s last-layer representation with a nearest-token embedding.

It stores tool responses in a separate text-only memory.

It maps the hidden state into an input-space vector for the next latent step.

$ check

Quiz

How does the generation interface enter and leave latent mode?

It enters after a fixed number of text tokens and exits after a preset latent length.

It enters whenever a tool call occurs and exits when the tool responds.

It stays in latent mode for the remainder of generation once entered.

It enters when `<latent>` is generated and exits when the state aligns with `</latent>`.

$ check