Sparse Mixture-of-Experts Text Encoder
Decoder-only models use one-way context by default; this encoder replaces causal masking with full bidirectional visibility and uses sparse expert routing to produce normalized pooled embeddings.
What it is
- Bidirectional encoder The model lets each token use information from both earlier and later tokens, rather than restricting it to earlier context.
- Sparse experts The largest encoder has many specialized expert networks, but routes each token to only a selected subset, alongside a shared expert.
- Pooled embedding The encoder averages token representations into one vector and normalizes it for comparing texts by cosine similarity.
Why it matters
This adapts a decoder-style model for whole-text representations while keeping expert computation sparse: the largest model has 10B total parameters, with approximately 1.8B active per token.
Before getting into the machinery, here is the short version: take a decoder-style model, let tokens see context on both sides, and turn all the token representations into one normalized vector. In the largest version, each token uses a small selected group of experts instead of the full bank. That gives you a text encoder built around both whole-input context and sparse model capacity.
-
Context: A decoder’s usual attention mask prevents a token from seeing later tokens. For an embedding of a complete text, that is an awkward restriction: the meaning of a word can depend on what follows it. The canonical example is an ambiguous word whose sense is clarified by the next phrase. Letting the token use right-hand context can change its representation; it does not mean the model has become a different kind of language model in every respect.
-
Capacity: A large expert model contains many routed experts, but a token need not use all of them. The largest encoder described here has 64 routed experts, one shared expert, and top-4 routing; its 10B-A1.8B name refers to 10B total parameters and approximately 1.8B active parameters per token. Think of a workshop where each job is sent to a few specialist benches while a common bench remains available. The analogy stops short of explaining the router: the architecture description gives the selection counts, not the routing equation.
-
One vector: Once each token has a context-aware state, the encoder needs to turn the sequence into something retrieval can compare. Mean pooling is like taking a summary across the token states, and normalization puts the resulting vector in the form used for cosine similarity. The recipe states these operations but does not specify a pooling or normalization equation here.
So the real question is not only how much context each token can use, but how much model capacity it needs to do useful encoding. Sparse routing answers the capacity side by selecting experts, while bidirectional attention answers the context side. The remaining practical question is how the router makes those selections for a particular token.
Read the deep dive
Why People Wanted It
A decoder-only language model naturally reads a sequence from left to right. An embedding encoder has a different job: represent the whole input, so a token should be able to use context on both sides. At the same time, a large expert model offers substantial total capacity, but running every expert for every token would throw away the point of sparse routing. The real question is how to turn that model into a whole-text encoder while retaining sparse computation.
The Core Move
Replace the causal attention mask with a fully visible bidirectional mask, then pool and normalize the token states; in the largest encoder, each token is routed to four of 64 routed experts alongside a shared expert. The paper gives these architectural settings, but does not provide a mathematical formula for the routing operation here.
Why Bidirectional Visibility Changes the Token States
Now the version for readers who want more of the machinery: a causal mask blocks a token from attending to positions on its right, whereas the replacement mask makes both left and right context available. This changes the context each token state can represent; it is not merely a change to how the final vector is collected. Mean pooling then aggregates the token states into a fixed-dimensional vector, and normalization prepares that vector for cosine-similarity scoring. The described recipe gives these operations, not a pooling or normalization equation.
Why Sparse Routing Keeps the Large Model Practical
Here is the part that gets technical: the largest model is a bidirectional DeepSeekMoE-style encoder with 10B total parameters, 64 routed experts, one shared expert, and top-4 routing. “A1.8B” denotes approximately 1.8B active parameters per token. The distinction is between total parameters present in the model and the active parameter count for a token: routing selects a subset rather than activating all routed experts. The shared expert is part of the stated architecture, but the available description does not specify its internal computation or a routing equation.
Minimal Causal Code
This sketch makes the described dataflow visible. The paper does not give routing or pooling equations, so the expert-selection interface and the ordinary mean-and-normalize steps below are illustrative implementation choices, not a reproduction of a paper-specified formula.
import torch
import torch.nn.functional as F
# Illustrative interface: the encoder supplies token states after
# bidirectional attention and sparse expert routing.
token_states = encoder(input_ids, attention_mask=fully_visible_mask)
# Keep valid tokens, average their states, then normalize the vector.
weights = valid_tokens.unsqueeze(-1).to(token_states.dtype)
pooled = (token_states * weights).sum(dim=1) / weights.sum(dim=1).clamp_min(1)
embedding = F.normalize(pooled, p=2, dim=-1)
What You Gain, and What You Lose
Bidirectional visibility lets token representations use context from both sides, which fits whole-input embeddings rather than next-token prediction. Sparse routing gives the largest model 10B total parameters while its release designation corresponds to approximately 1.8B active parameters per token. The tradeoff is architectural complexity: the encoder must combine a bidirectional attention mask, expert routing, pooling, and normalization. And the provided description does not specify the routing equation, so the exact router behavior cannot be reconstructed from these settings alone.
Make the whole sequence visible, then let each token use a selected slice of the model’s expert capacity.