Giga-Embeddings: Sparse MoE Text Encoder
Decoder-only language models cannot see both sides of a token, so Giga-Embeddings makes the encoder bidirectional and routes each token through a small subset of experts.
Read the paper ↗What it is
- Bidirectional encoder adaptation The model changes a left-to-right language model into an encoder whose tokens can use context from both earlier and later words.
- Sparse expert routing The largest model has many expert networks, but each token is routed through only a small selected group, plus a shared expert.
Why it matters
This design combines whole-input context with a large pool of model capacity without activating that entire pool for every token. It trades simpler dense execution for sparse routing and expert-serving complexity.
Before the details, here is the short version: Giga-Embeddings takes a decoder-style language model and changes it into an encoder that can read context from both sides of each token. Its largest version also has many expert networks, but only a selected few are active for any one token, so total model size and per-token active size are different things.
-
Context: A decoder reads left to right, which is useful for generating the next token but leaves later words unavailable to earlier token representations. For embeddings, I want the representation of a word to be able to use the whole input. The canonical example is a sentence where the word “bank” is disambiguated by a word that comes after it: a left-to-right view cannot use that later clue when processing “bank,” while a bidirectional view can. That does not mean every downstream task needs both directions equally, but it explains the encoder adaptation.
-
Capacity and activation: A large pool of expert networks can increase the model’s total capacity without running every expert for every token. The canonical example is Giga-Embeddings-10B-A1.8B: it has 10 billion total parameters, 64 routed experts, one shared expert, and top-4 routing; the A1.8B name denotes about 1.8 billion active parameters per token. In notebook terms, the model keeps a large library but consults a selected few pages for each token. The analogy breaks because experts are learned neural networks, not stored passages.
So the real question was never simply “how do we make a decoder into an embedder?” It was how to let tokens use both sides of the input while keeping the largest encoder sparse. The model replaces the causal attention mask with a fully visible bidirectional mask, then mean-pools token states and normalizes the vector for cosine comparison. In plain language: every token can read the whole sequence, and the encoder averages the resulting token representations into one normalized embedding.
What does sparse mean here in operational terms? Instead of running all routed experts for a token, the router selects four from the pool of 64, while the architecture also has one shared expert. The paper specifies top-4 routing and the expert counts, but its model description does not state a routing equation or the gate-scoring details. So I would not infer a particular formula or dispatch scheme from the word “top-4” alone.
This is a useful separation of accounting: 10B refers to total parameters in the model, while about 1.8B refers to active parameters per token. The model remains large to store and serve; sparse activation is not the same thing as shrinking it into a dense 1.8B model. Overall, I like the clarity of that distinction, but it also means sparse routing has to work well in the serving setup. What does the design buy, and what operational complexity does it add?
Read the deep dive
Why People Wanted It
A text embedding model has to turn a whole input into one vector that works for retrieval or semantic comparison. A decoder-only language model is trained to read left to right, but an embedding encoder can benefit from letting each token use both its left and right context. At the same time, a large model can carry substantial capacity without needing every parameter to run for every token.
So the real question is: can you keep a large pool of model capacity, make the representation bidirectional, and activate only part of that pool for each token?
The Core Move
Replace causal attention with a fully visible bidirectional mask, pool the resulting token states into an embedding, and use sparse expert routing in the largest model. The concrete model here has 10 billion total parameters, with top-4 routing across 64 routed experts plus one shared expert; its A1.8B designation means about 1.8 billion active parameters per token.
The paper does not give a routing equation in the cited model description, so there is no routing formula to reproduce here. Its stated mechanism is top-4 routing: each token uses four routed experts, alongside the shared expert.
Why Bidirectional Context Changes the Encoder
Here is the part that goes beyond the basics: the adaptation changes what information is available to each token, not just how the final vector is read out. A causal mask prevents a token from attending to later positions; the fully visible mask lets it use both earlier and later tokens. The model then mean-pools token states and normalizes the resulting fixed-dimensional vector before cosine-similarity scoring.
That combination turns a decoder-only backbone into an encoder for whole-input representations. The paper specifies the mask change, pooling, and normalization, but does not provide a new equation for this adaptation. What does bidirectional visibility buy if the encoder still has to compress the whole input into one vector?
Why Top-4 Routing Keeps the Model Sparse
Now the version for readers who want more of the machinery: the 10B-A1.8B model is described as a bidirectional DeepSeekMoE-style encoder with 64 routed experts, one shared expert, and top-4 routing. The total parameter count describes the full expert pool; the active-parameter designation describes the approximate amount active per token. These are different quantities, so “10B” does not mean that every token executes all 10 billion parameters.
The routed experts provide token-dependent expert selection, while the shared expert is present in addition to that routed selection. The source gives these architectural counts and the active-parameter designation, but no gate equation or detailed dispatch algorithm in the cited description; adding one would claim more than the stated mechanism. How much of the capacity advantage survives the cost of routing and serving a sparse model?
Minimal Causal Code
The following sketch makes the architectural distinction visible: bidirectional attention exposes all positions, while a top-k choice selects routed experts for a token. It is explanatory pseudocode, not a reproduction of the model’s undisclosed router implementation.
# x: token states for one sequence
# all positions are visible to every token
attention_mask = fully_visible_mask(x.shape[0])
x = transformer_blocks(x, attention_mask=attention_mask)
# Illustrative sparse dispatch; the cited description specifies top-4 routing,
# but does not specify this router's scoring or dispatch implementation.
for token_state in x:
scores = router(token_state)
expert_ids = top_k(scores, k=4)
routed = sum(
scores[j] * experts[j](token_state)
for j in expert_ids
)
token_state = shared_expert(token_state) + routed
embedding = mean_pool(x)
embedding = normalize(embedding)
What You Gain, and What You Lose
The model can retain a large total expert pool while activating only about 1.8 billion parameters per token, and its bidirectional mask lets token representations use both left and right context. That is the central tradeoff: sparse activation is not the same as a small model, and the design adds routing and expert-serving complexity rather than making the whole parameter store disappear.
The encoder’s final embedding is a fixed-dimensional pooled vector, so token-level information must be compressed before cosine comparison. The paper gives the 10B-A1.8B embedding dimension as 1536, but this module’s focus is the encoder adaptation and routing rather than the separate compact-model distillation recipe.
The encoder sees both sides of the text, but each token activates only a slice of the large expert pool.