Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev, Fyodor Minkin

arXiv 2608.23806 7 pages Read the paper

The short version

The problem: Text search and other language tools need to compare meaning, but accurate embedding models can be expensive to run and store. This makes it harder to serve large volumes of requests or use strong models on limited hardware.
The idea: The paper offers two ways to balance quality and cost. One model has a large pool of specialist components but uses only a small selected subset for each token, like calling only the relevant experts; another small model learns from a larger model’s pattern of which candidate texts are more similar, without needing their internal representations to have the same size.
What it got: The large model processes 25% more tokens per second than the family’s dense 3-billion-parameter model at 1,024-token inputs, while leading the family’s benchmark averages. The compact 480-million-parameter model scores 70.98 on Russian MTEB, just above FRIDA’s 70.95.

Modules from this paper

Dimension-Agnostic Similarity-Distribution Distillation

Compact embeddings need to inherit a stronger encoder’s ranking behavior without matching its vector size, so the student learns the teacher’s candidate-score distribution alongside contrastive learning.

embeddingsknowledge-distillationcontrastive-learningretrieval

Giga-Embeddings: Sparse MoE Text Encoder

Decoder-only language models cannot see both sides of a token, so Giga-Embeddings makes the encoder bidirectional and routes each token through a small subset of experts.

embeddingsmixture-of-expertstext-encoderssparse-models