REKI: Registering Knowledge in VLAs
REKI pairs a vision-language backbone with a flow-matching action expert and a single-stage pretraining recipe over curated public robot, human, and vision-language data.
Read the paper ↗What it is
- Separate perception and action A vision-language model reads camera views and instructions. A separate action model uses that context and the robot’s state to generate a short sequence of commands.
- Public data, explicit differences Robot demonstrations, human manipulation recordings, and image-language examples train one system. Robot datasets keep their own control conventions rather than being forced into one supposedly universal command format.
- One pretraining stage Broad training on the public-data mixture comes before separate adaptation to each benchmark. One pretraining stage does not mean there is no later fine-tuning.
Why it matters
An inspectable data-to-deployment pipeline gives engineers a concrete baseline for testing model and data choices without relying on private robot collections. The tradeoff is that curation, incompatible controls, and task-specific adaptation still require work.
REKI combines a model that reads images and instructions with a separate model that generates robot actions. It trains them using curated public robot demonstrations, human manipulation recordings, and vision-language examples. The useful idea is not that these sources are naturally compatible—it is that their differences can be handled explicitly in an inspectable recipe.
- Reproducibility: Private collections and elaborate training pipelines make it difficult to isolate what helped. A concrete example is changing the action model while keeping the public training mixture fixed: an open baseline makes that comparison possible.
- Control compatibility: One robot dataset may store joint-position targets while another stores changes in end-effector pose. Those commands can fit the same tensor shape without meaning the same thing.
- Broader supervision: Human wrist-and-hand motion offers manipulation targets without robot proprioception; spatial-grounding examples offer supervision about where things are. Neither should be mistaken for another robot demonstration.
So how does the model connect understanding to control? The vision-language backbone produces a sequence of context tokens, and the action expert consults that sequence while generating actions. For example, an overhead view and an instruction to move an object enter the context stream. Think of that stream as material the controller can look up, not as a finished motor command; the analogy stops at lookup because the representations are learned, not written facts.
Robot state takes a different route. For a joint-position state vector, a layout-specific mapping turns the values into a compact signal that modulates the expert's layers. The same signal is reused during action generation rather than recomputed at every refinement step. The action chunk changes as the expert refines noisy actions through Euler steps; the current frame's encoded context and state embedding do not. What, then, tells the controller which kind of command it is producing?
Prompts and state-layout projections supply the convention, while normalization and padding supply a common training container. Return to joint positions versus end-effector changes: each keeps its native meaning, each is scaled using its dataset's quantiles, and unused padded entries are excluded from the loss. I like this explicitness—matching tensor shapes is not the same as matching semantics. How much work remains outside the model?
Quite a bit. Curation removes static or near-constant episodes, retains commanded degrees of freedom, and corrects sign conventions. Human targets use wrist motion and hand articulation without proprioceptive state. Spatial-language examples are mixed in at a small ratio to preserve perception and spatial reasoning.
These data interfaces and curation choices are described in §5.1, p. 6; the two conditioning routes are described in §3.2, p. 4.
So why call the recipe single-stage? It uses one broad public-data pretraining stage, without a world model or extra pretraining stages, and then fine-tunes separately for each benchmark. For example, adapting the pretrained model to a benchmark is still a separate step—it has not vanished into the word “open.”
REKI is “pretrained in a single stage on public robot, egocentric human and vision–language data, and then finetuned on each benchmark.” — §8, p. 12
That leaves the practical question: which data source, architecture choice, or objective would you change while keeping the rest of this baseline fixed?
Read the deep dive
Why People Wanted It
A robot policy can look capable because it learned from a large private dataset, a costly multi-stage recipe, or both. That makes it hard to tell which ingredients matter—and hard for another team to reproduce the result. So the real question was never just whether another robot policy could perform well. Could it provide an open reference for testing data and model choices under comparable conditions?
The data itself is awkward. Robot datasets differ in action units, state layouts, and control conventions. Egocentric human demonstrations do not supply robot proprioception. Vision-language examples supply another kind of supervision again. Combining these sources requires explicit interfaces and curation, not just a larger collection of files.
“We release REKI with its full pipeline, from data curation to deployment, and the checkpoints of every benchmark.” — §8, p. 12
The Core Move
Keep perception and action generation separate: a vision-language model encodes camera views and instructions, while a flow-matching expert generates action chunks using that context and a separate robot-state signal.
The compact equation to retain is the state-conditioning interface:
In words, the robot state becomes a single embedding that modulates the action expert. Visual-language context instead remains a token sequence that the expert reads through cross-attention. This equation describes conditioning, not the flow-matching training loss.
“The observations and the instruction are variable-length token sequences.” Their encoded tokens are read “as keys and values in its cross-attention blocks.” The state “is pooled into one state embedding” that modulates adaptive normalisation. — §3.2, p. 4
Why Context and State Take Different Routes
Now the version for readers who want more of the machinery: context supplies a sequence of representations to attend over, whereas state supplies a compact modulation signal inside the expert.
The context consists of the final-layer hidden states of Qwen3.5-2B. The backbone reads a single stream containing the prompt and camera views. The prompt carries the instruction, embodiment, control mode, control frequency, and binned episode length. Learned camera-role tokens identify views such as overhead or wrist cameras. The action expert uses the resulting context as cross-attention keys and values, preserving a boundary between the encoder and action generator.
State follows a layout-specific linear layer with GELU, then a shared zero-initialized projection into the expert's width. Its output modulates adaptive normalization alongside the flow-time embedding. The important dependency is that state conditioning does not change with flow time: compute the state embedding once per frame and reuse it across Euler steps.
“We use Qwen3.5-2B” and “pack the prompt and the camera views into a single stream.” The state pool is “one linear layer with a GELU per state layout, followed by a shared zero-initialised projection”; its output “is computed once per frame for all Euler steps.” — §3.2, p. 4
The separation makes the conditioning routes explicit, but it does not resolve incompatible action meanings. How can joint-position targets and end-effector displacement targets enter the same training batch?
Why Public Data Needs a Shared Action Interface
Here is the part that gets technical: the shared interface is a masked, padded container for native controls, not a universal action space.
REKI retains each dataset's control convention. It normalizes action dimensions to using per-dataset quantiles, then zero-pads actions to a common width and masks the padded dimensions out of the loss. Padding makes the shapes compatible; it does not make the commands semantically equivalent.
States remain absolute. Actions are absolute joint positions or delta end-effector poses, and rotations are harmonized to one Euler parameterization. Prompts and per-layout state projections identify the convention associated with a sample. Curation keeps commanded degrees of freedom, discards static and near-zero-variance episodes, and corrects dataset-specific sign conventions.
“We keep each dataset’s native control convention rather than remapping to a shared action space, normalise each action dimension to [−1, 1] by per-dataset quantiles, and zero-pad actions to a common width under a loss mask.” — §5.1, p. 6
Human manipulation data uses a bimanual action target containing wrist motion and hand articulation, with the wrist expressed in the camera frame and no proprioceptive state. Spatially relevant vision-language and grounding data provide auxiliary supervision for perception and spatial reasoning.
Human targets contain “wrist motion and hand articulation per hand,” with “no proprioceptive state.” Vision-language supervision “preserves the backbone’s perception and spatial reasoning” and is mixed in “at a small ratio.” — §5.1, p. 6
These sources can share a training recipe without becoming interchangeable. What does “single stage” mean when their supervision differs?
Why One Stage Can Still Be a Serious Recipe
Now the deeper recipe detail: single-stage pretraining means one broad pretraining mixture before benchmark-specific fine-tuning, not identical inputs or supervision for every example.
REKI pairs Qwen3.5-2B with an action expert containing 10 blocks of width 768. Pretraining uses the public robot, egocentric human, and vision-language mixture for 200k steps on 32 H100 GPUs, with a global batch of 2,880 samples. The backbone's learning rate is one tenth of the expert's. Each benchmark is then fine-tuned separately.
“REKI pairs the Qwen3.5-2B vision–language backbone with an action expert of 10 blocks of width 768.” It is pretrained “for 200k steps on 32 H100 GPUs with a global batch of 2,880 samples, then finetuned on each benchmark separately.” The backbone uses “0.1 × the learning rate of the expert.” — §5.2, p. 6
The recipe uses no proprietary data, world model, or additional pretraining stages. The register-memory extension is a separate mechanism; the focus here is the backbone–expert interface and public-data recipe.
REKI is “pretrained in a single stage on public robot, egocentric human and vision–language data, and then finetuned on each benchmark.” It uses “no proprietary data,” “no world model” and “no additional training stages.” — §8, p. 12
That makes the setup an inspectable reference, not a claim that pretraining is cheap or that fine-tuning disappears. Which ingredient would you change first while holding the others fixed?
Minimal Causal Code
This schematic PyTorch inference loop exposes the action-chunk update. The encoded context and pooled state stay fixed while Euler steps update the noisy actions. The expert and flow-time schedule are supplied interfaces; this is not a specification of the training loss. The register-memory path is intentionally omitted.
“At inference, each frame needs one encoder pass over the current observation and ten Euler steps on the action tokens.” — Inference, p. 6
import torch
@torch.no_grad()
def generate_chunk(encoder, pool_state, expert,
views, prompt, robot_state,
noisy_actions, flow_times):
# Computed once for this frame.
context = encoder(views, prompt)
state_embedding = pool_state(robot_state)
actions = noisy_actions.clone()
# flow_times supplies the Euler schedule.
for tau, next_tau in zip(flow_times[:-1], flow_times[1:]):
velocity = expert(
actions,
context=context,
state_embedding=state_embedding,
flow_time=tau,
)
# The changing state is the action chunk itself.
actions = actions + (next_tau - tau) * velocity
return actions
What You Gain, and What You Lose
You gain an explicit separation between perception and action generation, plus a public-data recipe that keeps control conventions visible. The released curation-to-deployment pipeline and benchmark checkpoints make it possible to modify architecture, data, or objectives against a concrete reference.
You do not get uniform data simply by pooling it. Normalization, action masks, prompt metadata, state-layout projections, and source-specific corrections remain real engineering work. Human manipulation targets are not robot control traces, and padding does not erase that distinction. Single-stage pretraining still uses substantial compute and is followed by benchmark-specific fine-tuning.
The useful claim is therefore not “the same policy, but cheaper.” It is a reproducible baseline whose ingredients can be inspected and changed—without pretending that data compatibility or adaptation has been solved.
REKI separates visual-language context from state conditioning and combines curated public data without pretending that different robot control conventions are interchangeable.