User Behavior as a Foundation for Reinforcement Learning
A base model can struggle to learn user-facing tasks from reinforcement learning alone; Vela tests whether supervised training on real user behavior gives that same RL setup a stronger starting point.
Read the paper ↗What it is
- Behavior-trained initialization A model first learns from examples based on real user behavior, then serves as the starting checkpoint for reinforcement learning.
- Matched RL comparison Compare that checkpoint with the original base model while keeping the reinforcement-learning setup the same within each task domain.
Why it matters
The reported comparisons find higher final scores when reinforcement learning starts from the behavior-trained checkpoint on both VelaBench and ProfileSummary, suggesting that supervised experience can improve what later reinforcement learning builds on.
Before getting into the details, here is the short version: Vela trains a model on examples drawn from user behavior, then asks whether that model is a better launch point for reinforcement learning than the original base model. In the two reported comparisons, the behavior-trained starting point ends up with higher scores after the same RL setup.
-
The post-training question: Reinforcement learning is commonly used after supervised fine-tuning, so the useful question is not just whether the supervised stage improves a score. It is whether what the model learned changes the result of the next stage. The canonical concrete example here is VelaBench: take the base model or the Vela supervised checkpoint, run the same RL setup on the domain's training tasks, and compare on VelaBench. That makes initialization the contrast you care about. But does the same pattern appear outside that benchmark?
-
A second domain, not a universal claim: ProfileSummary gives the comparison another user-task setting. The Vela checkpoint is trained on 18.8K Vela-Instruct examples; the RL runs use 2,330 ProfileSummary training examples, and are evaluated on ProfileSummary. The base-model and Vela-initialized runs use the same RL data and configuration within the experiment. This is like giving two runners the same course and conditions but different preparation; the analogy stops at the starting point, because model training is not a controlled human race. So what do the actual scores say?
-
The reported outcome: On VelaBench, RL from the Vela checkpoint reaches 66.2, compared with 58.5 from the base model; on ProfileSummary, the scores are 37.6 and 29.9. In both cases, RL from the base model stays below the Vela supervised checkpoint, while RL from that checkpoint improves further. I read that as evidence that behavior supervision is more than a one-off score bump: it gives RL a stronger foundation in these tested settings. I would not turn two domains into a guarantee about every task or RL recipe. What happens when the setup changes is still open.
Read the deep dive
Why People Wanted It
Reinforcement learning is often applied after supervised fine-tuning, but that leaves a practical question: what should the supervised stage teach? If it teaches patterns drawn from real user behavior, does the resulting model merely score better before RL, or does it also learn more effectively once RL begins? Vela tests the second possibility: whether user-behavior supervision is a stronger starting point for downstream reinforcement learning.
The Core Move
Train a model on user-behavior examples first, then apply the same reinforcement-learning setup to that checkpoint and to the original base model; compare their results on the corresponding tasks. The paper gives no equation for this comparison, so the key is the controlled setup, not a new mathematical rule.
Why the Initialization Matters
Now the version for readers who want more of the experimental logic: the comparison asks whether the supervised checkpoint changes what the subsequent RL stage can achieve, not simply whether supervised training helps on its own. The two starting points are Qwen3.6-35B-A3B and Vela-35B-A3B, where the latter is the supervised-fine-tuning checkpoint trained on 18.8K Vela-Instruct examples. Each pair uses the same RL configuration and training data within its experiment, then is evaluated on the benchmark for that RL domain. The question this leaves open is whether any advantage is specific to one user-task domain.
Why the Result Is About a Stronger Starting Point
Here is the part that gets technical in the comparison: on VelaBench, RL from the Vela checkpoint reaches 66.2, versus 58.5 from the base-model initialization; on ProfileSummary, the scores are 37.6 versus 29.9. The supervised checkpoint itself is below its RL-trained result on both benchmarks, while RL from the base model remains below that checkpoint. In other words, the result is not just that supervised training raises the starting score: applying the shared RL setup to that starting point yields a further gain, and the final score is higher than with RL from the base model. The question still open is how far this pattern carries beyond these two tested domains.
What the Shared RL Setup Controls
Now for the machinery behind the comparison: both initializations within an experiment receive the same RL training data and configuration, which makes initialization the intended point of contrast. The setup uses Group Sequence Policy Optimization with 16 responses per prompt and no learned critic. The VelaBench run samples 2,000 training tasks; the ProfileSummary run uses 2,330 training examples. Training and evaluation data have no overlapping samples or users. That control supports the comparison, but the reported results remain results on these benchmark domains; what would happen under another RL recipe is a separate question.
Minimal Causal Code
This schematic loop makes the comparison visible: both models go through the same RL update procedure, but begin from different checkpoints. It is not a reproduction of GSPO or the paper's training infrastructure.
base_model = load_checkpoint("Qwen3.6-35B-A3B")
vela_model = load_checkpoint("Vela-35B-A3B")
for model in (base_model, vela_model):
for prompts in rl_training_batches:
responses = model.generate(prompts)
rewards = score(responses)
loss = rl_objective(model, prompts, responses, rewards)
loss.backward()
optimizer.step()
optimizer.zero_grad()
What You Gain, and What You Lose
The gain is a direct empirical case that user-behavior supervision can provide a stronger initialization for later RL: under the shared configuration, the Vela-initialized runs finish higher on both reported benchmarks, and improve over the supervised checkpoint. The limit is scope. These results establish the comparison for VelaBench and ProfileSummary under the tested setup; they do not show that every behavior-trained checkpoint, RL algorithm, or task domain will benefit in the same way. The paper's separate task-construction work is outside this module's focus.
A better supervised starting point can make the same reinforcement-learning setup finish stronger.