Choosing Midtraining Recipes After SFT

Why People Wanted It

A midtraining recipe is supposed to prepare a model for what comes next, not merely make it look better on a convenient base-model test. But a candidate can improve measures of code or repository understanding without those gains carrying through to agent use. The real question is whether the midtraining choice helps after post-training—and whether you can test that without confusing the recipe’s effect with a different fine-tuning setup.

The Core Move

Run the same supervised fine-tuning on both the midtrained candidate and its starting checkpoint, then compare their downstream results. The evaluation protocol states no new mathematical formula: its key control is identical SFT on both sides.

Why the Starting Checkpoint Needs the Same SFT

Here is the deeper evaluation logic: identical SFT makes the starting checkpoint a matched control, so the comparison asks what midtraining changed under a shared downstream treatment. In the protocol, each midtrained model is compared with its starting checkpoint after identical SFT. The post-training pipeline is the ultimate measure of success, while base-model post-trainability metrics and pretraining general-capability benchmarks are also tracked.

This does not isolate every possible interaction between midtraining and SFT; it gives a practical comparison under the same SFT recipe. The evaluation also checks general-capability benchmarks for regressions. So what if waiting for the full downstream run is too slow?

Why Short SFT Can Be an Early Signal

Now the more practical layer: short SFT is used as a quicker signal, not as a replacement for the full evaluation. The protocol compares both a short and a full SFT budget, and reports that gains seen after short SFT carried over to full SFT in the blend-validation experiment. That makes a short run useful for screening a recipe, while the full run remains the stronger check for the main claims.

The signal is not equally clean on every benchmark: Terminal-Bench was noisier in that experiment, and improved only in full-SFT pass@1. So when should you trust an early signal, and when should you wait for the fuller comparison?

Minimal Causal Code

This sketch shows the control visibly: both starting points receive the same SFT data and update rule, then their downstream scores are compared. The loop is illustrative, not a claim about the paper’s training implementation.

for label, model in [("candidate", midtrained_model), ("control", starting_model)]:
    for batch in identical_sft_batches:
        loss = assistant_sft_loss(model, batch)
        loss.backward()
        optimizer_step(model)
        zero_grad(model)

    score = evaluate_agent_tasks(model)
    print(label, score)

What You Gain, and What You Lose

You gain a downstream-facing comparison that asks whether the recipe helps after a shared SFT stage, rather than relying on base-model proxies that did not track post-training results closely enough to guide recipe decisions. Short SFT can make that comparison quicker, and the reported blend-validation gains carried over to full SFT.

You lose the certainty of treating a short run as a final verdict: benchmark behavior can be noisy, and the protocol’s short-SFT result is an early signal. You also pay for running SFT and downstream evaluations on both candidate and control. The point is not that one proxy or one short run settles the recipe choice; it is that the candidate must be judged in the same downstream conditions as its baseline.