Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe

The short version

The problem

Models that work on real software projects must handle large codebases and interact with tools over many steps. Ordinary pretraining rarely teaches those skills, while putting too much of that teaching into a long final training phase can make later learning less effective.

The idea

The paper adds a middle training stage before the final instruction-tuning stage. It teaches from long, mostly real-world software examples—such as connected repository files and project changes over time—plus a limited number of tool-use sessions. For instance, a model can learn how a change in one file relates to others and what to do after a test fails.

What it got

After the same final training, the larger model’s score on DeepSWE, a benchmark of extended software-engineering tasks, rises from 11.7 to 15.7. The approach also improves results across the tested agent interfaces, the software environments through which a model uses tools.