Natural Software-Engineering Data for Long-Context Midtraining

Why People Wanted It

A model can learn to write code without learning how a real project hangs together. Pretraining often presents code as individual files, while an agent must work across a repository and its history. The question is how to give the model that project-level experience before post-training, without making the middle stage mostly resemble the later supervised examples.

The move is a dedicated midtraining stage: train at 256K context on a blend dominated by natural software-engineering artifacts. This sits between pretraining and post-training, where the model can encounter whole repositories and long code changes before final tuning. The recipe also includes a bounded share of agent traces, but the focus here is the natural-data foundation.

The Core Move

Use long-context midtraining to expose the model to naturally occurring software work at project scale, rather than relying on isolated code files alone. The central setting is a 256K context; no mathematical formula is given for this recipe.

Why Long Context Changes What the Model Can Read

Now the version for readers who want more of the machinery: context length matters because repository knowledge is distributed across files, and a long window lets training examples place those dependencies within one sequence. In short-context pretraining, only a small part of a repository may fit; at 256K, the recipe can pack whole repositories, and the model can predict code that depends on files tens of thousands of tokens earlier. This is not a claim that context alone teaches every relationship: the training sequence still has to contain useful project structure.

The timing matters too. Midtraining comes after pretraining and before post-training. Its role is to bridge the gap between general language-and-code exposure and the later agent-oriented tuning, so the model can acquire project-level knowledge in a stage built for long sequences.

Why Natural Artifacts Form the Main Blend

Here is the part that gets technical about the data mix: the blend is led by natural software-engineering records, rather than being formatted entirely as conversations. In the 500B-token blend, issue–PR data is 40.8%, repository data 20.0%, diff data 3.0%, build and CI logs 1.2%, and long-context data 5.0%; pretraining replay contributes 20.0%. Agentic traces contribute 10.0% in the table. These are shares of training tokens, not a claim that every training run must use identical proportions.

The natural sources cover different views of work: issue–PR records connect requests and code changes; repositories expose code in project context; diff data captures changes; logs show system responses; and long-context data supports the long-window stage. The model sees more than polished answers: it sees artifacts produced while software is built, changed, and debugged. The recipe keeps non-trace sources in natural formats rather than applying chat templates to everything.

Minimal Causal Code

A plain loop makes the stage boundary visible. The model reads a long-context example, predicts its next token, and updates its weights; the next example comes from the data blend. This pseudocode shows the training idea, not the paper's full data pipeline.

for example in midtraining_stream:
    tokens = example.token_ids  # may contain repository-scale context
    logits = model(tokens[:-1])
    loss = cross_entropy(logits, tokens[1:])
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

The key choice is what examples enter midtraining_stream: natural software-engineering artifacts make up most of the blend, with replay, long-context data, and a bounded trace share also represented.

What You Gain, and What You Lose

You gain a stage explicitly aimed at project-level exposure: the model can read whole repositories and code histories in context, rather than seeing only disconnected files. The blend also gives it varied natural evidence of software work, not just agent conversations.

You pay for a long-context training stage and for building a diverse corpus whose sources need filtering and formatting. The recipe does not make post-training unnecessary; it prepares the model before that stage. Nor does a long window guarantee that every dependency will be learned: what the model can use depends on the examples and the structure they preserve. The trace-specific masking details belong to the separate module on curated agent traces.