Dependency-Aware Repository and Code-History Construction
File-by-file code misses the relationships and coordinated changes agents must handle, so this recipe selects connected repositories and turns linked edits and commit sequences into training examples.
Read the paper ↗What it is
- Connected repositories Keep repositories where code files often refer to other files in the same project, so examples contain real cross-file relationships.
- Dependency ordering Place the README first and arrange connected files near one another instead of sorting files alphabetically.
- Change histories Train on both coordinated edits across linked files and files shown through a sequence of real changes.
Why it matters
Agents need to work with code that is defined elsewhere and changes that span files or accumulate over time. Connected repository examples and history-based examples make those relationships and changes visible during training.
Before we get into the mechanics, here is the short version: instead of treating a repository as a pile of unrelated files, this recipe tries to put files with real dependencies next to each other. Then it adds examples where code changes across linked files, or evolves through a sequence of commits. That matters because a final snapshot tells you what the code is, but not necessarily how the pieces depend on one another or how they got there.
-
Cross-file context: In ordinary file-by-file training, a model may see a function without seeing the file it calls. A repository context can help, but only if the files in it actually connect. The canonical example is a Python module importing a helper from another file in the same project. Generalizing from that, the data construction parses references across supported languages and keeps repositories with enough within-repository imports. Most languages use a 20% connected-file threshold; Ruby uses 10% because Rails autoloading makes many references hard to see statically. So the real question was never just “How much code fits?” — it was “Does this context contain relationships worth learning?”
-
Order as useful context: Imagine a notebook where the pages you need to cross-reference are kept together rather than filed alphabetically. That is the intuition for dependency ordering: put the README first, then traverse the import graph so connected files sit near each other. The analogy has a limit: a sequence is still not a live repository, and proximity does not guarantee the model understands the dependency. Still, changing the order changes what can appear close together. The reported comparison says dependency ordering separates stronger from weaker models better than alphabetical ordering: bits-per-byte goes down for stronger models and up for weaker ones. What about code that is not just related, but changing?
-
Two kinds of change: The canonical multi-file example is a pull request that edits a function and a linked file it depends on. Training starts with affected files before the change and includes the updated versions; only edits whose files are linked in the import graph are kept. That teaches coordinated edits, rather than asking the model to treat each file as independent. Code evolution is a different task: picture following one file through a real commit history, like keeping a notebook while moving between towns and deciding what changed at each stop. The analogy captures continuity, but real commits can touch more than one idea. These examples provide the initial state, the ordered changes, and the final state. So if snapshots hide the path, what kind of signal should the training loss follow?
For multi-file edits, loss covers the original files, diffs, and changed lines in new files with two lines of context. For code-evolution documents, it covers commit messages, diffs, and the final state. In both, license blocks are masked. The important intuition is that the model gets a change process to follow, not merely a request to reproduce a finished repository. I like this distinction because “make a consistent edit” and “track a file’s state over time” sound similar but ask for different kinds of reasoning.
Read the deep dive
Why People Wanted It
A repository is not just a bag of files. A function may call code defined elsewhere, and a change that looks local can require edits in several linked files. If training mostly presents isolated files or a repository’s final snapshot, the model sees less of the structure and change process an agent encounters while working in a codebase.
So the real question was not simply how to include more code. It was how to make examples expose meaningful relationships between files, and how to show changes as changes rather than only as finished code.
The Core Move
Choose repositories with recoverable cross-file connections, serialize their connected files near each other, and build separate examples from linked edits and real commit histories. The paper gives no formula for this construction; its stated selection rule is a threshold on the fraction of code files that import another file in the same repository.
Why Connected Repositories Matter
Now the version for readers who want more of the machinery: repository selection is based on a parsed import graph, not on repository size or an informal judgment that a project looks coherent. For each repository, references between files are parsed across 14 languages. Keep it when at least 20% of code files have an import edge to another file in the repository; Ruby uses a 10% threshold because Rails autoloading hides many references from static parsing.
This filter aims at a specific failure mode: a repository can contain many files yet have little recoverable cross-file structure. Repositories that are collections of scripts, exercises, or snippets may add volume without making one file depend on another. Why is selecting connected repositories only half the problem? The order in which their files appear can still separate related code.
Why Dependency Ordering Changes What the Model Sees
Here is the part that gets technical: serialization follows the README with a depth-first traversal of the import graph, keeping connected components together and placing each file near the files it imports. Files begin with a repository-and-path header, which is masked from the loss. The main text also describes packing at 256K, splitting longer repositories between connected components rather than truncating them.
The comparison is with alphabetical ordering. Dependency ordering lowers bits-per-byte for stronger models and raises it for weaker models, a pattern interpreted as exposing cross-file structure that weaker models have not yet learned. The point is not that a particular ordering is universally best: the ordering changes which dependencies are nearby in the sequence. What does a repository snapshot still fail to teach, even when its dependencies are arranged well?
Why History Must Show Both Coupled Edits and Sequence
Now the deeper data-design distinction: multi-file edits and code-evolution documents answer different questions. A multi-file edit starts from the affected files before a merged pull request and trains on the change to those files afterward. Only changes whose files are linked in the repository import graph are kept, so the example couples the edit to a dependency relationship.
A code-evolution document instead gives a file’s initial state, the real commits that changed it from oldest to newest, and its final state. In this form, the model has to follow a sequence of changes, not infer it from a before-and-after pair alone. Histories range from a single commit to several dozen; the appendix reports a median of 3 and a maximum of 59. The loss treatment also differs by example type: for multi-file edits it covers original files, diffs, and changed lines of new files with two lines of context; for code evolution it covers commit messages, diffs, and final state. Which representation is useful depends on whether the challenge is coordinated edits across files or tracking a file through successive versions.
Minimal Causal Code
This small sketch makes the selection and ordering idea visible. It is not the paper’s parser or full data pipeline.
for repo in repositories:
graph = parse_imports(repo.files)
connected = sum(
1 for file in repo.code_files
if has_edge_to_another_repo_file(file, graph)
)
threshold = 0.10 if repo.language == "Ruby" else 0.20
if connected / len(repo.code_files) < threshold:
continue
yield repo.readme
for component in graph.connected_components():
for file in depth_first_import_order(component, graph):
yield file.header(repo.name, file.path) # masked from loss
yield file.contents
What You Gain, and What You Lose
Dependency-aware selection and ordering make cross-file relationships more visible than alphabetical file order, while linked multi-file edits and commit sequences add examples of coordinated change and state tracking that final snapshots cannot provide. These examples are still a constructed view of software work: static parsing can miss relationships, and the two history formats emphasize different kinds of change. They also require filtering and careful loss masking; they are not a substitute for interaction traces or the broader long-context data mixture.
Put related files near each other, then teach change either as a coordinated edit or as a sequence through time.