AutoEnvScaling: Building Training Environments in a Terminal
Useful training environments are costly to design by hand; AutoEnvScaling lets a proposer inspect solver failures, build and test complete tasks in a sandbox, then admit only tasks validated and calibrated for the current solver.
Read the paper ↗What it is
- Solver-guided task choice The proposer looks at what the solver gets wrong or already handles, then chooses a new task that targets that current capability boundary.
- Sandboxed environment building The proposer writes a complete executable task and runs its build, reference solution, and tests in isolated containers, repairing problems before submission.
- Validation and calibration The host checks that a task works, then checks whether the current solver sometimes succeeds and sometimes fails before adding it to training.
Why it matters
This turns environment creation from a fixed batch of prompts into an iterative engineering loop. Solver attempts guide what to build next, while validation and calibration keep broken or poorly matched tasks out of the active training pool.
Before getting into the moving parts, here’s the short version: AutoEnvScaling gives an agent the job of building training tasks as executable software. It watches where a solver gets stuck, builds a task aimed at that boundary, and checks that the task both works and is useful for that solver right now. That matters because a pile of generated instructions is not a training environment; somebody still has to make the environment run and decide whether its tests mean anything.
- Design labor: Useful tasks take more than writing a prompt. Someone has to provide a runnable setup, a reference solution, and tests. The proposer takes on that researcher-like terminal work inside a sandbox.
- Task choice: A fixed generator can keep producing tasks whether or not they match what the solver needs. Here the proposer sees the solver’s pass rate, common failure, and trajectories for a parent task. The canonical example is a failed PyTorch checkpoint-recovery task: the proposer noticed the solver treated buffers as weights and built a new environment around that mistake. That is one concrete failure mode, not a claim that every failure can be read so cleanly.
- Keeping the pool useful: A task can run correctly and still be a bad fit. If the solver always fails, it may be too hard; if it always succeeds, it may no longer teach much. So the host checks both whether the task works and whether the current solver lands in a useful middle ground.
So what does “build a task” actually mean? Think of a small software project with an instruction, an execution environment, a reference answer, and tests. The solver gets the instruction and the environment, while the answer and tests stay hidden. The proposer writes the files, starts child containers to build and test them, and repairs whatever goes wrong. The notebook analogy is useful here: rollouts are notes about what the solver understands and where it stumbled. But unlike a notebook, the resulting task has to execute correctly; a good insight does not make broken code trainable.
The proposer uses those notes to choose a direction. If the solver keeps failing, it can build a simpler variant; if the solver is succeeding, it can add a skill or stretch the task. In other words, the knob is not just “make it harder”: the proposer chooses which capability to target. So how does the system stop a plausible-looking task from slipping into training just because it built?
First the host checks the reference solution and an empty solution, and performs a contamination check; task review adds another check because the tests and reference solution could share a flaw. Then the current solver tries the candidate repeatedly. It is kept when its mean reward is between and and the attempts vary—in plain terms, the solver sometimes passes and sometimes does not. Those cutoffs are the admission rule here, not a universal definition of a good task.
At refresh, the updated solver is run on the pool again. Tasks outside the calibration band are removed, and the next round’s proposer gets fresh rollouts. I like this part of the design: it treats “useful training data” as something that can expire as the solver learns. The tradeoff is that a task’s value is tied to the current solver, and generated tests can still encode mistakes. What will the proposer learn from the next set of failures?
Read the deep dive
Why People Wanted It
A solver can only learn from tasks that exist, and useful tasks take real work to design. A generated prompt is not yet a training environment: it needs an executable setup, a reference solution, and tests that distinguish success from failure. So the real question was not merely how to generate more tasks. It was whether an agent could do the researcher’s end-to-end environment-building work, and use the solver’s actual attempts to decide what should come next.
The Core Move
Treat environment design as a terminal task: a proposer reads solver rollouts, builds a complete executable task in an isolated sandbox, and submits it only after it works and is appropriately difficult for the current solver.
A task includes an instruction, an execution environment, a reference solution, and tests. The solver sees the instruction and environment; the reference solution and tests remain hidden. Accepted tasks join the training pool, and the solver’s later rollouts guide the next round.
Why Solver Rollouts Change What Gets Built
Now the version for readers who want more of the machinery: the assignment is grounded in a parent environment and gives the proposer the solver’s pass rate and most common failure. The proposer inspects trajectories and verifier outputs to identify what the solver can already do and where it breaks.
That evidence guides the task choice. If the solver repeatedly fails at a capability, the proposer can make a simpler variant that isolates it; if the solver succeeds, it can create a harder variant, such as one with a longer horizon or an added skill. The failure is not just a score to optimize: it points to a concrete opportunity for a new task. The proposer is free to decide what to build and when it is finished, rather than following a fixed sequence of rewrite operations.
Why a Complete Task Needs Its Own Build-and-Test Loop
Here is the part that gets technical: a proposed environment is a small software artifact, not just text. The proposer writes the task files in its workspace, then launches child containers to build the environment image and run the reference solution and tests. If behavior is wrong, it revises the files and tries again; it can also run the current solver on the candidate to estimate difficulty.
The host independently validates submissions. It checks that the reference solution earns full reward and that an empty solution earns less than half, and it performs a contamination check against held-out tasks. Because the same proposer may write both tests and reference solution, those checks can miss shared mistakes or shortcuts; a task review against the benchmark rubric is used to catch issues of that kind. Passing the build loop is necessary, but does not by itself make a task useful.
Why Admission Depends on the Current Solver
Now for the exact admission logic: after validation, the current solver attempts the environment several times. The host keeps it only when mean reward lies between and and rewards vary across attempts. In words, the task should be neither reliably out of reach nor already reliably solved; variation indicates that the current solver sometimes succeeds and sometimes fails.
At pool refresh, the host reruns the updated solver on accepted environments, removes tasks that no longer meet the calibration band, and asks for replacements guided by the latest rollouts. That makes the training pool a moving target: tasks useful now may stop being useful as the solver changes. What should happen when the solver’s own improvement changes which tasks deserve to remain?
Minimal Causal Code
This plain loop sketches the visible control flow; validation and calibration are represented as functions rather than hidden inside the proposer:
pool = []
for round_id in range(num_rounds):
rollouts = run_solver_on_pool(solver, pool)
assignments = summarize_failures_and_pass_rates(rollouts)
for assignment in assignments:
task = proposer.build_task(assignment, sandbox=True)
while not task_builds(task):
task = proposer.repair(task, build_logs(task))
if not host_validation_passes(task):
continue
rewards = run_solver_repeatedly(solver, task)
if calibrated_for_current_solver(rewards):
pool.append(task)
solver = train_with_verifier_rewards(solver, pool)
What You Gain, and What You Lose
You get a workflow that can choose tasks from observed solver behavior, construct executable environments, and filter them for both correctness and current difficulty. Because the accepted pool is refreshed from new rollouts, the data-generation process can respond as the solver improves.
You also inherit the risks of executable task generation. Tests and reference solutions can share mistakes; validation can miss shortcuts; and calibration depends on the current solver, so a task’s usefulness is not permanent. The proposer must spend sandbox work building and repairing tasks, and admission checks remain a boundary between what the proposer proposes and what training actually accepts. The flywheel is useful precisely because it is selective—not because every generated environment is automatically good.
A useful generated task is not just written—it is built, checked, and admitted only when the current solver is ready to learn from it.