Co-Evolving Proposer and Solver

Why People Wanted It

A solver can only train on environments that someone has already built. If the proposer stays fixed while the solver improves, its task-generation behavior may stop matching what the solver now needs. The real question is whether the model being trained can also take over the proposer’s job—and whether training both jobs together gives the proposer a way to improve rather than merely repeating its initial habits.

AutoEnvScaling’s co-evolution setup addresses this by using one policy as both proposer and solver. The solver learns from accepted environments; its updated checkpoint then helps generate the next round, while proposer trajectories provide a second training signal.

The Core Move

Use a shared policy for both roles, but keep their reward and advantage calculations separate:

$$\pi_\theta: \text{proposer} \;\cup\; \text{solver}, \qquad A_P \ne A_S$$

The weights are shared; the role-specific learning signals are not.

Quiz

Why does the shared proposer-solver policy use separate advantage baselines?

Because proposer rewards are always higher than solver rewards.

Because proposer and solver trajectories measure different objectives and need role-appropriate comparisons.

Because solver trajectories are never used for policy updates.

Because the roles use unrelated model weights.

$ check

Why One Policy Can Play Two Roles

Now the version for readers who want more of the machinery: the same checkpoint alternates between proposing environments and solving accepted ones, so its solver-trained update becomes the starting point for the next proposer round. In the reported self-improvement setup, Qwen3.6-35B-A3B serves as both roles: it designs environments, trains on them, and uses the updated checkpoint for the next generation round. This removes the need for a stronger external proposer in that setup; it does not mean the two roles have identical objectives.

Why the Two Advantages Must Stay Separate

Here is the part that gets technical: proposer and solver rewards describe different outcomes, so a shared policy needs role-specific baselines. Let a role $q \in \{P,S\}$ have a group of trajectories $\tau_{q,i}$ with rewards $r_{q,i}$. The proposer’s rewards are centered within proposer groups; the solver’s rewards are centered across attempts on the same environment. In shorthand, each role compares its trajectory reward to a baseline formed for that role’s comparison group:

$$A_{q,i} = r_{q,i} - b_q$$

Here $b_q$ is the role-appropriate group baseline, not a shared proposer-solver average. This matters because a proposer trajectory is judged as environment construction, while a solver trajectory is judged against attempts at a particular environment. Groups with no reward variation are dropped, following the described DAPO-style handling.

Why Co-Evolution Happens Across Rounds

Now for the training schedule: the solver trains at every step, while the proposer generates new environments and trains on its resulting trajectories every $k$ solver steps. The proposer’s updated behavior persists through policy updates, so its next generation round starts from co-evolved weights rather than a permanently fixed proposer. This is distinct from harness optimization, where memory, skills, or tools can also be updated between rounds; the validation and calibration checks stay fixed. The open question is how reliably these coupled updates remain useful as both roles change.

Minimal Causal Code

This loop makes the shared checkpoint and separate role-specific advantage computation visible. The group-building and reward functions are placeholders for the corresponding training data and scoring logic.

policy = initialize_policy()

for step in range(total_solver_steps):
    solver_batch = collect_solver_trajectories(policy)
    solver_rewards = score_solver(solver_batch)
    solver_advantages = center_by_environment(solver_rewards)
    policy = update(policy, solver_batch, solver_advantages)

    if (step + 1) % proposer_interval == 0:
        proposer_batch = collect_proposer_trajectories(policy)
        proposer_rewards = score_proposer(proposer_batch)
        proposer_advantages = center_by_proposer_group(proposer_rewards)
        proposer_batch, proposer_advantages = drop_constant_reward_groups(
            proposer_batch, proposer_advantages
        )
        policy = update(policy, proposer_batch, proposer_advantages)

The essential visibility is the sequence: solver update, then—on its interval—proposer generation and proposer update, all on the same policy weights.

What You Gain, and What You Lose

A single policy can improve at solving and carry that newer capability into later environment generation. Proposer training also gives the environment-building role its own reward-driven path to improve; the reported self-improvement comparison includes a run without proposer training, which ends earlier because training collapsed. These results support co-evolution in the tested setup, not a guarantee that joint training is stable in every setting.

The tradeoff is coupled optimization. The roles have different rewards, and the environments change as both improve, so role-specific advantage baselines and a deliberate update schedule are needed. Shared weights also mean a change useful for one role can interact with the other. Co-evolution avoids relying on a stronger external proposer, but it does not make the proposer-solver feedback loop free of coordination or stability problems.