Generated Clarification Tasks
Agents often attempt underspecified tasks without asking what is missing; AutoEnvScaling generates tasks with withheld requirements so agents can practice asking useful questions before acting.
Read the paper ↗What it is
- Withheld requirements A task leaves out a detail that determines what a correct solution should do. The agent can obtain that detail by asking the user through a tool.
- Question quality It is not enough to ask often. A useful question must uncover information that changes how the task should be completed.
Why it matters
Agents often guess when instructions are incomplete. Training on tasks where missing details affect correctness gives them practice asking targeted questions, and evaluation can distinguish useful clarification from tool use alone.
Before getting into the machinery, here is the short version: AutoEnvScaling creates practice tasks where the instruction leaves out a requirement, but the agent can ask the user for it. The point is to teach an agent not just to use a tool, but to notice what it needs to know before it commits to a solution.
-
The gap: A task can be executable while still being underspecified. The canonical example is a software change where the user has not said which behavior they expect. An agent may produce code that runs and still miss the requirement; this is not the same as a broken environment, because the missing piece is the user's intent.
-
The retrieval path: The task stores the missing detail somewhere the agent can retrieve it by asking. Think of the requirement as a note in a notebook: the agent can request the relevant page before acting. The analogy breaks because the tool is a designed interface, not a general memory system. The task also includes tests that depend on the withheld detail, so getting the answer can change what counts as a correct solution.
-
The learning signal: An agent can learn to call the question tool without learning what to ask. In the reported setup, a cold-start phase used trajectories that ask about missing requirements, and later reinforcement learning used a reward based on whether questions covered needed information. Ask rate tracks how often the agent asks; Ask-F1 is about whether the questions cover the necessary details. Those are different knobs: one encourages the behavior, the other rewards its usefulness.
So the real question was never just whether the agent asks — it was whether the answer resolves a requirement that matters. On held-out software-engineering tasks, training with generated tasks raised both success and Ask-F1 compared with training on the official set, while ask rate increased more modestly between those two settings. I like this distinction because it keeps “use the tool” from masquerading as “understand what is missing.” But how well does that lesson transfer beyond this task format?
Read the deep dive
Why People Wanted It
An agent can be good at carrying out instructions and still fail when the instruction leaves out a requirement. In software work, guessing may produce a plausible implementation that violates what the user actually wanted. The real question is not simply whether an agent can finish a task: can it recognize what it does not know, ask for the missing detail, and then use the answer?
This matters because the agent may rarely ask on its own. If training rewards mostly reflect task completion, a model that guesses can receive little signal about when a clarification would have helped.
The Core Move
Generate a task whose instruction deliberately omits a requirement, make that requirement available through a user-question tool, and make the tests depend on it.
The task therefore has three linked pieces: an underspecified instruction, withheld details that an ask_human tool can reveal, and tests that check behavior against those details. A complete-instruction version and a no-tool version can be derived from the same task.
Why Withheld Requirements Make Asking Necessary
Now the version for readers who want more of the machinery: the missing information is not merely context; it is part of the task's correctness condition. Suppose a user asks for a software change but leaves out a required behavior. A registry stores that requirement, the clarification tool can reveal it, and the tests encode it. Without the answer, the agent cannot reliably infer which acceptable-looking implementation matches the intended one.
This creates a causal chain: the task instruction leaves a gap; a question can retrieve the hidden requirement; the requirement determines what passes the tests. The complete-information variant supplies the requirement up front, while the no-tool variant removes the channel for retrieving it. That comparison separates ordinary task execution from the behavior of asking.
Why Asking More Is Not the Same as Asking Well
Here is the part that gets technical: asking rate only records whether the agent used the tool. It does not tell you whether its question targeted the information needed to complete the task. Ask-F1 is shaped around whether the questions cover that needed information; task success separately checks whether the resulting work is correct.
That distinction explains why a model can ask often yet remain ineffective: a vague or irrelevant question may consume a turn without resolving the requirement. Training can first provide examples of trajectories that ask about missing requirements, then use reinforcement learning with a clarification reward based on Ask-F1. The cold start gives the model a reason and pattern for asking; the later reward can distinguish useful questions from mere tool use.
Minimal Causal Code
This loop is schematic: it makes the information flow visible rather than implementing a particular training harness.
def run_task(agent, instruction, hidden_requirements, tests):
answer = None
while agent.has_next_action():
action = agent.next_action(instruction)
if action.kind == "ask_human":
answer = reveal_matching_requirement(
action.question, hidden_requirements
)
agent.observe(answer)
else:
agent.apply(action)
return tests(agent.workspace, hidden_requirements)
The key is that the test receives the hidden requirements, while the agent sees them only if its question retrieves them.
What You Gain, and What You Lose
Generated tasks provide more practice cases than a small fixed training set, and the held-out evaluation can test whether asking behavior transfers beyond the official training tasks. In the reported software-engineering setup, training with generated tasks improved both task success and Ask-F1 compared with training on the official set. The ask rate rose only modestly between those two RL settings, suggesting the improvement was mainly in question quality rather than simply asking more often.
There are limits. These results concern software-engineering tasks and a particular benchmark setup; they do not establish that the same training recipe works for every domain or kind of user interaction. And a task can only teach useful clarification if its withheld details, tool answers, and tests are aligned: otherwise the agent may learn benchmark-specific questioning rather than a dependable habit of resolving real ambiguity.
A useful clarification task makes the missing answer matter to the tests, so asking well—not merely asking—can be learned and measured.