Contrastive Testing for Empirical Skill Dependence

Why People Wanted It

A task can come from a human-written skill and still be solvable without that skill. If you call every such task “skill-dependent,” you confuse where a task came from with what actually helped solve it. The real question is not whether the skill inspired the task, but whether an agent’s success changes when that skill is available.

There is a second concern: a task that fails this dependence test may still be a perfectly usable executable environment. Rejecting it would throw away verified practice merely because the skill’s contribution was not demonstrated under one particular comparison.

The Core Move

Keep feasibility and skill dependence as separate decisions: first require an executable task with a working verifier and a verifier-passed solution, then compare a reference agent’s outcomes with and without the target skill.

Let $F_i=1$ mean task $i$ passes the design, structure, end-to-end execution, verifier, and verifier-passed-solution checks. Let $r_i^+$ and $r_i^-$ be the verifier outcomes with and without the target skill. The contrastive label is triggered by $(r_i^+,r_i^-)=(1,0)$, conditional on $F_i=1$.

Quiz

A task passes feasibility checks and has a working verifier. The reference agent passes with the target skill and also passes without it. Which label follows the contrastive rule?

Verifier-Passed fallback, because the required with-success/without-failure contrast is absent.

Rejected, because both runs passed.

Skill-Dep., because the task was built from a skill.

$ check

Why Acceptance Has Two Levels

Here is the part that goes beyond the basics: acceptance checks two different claims, and passing the first does not imply the second. Feasibility asks whether the environment and its verifier work; dependency asks whether the skill changes the reference agent’s verified outcome.

A task first has to survive design, structure, and end-to-end execution checks, including a working verifier and a solution that passes it. That gives the task a valid outcome signal to compare. Only then does the with-skill versus without-skill test make sense: without a functioning verifier, a changed outcome could be a broken test rather than evidence about skill use.

This separation also prevents a category mistake. A task may be accepted as executable without earning the narrower Skill-Dep. label. The label describes evidence under the reference configuration, not a universal property of the task for every possible agent. What should happen to a verified task that clears feasibility but not the contrastive bar?

Why the With/Without Contrast Is the Dependency Test

Now the deeper comparison: the skill-dependent condition is not merely “success with the skill.” It requires a paired contrast, with verifier success when the target skill is present and verifier failure when it is absent.

The outcomes are binary: $r_i^+,r_i^-\in\{0,1\}$. In words, each run either passes the task’s verifier or does not. The qualifying pair is $(1,0)$; the other pairs do not establish the specified contrast. In particular, success in both runs means the reference agent solved the task without needing the supplied skill in that tested setup, while failure in both gives no evidence that the skill enabled success.

This is an empirical dependence test, not a proof of necessity. The comparison holds the reference-agent setup as its frame: it says what changed in those executions when the target skill was supplied, not what every agent must do. Why preserve the tasks that fail to show this contrast instead of discarding them?

Why Non-Qualifying Tasks Become Fallbacks

Here is the final layer of the machinery: the labels preserve the difference between “valid environment” and “demonstrated skill dependence.” Once accepted environments pass feasibility and outcome-verification checks, the contrastive condition sorts them into Skill-Dep. tasks and verifier-passed fallback tasks.

A task without the specified $(1,0)$ evidence does not become useless; it remains a Verifier-Passed fallback environment. The dataset analysis reports these as distinct groups, rather than treating a task’s human-skill origin as sufficient proof of dependence. This makes the label conservative about causal interpretation while retaining verified tasks for practice. What can the label tell you—and what can it not tell you—about a different agent?

Minimal Causal Code

This deliberately plain loop makes the two acceptance decisions visible. The verifier and run_agent are placeholders for the task-specific environment and reference-agent execution.

def accept_and_label(task, target_skill, run_agent, verifier):
    # Level 1: establish that the environment and its outcome check work.
    if not task.design_checks_pass():
        return "rejected"
    if not task.structure_checks_pass():
        return "rejected"
    if not task.end_to_end_checks_pass():
        return "rejected"
    if not verifier.is_working(task):
        return "rejected"
    if not verifier(run_agent(task, reference_solution=True)):
        return "rejected"

    # Level 2: compare the same reference setup with and without the skill.
    with_skill = verifier(run_agent(task, skill=target_skill))
    without_skill = verifier(run_agent(task, skill=None))

    if with_skill and not without_skill:
        return "Skill-Dep."
    return "Verifier-Passed fallback"

What You Gain, and What You Lose

The two-level rule avoids conflating executable correctness with demonstrated dependence, and the fallback label retains verified environments that do not meet the contrastive condition. The comparison is interpretable because the verifier supplies an outcome signal and the skill is the factor being contrasted.

But the conclusion stays narrow: Skill-Dep. means the reference agent passed with the target skill and failed without it under the tested configuration. It does not prove that the skill is necessary for every agent, or that no alternative skill or strategy could solve the task. And a fallback label is not a claim that the task is easy, low quality, or irrelevant; it says only that the specified dependence test was not met.