SkillGym: Turning Skill Guides into Verified Practice Environments
Human skill guides are too general to practice directly; SkillGym pairs skill cards with reusable task templates and outcome checkers to create executable practice environments.
Read the paper ↗What it is
- Skill card A structured record of a reusable procedure, including guidance and constraints that can inform a concrete task.
- Task template A reusable task scaffold for a category of work, with room for different inputs, instructions, outputs, and completion checks.
- Outcome verifier Code that checks whether a task's concrete requirements were met, rather than whether an agent copied a reference answer.
Why it matters
SkillGym connects human procedural guidance to executable practice: a card supplies the procedure, a template makes a task instance, and a checker tests its stated outcomes. This makes success inspectable while keeping the limits of the checker explicit.
Before getting into the machinery, here is the basic idea: take a human-written skill guide, pair it with a reusable task setup, and make the resulting task runnable with checks for its actual requirements. That matters because a guide can tell an agent how to work without giving it a concrete job or a dependable way to know whether the job is done.
-
Practice: A skill description is procedural knowledge, not an exercise. The canonical example is a deployment guide: knowing practices around connections, secrets, storage, and readiness is different from being handed a broken deployment to repair. SkillGym makes that gap into a task. More generally, a reusable skill needs a concrete setting in which its steps can affect an outcome. But does one task structure fit every skill?
-
Reusable structure: The answer is to separate the procedure from the scaffold. A skill card holds the target procedure and related guidance; a category-specific template provides a task blueprint, assets, file structure, runtime, and checker specification. For the deployment example, the guide says what practices matter, while a task template can provide a runnable service setup and a place for the agent to make changes. The card is like a recipe and the template like a kitchen setup; the analogy stops where real tasks need executable constraints and checks. Inputs and success conditions can vary across tasks that use a shared scaffold. So what counts as success: looking like a sample answer, or meeting the task's requirements?
-
Outcome checks: I find this distinction important: the checker tests what the task asks for, not whether the response imitates a reference verbatim. In a deployment task, that can mean checking files or running acceptance tests; content tasks can instead specify sections, formatting, keywords, or filenames. More generally, checks can inspect executable behavior, structured outputs, schemas, or final state. The mechanism is like a test suite: it makes a stated condition observable, but it cannot judge quality beyond what those tests encode. What if the environment or the tests are the broken part?
-
Refinement: SkillGym uses construction failures to guide revisions. If a task will not execute, the runtime may need work; if the checker is wrong, its logic may need repair; if the task is mismatched in difficulty or weakly connected to the skill, its requirements may need adjustment. Think of it as debugging a test environment as well as the agent's attempt—though environment debugging does not by itself prove that a skill was necessary. Revised tasks are checked again, and tasks without valid execution and verification are discarded. Executable tasks without the separate contrastive evidence of skill dependence can be kept as fallback environments. The open question is how to test that dependence; that is a separate module's concern, as are collecting trajectories for training.
The construction recipe itself is intentionally plain: pair card and template, instantiate an executable task, check outcomes, and revise failures. The source pages state this design in prose rather than giving a governing formula. The payoff is concrete, inspectable practice; the cost is that templates, task requirements, and verifiers need careful construction and maintenance.
Read the deep dive
Why People Wanted It
A human-written skill guide can describe a reusable procedure, but a description alone does not give an agent a concrete problem to solve or a reliable way to tell whether it solved it. If you want to practice procedural knowledge, you need an environment where the guide can be applied and where success is checked against the task—not against how closely an answer resembles a sample.
So the real question is: how do you turn a reusable skill into a specific, runnable task without making every task identical or trusting unconstrained generation to invent the whole environment?
The Core Move
SkillGym separates the procedure from the practice setting: it pairs a structured skill card with a reusable, category-specific task template, then instantiates the pair as an executable task with a code-based outcome verifier.
The source pages give this design in prose; they do not state a governing formula for this construction. A skill card records the target procedure, while a template supplies the reusable task structure and completion criteria. Together they define what the agent should do and how the environment can check the result.
Why a Skill Card Is Not a Task
Now the version for readers who want more of the machinery: the card preserves skill-specific procedural information, while the template captures the common workflow and executable scaffold for a sub-category. A card can consolidate descriptions, usage examples, pseudocode or scripts, and execution constraints. The corresponding template specifies a task blueprint, input assets, file structure, runtime requirements, checker specifications, and execution constraints.
That division matters because a reusable procedure and a concrete problem instance are different objects. Inputs, instructions, expected outputs, and success conditions can vary across instances even when they exercise a shared workflow. SkillGym organizes skills into categories and sub-categories, selects skills for structured cards, and manually constructs reusable templates for sub-categories; the template is a scaffold, not a claim that all tasks should be interchangeable.
Why the Checker Targets Outcomes
Here is the part that gets technical: the verifier checks encoded completion requirements, not whether the agent reproduces a reference response verbatim. For content-generation tasks, requirements can cover sections, length, formatting, keyword coverage, and file naming. For software, data-processing, and automation tasks, checks can include executable tests, file inspection, structured-output comparison, schema validation, or final-state checks.
A task instance combines an instruction, input assets, runtime and file constraints, and a code-based verifier. Tasks are packaged in Docker for reproducible execution, and the verifier can include checks against input tampering, answer copying, bypassed requirements, and fabricated outputs. The important boundary is that the checker establishes whether its encoded requirements are met; it is not a general judgment of output quality. What should happen when the task or checker itself is wrong?
Why Failures Change the Environment
Now one level deeper: construction failures are treated as signals for revising the environment, not merely as failed agent attempts. The reported failure groups include environment errors, checker errors, difficulty mismatch, and weak connection to the target skill. Construction traces, execution logs, verifier outputs, and reflection notes guide changes to runtime configuration, task requirements, and verifier logic; difficulty or dependency problems can lead to changes in task demands and their connection to the procedure.
The loop is concrete: revise an instance, check it again, and discard it if valid execution and verification cannot be established. Executable tasks that lack the specified contrastive evidence for skill dependence can remain as fallback environments; this module is about making the practice task and its outcome checks, not the separate contrastive test for empirical skill dependence. How do you keep the verification loop visible rather than hiding it behind an abstract pipeline?
Minimal Causal Code
This plain loop shows the construction logic: pair the procedure with a reusable scaffold, run the resulting task, inspect the checker result, and revise when a construction failure is found. It is explanatory pseudocode, not a quoted implementation API.
# skill_card: target procedure and its constraints
# template: reusable task scaffold and outcome-check specification
task = instantiate(template, skill_card)
while True:
run_result = execute(task)
check_result = task.verifier(run_result)
if run_result.is_executable and check_result.passed:
break
failure = diagnose(run_result, check_result)
task = revise(task, failure)
What You Gain, and What You Lose
You gain a repeatable way to turn human procedural knowledge into runnable practice, with completion criteria tied to task outcomes rather than reference-answer imitation. Separating cards from templates also lets a shared task structure support varied inputs and success conditions.
You do not get an automatic guarantee that the skill is necessary, that a passing artifact is high quality beyond the encoded checks, or that every generated task is valid. Template construction and refinement require design work; verifiers cover only the requirements they encode, and tasks that fail valid execution and verification are discarded. The practical tradeoff is explicit: more structure and inspectable checks in exchange for authoring and maintaining task-specific environments and checkers.
A skill guide becomes practice when a reusable task scaffold gives it a concrete setting and an outcome checker gives success a test.