Evaluating Independent Performance After AI Assistance

Why People Wanted It

If someone solves a problem with an assistant beside them, what can they do once the assistant is gone? Accuracy during assisted work mixes the person’s contribution with the tool’s contribution. The problem is interpreting that success: completing a task with help is not the same as developing the ability to complete it alone.

So the real question was never just whether AI helps people answer problems. It was whether AI-supported practice leaves them able—and willing—to solve problems independently afterward. The evaluation separates the practice environment from the later unaided test, and records skipping alongside correctness.

The Core Move

Randomly assign people to practice with or without AI, then remove assistance for a shared unaided test and compare both correct answers and skipped problems.

This is an experimental protocol, not a mathematical update rule; no formula is specified for it. The essential contrast is between randomized practice conditions, evaluated afterward under a common no-assistance condition. Assignment, rather than self-selected AI use, defines the comparison.

“At the beginning of the experiment, participants were randomly assigned to two conditions” and the final problems “were identical across conditions and served as the primary measure of independent performance.” — p. 3

Quiz

Why remove the AI assistant before the final test?

To guarantee that everyone learned the same amount during practice.

To prevent the control group from seeing the test problems.

To measure what participants can do when both groups face the same no-assistance condition.

To measure how quickly participants can use the AI interface.

$ check

Why Withdrawal Separates Help from Independent Ability

Now the version for readers who want more of the machinery: withdrawal changes the availability of assistance while the final problems are held common across conditions.

In the fraction task, the AI sidebar was pre-prompted with each problem and its solution. Participants could obtain an answer with little effort. Assisted correctness therefore cannot distinguish independent solving from obtaining the solution through the assistant. Removing that assistant changes what a correct response demonstrates: the participant now has to respond without the supplied tool.

“The AI assistant was pre-prompted with each problem and its solution, allowing participants to receive immediate, accurate answers with minimal effort.” — p. 3

The first experiment used 15 fraction problems: 12 during practice and three after withdrawal. Controls received the same problems without assistance. Both conditions also saw the correct solution after an incorrect response, so the control condition was not simply practice without feedback.

“Participants in the control condition were presented with the same 15 problems with no assistance.” In both conditions, “if a participant submitted an incorrect answer, the correct solution was shown on the same screen.” — p. 3

This isolates a useful outcome: performance on the chosen problems without the supplied assistant. It is not a complete measure of learning or durable skill. And if withdrawing the assistant also changes the interface, could that transition contribute to the result?

Why Skipping Is a Separate Outcome

Here is the part that gets technical: solve rate and skip rate describe different observed behaviors, even though neither identifies a psychological cause by itself.

Solve rate records successful answers across the test problems. Skip rate records problems participants chose not to attempt through the available skip action. An incorrect submitted answer and a skipped problem both fail to produce a correct solution, but only the latter explicitly records non-engagement through that action.

The task instructions matter for interpreting this distinction. Participants were told that wrong answers carried no penalty and that payment did not depend on correctness. Under those rules, skipping was used as an indicator of motivation and persistence, rather than treated as just another wrong answer.

“Participants were explicitly informed that there was no penalty for providing wrong answers” and “their payment didn’t depend on how many questions they solve correctly.” Participants could skip using a “skip” button. — p. 3

The reading-comprehension experiment extended the operational definition: responses submitted in less than five seconds also counted as skips because that interval was too short to read the texts.

“We also recorded a problem as ‘skipped’ if the participant provided their answer in less than 5 seconds.” — p. 8

That makes the measurement rule part of the result’s meaning. A skip is informative about engagement under these instructions, not a diagnosis of why someone stopped. How do we keep differences in starting skill from being mistaken for consequences of AI-supported practice?

Why Random Assignment Matters—and What It Cannot Fix

Now the causal contrast gets more technical: random assignment makes groups comparable in expectation before practice, but exclusions based on later performance can undermine that comparison.

The first experiment had a specific vulnerability. Excluding people who could not solve basic fractions could remove low-skill controls while retaining similarly low-skill participants who obtained correct answers through AI. The retained groups could then differ in starting ability despite the original randomization.

“Our exclusion criteria removed participants unable to solve basic fraction problems but didn’t account for participants in the AI condition who were similarly unable yet submitted correct answers via AI.” — p. 5

The replication addressed this with an unaided pretest and exclusions based on that pretest rather than performance during the intervention. It also gave controls a sidebar containing previously seen pretest solutions, then removed it for the test. This made sidebar removal common to both conditions without giving controls new solutions during practice.

“We added a pretest of easy one-step fraction problems and used pretest performance for exclusions, rather than in-experiment performance.” Controls received “a sidebar displaying pretest solutions” to address the interface asymmetry. — p. 5

The resulting comparison concerns assignment to an AI-supported practice environment. It does not isolate the causal effect of choosing hints versus requesting answers, because those choices were not themselves randomized. Nor does it establish what would happen with a different assistant or a longer learning schedule. What should an evaluation optimize when immediate completion and later independence can diverge?

Minimal Causal Code

This plain PyTorch loop illustrates the protocol, not a fitted model or a simulated experiment. It takes randomized assignments and the shared problem lists as inputs; the callbacks represent participant interactions. The visible state updates accumulate correct responses and skips without inventing a sample size or an assignment ratio.

import torch

results = {"ai_practice": [], "unaided_practice": []}

for person, group in randomized_assignments:
    # Practice access differs by randomized condition.
    for problem in practice_problems:
        practice(
            person,
            problem,
            assistant_available=(group == "ai_practice"),
        )

    # Withdrawal: the supplied assistant is unavailable to everyone.
    state = {
        "correct": torch.tensor(0),
        "skipped": torch.tensor(0),
    }
    for problem in shared_test_problems:
        answer, skipped = respond_without_assistant(person, problem)
        correct = (not skipped) and is_correct(answer, problem)

        # Explicit accumulation of the separate outcomes.
        state["correct"] = state["correct"] + int(correct)
        state["skipped"] = state["skipped"] + int(skipped)

    results[group].append(state)

For the reading task, the response callback must also apply the response-time skip rule. The comparison uses the same test denominator for both groups and retains correctness and skipping as separate outcomes.

What You Gain, and What You Lose

You gain a direct test of performance without the supplied tool, rather than inferring independence from assisted answers. Random assignment supports a cleaner comparison than grouping people by voluntary AI use. Recording skips alongside correctness also makes deliberate non-engagement visible.

In the fraction replication, assistance improved performance during practice, but solve rates dropped and skips increased when it was removed. On the shared unaided test, the AI-assigned group had a lower solve rate than controls. The reading-comprehension experiment also found lower final solve rates in the AI-assigned group.

“AI assistance improved performance during the learning phase, but solve rates dropped and skip rates increased once the AI was removed”; the AI condition had a “lower solve rate” than controls on the final test. — p. 5

“Participants in the AI condition had a lower solve rate” than controls on the final reading-comprehension problems. — p. 8

What you lose is breadth: this test measures performance after a particular practice setup on particular follow-up tasks, not every dimension of learning or long-term capability. Exclusion rules and interface transitions require care, and skipping remains an operational indicator rather than a complete explanation of motivation. You gain a sharper evaluation of independence, not a universal verdict on AI assistance.

Quiz

Why measure skipping as well as correctness?

Because correctness is not observable without AI.

Because skipping is a distinct persistence-related outcome, not just an incorrect answer.

Because skip rates directly reveal participants’ underlying ability.

Because every skipped problem should be counted as correct.

$ check