Teaching an Assistant When to Stay Silent

Why People Wanted It

A proactive assistant can have the right memory and still interrupt at the wrong moment. It may fire before an intention is actionable, keep speaking after it has already reminded you, or stay silent when the reminder is due. The real question is not simply whether the model can write a good reminder: can training teach it when a reminder is warranted, and when the right action is no action?

The retrieval system supplies evidence, but the generator still chooses what to do with it. In the reported setup, even correct evidence at every step did not prevent the generator’s timing performance from collapsing. Post-training targets that firing policy rather than asking retrieval to solve a decision it cannot make.

The Core Move

Train the generator inside its tool-using loop with a binary reward for the combined decision: speak once, in the trigger window, with an acceptable reminder—or stay silent when that is the correct action.

At time $t$, the gold action is fire only if $t$ is in the trigger window and no earlier fire has occurred; otherwise it is silent. A correctly timed, high-quality reminder and a correct silence receive positive credit, while unsuccessful outcomes receive the other reward. The evidence returned by the tool is context, not something the generator should learn to reproduce.

Why Evidence Is Context, Not an Action to Learn

Here is the part that gets technical: the policy is optimized over a tool-integrated rollout, but the retrieved evidence is injected by the harness rather than generated by the model. The generator observes the current frame, calls retrieval using the object in focus, receives refined evidence or none, then produces its reasoning and answer. The policy loss therefore masks the evidence tokens and applies only to generator-produced tokens: the observation call, reasoning, and answer. This keeps the learning signal attached to the choices the generator controls, instead of rewarding it for copying tool output.

Why the Reward Makes Timing and Repetition One Decision

Now the version for readers who want more of the machinery: the gold action depends on both the trigger window and the history of previous fires. If $W$ is the trigger window and $y_{t'}$ records an earlier action, then $g_t=\text{fire}$ exactly when $t\in W$ and there is no earlier $t'<t$ with $y_{t'}=\text{fire}$. In words, fire only while the reminder is due and still unsaid; every other step has the gold action silent.

The reward is binary and benchmark-verifiable. Correct silence is rewarded; a fire is rewarded only when it is correctly timed and its reminder passes the content-quality judgment. A miss, a false or repeated fire, low-quality content, or malformed generation falls into the unsuccessful outcome. A wrongly timed fire is not sent to the content judge. This makes non-repetition part of the action target, not a separate preference that can be traded away for more fluent reminders.

Why Group-Relative Optimization Fits This Reward

The deeper optimization move is to sample multiple responses for the same state and compare their rewards within that group. With $G=4$ rollouts per state, GRPO uses those relative outcomes to update the generator without requiring a separate learned value model in this described setup. The binary reward makes the comparison concrete: among attempts at the same decision, the policy is pushed toward the ones that get the action and reminder right, and away from early, repeated, or otherwise unsuccessful interventions. What remains is whether this policy improvement actually shows up beyond the harness’s own evaluation setting.

Minimal Causal Code

This deliberately plain loop makes the state update visible. gold_action implements the benchmark target; binary_reward stands for the benchmark’s action-and-content check. A real GRPO trainer also groups rollouts and computes policy-gradient updates.

for state in training_states:
    group = []
    for _ in range(4):
        object_query = generator.observe(state.frame)
        evidence = harness.retrieve_and_gate(object_query, state.frame)
        reason, answer = generator.decide(state.frame, evidence)

        gold = "fire" if (
            state.in_window and not state.already_spoken
        ) else "silent"
        reward = binary_reward(
            action=answer.action,
            reminder=answer.reminder,
            gold_action=gold,
            content_check=state.content_check,
        )
        group.append((object_query, reason, answer, reward))

    # Update the policy from group-relative rewards.
    generator.grpo_update(group, mask_tool_evidence=True)

    # The next decision sees whether this rollout has already fired.
    state.already_spoken = state.already_spoken or any(
        item.answer.action == "fire" for item in group
    )

What You Gain, and What You Lose

The reported comparison adds GRPO on top of the ProPerAgent harness. On the windowed setup, Perfect rises from 37.5% to 42.4%; on streaming, it rises from 33.3% to 40.6%. The breakdown is not a uniform win: windowed Hit moves from 54.6% to 54.0%, while over-triggering falls from 22.3% to 13.5%; streaming Hit moves from 60.0% to 60.4%, while over-triggering falls from 32.8% to 25.3%. The useful story is a better balance of timing and restraint, not that every metric improves.

The reward also has a cost: content quality depends on a judge, and a binary signal does not say how close a failed response came to being useful. Training is tailored to the benchmark’s trigger windows, history, and reward checks; it does not by itself guarantee good intervention policy in situations those checks do not cover. And because the tool supplies the evidence, this stage teaches the generator’s behavior around evidence—it does not repair retrieval or decide which memories deserve to surface.