ProPerMem: Benchmarking Proactive Assistance

Why People Wanted It

A proactive assistant is not answering a question someone chose to ask. It watches what is happening, consults personal history, and has to decide whether this is a moment worth interrupting. That makes a sensible reminder insufficient: the same reminder can be helpful in a narrow window and annoying outside it.

So the real question was never just whether a system can produce a relevant sentence. Can it find the right personal evidence, ground a reminder in the current scene, say it at the appropriate time, and otherwise stay silent?

The Core Move

ProPerMem turns that question into a benchmark: pair a long personal history with an egocentric video clip and score whether the assistant makes exactly one grounded intervention in the allowed window—or none when there is no reason to intervene.

For a clip with trigger window $W=[t_s,t_e]$, the primary success condition is: exactly one fire, inside the permitted window, with acceptable content. A relevant sentence outside that window is not a perfect result.

Quiz

What does ProPerMem treat as a perfect result on a clip with a trigger window?

Several reminders, as long as one is correct.

Exactly one grounded, acceptable reminder in the permitted window, with no other fires.

A reminder whenever relevant personal history is retrieved.

At least one relevant reminder at any time during the clip.

$ check

Why A Long History Makes The Reminder Hard To Find

Now the version for readers who want more of the benchmark’s setup: each example combines a video clip with a realistic, year-long stream of timestamped first-person captions, mixing useful events with ordinary distractors. The mean history is about 618K tokens, while target evidence averages about 0.19% of that history. Those figures describe the benchmark’s histories, not a guarantee about every example.

The target is not simply a stored fact. The assistant must locate evidence that justifies a reminder and connect it to the current frame. For example, prior coffee-maker mishaps can support a clean-mug reminder when the video shows the relevant setup. The target memory supplies the evidence for the window; the current observation supplies the situation in which that evidence matters.

The benchmark gives each clip one trigger window. If a matter appears intermittently, the window is defined by its first appearance; later reappearances do not create another opportunity to repeat the reminder. That constraint distinguishes a timely first intervention from treating every recurrence as a fresh prompt.

Why A Hit Is Not The Same As Success

Here is the part that gets technical: the metrics separate “did it ever say something correct?” from “did it behave correctly on the whole clip?” A fire is a hit only when it is within the trigger window, allowing the stated timing tolerance, and its content is judged sensible given the window’s evidence and the current observation.

The benchmark then tracks early fires, late fires, and over-triggering separately. Its primary Perfect measure requires exactly one in-window fire with good content, and no extra fires. A correct reminder repeated three times is not perfect: the extra interruptions count as a failure even though one utterance was useful.

The default timing tolerance is two seconds, matching the benchmark’s two-second decision steps: a fire just after the annotated window can still count as on time. Negative clips remove the target memory, making silence the correct behavior; any fire on those clips is a cost. What does a system’s score mean if it only measures the times it chose to speak?

Minimal Causal Code

This loop makes the benchmark’s decision shape visible; it is illustrative evaluation logic, not the benchmark’s full model or judge implementation.

fires = []

for t, frame in video_stream:
    recent_video = update_recent_window(frame)
    memory_evidence = retrieve(personal_history, recent_video)
    action = policy(recent_video, memory_evidence)

    if action is not None:
        fires.append((t, action))

in_window_good = [
    (t, text) for t, text in fires
    if window_start <= t <= window_end + tolerance
    and judge(text, evidence, frame_at(t))
]

perfect = (
    len(fires) == 1
    and len(in_window_good) == 1
)

What You Gain, and What You Lose

You gain a score that does not let a well-worded but mistimed reminder pass as success. The separate early, late, repeated, and hit measures help distinguish different failure modes, while negative clips make restraint measurable. The setup also tests retrieval against long histories rather than supplying only a short relevant note.

You lose some simplicity. A strict all-or-nothing primary metric can hide partial competence: a system that finds the right memory and fires correctly, then repeats itself, is not perfect. Content quality is judged against evidence and the current frame, so the benchmark’s content criterion depends on that judging procedure. And the benchmark evaluates a defined video-and-memory setting; it does not by itself establish how an assistant will behave across every real-world situation.

The benchmark’s useful question is therefore not “Can it remind?” but “Can it justify one interruption at the right moment—and refrain from inventing another?”