Graph Retrieval and the Intervention-Worthiness Gate

Why People Wanted It

A reminder system has two ways to fail: it can surface the wrong memory, or surface a useful memory at the wrong moment. A search result can be related to the scene and still be a bad reason to interrupt someone. So the real question was never just “Which memories match this object?” It was “Which memory is both connected to what is happening and worth acting on now?”

ProPerAgent addresses that question with two linked moves: store personal history in an episodic–semantic graph keyed to objects, then have a gate decide whether retrieved candidates justify intervention. The gate can return refined evidence—or nothing—before the generator decides whether to speak.

The Core Move

Use the objects in the current view to retrieve linked personal facts, then pass those candidates through an intervention-worthiness gate before they reach the assistant. In compact form:

$$\text{view objects} \rightarrow \text{graph candidates} \rightarrow \text{gate} \rightarrow \text{evidence or none}$$

The gate is not another relevance score: it asks whether a memory warrants a reminder in this scene.

Quiz

What question does ProPerAgent’s gate answer that graph retrieval alone does not?

Whether every retrieved fact should be shown to the user

Whether the generator should permanently delete a memory

Which object has the highest pixel count in the observation

Whether a retrieved memory justifies intervening in the current scene

$ check

Why Objects Make Personal History Retrievable

Now the version for readers who want more of the machinery: the graph separates event-specific episodic memories from reusable semantic facts, while object links give an observation a path into both layers. Episodic memories represent past events around entities; semantic memories hold standing facts such as preferences and constraints. The visible object is therefore a retrieval handle, not merely a keyword in a passage.

For example, seeing a milk carton can seed retrieval of an event in which it was nearly empty and the user intended to buy more. That intention can also be mirrored as an object-linked fact such as the carton needing restocking. This makes the open loop reachable from the object in view. The source event remains attached as provenance, so retrieval can provide a compact fact or its surrounding narrative.

A memory may carry an open-loop flag when it expresses an unresolved intention, constraint, or object state. The flag is a retrieval cue, not an automatic instruction to interrupt: an intention can be resolved, or simply not matter in the current scene. How can retrieval reach useful facts when the visible object is not named exactly the same way?

Why Retrieval Goes Beyond One Hop

Here is the part that gets technical: object names are embedded and matched to entity nodes, which seed complementary retrieval paths. One-hop lookup returns episodic triples attached to matched entities and semantic facts selected through entity links or direct similarity. Episodic ranking uses entity similarity, open-loop cues, and recency.

A second path runs Personalized PageRank from the entity seeds. It can reach related event passages beyond one hop through co-occurrence, semantic co-mention, and synonym edges, while down-weighting generic hub entities. The point is not to treat every graph neighbor as a reminder; these paths generate candidates with useful context. Why should a candidate still face another decision after retrieval?

Why Relevance Is Not Permission to Interrupt

Now the deeper distinction: the gate reads candidate memories against the observed object, keeps those worth intervening on, and refines them into one clear evidence item—or abstains. Mere association is insufficient: an object can co-occur with a past mention, and an open loop can already be resolved. Feeding such candidates straight to the generator risks distracting it or provoking a false trigger.

The gate’s output fills the evidence slot in a short loop: the generator identifies objects in focus, retrieval and the gate return refined evidence or none, and the generator uses that result with the current view to choose a grounded reminder or silence. Evidence presence is one signal, not the final speaking decision; the generator still decides. What does this extra filter trade away in exchange for cleaner evidence?

Minimal Causal Code

This plain loop makes the state transition visible. retrieve_candidates and gate stand in for the graph search and gate call; neither is a speaking decision.

memory_graph = build_personal_graph(events, standing_facts)

for observation in stream:
    objects = identify_objects_in_focus(observation)
    candidates = retrieve_candidates(memory_graph, objects)
    evidence = gate(candidates, objects)  # refined evidence, or None
    answer = generator(observation, evidence)
    if answer.should_speak:
        speak(answer.reminder)

What You Gain, and What You Lose

Object-centered graph retrieval connects what is visible to both event history and standing facts; multi-hop propagation can recover related passages even when names do not match exactly. The gate then filters and consolidates candidates, including abstaining when a memory is not worth surfacing. This is a deliberate separation of jobs: retrieval finds plausible evidence, while the gate judges intervention-worthiness and the generator retains the final say.

The cost is extra structure and another model call in the retrieval path. The result also depends on the graph links and candidate evidence being useful: the gate can only judge what reaches it. An open-loop flag helps retrieval find unresolved matters, but it cannot by itself establish that an interruption is appropriate. Graph retrieval plus filtering therefore does not make speaking automatic; it gives the generator a more selective evidence channel.

Quiz

How should an open-loop flag be interpreted in this memory graph?

As a replacement for object-centered retrieval

As proof that the user still needs a reminder

As a retrieval cue for a potentially unresolved matter, not an automatic intervention decision

As a guarantee that the generator will speak

$ check