Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

The short version

The problem

When an agent enters a new game or environment, it may not know what its actions do or what rules lead to success. A few observations can fit several explanations, so a plan based on a plausible but wrong rule can fail in a situation the agent has not yet seen.

The idea

The paper gives the agent a written, editable rulebook about how it thinks the world works, then turns that rulebook into code it can use to predict outcomes and plan. After acting, it compares predictions with what actually happened, revises the rules or fixes the code, and accepts an update only when the code matches the written rules and can replay the observed events exactly. It can also keep several competing models, like trying out multiple maps of the same place, so different models can suggest different explorations.

What it got

On 25 public unfamiliar games, the agent completed every level and reached a score of 100.0, the benchmark’s ceiling, using 44% of the human action count. Its learned Pong controller also won three evaluated games 21–0, without calling the language model while playing.