Population-Based Exploration with Competing World Models

Why People Wanted It

A single model can be internally consistent and still be wrong. Worse, if its chosen actions never expose the mistake, its predictions may remain unchallenged while the agent follows a narrow path. The practical question is: can the agent preserve alternatives long enough for their different plans to reveal different evidence?

Memento 3 answers by keeping multiple rulebook–executable hypotheses, checking them against one shared interaction history, and choosing among behaviorally different plans. The other module on reflective rulebook-guided model construction covers how each individual model is built and repaired.

The Core Move

Keep several candidate world models against the same history, then sample among the distinct nonempty action plans they predict. The population is written as $\Phi_t$; the learning state is the shared history and that population, $(H_t,\Phi_t)$. The history is shared, not replayed in separate environment copies.

Quiz

In the population extension, what is shared across retained world models?

Only the most recent observation, with earlier history discarded

A separate environment instance for each model

One identical executable and reconstructed state

The interaction history and observed trajectories

$ check

Why Multiple Models Can Reveal Different Evidence

Now the version for readers who want more of the machinery: the population addresses a specific failure mode of single-model control—self-confirming behavior. For a fixed model, imagine following only the actions its own plan selects. The paper defines $R(\theta)$ as the histories reached under that model’s plan-induced policy. A model can be wrong on actions it never chooses yet remain unfalsified along its own path. Retaining alternatives matters because their plans can take the agent to different histories, where those hidden disagreements become observable.

The population contains $N$ retained hypotheses; $N=1$ recovers the single-model case. Each member has its own rulebook and executable, but every real observation extends the same $H_t$. This is not a set of parallel environment runs: it is a set of competing explanations tested against one record. What prevents the population from merely keeping many textual variants of the same behavior?

Why Verification Uses the Common History

Here is the part that gets technical: each real interaction is checked against every member using the accumulated trajectories, not just the newest frame in isolation. A prediction that contradicts an observation falsifies that executable; affected members then undergo the reflective update. Members that have not contradicted the record remain candidates. The population therefore filters by shared evidence while preserving multiple survivors when the record does not distinguish them.

The source describes proposal, repair, and plan/act as the population loop: proposals are retained when admissible; falsified members are repaired; and action outcomes are added to the common history. The important population-level constraint is that no member gets a private evidence stream. How should the surviving candidates be made useful for action selection?

Why Plans, Not Programs, Define the Alternatives

Now the deeper selection rule: at a shared history, each model reconstructs its own internal state and supplies a plan if it has a nonempty one. Two members count as behaviorally equivalent when their action sequences are identical. The system samples uniformly over distinct nonempty plan classes for each planning episode, then follows a representative plan until success, falsification, or replanning.

This distinction avoids spending exploration opportunities on different programs that prescribe the same actions right now. It also does not promise that every sampled plan is correct: the point is to give retained behavioral alternatives opportunities to produce new evidence. If no member has a nonempty plan and the task remains unfinished, the control operator explores or revises before sampling resumes. Which tradeoff does this design make between diversity and commitment?

Minimal Causal Code

This plain loop makes the shared record and plan-class choice visible; it omits proposal and repair internals.

history = []
models = initial_models

while not finished:
    for model in models:
        if model.contradicts(history):
            model.repair_from(history)

    classes = {}
    for model in models:
        plan = model.plan(history)
        if plan:
            classes.setdefault(tuple(plan), []).append(model)

    if not classes:
        models = explore_or_revise(models, history)
        continue

    plan, representatives = choose_uniformly(list(classes.items()))
    model = choose_representative(representatives)
    for action in plan:
        observation = environment_step(action)
        history.append((action, observation))
        if model.contradicts(history) or is_success(observation):
            break

What You Gain, and What You Lose

The gain is broader model-guided exploration: distinct surviving plans can produce distinct histories, reducing dependence on one early model choice. Shared-history checks also let every retained member be tested by the same accumulated interactions.

The cost is maintaining and checking a population, and the population is not a guarantee of correctness. If candidates agree on a plan, or their chosen actions fail to expose their disagreements, the agent can still miss important errors. Sampling distinct plans trades commitment to one model’s route for opportunities to test alternatives; it does not make each action safer or each hypothesis equally plausible.

Quiz

When does the population treat two models as behaviorally equivalent for plan sampling?

When they have equal numbers of parameters

When their rulebooks use the same words

When their nonempty predicted action sequences are identical

Whenever both models have survived verification

$ check