Synthetic Digital-Life Environments for Agent Training
Real users and tool histories are costly and sensitive to collect; this pipeline creates coherent simulated digital lives and filters their task trajectories for grounded, privacy-aware, and safe agent practice.
What it is
- Populated digital lives Create a simulated user with contacts, messages, calendar events, documents, and other application history that fit together.
- Grounded scenarios Generate tasks using only tools and people that exist in that simulated world, then check whether the requested outcome actually happened.
- Safety traces Use filtered examples to practice sharing only appropriate information and ignoring malicious instructions hidden in tool results.
Why it matters
Agents need practice with tool interactions that have real context and checkable outcomes. Coherent simulated histories make that practice scalable, while filtering can teach grounded completion, privacy-sensitive sharing, and resistance to hostile tool content.
Before we dive into the machinery, here is the short version: give an agent a believable little digital life to work in, then ask it to do tasks that depend on what is actually in that life. The important bit is that the messages, people, and tools fit together, and that a trajectory only counts when its result can be checked. That gives you a way to practice tool use without treating every plausible-sounding answer as success.
- Realistic practice: Tool use depends on context. An email task is more than a command; the recipient, previous messages, and documents can change what a sensible action is. The pipeline builds those pieces into one simulated world.
- Grounding: If a task mentions a contact or application that was never set up, the agent is being tested on a hallucination trap rather than the intended work. So scenario generation is constrained by the world’s actual contacts and tools, and the resulting task gets a rubric.
- Safer behavior: Personal-assistant tasks can involve deciding what to share, or spotting hostile text in data returned by a tool. The training examples need to reward doing the legitimate job while avoiding the unsafe move—not just refusing anything that looks complicated.
The canonical example is a single simulated user with a network of contacts and a busy inbox. A persona starts from demographic and personality attributes and is expanded with a job, hobbies, and habits; the surrounding people form a relationship graph. That graph is useful because a calendar event or message can refer to established people rather than a random name pasted into the prompt. The environment described here uses a persona with 10–14 contacts and weeks of correspondence, events, documents, and messages. Think of it as a notebook shared across apps: entries can refer back to each other. The analogy breaks because the environment also has tools that can change state, not just pages to read.
The mock tools are populated in dependency order. In plain terms, you establish the people before writing emails about them, and establish event participants before creating the events. The tool interfaces use argument shapes and JSON-style returns like real counterparts, while the simulated state makes those calls meaningful. That is what makes “send a note to Alice” different from inventing Alice on the spot.
So how do tasks emerge from that world? First, a generator sees a concise summary and proposes objectives using only available tools and contacts. Then it turns those objectives into runnable user requests, adds setup material if a scenario needs it, and creates a rubric. The key intuition is a closed loop: the task is grounded in the world, and the result is checked against that same world. Post-processing removes failures, incoherent or duplicated traces, unverified outcomes, and unwanted behaviors; even a trajectory that reaches its goal can have bad tool calls masked.
Privacy and injection resistance use the same broad synthetic-data approach. For privacy, traces teach the agent to consider who is sending information, who receives it, what kind of information it is, and the conditions of sharing. The useful mental model is deciding what belongs in the message—not dumping the whole notebook into it. For adversarial content, a tool result might contain an instruction that should not override the user’s legitimate goal. The desired behavior is to ignore that instruction, refuse an unsafe subtask when needed, and continue safe work. The boundary matters: too much caution can become an over-refusal. So the real question is not whether the agent can say no, but whether it can tell which parts of a task are safe to complete.
Read the deep dive
Why People Wanted It
An agent can only learn to use tools well if its practice looks like tool use: messages refer to real contacts in the environment, calendar tasks depend on existing events, and a successful-looking answer is checked against what actually happened. Collecting enough realistic personal histories is costly, and personal data brings its own risks. The real question is how to create many coherent practice worlds without teaching the agent to invent details, overshare, or obey hostile text it encounters along the way.
The Core Move
Build a populated simulated digital life first, then generate tasks and keep only trajectories whose actions and outcomes are grounded in that world. A universe is a persona plus a network of contacts and populated application state; a scenario is an objective and a verifiable way to check whether it was completed.
Why Relationships Make the World Coherent
Here is the part that gets more specific: a persona is not just a name and a biography, but an anchor for a graph of people whose relationships support interactions across tools. Personas begin with sampled demographic and Big Five personality attributes, which are expanded into profiles with occupations, hobbies, and habits. Some profiles are grounded in sampled text to broaden the range beyond temperature-based generation.
The graph matters because a message or calendar event needs a plausible counterpart. The described universes use a persona and a network of 10–14 contacts, with weeks of correspondence, events, documents, and messaging history. The count is a property of this design, not a general requirement for synthetic environments. Why stop at a graph of people rather than populate their shared tools too?
Why Tool State Is Built Before Tasks
Now the version for readers who want more of the machinery: the pipeline materializes a suite of mock tool states in dependency order. It selects a realistic subset of applications, then populates earlier state before later content that depends on it—for example, contacts before emails that refer to them, or participants before calendar events. Workspace files and memory entries are generated as part of that universe.
The tools use argument structures and JSON return formats modeled on real counterparts, while the simulated environment supplies the world state behind those calls. This makes a task about searching a message or scheduling with a contact refer to things that are actually present. Once that world exists, objectives can be constrained to its tools and contacts instead of asking an agent to act on invented resources. How do you turn a coherent world into a trajectory you can trust?
Why Grounding Filters Matter
Here is the technical hinge: scenario generation separates proposing an objective from making it runnable. Given a concise summary of contacts and recent history, the generator proposes objectives across execution, search, adaptability, and time management, limited to available tools and contacts. It then rewrites objectives as natural user requests, plans missing setup content such as a needed email thread or invite, and produces verifiable rubrics.
Post-processing removes failed or incoherent trajectories, duplicates, and “fourthwall” breaks. A further grounding check discards trajectories with hallucinations, false success claims, or outcomes that were not validated. Parts containing undesirable behavior—such as bad tool calls or memory-search queries—can be masked from training even when the overall goal was achieved. So what should these carefully grounded traces teach beyond ordinary task completion?
Minimal Causal Code
This sketch shows the dependency that matters: first establish the world, then generate a task against its actual contents, then retain only a checked trajectory.
for persona in personas:
graph = make_relationship_graph(persona)
universe = populate_tools_in_dependency_order(persona, graph)
objectives = propose_objectives(
summary=universe.summary(),
available_tools=universe.tools,
available_contacts=universe.contacts,
)
for objective in objectives:
scenario = make_runnable_scenario(objective, universe)
trajectory = run_agent(scenario, universe)
rubric = make_verifiable_rubric(scenario, universe)
if passes_quality_checks(trajectory) and is_grounded(trajectory, rubric):
keep_for_training(trajectory)
The helper names are conceptual; the visible causal order is the important part.
What You Gain, and What You Lose
A populated universe gives tasks shared context across email, messaging, calendars, documents, and other tools, while constrained objectives and outcome checks make unsupported success harder to pass off as competence. It also gives a route to training privacy and injection behavior: traces can teach context-sensitive sharing, and adversarial examples can reward ignoring hostile instructions embedded in returned data while still completing safe work.
But simulated histories are still generated histories. A coherent mock world cannot guarantee that every interaction captures real-world nuance, and quality depends on the checks that catch fabricated or undesirable behavior. There is also a deliberate tension: privacy behavior should not mean refusing every request. The training traces discard oversharing, omission of necessary information, and incomplete tasks; safety traces likewise discard both obedience to adversarial instructions and over-refusal of legitimate work. The target is not blanket caution, but sharing and acting appropriately for the context.
A useful synthetic trajectory starts with a world that has real internal state and ends with an outcome that can be checked against it.