Evidence-Checked Orchestration for Computational Research
Computational research campaigns risk turning loosely coordinated experiments into unsupported claims; a staged, contract-driven agent workflow connects validated tasks, parallel investigation, retrospective evidence, and audited manuscript writing.
Read the paper ↗What it is
- Validated task Before agents explore, the task defines the candidate program, how it will be evaluated, and what counts as a valid result.
- Separate research gates One check asks whether an experiment followed its stated plan; another asks whether it improved the score. Those outcomes are tracked separately.
- Evidence-linked writing The final report is built from experiment outputs, and its claims are checked against the underlying results and artifacts.
Why it matters
Parallel agents can explore different ideas, but research is only useful when experiments are checkable and conclusions trace back to what ran. This workflow adds explicit handoffs and independent checks from task setup through manuscript review.
Before getting into the machinery, here is the short version: this is a way to run a computational research campaign with several agents without treating their conversation as the evidence. The workflow validates a task, lets workers explore in parallel, then uses the selected result and its artifacts to build comparisons and a manuscript. The point is not to automate scientific judgment away; it is to make the handoffs visible enough that people can steer and inspect them.
-
Executable task: A research brief is not yet an experiment. The task needs a candidate interface and evaluation procedures, plus programs that make the baseline and candidate checks concrete. The canonical example is a search evaluator: it decides whether a candidate is correct, gives the score used to rank candidates, and records metrics and artifacts. In the task-drafting path, a design agent proposes the method and protocols, then a drafting agent implements the task; review, smoke tests, and alignment checks happen before workers start. So what does “ready to explore” mean? It means the task has passed checks, not merely that an agent understands the brief.
-
Parallel exploration with memory: Think of several engineers taking turns from a shared queue, each working in a separate branch and leaving a concise handoff note for the others. That is close to how the workers operate: they keep separate run directories and local best programs, while sharing distilled conclusions at cycle boundaries. The analogy breaks because these workers are agents following a managed cycle budget, not people freely negotiating priorities. Each cycle separates intent from action: an ideation agent records a falsifiable hypothesis, plan, and target rubrics; an execution agent edits and evaluates the candidate. Why not let the same agent simply describe what it tried? Because a structured intent record makes it possible to check the implementation against the plan.
-
Evidence after exploration: A useful notebook analogy is to make a copy of the selected program before testing what each component contributes: the comparisons should refer to a stable candidate, not a moving target. The workflow adds metrics, proposes baselines and one-change ablations, and runs them alongside the initial and selected programs. Post-hoc diagnostics can inspect logs or checkpoints, but they must not change the selected program. Then the writing stage uses staged results and raw artifacts as sources for claims; a separate audit checks whether the manuscript’s numbers and conclusions match those sources. The notebook analogy has a limit: reproducibility still depends on the actual evaluator, artifacts, and task. So the real question is whether the final story can be traced back to the runs that support it.
Read the deep dive
Why People Wanted It
A coding agent can propose an experiment, edit a program, run an evaluator, and describe the result. But a research campaign needs more than a sequence of plausible actions: the task must be executable, experiments must be comparable, and manuscript claims must point back to actual artifacts. The real question is how to coordinate exploratory agents without letting intent, correctness, and evidence blur together.
The Agentic Scientist treats computational research as a sequence of linked phases: task setup, parallel investigation, retrospective evidence generation, and grounded paper writing. Each phase hands concrete programs, evaluation outputs, or artifacts to the next, so the final manuscript is constrained by what the campaign actually produced.
The Core Move
Make each stage produce checkable artifacts that become the next stage’s inputs: validate the task first, separate idea contracts from execution, then generate evidence and audit claims against it.
No mathematical formula is needed to describe this orchestration; the mechanism is defined by its task contracts, evaluator gates, phase boundaries, and artifact trail.
Why the Task Is More Than a Prompt
Here is the part that gets technical: a task specification is an executable evaluation arrangement, not just a natural-language goal. It contains an initial program, a search evaluator, an optional final evaluator, and a machine-readable contract defining the candidate interface and evaluation procedures. The initial program supplies a deterministic baseline; the search evaluator decides correctness, returns the scalar ranking score, and records metrics and artifacts for later analysis.
When a research brief is used, a design agent turns it into a proposed experimental design, and a drafting agent implements the programs and contract. Independent design review, contract checks, smoke tests, an alignment judge, and—when running in containers—a production check stand between that draft and parallel investigation. The alignment judge checks scientific fidelity and that search and final evaluators are independent and use disjoint data. What must be true before a candidate is safe to explore? The workflow makes that a validation question, not an assumption.
Why Idea and Score Gates Stay Separate
Now the version for readers who want more of the machinery: workers take a fixed budget of research cycles from a synchronized queue, explore in separate workspaces, and share distilled conclusions at cycle boundaries. Each worker retains local runs and a best search-score program; after workers finish, an aggregator selects the best correct program for retrospective study.
Within a cycle, an ideation agent emits a structured bundle: a falsifiable hypothesis, ordered implementation plan, and target rubrics. Missing fields or mismatched rubric identifiers are rejected. The execution agent then tests the accepted contract against the evaluator in an edit–evaluate–verify loop, producing metrics, correctness checks, and rubric evidence.
The two decisions are deliberately different. Passing every target rubric completes the idea contract; updating a worker’s best program instead requires a correct candidate with a strictly higher score. A candidate can therefore satisfy the stated intent without becoming the worker’s best-scoring program. Why preserve that distinction? Because “did it do what we intended?” and “did it improve the search score?” answer different questions.
Why Retrospective Evidence Must Hold the Program Still
Here is the deeper boundary: retrospective work asks what the selected program does, rather than continuing to change what it is. The selected program is instrumented with additional metrics, then baselines and ablations are proposed from its code difference against the initial program. Each ablation is meant to isolate one introduced innovation; a separate filter checks relevance, feasibility, and single-change scope before an isolated implementation is evaluated alongside the selected and initial programs.
Post-hoc diagnostics can inspect existing artifacts or run new inference, but checks reject proposals that retrain or otherwise alter the selected program, or depend on unavailable artifacts. If checkpoint saving is needed, an optional bootstrap reruns the evaluator; subsequent analyses use the resulting version as frozen. This boundary makes comparisons interpretable, but what connects all those results to a trustworthy written claim?
The writing workflow stages aggregate outputs and raw artifacts, while a memory summary guides interpretation rather than serving as quantitative evidence. A writer produces a manuscript, claim-to-source map, and figure manifest; validation checks provenance, compilation, and structure, while an independent audit flags numbers, comparisons, or conclusions that overclaim or contradict canonical evidence.
Minimal Causal Code
This plain Python sketch shows the phase handoff: the next stage receives the selected program and its recorded artifacts, rather than an untracked conversational summary.
# Conceptual orchestration sketch; the named functions stand for campaign tools.
task = draft_or_load_task(research_brief)
validated_task = validate_contract_and_evaluators(task)
workers = run_parallel_cycles(validated_task)
selected_program, investigation_artifacts = aggregate_workers(workers)
evidence = generate_baselines_ablations_and_diagnostics(
selected_program=selected_program,
initial_program=validated_task.initial_program,
artifacts=investigation_artifacts,
)
manuscript = write_from_staged_evidence(evidence)
final_manuscript = audit_claims_and_repair(manuscript, evidence)
What You Gain, and What You Lose
You gain explicit handoffs, isolated worker workspaces, shared distilled learning, and a trail from candidate intent through evaluator results to manuscript claims. Independent gates can catch different failures: a rubric pass does not imply a higher score, and a higher score does not by itself establish that the intended hypothesis was implemented. Retrospective baselines, ablations, and diagnostics give the selected program a more controlled evidential context.
You also pay for the machinery. Task drafting and validation add setup work; parallel workers need coordination and context management; retrospective comparisons and audits require usable evaluators and artifacts. A structured workflow cannot make a weak task scientific or guarantee that an audit catches every overclaim. Its contribution is narrower and practical: it makes the path from research intent to reported evidence inspectable and repairable.
A trustworthy research pipeline keeps intent, score, evidence, and claims connected—but never mistakes them for the same thing.