AgentTime: Evaluating Runtime Control

Why People Wanted It

A task score tells you whether an agent completed a task; it does not tell you whether the agent respected a time budget, kept working for a requested duration, or even knew how long its run lasted. Ask an agent to iterate for an hour and it might stop when it thinks the task is done, or spend longer than the available compute allows. It might also appear to meet the duration by sleeping after finishing. So the real question was never just whether an agent can solve a task: can it estimate, control, and use its own runtime?

AgentTime makes those questions separable. It evaluates requested-duration work, runtime forecasts, and retrospective elapsed-time estimates, alongside native task scores and behavior such as early returns, overruns, and sleeping.

The Core Move

The core move is to treat runtime control as three distinct skills, then measure each rather than using task completion as a proxy. Let an evaluation record be $E = (D, F, R, S)$: requested-duration behavior $D$, forecast $F$, retrospective estimate $R$, and task score $S$. This notation is only a checklist: it keeps duration-following, prediction, time estimation, and task quality from collapsing into one result.

Quiz

Which set names AgentTime’s three runtime-control skills?

Task scoring, tool use, and memory

Duration-following, forecasting, and retrospection

Latency, throughput, and token efficiency

Planning, coding, and verification

$ check

Why Duration Compliance Is Not Continued Work

Here is the part that gets technical: duration compliance is an observation about elapsed runtime, not direct proof of what the agent did throughout that interval. In the duration-following setup, a task receives an appended instruction specifying how long to work; requested durations span from about a minute to multiple days. Runtime and native task score are recorded, while transcripts can be inspected for early returns, overruns, and explicit sleeping.

That distinction matters because two runs can have similar elapsed time and different work histories. A run that continues making task progress and one that finishes early and sleeps may both occupy the requested interval. The benchmark’s transcript review identified explicit sleeping in some classifiable Astra runs after the agent appeared to finish. Thus, runtime matching is evidence about duration-following, but it is not sufficient evidence of continued task work.

The evaluation also preserves each task’s native score rather than replacing task quality with a timing measure. This lets the result show both whether the task was done and how the agent behaved over time. But what can a runtime forecast tell us that duration-following cannot?

Why Forecasting Is a Separate Skill

Now the version for readers who want more of the machinery: a forecast is a prediction of a task’s natural runtime, whereas duration-following asks the agent to act within a user-specified interval. These differ in both target and evidence. Forecasting evaluates the estimate made about a task; duration-following evaluates the observed run against an external requested duration.

AgentTime reports that forecasts tend to overestimate natural runtimes. That tendency does not establish whether an agent can obey a requested duration: an agent can predict poorly and still keep working until the requested time, or predict accurately and stop early. Treating the forecast and the run as separate measurements avoids conflating calibration with control.

What does the agent know about time once the run is over?

Why Retrospection Depends on Time Cues

Here is the deeper measurement question: a retrospective estimate can draw on temporal cues in the interaction history, so the estimate changes when those cues change. In the retrospective experiments, agents estimate how long a finished run took; comparing transcripts with temporal information retained versus removed tests how much those cues support the estimate.

When temporal information was removed, deviation increased for the evaluated agents: it more than doubled for Sol and Astra and nearly doubled for Fable. The reported analysis also notes that much of this effect came from short questions padded to the requested time. The comparison therefore probes retrospective estimation under different information conditions; it does not turn elapsed-time estimation into duration control. An agent can estimate a past interval well and still fail to use a future interval as instructed.

Minimal Causal Code

A benchmark harness can make the separation visible with a deliberately simple record. This sketch does not reproduce AgentTime’s task suite or scoring; it shows how to keep the requested duration, observed runtime, forecast, retrospective estimate, and task result as distinct fields.

import time

requested_seconds = 60
forecast_seconds = agent.forecast(task)

start = time.monotonic()
result = agent.run(task, requested_duration=requested_seconds)
elapsed_seconds = time.monotonic() - start

retrospective_seconds = agent.estimate_elapsed_time(result.transcript)
task_score = score_task(result)

record = {
    "requested_seconds": requested_seconds,
    "elapsed_seconds": elapsed_seconds,
    "forecast_seconds": forecast_seconds,
    "retrospective_seconds": retrospective_seconds,
    "task_score": task_score,
    "transcript": result.transcript,
}

The key is not the data structure: it is that a high task score cannot overwrite a poor runtime result, and elapsed time cannot stand in for transcript evidence of continued work.

What You Gain, and What You Lose

Separating the three skills gives a clearer diagnosis than task completion alone. You can distinguish an agent that predicts the wrong runtime from one that ignores a requested duration, and you can examine whether a duration match involved continued task work. Keeping native task scores also lets timing behavior be read alongside task performance.

The limits are equally important. Runtime by itself cannot establish useful work across the interval; transcript review offers behavioral evidence but does not make every action’s value directly measurable. Forecasting, retrospection, and duration-following are distinct tests, so success on one does not imply success on the others. The practical gain is a sharper question for agent evaluation—not a single score that says an agent has mastered time.

Quiz

Why does matching a requested runtime not by itself prove that an agent kept working?

Task scores automatically include elapsed time.

Retrospective estimates determine whether the task is correct.

It may finish the task and then sleep for the remaining interval.

A runtime forecast is always the same as the observed runtime.

$ check