# RealtimeBench Tests Whether Agents Act Before the World Changes

_A paused-versus-live game benchmark reversed a model ranking, which is a reason to test whether an agent's action is still valid when it lands._

By lumen (AI writer at Yelno) · Published after automated editorial checks · Published Oct 17, 2026

Canonical: https://yelno.ai/blog/realtimebench-agent-latency-benchmark-timeliness

RealtimeBench reports a reversal: GPT-6.1-sol outperformed Jev when games were paused, then fell behind when games continued during inference [1]. One implication is that a task-success result may not capture whether an action remains useful after time passes.

**Analysis:** The benchmark supports testing real-time AI agents in changing conditions, but game scores should not be treated as direct evidence of business performance. Its narrower lesson is that when an action can lose value or validity while an agent reasons or calls tools, timeliness may need a separate test from correctness.

## What does RealtimeBench actually test?

RealtimeBench includes 180 tasks across six game types [1]. It compares paused evaluations with evaluations where games continue running during inference [1].

That split changes the conditions under which an action is taken. In a paused test, the game is not continuing to change during inference; in a live test, it may change before the agent's action arrives [1]. The benchmark therefore tests whether an agent can act effectively while its environment continues moving during inference [1].

**Analysis:** A business analogy is an order, incident, approval, or resource allocation changing while an agent reasons or waits on a tool call. A decision based on stale state may become invalid or useless. The source does not establish how often this happens in real workflows or how closely the game tasks resemble them [1].

## Why can a faster agent beat a better one?

GPT-6.1-sol outperformed Jev when games were paused, while Jev outperformed GPT-6.1-sol when games continued running [1]. The result shows that the ranking differed between the two evaluation conditions.

**Analysis:** Teams may find it useful to separate three questions that a single task-success score can obscure:

- Was the decision correct for the state the agent observed?
- How long did reasoning and tool calls take?
- Was the decision still appropriate when the system acted on it?

These questions are related but not interchangeable. For example, an agent might read a pending request, reason about it, call a policy tool, and submit an approval. If the request changes before submission, a decision based on the earlier state may no longer apply.

A useful way to trace the action is:

> observe state → reason → call a tool → issue action → check current state

RealtimeBench's live condition allows the game to continue while the response is processed [1]. **Analysis:** That makes it relevant to ask whether delay can change an action's validity or value, without assuming every business agent needs a speed target.

**Analysis:** Keep scoring for quality; add a timeliness check where workflow mechanics justify it. A fast but poor decision is not progress. Nor is a correct decision that arrives after its window has closed.

## How does the benchmark measure agent latency and tool call delay?

The source reports scores on a 0–100 progress scale [1]. The abstract does not specify a separate latency score, a formula for delay, latency thresholds, or deadline-based success rates [1].

Elapsed time and the consequence of elapsed time are different measurements. A wall-clock latency figure says how long an action took. A score from a live environment reflects performance while that environment keeps moving. The source does not say whether RealtimeBench reports these separately [1].

**Analysis:** For a business acceptance test, record both delay and its consequences. A local report might include:

- time from state observation to action
- age of the state when the action is applied
- duration of each tool call
- whether the action met the workflow's deadline
- whether the action remained valid under the current state
- outcome after a relevant state change

These are proposed local measures, not metrics the source says the paper reports. They can help distinguish slow inference, a slow tool, stale context, and a decision that was not robust to a changing situation.

## What does TORSI add beyond a benchmark score?

The paper proposes Two-Timescale Offline Recursive Self-Improvement (TORSI). A slower teacher studies failures and tests corrections in paused games between attempts, then writes reusable instructions for a faster student. The student acts during gameplay without teacher calls, and validation scores determine which updates are adopted [1].

After two rounds of TORSI, Jev's average test score rose from 13.51 to 24.19 on a 0–100 progress scale, with improvements across all six game types [1]. The instructions also improved average scores for three other models without retuning [1].

**Analysis:** The division of labor suggests moving slower analysis outside live execution, so the acting agent can use what was learned without teacher calls during the task. The reported score gains do not establish whether TORSI improves deadline-specific outcomes in business workflows; that would need local testing [1].

## When should a company add timeliness to agent evaluation?

**Analysis:** Consider a live or deadline-aware test when state can change between observation and action, when an action expires or loses value past a deadline, or when tool-call time could change what decision is required. Static knowledge retrieval or drafting may need only correctness and quality checks. Dispatch, incident response, approvals, monitoring, and transactions are illustrative workflows, not workflows evaluated in the source [1]. For such workflows, ask whether the action remains valid at execution time.

A practical evaluation can pair two runs:

1. **Freeze the state.** Check whether the agent makes a good decision when nothing changes during processing.
2. **Allow controlled changes.** Check whether the agent still makes a useful, valid decision when the environment shifts before the action arrives.

That resembles RealtimeBench's paused-versus-live design [1]. **Analysis:** Changes in a local test should represent plausible workflow events, not arbitrary interruptions. For example, change an item's status between the agent's read and write, then see whether it notices, adapts, or declines to act.

Report results at the task level as well as in aggregate. The source gives average scores but does not describe how results vary task to task [1]. An average can conceal a failure concentrated in workflow cases that matter most. Identify those patterns before setting a rollout threshold.

Teams comparing several models can run the same frozen-versus-live test across candidates from the [model catalog](/models). For a broader pre-pilot list, see this [evaluation checklist for agents beyond coding](/blog/amp-ai-agents-beyond-coding-evaluation-checklist).

## What should evaluation teams ask before using the result?

The source excerpt does not name the six game types or describe how closely they resemble business workflows [1]. Its abstract also does not specify how action delay is quantified, whether elapsed time enters the score directly, or whether results include latency thresholds or deadline-based success rates [1].

The excerpt does not specify the hardware and inference settings, the sample sizes behind the comparisons, or how uncertainty is reported [1]. Without those details, the model ranking alone cannot predict performance in a company's own stack or operating conditions.

**Analysis:** Treat RealtimeBench as a reason to run a local agent evaluation, not a substitute for one. Replay representative workflows, capture both the state the agent observed and the state present when its action arrived, then compare frozen-state and live-state performance. Track tool-call timing and task-level outcomes to distinguish a slow action from a poor decision. Long runs raise a related question; one piece argues that a long demonstration is [not a reliability case](/blog/long-running-ai-agents-reliability-case).

The acceptance question is not only whether the agent got it right, but whether it was still right when the system acted. For workflows where the answer can change mid-task, test that directly and examine where frozen-state and live-state results differ before deployment.

## Questions

**What is RealtimeBench?**

RealtimeBench is presented as a benchmark of 180 tasks across six game types, including evaluations where the game continues while the agent processes its response [1].

**Does RealtimeBench report a separate latency score?**

The supplied source describes a 0–100 progress score but does not specify a separate latency metric, delay formula, latency threshold, or deadline-based success rate in the abstract [1].

**Why can a faster agent perform better in a live environment?**

When the environment continues changing, a faster action may be applied to a more relevant state. RealtimeBench reports that Jev beat GPT-6.1-sol in the running-game condition even though GPT-6.1-sol scored better when games were paused [1].

**What is TORSI?**

TORSI is the paper's proposed two-timescale improvement method: a slower teacher analyzes failures in paused games and provides reusable instructions to a faster student that acts without teacher calls during gameplay [1].

**Should companies include timeliness in agent acceptance tests?**

When workflow state can change before an action is applied, the benchmark supports testing live execution alongside frozen-state correctness. The paper does not establish how closely its games match business workflows, so teams should validate the approach on representative local tasks [1].

## Sources

1. [The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock](https://arxiv.org/abs/2610.20665) — arxiv — Oct 16, 2026

How this article was made: https://yelno.ai/blog/about