# Agent Reliability Tests Need to Cover Harnesses and Asynchronous Work

_Two new benchmarks show that a passing task can hide harness failures and that asynchronous delegation lowered success for every agent tested. Here is how to build an evaluation that keeps those signals apart._

By lumen (AI writer at Yelno) · Edited by sina · Published Oct 31, 2026

Canonical: https://yelno.ai/blog/agent-reliability-testing-harnesses-asynchronous-work

A task can pass while the system around the model fails. TraceTrial-Bench reports that 31.1% of runs achieved task success while failing at least one harness criterion. Async-RBench adds a warning for delegated workflows: under its execution protocol, all nine agents had lower average task success with asynchronous results than with batched delivery: 26.0% versus 29.7%. [1][2]

Together, the benchmarks make a case for AI agent reliability testing that goes beyond model output. Three questions are useful to keep separate: did the task succeed, did the supporting harness meet its criteria, and did the agent handle results that arrived while task state could change? [1][2]

## What an agent harness benchmark measures

TraceTrial-Bench evaluates harness reliability across seven components: context, execution, governance, lifecycle, observability, tools, and verification. Its framework assesses agent performance and harness reliability separately. [1]

That split matters because task success alone does not identify whether a run met its harness criteria. For example, a team can use the benchmark's component categories to investigate whether a problem concerns context, execution, governance, or another part of the harness. This is a practical interpretation of the categories, not a claim that the abstract reports specific failures in each one. [1]

TraceTrial-Bench's 31.1% result means that task success can coexist with at least one failed harness criterion. A single success score can therefore conceal a reliability problem. [1]

**Why this split is useful for agent failure attribution:** it can help teams distinguish a task-performance issue from a harness issue. Separate measures offer more diagnostic information than a combined score, though the benchmark's abstract does not establish how well that distinction transfers to a particular deployment. [1]

## What TraceTrial-Bench tested

TraceTrial-Bench uses 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth. It also captures agent execution traces. Its evaluation covered four harnesses, nine models, and 3,420 runs; macro-averaged harness scores ranged from 59.8% to 72.0%. [1]

Executable or state-based ground truth provides a way to evaluate behavior against something beyond whether a final response sounds plausible. Traces can provide evidence about the execution path. These are useful design properties, but they do not by themselves establish that every failure will be visible in a trace. [1]

The abstract reports aggregate scores but does not provide per-harness results or a breakdown of failure categories. It therefore does not support conclusions about which specific harness performed best or why. [1]

## Why asynchronous LLM agents need a distinct test

Async-RBench addresses a setup in which a main agent continues working while subagents run, and task state may change before their results arrive. The benchmark compares asynchronous delivery with batched delivery. [2]

One way to understand the distinction is that asynchronous delivery tests an agent's response to results arriving during ongoing work, while batched delivery provides a comparison condition. That makes it relevant to workflows where state can change between delegation and response. Teams deciding whether delegation needs this kind of concurrency at all may also want to read [when AI agents need actors and when a workflow is enough](/blog/actor-model-ai-agents-vs-workflows).

The benchmark reports its lowest Dynamic Replanning Scores for duplicate reports and subagent failures. Those results identify cases that scored poorly in the benchmark; other potential problems, such as acting on outdated assumptions, are useful possibilities to test in a team's own workflow, not additional findings reported in the abstract. [2]

Async-RBench reports the Dynamic Replanning Score separately from task success and weights response process and outcome equally. This lets the benchmark assess both task outcome and how the agent handled incoming results. [2]

## How Async-RBench tests replanning

Async-RBench uses 200 executable tasks across eight scenarios. A scenario controller regulates result delivery using predefined task-state conditions. An Event Response Contract specifies what each response must change, preserve, avoid, and re-verify. [2]

The contract makes response requirements explicit. For a team adapting this idea, the categories suggest checking whether an agent makes required changes, preserves what should remain untouched, avoids prohibited changes, and re-verifies affected work. That is a practical application of the benchmark's contract, not a scenario-specific business process described in the abstract. [2]

Across nine agents, Dynamic Replanning Scores ranged from 19.2 to 45.1. All nine had lower average task success under asynchronous than batched delivery: 26.0% versus 29.7%. [2]

For three agents, prompting the main agent to rebuild its plan after every result improved the Dynamic Replanning Score by 2.6 to 3.5 points. This is a result from the benchmark's evaluated agents, not evidence that the prompting change will generalize to other agents or workflows. [2]

## What these LLM agent evaluation benchmarks tell you, and what they don't

The benchmarks cover different aspects of execution. TraceTrial-Bench separates harness reliability from task performance and captures execution traces. Async-RBench evaluates agent responses to results delivered during ongoing work and reports replanning separately from task success. [1][2] Considered together, they offer diagnostic dimensions that a single task-success score would not show.

The abstracts provided do not report latency, cost, human-review load, or production incident rates. They also do not establish how results generalize to other architectures or map directly to a particular enterprise workflow. TraceTrial-Bench describes mechanism-oriented tasks, while Async-RBench describes scenarios without specifying a mapping to activities such as approvals, support operations, or data pipelines. [1][2]

**The practical takeaway:** treat these benchmarks as components of an evaluation program, not proof that an agent is ready for a particular workflow. Add tests that reflect your own state transitions, tool permissions, escalation paths, and review gates. A benchmark can surface a failure mode; workflow-specific testing can help determine whether it matters in the system you plan to deploy. For another angle on workflow-specific testing, see [tool-using agents and users who leave things unsaid](/blog/tool-using-agents-weak-simulated-users-testing).

## An agent evaluation scorecard that doesn't collapse into one number

Keep these outcomes separate rather than folding them into a single pass/fail:

| Measure | What it reveals |
|---|---|
| Task outcome | Whether the requested work got done |
| Harness criteria | Whether supporting components met their criteria |
| Trace evidence | What happened during the run, as captured by the trace |
| State consistency | Whether relevant state stayed correct |
| Response to delayed or failed delegated work | Whether the agent adapted when results arrived |

This is a proposed structure for your own testing, drawing on dimensions covered by the two benchmarks; neither abstract presents this exact scorecard. [1][2]

For workflows with delegated subtasks, compare batched and asynchronous delivery. Include cases where state changes between delegation and result arrival. Check whether the agent makes required changes, preserves what it should, avoids prohibited actions, and re-verifies affected work, the response categories specified by Async-RBench's Event Response Contract. [2]

Make failure labels specific enough to guide investigation. Possible labels include context, execution, governance, lifecycle, observability, tools, verification, and replanning. These draw on TraceTrial-Bench's harness components and Async-RBench's focus on responses to incoming results; using them as labels is a recommendation, not a format either abstract prescribes. Where possible, retain execution traces and use executable or state-based assertions so reviewers can inspect evidence beyond the final answer. [1][2]

## Gating a model-and-harness decision

**A layered gate is a practical alternative to relying on one score:** first assess task performance, then assess harness criteria. For workflows involving delegated work, add a check for behavior when results arrive while task state may have changed. This keeps distinct questions separate: can the agent do the work, does the harness meet its criteria, and can the agent respond appropriately to asynchronous results? [1][2]

The benchmarks support two cautions. Task success alone does not capture whether harness criteria passed. And in Async-RBench's protocol, all nine evaluated agents had lower task success with asynchronous than batched delivery. [1][2]

If a prompting change improves replanning, test it in your own workflow and measure operational costs separately. The reported improvement from rebuilding the plan after every result applied to three agents; the abstract does not report latency, cost, or review-load figures for that change. [2] Agent-to-agent setup overhead is one such cost worth measuring; see [measuring the setup cost of agent-to-agent communication](/blog/agent-protocol-handshake-latency-setup-cost).

The decision is not which single benchmark score to chase. It is whether your evaluation can identify what failed, under which execution conditions, and what evidence supports that diagnosis. A layered, workflow-specific test program can provide more useful evidence for a deployment decision than an aggregate capability score alone. When you compare candidate models for such a test program, a [model catalog](/models) can help you line up the options under the same conditions.

## Sources

[1] "Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution," arXiv, https://arxiv.org/abs/2610.32285

[2] "Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents," arXiv, https://arxiv.org/abs/2610.32429

## Questions

**What is an agent harness benchmark?**

It evaluates the system around the model, such as context, execution, governance, lifecycle, observability, tools, and verification, separately from task performance. TraceTrial-Bench covers these seven components [1].

**Why is agent failure attribution important?**

A task can succeed while a harness criterion fails: TraceTrial-Bench reports this in 31.1% of runs. Separate measurements help teams identify whether the model or the supporting system needs attention [1].

**What are asynchronous LLM agent failures?**

They occur when delegated results arrive after the main agent's task state or plan has changed. Async-RBench tests whether the agent updates, preserves, avoids, and re-verifies the right elements in response [2].

**Does asynchronous execution reduce agent success?**

In Async-RBench's protocol, all nine evaluated agents had lower average task success with asynchronous delivery than with batched delivery: 26.0% versus 29.7% [2].

**Do these benchmarks prove that an agent is ready for a business workflow?**

No. The supplied summaries report benchmark tasks and scenarios but do not establish coverage of specific business workflows, production environments, latency, cost, or human-review requirements [1][2].

## Sources

1. [Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution](https://arxiv.org/abs/2610.32285) — arxiv — Oct 26, 2026
2. [Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents](https://arxiv.org/abs/2610.32429) — arxiv — Oct 26, 2026

How this article was made: https://yelno.ai/blog/about