When Agents Run Unattended: Testing for Retries That Cost Money and Outputs That Vary

A scheduler running an agentic workflow hundreds of times overnight may have no one watching when something goes wrong. ¹ Traditional enterprise systems were built around assumptions that jobs finish quickly, retries are nearly free, and the same input produces the same output. Agentic workloads break those assumptions, and pilots can conceal the operational problems that emerge when workflows run unattended. ¹

An agent task can take minutes, long enough to exceed timeout thresholds a system has not encountered before. ¹ In a pilot, a person may review the output, retry a failed step, or correct an error by hand. In production, that manual recovery may be absent. The practical question is whether the workflow can detect a bad result, contain it, and leave enough information to explain what happened.

What breaks when production AI agents run unattended

Traditional systems often treat retries as nearly free and repeated runs as predictable. For agents, attempts against a metered model consume compute whether or not the result is usable, and failures may not be reproducible by simply rerunning the task. ¹

That changes what counts as evidence that a workflow is ready. A pilot where a human intervenes shows that the task can work; it does not establish that it works unattended. People may notice problems and try again, while automated workflows can run headlessly and allow unrecorded failures to affect downstream systems. ¹

Treat the move to production as a change in operating conditions, not just a jump in volume. Test what happens when a run takes longer than expected, an attempt fails, or output quality is inadequate, and no one is there to compensate.

Testing agentic workflows: build tests around failure, not success

Test failure paths deliberately: long-running execution, timeout boundaries, and repeated attempts are useful starting points. Agent tasks can run long enough to exceed existing timeout thresholds, but the source does not specify a universal timeout value. ¹ Set a workflow-specific boundary, then verify what happens when the workflow reaches it.

AI agent retries need their own test cases

A failed attempt against a metered model still consumes compute, and automatic retries can accumulate costs that become visible later. ¹ Choose an explicit attempt cap and verify that it holds when failures repeat. The source identifies capping attempts as a question teams should address, but does not prescribe a universal limit. ¹

For every retry, consider recording why it happened:

timeout
transport failure
tool error
output-quality failure

These are possible categories for your own workflow, not a taxonomy prescribed by the source. Distinguishing triggers can help teams choose different responses, for example adjusting a timeout or stopping and escalating an output-quality problem.

Compare what a human does in the pilot (reviewing output, retrying, and correcting errors) with the automated check or system that will take over in production. ¹ Where no dependable check exists, consider a deliberate stop or human-review step. This makes visible the manual recovery work that a pilot can conceal. ¹

Measure cost per completed unit, not cost per call

Cost per API call can hide discarded attempts. The source recommends asking for cost per completed unit instead, because the per-call figure excludes attempts that did not produce usable results. ¹

Define "completed unit" in business terms before measuring. A system that looks inexpensive per call may still cost more per accepted output if many attempts are discarded. This is why cost should be considered alongside output quality.

The source recommends recording retry counts, token cost, execution latency, and the final output artifact as part of the job data. ¹ Tracking these fields together can help teams assess the cost and outcome of a workflow rather than considering call cost alone.

Report successes and failures separately, and test what happens at the attempt cap. An unbounded retry loop against a vague quality standard can drive overspending; the source asks teams to establish whether attempts are capped but gives no universal limit. ¹ Set workflow-specific cost, latency, retry, and escalation limits before rollout, and include them in the test plan. Where model calls pass through an enterprise AI gateway, budgets and spend controls can add a limit outside the workflow, though they are no substitute for a retry cap inside it.

Non-deterministic AI testing: when the same input gives different outputs

Write acceptance criteria before running an evaluation. The source identifies programmatic criteria, including assertion checks, LLM-as-a-judge rules, and semantic benchmarks, as ways to assess output quality. ¹ Format is not the same as substance: an output can be structurally valid while containing bad data that downstream systems accept. ¹

As a practical evaluation, run representative inputs more than once and compare decisions, tool calls, formatting, and final artifacts. This can characterize variation, but it cannot establish that every future run will be identical. Combining methods may help: deterministic assertions can check known constraints, semantic checks can assess meaning, and an LLM judge can assess less structured outputs. The source lists these approaches without ranking them. ¹

Decide in advance what each check does: which failures block downstream processing, which trigger a capped retry, and which require a human. That turns acceptance criteria into a control path rather than a score reviewed only after a run has affected another system.

What an AI agent run record should contain

A run-level record worth considering includes:

input payload
prompt and model version
timestamp and execution latency
retry count and token cost
decisions, tool calls, and API calls
final output artifact

The source specifically recommends recording the input payload, prompt and model version, timestamp, execution latency, retry counts, token cost, and final output artifact. ¹ It also explains that without a step-by-step trace of agent decisions, tool calls, and API runs, investigating a failure becomes speculative. ¹ Keeping a sufficiently detailed record can help teams trace an artifact back to the inputs and decisions behind it.

What should qualify a pilot for production AI agents

Good results from a person-led pilot do not, by themselves, show that a workflow is ready to run unattended. Before rollout, seek evidence on these fronts:

• Unattended execution: the workflow completes or stops safely without a person retrying or correcting it. Interactive pilots can hide that intervention. ¹
• Predeclared output quality: criteria are written before the run and address outputs that may be structurally valid but substantively wrong. ¹
• Cost per completed unit: account for discarded attempts, not just successful calls. ¹
• Capped retries and a recovery path: establish an attempt limit and keep run records that support investigating decisions, tool calls, and API activity. ¹
• Version-change testing: test model updates in staging before production, because updates can change formatting, accuracy, or reasoning. ¹

The source does not set a single numeric bar for production readiness. ¹ Set workflow-specific pass criteria instead: agree on acceptable quality, cost, latency, retry behavior, and escalation thresholds before evaluation, then make the rollout decision against those criteria. For a broader view of what evidence is enough before an agent can take action, see our governance piece, and for buyers comparing platforms, production controls as the buying test covers the orchestration side.

Before you flip the switch on automation

Start with an inventory of tasks a person currently performs in the pilot: reviewing, retrying, correcting, and judging acceptability. ¹ For each one, identify the automated check or system that will take over. If nothing reliable exists, consider keeping a deliberate stop or review step rather than assuming the gap will resolve itself. This addresses the manual recovery work that interactive pilots can conceal. ¹

Then test the workflow under its intended operating conditions: unattended, with variable outputs, bounded retries, and acceptance rules written down in advance. Keep run history that lets the team examine quality, latency, retries, and cost, and test model version changes in staging before production. ¹ Include the final artifact and enough information about the inputs and decisions behind it to investigate failures. ¹

Before approving rollout, ask whether the evidence shows that a failure will stop safely or recover within agreed limits, and whether the team can explain afterward what happened. If not, continue testing and address the gaps before shipping.