How to Test Whether Agents Specify Engineering Simulations Correctly

A solver can run to completion and still solve the wrong problem. A recent paper evaluates agent-generated simulation specifications separately from solver execution, reporting 98.3% correct first-turn specification and decision outcomes across 120 template-related requests for its full workflow ¹.

That distinction matters when agents enter design workflows. A completed run is not, by itself, evidence that the model represents the intended problem. Teams need to assess whether a specification captures the intended geometry, physics, conditions, and numerical settings, and whether the resulting solution is credible.

What counts as a correct agent-generated simulation specification

The agents studied turn engineering requests into complete, executable JSON specifications for automated geometry creation, meshing, PDE solving, and result export ¹. The output is an artifact intended to drive a simulation pipeline, not just a natural-language plan.

For review purposes, that artifact can be broken into geometry; the physical model and its assumptions; boundary and initial conditions; mesh requirements; solver settings; and requested outputs. This is a practical way to organize the specification's scope, not a review protocol reported in the paper ¹.

The central risk is a mismatch between what a request means and what its specification encodes. A specification might be executable while still representing the wrong problem. The paper evaluates specification correctness, execution completion, and numerical verification separately; success at one does not establish success at the others ¹.

The evidence for the workflow

Across 120 template-related requests, the full workflow achieved 98.3% correct first-turn specification and decision outcomes, compared with 92.5% for schema-assisted single-shot use and 82.5% for model-only use ¹.

The workflow combines model extraction with deterministic scientific and capability checks, explicit clarification, and validated local patches that preserve unrelated accepted fields ¹. In practical terms, it checks extracted information, requests clarification when needed, and applies scoped repairs rather than regenerating the whole specification ¹.

The workflow made zero false-ready decisions among 42 requests requiring clarification or rejection ¹. This is evidence about the evaluated cases, not proof that an unsuitable setup will never be marked ready on a different workload.

The abstract reports that component ablations identify contributions from scientific checks and conditional repair ¹. It does not provide enough detail to quantify each component's individual effect. The reported comparison supports the combined workflow, but does not establish how much each check contributes on its own.

How the paper tests execution and numerical credibility in automated PDE simulation

The evaluation keeps three questions separate:

1. Specification: Does the generated setup represent the requested problem?
2. Execution: Can the workflow run to completion?
3. Numerical verification: Do the results pass the study's numerical checks? ¹

That separation is useful because the questions address different potential failures. A setup can be semantically wrong before execution, fail during a run, or produce results that do not pass numerical verification.

The reported downstream studies span 1D, 2D, and 3D steady and unsteady problems; linear and nonlinear cases; flow and elasticity; and complex 3D wing geometries ¹. A separate 30-case gallery demonstrates automated execution from retained JSON specifications ¹.

The paper also reports 12 selected live user-to-results tests. Nine passed specification, execution, export, and numerical-reference checks together, a joint success rate of 75.0% ¹. This end-to-end result measures a different part of the workflow from the 120-request first-turn comparison ¹.

Why 75.0% isn't a production-readiness score

The 120 template-related requests measure first-turn specification and decision outcomes, while the 12 live tests measure joint success from request to results ¹. Treating them as competing scores, or combining them into one headline rate, would obscure what each evaluates. The same caution applies to any single number; behavioral diagnostics, not just aggregate scores, are a better basis for judging a model or agent.

The cited abstract does not specify the models, prompts, software versions, exact checks, test-selection method, numerical-reference tolerances, or task-by-task breakdown ¹. It reports several problem types, but does not establish that those cases represent any particular company's workload or every simulation domain ¹.

The authors identify consistency across fresh-response regressions as a priority for further improvement ¹. One implication is that repeated responses to the same requirement could produce inconsistent specifications or decisions. The abstract does not quantify how often this occurs, its severity, or its causes ¹.

Analysis: These results support evaluating an agent as one component inside a controlled engineering workflow, not treating its output as a substitute for engineering judgment. The deployment question is whether the system handles a team's requirements, failure modes, and acceptance criteria consistently, not whether a published aggregate rate clears an unspecified bar.

Review gates for agent-generated boundary conditions and setups

The abstract reports separate measures for specification correctness, execution, and verification, but does not lay out a full engineer-review protocol or universal tolerances ¹. The gates below are proposed controls for applying that separation.

Intent gate. Map each user requirement to an explicit field in the specification. Flag anything missing, ambiguous, or contradictory before execution. This tests whether the agent represented the requested problem, not merely whether it produced something runnable.

Physics gate. Check geometry, governing equations, material or constitutive assumptions, dimensionality, units, and boundary and initial conditions against the intended physical problem. A successful solver run does not waive this step: the paper evaluates specification correctness and execution separately ¹.

Numerical gate. Inspect mesh adequacy, discretization and solver settings, convergence behavior, and stability choices, then compare results against an established reference or invariant. The abstract reports numerical verification but does not specify a universal procedure or tolerance ¹.

Artifact gate. Retain the JSON specification, validation findings, patches, clarification history, solver outputs, and exported results. The paper's 30-case gallery demonstrates automated execution from retained JSON specifications ¹. Keeping the decision trail is a proposed way to support later review and reproduction.

Repair gate. Define when the system marks a request ready, asks for clarification, or rejects it. As a proposed control, accept a local patch only when its scope is explicit and unrelated accepted fields remain untouched; preserving those fields is a stated purpose of the workflow's validated local patches ¹.

What to measure in your own deployment

Track specification correctness, execution completion, and numerical verification as three separate measures, following the paper's evaluation structure ¹. A single "successful run" metric can hide which stage failed.

Measure false-ready decisions: cases where the system marks a setup ready when it should have asked for clarification or rejected it. The paper reports zero among 42 applicable requests, a benchmark teams can test against their own evaluation set ¹.

Rerun the same requirements with fresh responses and compare the specifications, decisions, and numerical results. This targets the consistency issue the authors identify for further work ¹. Report results by problem family and failure mode rather than only as an aggregate; the cited abstract describes multiple problem types but does not provide task-by-task results ¹.

Keep a human in the release path for consequential design work. The authors describe their results as supporting traceable, engineer-reviewed assistance within the tested scope ¹. That supports human-reviewed use; it is not evidence for autonomous approval.

Before connecting an agent to your simulation pipeline

Analysis: Build a locked evaluation set that includes clear, incomplete, ambiguous, physically inconsistent, and unsupported requests. This is a proposed test design, not a feature reported by the paper. For each request, specify the expected outcome (ready, clarify, or reject) and the evidence that would justify it.

Then walk the full path:

• Does the specification preserve the request's intent?
• Do the scientific and capability checks catch the problems you expect them to catch?
• Can retained artifacts drive execution on their own?
• Do outputs pass your numerical checks?
• Do fresh responses to the same request produce acceptably consistent specifications and decisions?

The paper supports separating these questions while leaving implementation details and acceptance criteria to deploying teams ¹. Before agent-generated setups enter a real design workflow, require a human sign-off record covering intent, physics, numerical settings, and reference comparisons. Decide in advance which cases should be clarified or rejected.

The same verify-first habit applies to other agent claims; see what to verify before you buy a world model for agents. Controlling which internal tools an agent can reach is a separate question, covered in Uber's MCP Gateway: separating registration from runtime.

Source

¹ "From 1D Benchmarks to Complex 3D Geometries: Can Agents Reliably Specify Problems for Automated PDE Simulation?" arXiv. https://arxiv.org/abs/2610.41267