Tool-Using Agents Need Tests for Users Who Leave Things Unsaid

A Qwen3.5-2B agent trained with weak simulated users reached 76.8% success on ToolWOZ against weak users — 4.8 points above GRPO, 5.4 points above GRPO trained with strong users, and 2.8 points above the strongest baseline. ¹ The researchers created weak users by masking sentences in simulated users’ replies while keeping the task and the user’s knowledge fixed. ¹

For teams building support or internal agents, task completion alone may hide a failure: an agent can perform well when every detail is supplied and still struggle when a user leaves out information a tool call needs. The paper offers a way to probe that behavior, though sentence masking is a stress test, not proof that an evaluation reflects real users.

Why clear simulated users hide tool-using agent failures

Tool-using agents are trained and evaluated almost exclusively with strong simulated users who state details clearly, while real users are often weaker. ¹ That creates a blind spot: an evaluation may show whether an agent completes a task after receiving the required details, but not whether it notices when details are missing.

The distinction matters anywhere an agent acts through a tool. A complete request gives it the information it needs to proceed. An underspecified user request forces a decision: continue, ask, or wait.

Analysis: Separate “did it finish the task” from “did it get the information it needed.” Check whether the agent identifies what is missing, asks a targeted question, and resumes once it has enough — without filling the gap with a guess. These are proposed evaluation criteria, not findings from the paper.

How the weak-user test works

The researchers created weak simulated users by masking sentences from replies while holding the underlying task and the user’s knowledge fixed. They cross-evaluated agents trained with strong and weak simulated users on tau2-bench and ToolWOZ. ¹

Holding task and knowledge constant isolates one variable: how much the user’s reply says. Analysis: If an agent’s score shifts between complete and masked versions of a case, that suggests its behavior responds to user informativeness rather than a different task definition.

This is a controlled probe, not a model of human conversation. The source excerpt does not specify how closely sentence deletion matches real users’ omissions, how masking rates or target sentences were chosen, or whether results transfer to a particular business workflow. Those are open questions for anyone adapting the method, not questions the paper claims to have settled. ¹

What breaks when the user says less

Weak users lowered the success of every policy tested. ¹ Training on weak users was not enough by itself: policies trained with fixed masks, mixtures, or a curriculum did not serve weak users better than the strong-trained policy beyond run-to-run noise. Prompting agents to ask more questions did not help either. ¹

The counterintuitive result is that a policy trained only on strong users served weak users at least as well as GRPO trained with weak users. ¹ Exposure to incomplete requests, on its own, was not enough.

Analysis: Asking more questions is not the same as asking the right one. A question that misses the actual blocker adds friction without resolving it. Evaluation should check whether a question targets the real gap, not just count how often the agent asks.

This also argues against trusting one aggregate score. A system can behave differently depending on whether the simulated user is strong or weak. ¹ Reporting results separately for both conditions makes that sensitivity visible instead of averaging it away.

What StaGPO adds beyond masking

The paper proposes StaGPO, in which the agent records what the task requires, what is missing, and how it is filling the gap. GRPO then compares turns across different rollouts that reach the same situation, assigning credit at the step level rather than only at task completion. ¹

Analysis: This shifts attention from the final tool call to the decisions leading up to it. In a workflow where acting on incomplete information is risky, you can inspect whether the agent flags the blocker and chooses a defensible next move; a correct final answer alone will not show that.

The paper reports results within a standard recipe of a small supervised fine-tuning cold start followed by reinforcement learning: with StaGPO, weak-user training became useful in its experiments. That result is tied to the reported setup, not a guarantee that the same approach transfers to another workflow. ¹

How strong is the evidence

On ToolWOZ, the Qwen3.5-2B policy trained with weak users scored 76.8% against weak users — 4.8 points above GRPO, 2.8 points above the strongest baseline, and 5.4 points above GRPO trained with strong users. It also generalized to a mask rate it had not seen during training. ¹

On tau2-bench, StaGPO scored highest against weak users for both models tested, though the margin was small for Qwen3.5-4B. ¹ These results are encouraging for the reported setup, but they do not establish that weak-user training is generally superior: on ToolWOZ, the strong-user-trained policy served weak users at least as well as GRPO trained with weak users. ¹

The source describes the work as a 22-page preprint under review. ¹ The results are scoped to the reported models, benchmarks, and masking procedure; they do not establish how well these benchmark results predict performance in a particular support or internal-tooling workflow.

Analysis: Treat the benchmark result as a reason to run your own test, not as a stand-in for production readiness. The practical question is whether your test surfaces omissions that occur in your workflows and whether the agent handles them safely.

How to build your own AI agent evaluation for incomplete requests

Start with a task inventory. For each workflow, write down what information a tool call requires, what users tend to omit, what question would close the gap, and what action should wait until it is closed. This adapts the paper’s focus on task requirements and missing information to your own process. ¹

Then build paired cases: the same task and simulated-user knowledge, but a complete reply versus versions with specific sentences removed. That pairing follows the paper’s design. ¹ Label which omissions should trigger a clarifying question and which can be safely inferred. That labeling step is a recommended addition, not something the paper reports doing.

Measure more than completion. For each case, record whether the agent:

• Identified the missing information
• Asked a question that targets the actual gap
• Avoided taking an unsupported tool action
• Resumed correctly once the user answered
• Completed the task

These are proposed measurements, not results from the paper. They can help distinguish “asks good questions” from “asks more questions.”

Test interventions as separate conditions instead of assuming one approach will win. Strong-user training, weak-user training, mixtures, curricula, and question-prompting behaved differently in the paper: weak-user exposure alone did not beat strong-user training beyond noise, and prompting for more questions did not help. ¹ Vary your masking procedure and hold some conditions out for testing. The paper reports generalization to an unseen mask rate, but the source excerpt does not provide enough detail to prescribe specific rates or sentence-selection rules for your data. ¹

If you want to run the same paired cases across several models, a single API key with an OpenAI- and Anthropic-compatible interface across many providers keeps the test harness simple; the model catalog lists what is available.

What to require before deploying tool-using agents

Ask for results under both complete and incomplete user conditions, with the task set, models, training method, masking procedure, and run-to-run variance documented. The paper’s cross-evaluations show why a score under one informativeness level does not tell the whole story. ¹

Review failures in both directions: agents that act when they should have asked, and agents that ask when they already had enough. The finding that question-prompting did not help is a reason not to treat question count as a quality signal on its own. ¹

Treat sentence masking as one layer, not the whole evaluation. Pair it with review of real or realistically reconstructed interaction traces before drawing conclusions about how an agent will behave in your workflow. The paper does not establish that masked replies reproduce real-user omission patterns. ¹ For a related look at how the setup around an agent shapes what it can be trusted to do, see A Real PMID Is Not Enough: What Biomedical Claim-Checking Agents Need From Their Harness.

Frequently asked questions

What are weak simulated users?

Simulated users whose replies have sentences masked out while the underlying task and user knowledge stay fixed, making them less informative. ¹

Does training on incomplete simulated users always help?

No. Fixed masks, mixtures, and curricula did not make weak-user-trained policies serve weak users better than the strong-trained policy beyond run-to-run noise. Prompting agents to ask more questions did not help either. ¹

What is StaGPO?

A method in which the agent records task requirements, missing information, and how it is filling the gap; GRPO then compares turns across rollouts that reach the same situation to assign step-level credit. ¹

Does sentence masking represent real users?

That is unproven. It creates a controlled weak-user condition, but the paper does not establish how closely it tracks the way real users leave things out. ¹

If you are building an agent evaluation, a practical next step is to create paired cases from tasks in your workflow: remove a load-bearing sentence from a user turn and check where the agent guesses instead of asking. Treat that as a probe, not a measure of production readiness.