When Should an Agent Ask for More Information Before Deciding?

Across four evaluated frontier LLMs, initial plans were infeasible on 58–98% of tasks when information was missing. When all facts were provided upfront, that rate was 5–28%. ¹ For companies deploying agents to plan or allocate resources, this points to a capability worth evaluating: knowing when to stop and ask a person for information before acting.

Ask2Decide is a benchmark designed to test whether an agent can identify missing facts that matter to a decision, ask for them, and distinguish them from details that would not change the plan. ¹ The practical lesson for deployment teams is to evaluate the question and the resulting decision together. A fluent exchange alone does not show whether the plan it produces is feasible.

What the Ask2Decide benchmark actually measures

The benchmark contains 43 planning tasks. An agent receives an incomplete request or incomplete supporting records, while an operator holds facts absent from the agent's information. ¹ The benchmark tests whether the agent asks for missing information that matters to its decision.

To determine what matters, the benchmark uses optimization solvers to identify which hidden facts change the optimal plan. ¹ This gives relevance a decision-based meaning: a fact is relevant in this setup if knowing it changes the optimal plan.

The benchmark also includes similarly worded hidden facts that leave the optimal plan unchanged. ¹ An agent that asks about every plausible detail may appear thorough without improving the decision. Ask2Decide tests whether a model can distinguish information that changes the plan from information that merely resembles it.

The paper describes a gap in prior evaluations: existing benchmarks typically provide fully specified problems or use uncalibrated LLM-simulated users, leaving agents' ability to ask for missing information insufficiently tested. ¹

Analysis: This is a harder test than counting questions asked. More questions are not automatically better. A useful evaluation should connect each question to a possible change in the action, constraint, assignment, or schedule, then check whether the agent asks when that connection exists—and avoids asking when it does not.

Why LLM agents asking the right question is harder than producing a plan

Across four evaluated frontier LLMs, initial plans were infeasible on 58–98% of tasks with missing information, compared with 5–28% when all facts were provided upfront. ¹ In this benchmark, missing information is associated with a much higher rate of infeasible initial plans.

A second result exposes a subtler weakness. Agents uncovered facts that changed the optimal plan at rates of 39–52%, and facts that left it unchanged at rates of 32–41%. ¹ The reported ranges suggest agents found decision-changing facts only slightly more often than facts that did not change the plan.

That distinction matters when evaluating an agent. “What information is available?” is not the same as “which missing fact could change the decision I'm about to make?” The latter is the capability Ask2Decide is designed to test. ¹

Analysis: Scoring only how often an agent asks, or how many facts it retrieves, rewards activity rather than judgment. Score whether it found the consequential fact and used it—and whether it skipped questions with no decision value.

What the human sessions reveal about human-agent interaction

The researchers collected 79 interaction sessions from 19 participants. ¹ After an incorrect plan, human replies omitted the fact needed to correct it 43% of the time. Replies from an uncalibrated simulated user omitted that fact 7% of the time. ¹

That gap matters for test design. A simulated operator that reliably supplies the exact correction an agent needs could make the agent look better at gathering information than it would in interactions with people. The researchers used the collected sessions to calibrate a simulated operator to more closely match human reply patterns. ¹

Treat this as a warning, not a finished model of workplace interaction: the paper describes the work as ongoing. ¹ The source summary does not establish how well the calibrated operator generalizes beyond the sessions used in the work.

Analysis: Test an agent's information-gathering against replies that are partial, ambiguous, or non-corrective—not just ideal answers. Otherwise, the evaluation may measure how well the agent handles a cooperative test setup rather than how it responds when a person does not provide the key fact.

How to evaluate AI agents with incomplete information in your own workflow

Start with decisions where a missing or outdated fact could change the plan. Write down which facts can change the preferred plan, then map what the agent can and cannot see at decision time. This applies the benchmark's structure to your own process. ¹

Build paired cases: one where you withhold a fact that changes the best plan, and one where you withhold a similarly worded fact that does not. This tests the distinction Ask2Decide draws between consequential information and plausible noise. ¹

Then score the whole decision cycle, not just the dialogue:

• Does the agent ask before acting when a missing fact could change the plan?
• Does the question target that fact rather than request broad background?
• Does the plan change when new information should change it?
• Does the agent skip unnecessary questions when a missing detail does not affect the decision?
• Can it recover when a reply does not provide what it needs?

These are deployment measures built on the benchmark's logic, not results reported by it. The first two reflect its focus on information gathering; the others extend that focus to whether the interaction improves the final decision.

Score the plan, not just the conversation. A longer, more polite exchange is not evidence of success if the resulting plan is infeasible or the agent ignores a decision-changing answer. Feasibility and changes to the optimal plan are useful evaluation anchors because the benchmark studies both plan feasibility and whether hidden facts change the optimal plan. ¹

Include realistic, incomplete human replies in your test set. In the benchmark, people omitted the corrective fact 43% of the time after an incorrect plan. ¹ That is a reason to test for incomplete answers, not a rate to expect in your own operation.

What an agent is given to work from also deserves scrutiny; our piece on how stale repository notes can distract coding agents looks at a related problem.

What the benchmark doesn't tell you

The source summary reports ranges across four frontier LLMs but does not provide a model-by-model breakdown. ¹ It also does not describe every planning task or operational setting represented by the 43 tasks. ¹ Check whether the benchmark's task structure resembles your own decisions before drawing conclusions about likely performance.

The simulated operator was calibrated against the collected sessions, but the source summary does not establish how well it matches people outside those sessions. ¹ Use it to widen test coverage, while comparing its behavior against real users before relying on it as a stand-in.

Analysis: Treat Ask2Decide as a test-design pattern, not a production guarantee. The paper concludes that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning. ¹ That identifies a failure mode; it does not establish how well any particular agent will ask questions in your workflow.

What to do next

Pick one workflow where an incorrect plan has a clear operational cost. List the hidden facts that can change the preferred plan, and make the decision boundary explicit before evaluating the agent.

Build a small evaluation set with three kinds of cases: missing facts that change the plan, missing facts that do not, and replies that fail to reveal the fact the agent needs. These categories reflect the benchmark's tests of relevance and human interaction. ¹ Compare performance with information missing against performance with all facts supplied upfront, as the benchmark does. ¹

As a diagnostic, ask the agent to state what decision it is making and why a particular question could change it. This is a review aid, not a benchmark result; it can help reviewers distinguish purposeful information gathering from generic clarification.

Track four things: plan feasibility, whether new information changes the decision when it should, the rate of unnecessary questions, and recovery after incomplete replies. When a case fails, classify why: the agent did not ask, asked about the wrong fact, misread the answer, or failed to revise its plan. That breakdown can make the evaluation more actionable.

For related reading on keeping agents reliable once they are running, see Can Pitfall-Anticipation Models Help Steer Black-Box Agents? and A Physics Experiment Co-Pilot Is a Test of Operational AI, Not an Autonomy Demonstration.

The question is not whether an agent can keep a conversation going. It is whether it can identify a fact that could change its decision—and produce a sound plan when the answer it receives is incomplete. Ask2Decide offers a way to test that. Running a similar test with your own facts, people, and consequences is up to your team.