A Correct SQL Query Can Still Answer the Wrong Question

A text-to-SQL paper introduces AIR, an ambiguity-resolution framework, and reports that it outperforms IntentRL on the AMBROSIA benchmark. Its underlying point is that one natural-language question can have multiple valid interpretations that produce different outputs. ¹ For teams deploying conversational analytics, that reframes the problem: it is not only whether generated SQL runs, but whether the system notices when the request can be interpreted in different ways.

The real failure mode: text-to-SQL ambiguity and a confident wrong answer

A SQL query can execute correctly and still reflect the wrong interpretation of a request. The paper's premise is that natural-language queries can admit multiple valid interpretations that yield distinct outputs. ¹

That makes ambiguity an intent-resolution problem as well as a query-generation problem. A system that silently chooses one interpretation could return an answer to a question the user did not intend. This is a practical risk suggested by the paper's framing; the abstract does not report a production incident. ¹

The buyer question follows: can the system identify when interpretations lead to different results and ask for clarification? AIR is the paper's approach. Its benchmark results are evidence about performance on the tested benchmarks, not proof of readiness for consequential decisions. ¹

How AIR resolves ambiguous intent in natural language to SQL

"Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL" was published on arXiv on October 20, 2026. ¹ AIR is a multi-agent framework designed to probe and resolve ambiguity without task-specific training. It constructs competing interpretations, checks through SQL execution whether they produce different results, then generates targeted clarification questions grounded in those differences. ¹

The SQL-execution step is central to the approach: AIR checks whether competing interpretations produce genuinely different results before grounding its clarification questions in those differences. ¹ This makes clarification evidence-driven. The abstract does not report the cost of running multiple candidate queries per request or describe safeguards against unsafe queries. ¹

What the AMBROSIA numbers show, and what they don't

On AMBROSIA, AIR outperformed IntentRL across all reported metrics under schema-only context. The abstract gives this F1 comparison: ¹

Framework	AMBROSIA F1 (schema-only context)
AIR	0.679
IntentRL	0.618

The paper describes schema-only context as a realistic production setting where exposing the full database is infeasible. It reports that AIR's performance degradation without the full database was about 7%, compared with about 20% for IntentRL. ¹ This suggests AIR degrades less under that context restriction on the reported benchmark; it does not establish how it will perform on another organization's schemas, queries, or users.

The abstract does not provide an example count, domain breakdown, confidence intervals, or cost and latency figures. ¹ Those omissions limit what can be inferred about the stability of the reported result or its relevance to a particular deployment.

Second benchmark, same gap in the evidence

The paper also evaluates AIR on AmbiQT, where ambiguity arises from schema design rather than linguistic structure. It reports that AIR achieves higher F1 than the comparison system under both context settings, without benchmark-specific adaptation. ¹

The abstract does not give the AmbiQT scores. ¹ So it establishes the direction of the comparison, but not the size of the difference. Benchmark results alone also do not establish how the approach will perform on other kinds of schema or language ambiguity.

The gaps that matter for LLM SQL evaluation in deployment

The abstract does not state how many examples or schemas were tested, how ambiguity types were distributed, or how the quality of AIR's clarification questions was evaluated. ¹ Detecting that two queries return different results is distinct from determining which result the user intended. A benchmark score alone does not establish that the final interpretation matches a user's intent.

The abstract also does not report whether real users accept clarification questions, whether AIR's final interpretations match user intent outside the benchmarks, or the latency and compute costs of the multi-query approach. ¹

Analysis: F1 results on AMBROSIA and AmbiQT do not, by themselves, measure the consequences of a wrong interpretation in a financial report or operational decision. ¹

SQL agent clarification: what to require before trusting generated SQL

Treat ambiguity detection as a control, not just a feature. A deployed system should surface competing interpretations or ask for clarification rather than silently choosing one. That is a recommendation based on AIR's design, not a claim that the paper provides a complete production safeguard. ¹

Before executing candidate SQL, consider requiring read-only access, row and query limits, an approved-schema allow-list, and authorization checks independent of the model. Log the request, candidate interpretations, SQL executed, result differences, clarification shown, and the user's selection. These are deployment recommendations; the abstract does not claim AIR includes them. ¹

If your model calls run through an enterprise AI gateway, its audit logs and spend controls can support part of that record, but the database-side limits above still have to be enforced outside the model. A related argument about wrapping agents in verification checks appears in A Real PMID Is Not Enough: What Biomedical Claim-Checking Agents Need From Their Harness.

For consequential uses, keep a human involved until you have measured intended-interpretation accuracy, false reassurance, latency, and query-execution safety in your setting. ¹

An evaluation plan for AI analytics queries, not a leaderboard number

Before setting an acceptance bar, build a test set from your own schemas and queries. Label the intended interpretation, acceptable alternatives, and cases that should trigger clarification. Include both phrasing ambiguity and schema-design ambiguity, the types described in the paper. ¹

Measure more than whether the SQL runs. Track whether the system detects ambiguity, whether its clarification questions are useful, whether its final answer matches user intent, and whether it avoids executing unauthorized queries. The abstract's reported F1 results do not establish these outcomes. ¹

Estimate the cost of the full interaction, including candidate generation, SQL executions, clarification exchanges, end-to-end latency, and database load. The abstract describes AIR as efficient but provides no numerical cost or latency results. ¹ If you want to compare several underlying models on the same test set, a model catalog makes it easier to vary the model while holding the evaluation fixed.

Use AIR's 0.679 versus 0.618 AMBROSIA F1 result under schema-only context as a reference point, not a target. It is a result on one benchmark and context setting, not a guarantee for your schemas or users. ¹

The practical test is whether a text-to-SQL system notices when plausible interpretations produce different results, surfaces that difference, and gives the user a way to resolve it. AIR's reported benchmark results suggest one approach to that problem. Whether it works on your data requires evaluation in your own setting. ¹

Source

¹ "Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL," arXiv, published October 20, 2026. https://arxiv.org/abs/2610.24248