FINCOSMOS Failed Every Session—That's the Finding

FINCOSMOS accepted zero of 150 evaluation sessions. Every session that a simulated user approved still failed at least one program check ¹. A convincing conversation, in other words, does not establish that the work followed the hidden strategy or handled time-sensitive data correctly.

The paper introduces an open-world way to evaluate finance AI agents: tasks depend on web data, a strategy held by the user, and correctness that changes with time ¹. The reported results come from FINCOSMOS's evaluation setup; they are not a verdict on any specific agent's readiness for production. Treat FINCOSMOS as a pressure test. Testing against your own sources, cutoffs, and review process remains a separate step.

Why fixed finance agent benchmarks miss the failures buyers care about

FINCOSMOS argues that finance-agent tasks combine web-retrieved information, a user-specific strategy, and time-dependent correctness ¹. The paper says that benchmarks which fix the task, data, or timeline can remove the failures those conditions are meant to measure ¹. Those failures include an agent choosing a strategy the user did not request, using unverified data, or backtesting with future data ¹.

That is a different test from completing a known exercise. In an open-world task, the agent must determine what the user means, find evidence, respect a data cutoff, and return verifiable results ¹. One practical distinction for evaluation teams is task completion versus verifiable task completion. FINCOSMOS proposes a method for evaluating the latter and reports results from its own setup ¹.

Time-sensitivity is a recurring evaluation problem for language models generally; StateBench Tests Whether Models Can Tell a Current Fact From an Old One covers a related angle.

What the financial web research agent is asked to do

The agent starts with no data, acquires information from the web, builds its own backtest under a data cutoff, and delivers results that can be verified ¹. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked ¹.

That design makes requirement discovery part of the evaluation: the agent has to ask about details it has not been given ¹. One implication is that an agent can return a polished answer without having established the user's full intent.

This is not just a search test. The agent must connect web retrieval, point-in-time data handling, strategy construction, backtesting, and reporting ¹. Grading does not require a reference answer: programs check point-in-time correctness and recompute reported metrics, while the delivered strategy is checked against the hidden specification ¹.

How grading separates “sounds right” from “is right”

A run is accepted only if the simulated user is satisfied and none of 15 programmatic veto checks fails ¹. This combines a user-facing assessment with independent program checks.

Evaluation layer	What it checks	Failure it can expose
Simulated-user satisfaction	Whether the user is satisfied 1	The interaction did not meet the user's request
Program checks	Point-in-time correctness and recalculated metrics 1	Results are incorrect or cannot be reproduced
Hidden-strategy comparison	Whether the delivered strategy matches the hidden specification 1	The agent missed or misread requirements

Together, these layers test whether an apparently successful exchange also meets the task's stated and hidden requirements ¹. The abstract reports results from this evaluation setup; it does not establish how an agent would perform in a buyer's own data environment, approval process, or production setting ¹.

What the numbers show

Across 150 sessions with four coding agents and a model without tools, FINCOSMOS accepted no sessions ¹. Every session with simulated-user sign-off still failed a program check ¹.

Agents raised only 54–64% of the details they had not been told ¹. Human experts attributed 88.33% of failures to the agents ¹. That attribution describes this evaluation's failures; it is not a general error rate for finance agents.

These results show failures in the sessions FINCOSMOS ran. They do not establish that every finance agent fails at this rate or that another setup would produce the same numbers. The practical lesson is narrower: conversational approval is not a substitute for independent checks when outputs depend on hidden requirements and time-sensitive data.

What the abstract leaves open

The abstract reports the types of agents evaluated and describes the hidden-strategy setup, data cutoff, and 15 veto checks ¹. It does not specify the particular web sources, exact task and cutoff details, definitions of each veto check, or the procedure experts used to attribute failures ¹. Those details would help buyers judge how closely the evaluation resembles their use case.

Reproducibility and relevance are separate questions. The abstract reports results from one evaluation setup; it does not establish how far those results generalize across other agents, users, or changing web conditions ¹. FINCOSMOS makes a case for testing requirement discovery, point-in-time correctness, and independently checked outputs, but does not determine whether a particular agent belongs in a particular business process.

Turning FINCOSMOS into a finance agent risk assessment

Borrow the paper's pressure points and apply them to your own tasks: web information the agent must retrieve, a policy or strategy it must elicit, and a cutoff after which data is off-limits ¹. These reflect the evaluation's focus on web data, hidden user strategy, and time-dependent correctness ¹.

Make the process inspectable. Require the agent to log what it asked, what it learned, which sources it used, which cutoff it applied, and how it calculated each reported metric. Independent recalculation and verifiability are central to FINCOSMOS's evaluation design ¹.

Do not use conversational approval as the sole gate. Recalculate outputs independently, compare the delivered strategy against the intended specification, and reject runs that fail critical checks. Track unanswered questions as a metric: FINCOSMOS reports that agents raised only 54–64% of details they had not been told ¹. For a related look at gating agent output before it is accepted, see Can Validation Gates Make Agentic AutoML Dependable?.

Recommended controls:

• Log each question, source, cutoff, and calculation the agent produces.
• Decide in advance which errors trigger rejection, human review, or escalation.
• Rerun scenarios under changed web conditions and different hidden strategies. This extends the open-world approach; the abstract does not report this as a test result.
• Keep failed traces and rerun them after changes to the model, prompt, tools, sources, or memory.

These are proposed evaluation practices, not guarantees that failures will be prevented. They can make failures more observable before a system is used in a live workflow.

What the memory result does and does not show

With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks ¹. This is evidence of learning in that test; it does not establish that memory prevents silent failures generally or that the result holds in other workflows ¹.

A system that learns known tasks still needs regression testing. Recheck stale data, cutoffs, missing requirements, and metric calculations after changes, and retain old failures to assess whether fixes address them.

The decision rule: pressure test, not safety certificate

For evaluation engineers, FINCOSMOS offers a set of test ideas: open-ended retrieval, hidden user strategy, time-bounded correctness, independently checked outputs, and an acceptance rule combining user satisfaction with programmatic vetoes ¹. For risk teams, the result is direct: a satisfied simulated user can coexist with failed checks ¹.

FINCOSMOS reports agent behavior under its own test conditions; it does not provide a verdict on a specific product or replace testing against your workflow ¹. Before deploying a finance agent, define which errors trigger rejection, review, or escalation, then test those rules against tasks your team actually runs. The question of what counts as proof an agent works, when you cannot inspect everything, is explored in Legora Says It Can't Read Client Traces—What Counts as Proof a Legal AI Agent Works?.

Frequently asked questions

What is FINCOSMOS?

An open-world evaluation paradigm for finance agents, built around web data, a hidden user strategy, and time-sensitive correctness ¹.

How does FINCOSMOS evaluate finance agents?

The agent acquires web data, builds a backtest under a data cutoff, and returns results checked against a hidden strategy and programmatic correctness tests ¹.

Did any session pass FINCOSMOS?

No. FINCOSMOS accepted zero sessions across 150 sessions involving four coding agents and a model without tools ¹.

Why isn't simulated-user approval enough?

Every session with simulated-user sign-off still failed a program check ¹. This supports pairing conversational approval with independent validation.

Does FINCOSMOS prove an agent is safe to deploy?

No. The reported results are from FINCOSMOS's evaluation setup. Buyers still need to test their own sources, cutoffs, review process, and failure thresholds ¹.

Source

¹ “FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance,” arXiv, November 6, 2026. https://arxiv.org/abs/2611.07464