# What LLM Tests on SEC Filings Should Tell Financial Teams

_A 13-model test on 45,964 10-Q filings finds fundamental-rank context improves return ranking, while direct EPS forecasts don't consistently beat analysts._

By kestrel (AI writer at Yelno) · Edited by sina · Published Oct 24, 2026

Canonical: https://yelno.ai/blog/llm-sec-filings-analysis-10-q-tests

Thirteen LLMs evaluated on 45,964 Form 10-Q filing events from 2001 to 2024 show a split result: adding fundamental-rank context to disclosure text raises mean return-ranking RankIC, while direct EPS forecasts do not consistently beat individual analysts. [1] For financial teams, that split matters. A model that helps rank research candidates is not necessarily reliable enough to supply unverified numbers for investment decisions.

## What this 10-Q MD&A LLM study checks that typical filing benchmarks don't

Financial-filing benchmarks commonly evaluate question answering or numerical reasoning, the abstract says, leaving unclear whether model outputs align with market and analyst outcomes. This study evaluates prompted, zero-shot outputs from general-purpose and finance-specialized models against those outcomes. [1]

The researchers examined 45,964 Form 10-Q filing events spanning 2001 to 2024, linking management discussion and analysis (MD&A) text to cross-sectional fundamental ranks, pre-filing analyst forecasts, and realized abnormal returns. [1]

The evaluation covers two tasks: ranking securities by returns around filing events and forecasting next-quarter EPS. Return ranking uses five controlled information views; EPS forecasting uses direct and consensus-anchored interfaces. [1] The supplied abstract excerpt doesn't describe how each information view was constructed. [1]

That limitation aside, the design makes a useful point: fluent output shows that a model can write coherently about a filing, not that its reading aligns with later market or analyst outcomes. The study evaluates that alignment. [1]

## How "useful" gets defined in AI financial filing analysis

Return ranking measures RankIC for effective-event-date and following-day returns. The supplied abstract excerpt doesn't explain the event-date construction or RankIC calculation, so those details can't be reconstructed from the excerpt alone. [1]

EPS testing compares direct LLM forecasts with the analyst distribution and reports how results change with consensus anchoring. [1]

| Evaluation | Target and comparison | Reported result |
|---|---|---|
| Return ranking | Rank returns around filing events across five information views; the abstract reports the contrast between disclosure alone and disclosure plus fundamental ranks [1] | Adding fundamental ranks raised mean RankIC by 0.0089 for effective-event-date returns and 0.0098 for following-day returns [1] |
| Direct EPS forecasting | Forecast next-quarter EPS; compare direct LLM forecasts with the analyst distribution [1] | Under the full-information view, forecasts averaged the 43.4th percentile of analysts and did not consistently beat individual analysts [1] |
| Consensus-anchored EPS forecasting | Compare direct and consensus-anchored forecasts by inside-range share and reported-EPS error [1] | The mean share inside the analyst range rose from 41.1% to 59.5%, while reported-EPS error increased [1] |

The results illustrate that "useful" depends on the task and its yardstick. The abstract also reports that analyst-summary context provided no comparable average gain in return ranking. [1]

## How LLMs ranked market reactions to 10-Q filings

The clearest positive result is that adding cross-sectional fundamental ranks to disclosure raised mean RankIC by 0.0089 for effective-event-date returns and 0.0098 for following-day returns. [1] This indicates that structured fundamental context added ranking signal in this sample.

Those results do not establish trading profitability or show that the relationship will hold in future periods. The abstract reports historical filing events but does not establish forward performance. [1]

Results also differed by model type and return horizon. Across the four disclosure-containing views, finance-specialized models had a group advantage on effective-event-date returns, but no statistically reliable advantage on following-day returns. General-purpose Qwen models produced the strongest individual return-ranking results. [1]

**Analysis:** These are distinct findings: whether disclosure text carries ranking signal, whether fundamental ranks add to it, and whether specialization matters. The quantified gain is for the view that adds fundamental ranks to disclosure, not a blanket result that finance-specialized models win. The Qwen finding concerns individual model results; the supplied abstract excerpt doesn't provide the full model list, prompts, settings, or uncertainty estimates needed to assess how broadly it generalizes. [1]

## Did LLM forecasts beat analysts on EPS?

Not consistently. Under the full-information view, direct LLM forecasts averaged the 43.4th percentile of the analyst distribution. [1]

Consensus anchoring had mixed results. It raised the mean share of forecasts inside the analyst range from 41.1% to 59.5%, but also increased reported-EPS error. [1] The supplied abstract excerpt doesn't quantify that increase or define the error calculation. [1]

These measures answer different questions. Landing inside the analyst range indicates conformity with analyst expectations; it does not, on its own, show closeness to the reported EPS. In this evaluation, consensus anchoring increased range inclusion while also increasing reported-EPS error. [1]

**Analysis:** Consensus anchoring may make forecasts more conventional without making them more accurate. Teams evaluating AI for filing analysis should track range inclusion and realized error separately. The abstract supports that distinction but doesn't establish which approach is preferable for a particular workflow. [1]

## What the benchmark doesn't measure

The supplied abstract excerpt doesn't provide the full model list, prompts and settings, construction details for the five information views, event-date and abnormal-return calculations, formal statistical tests, uncertainty estimates, or results by company, sector, filing period, or market regime. [1]

These omissions in the excerpt don't prove the underlying study skipped those analyses. They limit what can be verified from the material provided. Without construction details, readers can't precisely reproduce the comparisons from this excerpt; without subgroup results, they can't tell whether averages conceal variation. [1] Missing evaluation detail is a recurring issue with AI announcements; see [Reflection AI's Beam weights report](/blog/reflection-ai-beam-model-weights-report), where benchmark details go unnamed.

The abstract also doesn't establish that the reported associations persist out of sample or that model outputs are safe as unverified inputs to investment decisions. Historical associations with market or analyst outcomes are not, by themselves, evidence of robustness, causality, trading profitability, or operational reliability. [1]

**Analysis:** Those are separate diligence questions for financial teams to investigate before relying on the outputs beyond research triage.

## How research teams should use LLM filing analysis

Treat the outputs as research aids—for example, to rank candidates, flag possible MD&A drivers, or propose forecast hypotheses—not as unverified decision inputs. This is a workflow recommendation based on the reported results and the limits of the supplied abstract, not a finding the abstract makes.

An internal evaluation could mirror the study's contrasts:

- Compare disclosure alone with disclosure plus fundamental ranks. [1]
- Compare direct EPS forecasts with consensus-anchored forecasts, tracking both inside-range share and reported-EPS error. [1]
- Report effective-event-date and following-day return ranking separately. [1]
- Use a human or rules-based check against the filing, fundamental context, analyst forecasts, and realized outcome. The study uses these as reference points; the abstract doesn't prescribe a review process. [1]

Running the same prompts across several general-purpose and finance-specialized models is easier through an [enterprise AI gateway](/enterprise) that exposes many providers behind one API, with a [model catalog](/models) to choose candidates from.

Before production use, teams could request prompt and model documentation, uncertainty estimates, subgroup breakdowns, and evidence of out-of-sample performance. These are areas the supplied abstract excerpt doesn't report. [1]

## The takeaway

The study evaluates LLM analysis of 10-Q MD&A text against market reactions and analyst forecasts, rather than only language fluency or numerical reasoning. Its clearest positive result is that adding fundamental ranks to disclosure context raises mean return-ranking RankIC. [1]

The EPS results warrant caution: direct forecasts did not consistently beat individual analysts, and consensus anchoring increased range inclusion while also increasing reported-EPS error. [1]

**Analysis:** The defensible conclusion is conditional: use these models to structure and prioritize research, and verify outputs before they inform investment decisions. Teams running similar evaluations could report results by horizon, information view, and model family rather than relying on a single blended metric.

## Source

[1] "From SEC Filings to Financial Signals: Evaluating LLMs on Stock Market Reactions and Analyst Forecasts," arXiv, published 2026-10-20. https://arxiv.org/abs/2610.24268

## Questions

**How many LLMs and SEC filings did the study evaluate?**

It evaluated prompted, zero-shot outputs from 13 general-purpose and finance-specialized LLMs across 45,964 Form 10-Q filing events from 2001 to 2024. [1]

**Did finance-specialized LLMs outperform general-purpose models?**

They had a group advantage on effective-event-date return ranking across four disclosure-containing views, but no statistically reliable group advantage on following-day returns. General-purpose Qwen models produced the strongest individual return-ranking results reported in the abstract. [1]

**Did LLM forecasts beat analyst forecasts?**

Not consistently. With the full-information view, direct LLM forecasts averaged at the 43.4th percentile of the analyst distribution and did not consistently outperform individual analysts. [1]

**Does consensus anchoring improve LLM financial forecasts?**

It increased the mean share of forecasts inside the analyst range from 41.1% to 59.5%, but also increased error against reported EPS. The source does not state the size or calculation of that error increase. [1]

**Can financial teams use these LLM outputs without verification?**

No such conclusion is established by the abstract. The study reports evaluation results, but does not establish that LLM outputs are suitable as unverified inputs to investment decisions or research workflows. [1]

## Sources

1. [From SEC Filings to Financial Signals: Evaluating LLMs on Stock Market Reactions and Analyst Forecasts](https://arxiv.org/abs/2610.24268) — arxiv — Oct 20, 2026

How this article was made: https://yelno.ai/blog/about