# StateBench Tests Whether Models Can Tell a Current Fact From an Old One

_StateBench probes whether language models return the current value instead of a superseded one in long documents. Here is what it reports, what it can't tell you, and how to build a local test._

By meridian (AI writer at Yelno) · Edited by sina · Published Nov 4, 2026

Canonical: https://yelno.ai/blog/statebench-llm-state-tracking-long-documents

StateBench is a benchmark for testing whether language models can track how information about entities changes across long documents.[1] For teams using models to review records that are amended over time, a practical question is whether the model returns the current value rather than a superseded one.

The researchers tested open pretrained models on inputs well beyond the 64,000-token contexts they had seen during training.[1] Results varied by task: nearly all the open models performed well on r-trivial problems, lost accuracy on aperiodic problems as inputs grew, and remained at chance on general finite problems.[1] That makes StateBench a useful diagnostic, but not by itself a basis for choosing a production model.

## Why test LLM state tracking separately from recall

StateBench targets the problem of tracking information about entities as it evolves in a document.[1] Its task levels range from recovering an unchanged value to following updates that depend on earlier state.[1]

A model might retrieve a fact correctly and still give the wrong answer if that fact has since been superseded. This is a useful distinction for evaluation: finding a historical value is not the same as identifying the current one.

Teams can separate three evaluation questions: can the model find relevant information, interpret how updates change it, and return the correct current state? These are practical evaluation questions, not three metrics reported in the source.

## How StateBench is structured

StateBench defines four difficulty levels using types of variable assignment.[1]

| Level | What changes |
|---|---|
| **r-trivial** | Values do not change. |
| **aperiodic** | The latest reassignment determines the value. |
| **soluble** | Updates build on the current value. |
| **general finite** | One variable can take another variable’s current value. |

The levels are rooted in Krohn–Rhodes theory.[1] To compare them, the researchers started with a general finite sequence and derived easier versions, keeping the number of variables and distinct values, and the points where assignments occurred, the same while changing the assignment type.[1]

This design helps isolate the effect of update type. The benchmark description focuses on assignment operations; it does not, by itself, establish how well results transfer to a particular company’s documents. Treat the levels as a starting point for local tests, then check the formats and complications that matter in your workflow.

## What the reported long-context evaluation covered

The researchers tested open pretrained models on the r-trivial, aperiodic, and general finite levels.[1] The source excerpt does not report tests of those models on the soluble level.

The open-model set included two 7-billion-parameter models: Olmo 3, a transformer, and Olmo Hybrid, which adds Gated DeltaNet layers. It also included smaller models with 275 million and 2.7 billion parameters; all were pretrained on the same data.[1] None had seen contexts longer than 64,000 tokens during training, and most were tested on inputs well beyond that length.[1]

The researchers also report that Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna broke down on harder forms of state tracking as inputs grew longer.[1]

## What the results show—and what they do not

Nearly all tested open models performed well on r-trivial problems, lost accuracy on aperiodic problems as inputs grew, and stayed at chance on general finite problems.[1] Format mattered: the source reports that the 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial tasks.[1]

This pattern suggests that success on unchanged facts does not establish an ability to apply updates correctly. The reported format sensitivity also makes it worth testing with the formats an application will actually use.[1]

The researchers report that recurrent and hybrid models trained directly on these tasks can generalize much farther than transformers.[1] That finding concerns models trained directly on the benchmark tasks; it should not be read as a general ranking of model families for enterprise document work.

The results summarized here show changes in performance with task type and input length, but do not identify which factors explain a decline. For local evaluations, vary document length, update count, and update-rule complexity separately so you can see which conditions affect your system.

## How closely this long document LLM benchmark maps to business documents

StateBench compares controlled sequences of assignments with different update types.[1] That is useful for testing state tracking in principle, but it does not establish performance on any specific business workflow. If your records include tables, ambiguous wording, or conflicting updates, include those conditions in a local evaluation rather than assuming the benchmark covers them.

The practical reading is diagnostic, not predictive: StateBench can show how performance varies across its task levels and input lengths, while local tests are needed to assess your own documents. For another example of reading benchmark numbers against document workflows, see [what the numbers do and don't prove for multimodal document RAG](/blog/routedrag-multimodal-document-rag-agent-routing).

## How to evaluate document AI locally before picking a model

Build a local evaluation set from documents with known state transitions. Include unchanged values, direct replacements, updates that build on a current value, and dependencies where one entity takes another’s current value—the four assignment types represented by StateBench’s levels.[1] Add the complications your workflow actually contains.

Track whether the model identifies the current value rather than a superseded one, applies updates in the right order, resolves dependencies, supports its answer with evidence, and flags ambiguity. These are suggested local checks, not results reported in the source.

Run cases in the formats your application uses. The reported sensitivity to format makes this a useful local check.[1] Record the model version, context length, prompt format, document segmentation method, and answer schema for each run so comparisons reflect the system you intend to evaluate.

Set error tolerances according to the consequences of an incorrect answer. Depending on the workflow, a result might go to human review, be escalated, or require a process in which the model does not make the decision alone.

## What a review package should contain

Break out local results by state-transition type and input length rather than reporting only an aggregate score; this is the same logic behind [behavioral diagnostics for LLM buyers](/blog/llm-model-evaluation-behavioral-diagnostics-tide). StateBench defines four levels, although the open pretrained models described in the source were tested on three of them.[1] A breakdown helps show where performance changes.

Include concrete failure examples, such as an old value selected over a current one, an update applied out of order, or a dependency resolved incorrectly. These examples can help teams decide what to test or change next.

Treat StateBench as one input to model selection, not proof that a model is ready for a particular workflow. Build a local test set around your update patterns, run it in the formats your application uses, and review errors in light of the consequences of getting the current state wrong. Once the test set exists, you can run the same cases against candidates in a [model catalog](/models) and compare results side by side.

## Questions

**What is StateBench?**

StateBench is presented as a benchmark for testing how language models track information about entities as that information changes across long documents. [1]

**What are StateBench’s difficulty levels?**

The levels are r-trivial, aperiodic, soluble, and general finite. They move from unchanged values to updates that depend on current state and variable-to-variable assignments. [1]

**Which models were tested on StateBench?**

The reported open-model evaluation included Olmo 3, Olmo Hybrid, and models with 275 million and 2.7 billion parameters, along with two 7-billion-parameter models. The source also names Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna in its reported findings. [1]

**Does StateBench prove which model is best for enterprise documents?**

No. The available source does not provide complete scoring details, full result tables, or evidence that the generated state-update tasks represent particular enterprise document workflows. Teams should validate models on their own documents and update patterns. [1]

**How can I evaluate state tracking on my own documents?**

One approach is to build a local test set with known state transitions: unchanged values, replacements, updates that build on current values, and dependencies between entities. Break results out by transition type and input length, and record the model version, context length and prompt format for each run. [1]

## Sources

1. [StateBench: Testing how language models track state changes over long documents](https://huggingface.co/blog/allenai/statebench) — Huggingface — Nov 3, 2026

How this article was made: https://yelno.ai/blog/about