StateBench: When Long Context Loses Track of Changes

Ai2's StateBench separates finding a fact from following changing state. What its results mean for long-document and agent evaluations.

An assistant reading a project history needs to know which decisions still apply. Finding an old deadline is easy to score; following several revisions, dependencies and reversals needs a different test. That is the useful question behind StateBench, introduced by Ai2 on November 3, 2026. This November 9 assessment looks at its implications for evaluation, rather than treating context-window size as evidence of reliable memory. Ai2 announcement

Four kinds of change

The paper by Jackson Petty, William Merrill and Ashish Sabharwal, submitted November 2, divides state tracking into four levels. These short examples illustrate the distinction; they are not benchmark samples.

LevelExampleWhat the model must retain
R-trivialA record gets a permanent identifierThe original assignment
AperiodicA setting changes from 4 to 2The latest assignment
SolubleA counter repeatedly increasesThe accumulated value
General finiteOne variable takes another’s current valueDependencies across updates

Those operations give researchers a way to separate different failure modes inside an apparently simple task: read a sequence and report its final state. The underlying hierarchy is mathematical; the examples above are an introduction, not a complete definition of each class.

Training and architecture both matter

Ai2 reports that Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels well, but fell to 13% accuracy or less on the longest general-finite problems. That is a result for these benchmark conditions, not a failure rate for everyday conversations. The release also notes a limitation: its framework keeps the number of variables fixed, whereas real conversations can introduce new entities. Ai2’s results and limitations

Separate experiments trained 275-million-parameter models directly on the task, using sequences of up to 256 assignments and testing up to 64,000. Gated DeltaNet and hybrid models generalized much farther on the easier classes; the hardest class still exposed differences between variants. These controlled training results do not establish that an off-the-shelf hybrid model will outperform a frontier assistant in your workflow. Paper, direct-training experiments

How to evaluate an agent’s memory

Our guide to interpreting AI benchmarks explains why a score needs its task and evaluation conditions. For a project assistant, we would add a small set of state-change tests drawn from the actual workflow:

  • Replace a deadline twice, then ask for the current deadline and the evidence that superseded the earlier dates.
  • Transfer a task between owners, then ask who is responsible after the final handoff.
  • Cancel an earlier decision, add unrelated discussion, then check whether the assistant still recommends the cancelled action.

These are proposed checks, not experiments S5 Labs has run. Keep an authoritative answer for each case and increase the number of consequential changes separately from the amount of irrelevant text. Record the first update the system misses, not just whether its final response sounds plausible.

For reproducible research, Ai2 provides code and generation configurations and a published dataset. The repository distinguishes assignment depth from filler and documents an important detail: copying a variable uses its value at that moment; it does not create a permanent alias. Pin the data revision, format and scoring procedure when comparing runs.

A system that can find the right passage can still reconstruct the wrong current state. Test both before relying on a long conversation as the record of what an agent should do next.

Continue reading.

Insight22 min read

GPT-6.2 Astra Drops to $8/$40, Sol Holds at $2/$10 and Cyber Safeguards Tighten

GPT-6.2 Astra costs $8/$40 per million tokens, 20% below GPT-6 Astra; Sol stays at $2/$10. Migration steps, benchmark caveats and tighter cyber refusals.

Insight7 min read

GPT-6 Astra Brings Better Computer Use and a Higher Bar for Delegating Work

OpenAI's GPT-6 Astra improves computer use and complex work. A closer look at launch benchmarks, API costs, limited rollout, and safety monitoring.

Insight10 min read

DeepSeek Open-Sourced the Harness That Graded Its Own Benchmarks

DeepSeek Harness (dsh) is MIT-licensed and model-agnostic. It is also the software that produced V4-Pro's Terminal-Bench and DeepSWE scores.