An assistant reading a project history needs to know which decisions still apply. Finding an old deadline is easy to score; following several revisions, dependencies and reversals needs a different test. That is the useful question behind StateBench, introduced by Ai2 on November 3, 2026. This November 9 assessment looks at its implications for evaluation, rather than treating context-window size as evidence of reliable memory. Ai2 announcement
Four kinds of change
The paper by Jackson Petty, William Merrill and Ashish Sabharwal, submitted November 2, divides state tracking into four levels. These short examples illustrate the distinction; they are not benchmark samples.
| Level | Example | What the model must retain |
|---|---|---|
| R-trivial | A record gets a permanent identifier | The original assignment |
| Aperiodic | A setting changes from 4 to 2 | The latest assignment |
| Soluble | A counter repeatedly increases | The accumulated value |
| General finite | One variable takes another’s current value | Dependencies across updates |
Those operations give researchers a way to separate different failure modes inside an apparently simple task: read a sequence and report its final state. The underlying hierarchy is mathematical; the examples above are an introduction, not a complete definition of each class.
Training and architecture both matter
Ai2 reports that Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels well, but fell to 13% accuracy or less on the longest general-finite problems. That is a result for these benchmark conditions, not a failure rate for everyday conversations. The release also notes a limitation: its framework keeps the number of variables fixed, whereas real conversations can introduce new entities. Ai2’s results and limitations
Separate experiments trained 275-million-parameter models directly on the task, using sequences of up to 256 assignments and testing up to 64,000. Gated DeltaNet and hybrid models generalized much farther on the easier classes; the hardest class still exposed differences between variants. These controlled training results do not establish that an off-the-shelf hybrid model will outperform a frontier assistant in your workflow. Paper, direct-training experiments
How to evaluate an agent’s memory
Our guide to interpreting AI benchmarks explains why a score needs its task and evaluation conditions. For a project assistant, we would add a small set of state-change tests drawn from the actual workflow:
- Replace a deadline twice, then ask for the current deadline and the evidence that superseded the earlier dates.
- Transfer a task between owners, then ask who is responsible after the final handoff.
- Cancel an earlier decision, add unrelated discussion, then check whether the assistant still recommends the cancelled action.
These are proposed checks, not experiments S5 Labs has run. Keep an authoritative answer for each case and increase the number of consequential changes separately from the amount of irrelevant text. Record the first update the system misses, not just whether its final response sounds plausible.
For reproducible research, Ai2 provides code and generation configurations and a published dataset. The repository distinguishes assignment depth from filler and documents an important detail: copying a variable uses its value at that moment; it does not create a permanent alias. Pin the data revision, format and scoring procedure when comparing runs.
A system that can find the right passage can still reconstruct the wrong current state. Test both before relying on a long conversation as the record of what an agent should do next.
