When Do Memories Break?

Background

Long-term memory is becoming a core component of persistent LLM agents. Yet these systems still forget user preferences, contradict earlier statements, and fail to use information that appeared much earlier in a conversation.

Most benchmarks evaluate only the final answer. That single outcome cannot explain whether a failure began when the system wrote a memory, modified it, retrieved it, or tried to reason with it. The result is an engineering blind spot: the same incorrect answer can point to four very different components.

This project introduces MemEval, a provenance-aware diagnostic framework for locating likely failures across the memory lifecycle. It is a diagnostic method rather than a new memory architecture.

Diagnostic Framework

MemEval instruments observable intermediate states and follows each relevant fact from the source dialogue to the generated answer. It assigns the root cause to the earliest stage that deviates, preventing one upstream error from being counted repeatedly as several downstream failures.

Together, these stages define an 11-class error taxonomy. The pipeline view makes each diagnosis actionable: an extraction failure calls for better information preservation, while a utilization failure calls for stronger evidence-grounded reasoning.

Evaluation Scope

The study separates the systems being diagnosed from the models used inside those systems and from the models serving as evaluators.

• Memory systems: mem0 is the primary testbed; A-mem, MemoryOS, and OpenClaw test cross-architecture transfer.
• Base models in mem0: GPT-4.1-mini, GPT-4o-mini, Qwen3.5-35B, and Qwen3.5-122B test whether greater model capacity improves memory reliability.
• Meta-evaluators: GPT-5, DeepSeek V3.2 Exp, and GPT-4.1 approximate expert stage attribution through single-model inference, multi-agent discussion, or ensemble voting.

Expert annotators establish the reference labels by examining each question, answer, supporting dialogue evidence, initial memories, update chains, and retrieved memories. They label only the earliest causal failure and resolve disagreements through discussion.

Datasets

LoCoMo

The primary benchmark contains 1,540 dialogue-grounded QA instances: 282 Multi-hop, 321 Temporal, 96 Open-domain, and 841 Single-hop questions. These categories stress different components, from retrieval-heavy evidence composition to temporal reasoning and direct recall.

LongMemEval-S

A second evaluation uses all 500 long-term-memory QA instances from LongMemEval-S. Its knowledge-update, cross-session, preference, and temporal tasks test whether the taxonomy transfers beyond LoCoMo's distribution.

Experimental Results

Automated diagnostic consistency

The 81.62% Stage Match measures agreement between automated diagnosis and expert annotation. It is not the QA accuracy of a memory system.

Ensemble voting is more stable than a single evaluator or multi-agent discussion. Stage 1 is recognized reliably, but the evaluator has an upstream bias: 40.4% of true Stage 3 failures are attributed to extraction. Multi-hop and open-domain cases therefore continue to benefit from human review.

Base-model scaling in mem0

Capacity produces only marginal gains. Stronger models reduce some extraction or retrieval errors but can introduce more ignored-evidence and response-detail errors, shifting the bottleneck downstream rather than removing it.

Architecture and dataset transfer

On the complete LoCoMo set, OpenClaw reaches 73.70% QA accuracy: 1,135 correct answers and 405 diagnosed failures from 1,540 instances. Missing extraction accounts for 119 failures and missed retrieval for 115.

On LongMemEval-S, mem0 with GPT-4.1-mini reaches 49.60% QA accuracy: 248 correct answers and 252 errors. Missing Critical Information accounts for 166 errors, making extraction even more dominant in cross-session and temporal settings.

Main Findings

1. Extraction is the common early bottleneck. Summary-style compression frequently removes low-frequency entities, qualifiers, implicit evidence, and temporal anchors before they enter persistent memory.
2. Downstream bottlenecks depend on the task. Multi-hop questions expose missed retrieval; temporal tasks expose reasoning errors; some single-hop failures occur because the model ignores evidence that was successfully retrieved.
3. Scaling does not repair the pipeline. Larger backbones change where failures appear, but do not eliminate the structural weaknesses of memory extraction, retrieval, and evidence utilization.
4. End-to-end accuracy is not enough. Two systems with similar QA accuracy may require entirely different engineering interventions.
5. Automated diagnosis is useful at the trend and stage level. It can reduce the cost of large-scale analysis, while experts remain necessary for subtle cascading and ambiguous cases.

Limitations

• MemEval requires observable intermediate traces and is not directly applicable to fully black-box memory systems.
• The automated evaluator favors upstream explanations and can confuse retrieval failures with extraction failures.
• Multi-hop and open-domain attribution is less reliable than temporal and single-hop attribution.
• The framework diagnoses failures but does not itself introduce stage-specific fixes.
• The main cross-architecture table does not report the sampled subset size for A-mem and MemoryOS, so those percentages should be treated as preliminary transfer evidence.

Conclusion

Reliable long-term memory requires more than a stronger language model or a better final QA score. It requires visibility into the entire path from dialogue to stored representation, subsequent updates, retrieval, and final reasoning.

MemEval turns memory reliability into a pipeline engineering problem. Its central result is that information is often lost before it reaches persistent memory, while complex tasks continue to fail after storage through retrieval and utilization. Stage-wise diagnosis reveals those differences and supports targeted optimization in place of brute-force scaling.