Memora Tests Whether AI Agents Forget

A paper presented at Findings of ACL 2026 introduces Memora, a benchmark for testing whether personalized AI agents update or discard outdated memories, not just recall facts. The authors of “From Recall to Forgetting” argue that existing tests mostly measure recall: 94% of LoCoMo questions and 85% of LongMemEval questions can be answered from at most two past sessions, while major benchmarks average about one session of history per question. LongMemEval allows at most two knowledge updates before evaluation, and PersonaMem tests changes across no more than three sessions.

Memora simulates 10 professional personas across weekly, monthly and quarterly histories, averaging about 155, 615 and 1,991 sessions per persona, respectively, with roughly 16 turns per session. In the quarterly setting, each persona’s memory undergoes an average of about 1,171 operations: 63% additions, 18% updates and 19% deletions. Explicit traces track which facts remain current. The benchmark tests remembering facts, reasoning across memories and recommending according to current preferences. Its FAMA score (accuracy that also penalizes outdated facts) checks both required and prohibited information; three language-model judges agreed with human annotators 88.3% of the time.

The researchers tested four language models reading conversation histories directly, with and without reasoning tokens, and six dedicated memory agents using GPT-4o-mini to answer. Because the memory agents shared that smaller backend, the results compare memory systems on a fixed model rather than memory and no-memory setups at equal model strength. Aggregated FAMA scores across the three timeline settings total up to 300 and should be read as relative scores, not percentages. Memory agents scored 119.5 on recall, nearly twice the language models’ 65.7. On recommendations, plain language models scored 144.7 to 153.3, compared with 138.4 for memory agents. The authors say recommendations may be possible from approximate preferences inferred from persona cues.

Reasoning was weak across all systems: memory agents had the best average score at 27.6, while language models scored 12.4 to 13.9. In a manual review, every sampled reasoning error involved partial retrieval followed by an answer; reasoning tokens made little difference. The authors’ example involved agents saying a user was “within budget” instead of calculating the remaining amount after logged expenses.

The stale-memory penalty moved in opposite directions as histories grew. For language models reading raw histories, it fell from 32.6 points in the weekly setting to 17.8 in the quarterly setting because older conversations dropped out of the context window. For memory agents, the penalty rose from 18.2 to 29.5 as stores accumulated more history. In the monthly setting, MemoryOS scored 112.8 and A-Mem 112.0 on plain memory accuracy, ahead of Nemori at 105.4. But Nemori’s forgetting penalty was 15.4, compared with 28.4 for MemoryOS and 29.5 for A-Mem, putting Nemori ahead once FAMA counted stale memories. Of sampled recommendation errors, 64% involved outdated memories that had not been invalidated; one example was an agent recommending a historical biography after the user had switched from preferring nonfiction to contemporary fiction.

The authors recommend treating memory updates as replacements for old facts, testing agents with corrections and deletions, scoring information that should be excluded, and not expecting a memory layer to fix reasoning. They caution that the conversations were generated by language models rather than collected from users, that all six memory agents used GPT-4o-mini, and that the evaluation relied on language-model judges despite three-way voting and a human agreement study. The benchmark covered 10 personas and three memory types: preferences, activities and goals. The authors describe its synthetic results as a lower bound because real interactions add ambiguity and contradiction.

A paper presented at Findings of ACL 2026 introduces Memora, a benchmark for testing whether personalized AI agents update or discard outdated memories, not just recall facts. The authors of “From Recall to Forgetting” argue that existing tests mostly measure recall: 94% of LoCoMo questions and 85% of LongMemEval questions can be answered from at most two past sessions, while major benchmarks average about one session of history per question. LongMemEval allows at most two knowledge updates before evaluation, and PersonaMem tests changes across no more than three sessions.

Memora simulates 10 professional personas across weekly, monthly and quarterly histories, averaging about 155, 615 and 1,991 sessions per persona, respectively, with roughly 16 turns per session. In the quarterly setting, each persona’s memory undergoes an average of about 1,171 operations: 63% additions, 18% updates and 19% deletions. Explicit traces track which facts remain current. The benchmark tests remembering facts, reasoning across memories and recommending according to current preferences. Its FAMA score (accuracy that also penalizes outdated facts) checks both required and prohibited information; three language-model judges agreed with human annotators 88.3% of the time.

The researchers tested four language models reading conversation histories directly, with and without reasoning tokens, and six dedicated memory agents using GPT-4o-mini to answer. Because the memory agents shared that smaller backend, the results compare memory systems on a fixed model rather than memory and no-memory setups at equal model strength. Aggregated FAMA scores across the three timeline settings total up to 300 and should be read as relative scores, not percentages. Memory agents scored 119.5 on recall, nearly twice the language models’ 65.7. On recommendations, plain language models scored 144.7 to 153.3, compared with 138.4 for memory agents. The authors say recommendations may be possible from approximate preferences inferred from persona cues.

Reasoning was weak across all systems: memory agents had the best average score at 27.6, while language models scored 12.4 to 13.9. In a manual review, every sampled reasoning error involved partial retrieval followed by an answer; reasoning tokens made little difference. The authors’ example involved agents saying a user was “within budget” instead of calculating the remaining amount after logged expenses.

The stale-memory penalty moved in opposite directions as histories grew. For language models reading raw histories, it fell from 32.6 points in the weekly setting to 17.8 in the quarterly setting because older conversations dropped out of the context window. For memory agents, the penalty rose from 18.2 to 29.5 as stores accumulated more history. In the monthly setting, MemoryOS scored 112.8 and A-Mem 112.0 on plain memory accuracy, ahead of Nemori at 105.4. But Nemori’s forgetting penalty was 15.4, compared with 28.4 for MemoryOS and 29.5 for A-Mem, putting Nemori ahead once FAMA counted stale memories. Of sampled recommendation errors, 64% involved outdated memories that had not been invalidated; one example was an agent recommending a historical biography after the user had switched from preferring nonfiction to contemporary fiction.

The authors recommend treating memory updates as replacements for old facts, testing agents with corrections and deletions, scoring information that should be excluded, and not expecting a memory layer to fix reasoning. They caution that the conversations were generated by language models rather than collected from users, that all six memory agents used GPT-4o-mini, and that the evaluation relied on language-model judges despite three-way voting and a human agreement study. The benchmark covered 10 personas and three memory types: preferences, activities and goals. The authors describe its synthetic results as a lower bound because real interactions add ambiguity and contradiction.