Study Finds Agent Memory Can Underperform RAG

A study by researchers from the University of Illinois at Urbana-Champaign finds that many dedicated memory systems for long-horizon AI agents perform worse than a simple retrieval-augmented generation (RAG) baseline. The team evaluated systems across multiple benchmarks and model backbones, and says some also incur higher costs. The paper was published on October 25 and submitted to Hugging Face by Xuan He on October 28.

The researchers define memory’s net value relative to direct retrieval from raw conversation history. They call performance gains from reorganizing facts or distilling preferences “memory benefit,” and losses from removing or obscuring evidence “memory tax.” Because memory designs aim to improve tasks, the study focuses on the less examined tax and how it arises from system design and evaluation choices.

In one commenter-cited result, two-round RAG with Qwen3.6-27B scored 81.4 on LongMemEval-s, compared with 76.6 for MemOS and 78.2 for HippoRAG2. The commenter also cited a fact-tracing result in which MemoryOS had no loss for 98.4% of facts, while 38.0% of sampled facts for OpenViking had no corresponding memory unit. These figures appear in comments on the paper page.

The page’s commenters also cite a cost comparison for LightMem with GPT-5.6 Luna: building memory used 5.22M tokens across 100 histories, or about 52.2K per history; question answering used 0.09M tokens versus 0.27M for one-round RAG, about 0.9K versus 2.7K per query. One commenter estimates token costs cross at about 29 queries per history, while reporting accuracy of 77% for LightMem against 82% for RAG. The authors say they will open-source the code and results and hope the framework will help guide memory design and evaluation.

A study by researchers from the University of Illinois at Urbana-Champaign finds that many dedicated memory systems for long-horizon AI agents perform worse than a simple retrieval-augmented generation (RAG) baseline. The team evaluated systems across multiple benchmarks and model backbones, and says some also incur higher costs. The paper was published on October 25 and submitted to Hugging Face by Xuan He on October 28.

The researchers define memory’s net value relative to direct retrieval from raw conversation history. They call performance gains from reorganizing facts or distilling preferences “memory benefit,” and losses from removing or obscuring evidence “memory tax.” Because memory designs aim to improve tasks, the study focuses on the less examined tax and how it arises from system design and evaluation choices.

In one commenter-cited result, two-round RAG with Qwen3.6-27B scored 81.4 on LongMemEval-s, compared with 76.6 for MemOS and 78.2 for HippoRAG2. The commenter also cited a fact-tracing result in which MemoryOS had no loss for 98.4% of facts, while 38.0% of sampled facts for OpenViking had no corresponding memory unit. These figures appear in comments on the paper page.

The page’s commenters also cite a cost comparison for LightMem with GPT-5.6 Luna: building memory used 5.22M tokens across 100 histories, or about 52.2K per history; question answering used 0.09M tokens versus 0.27M for one-round RAG, about 0.9K versus 2.7K per query. One commenter estimates token costs cross at about 29 queries per history, while reporting accuracy of 77% for LightMem against 82% for RAG. The authors say they will open-source the code and results and hope the framework will help guide memory design and evaluation.