A research team led by Wei Zhou from Shanghai Jiao Tong University released a study on June 23 proposing an analytical framework for evaluating memory systems in large language model (LLM) agents. The authors argue that current evaluation methods rely too heavily on end-to-end task success metrics like F1 and BLEU, which treat memory as a monolithic black box while ignoring system-level performance characteristics. The proposed framework decomposes these memory systems into four core modules: memory representation and storage, extraction, retrieval and routing, and maintenance.

Researchers evaluated 12 representative memory systems alongside two reference baselines using five benchmark workloads spanning 11 datasets. The results indicate that no single architecture excels across all scenarios, as effectiveness is largely determined by how well a memory structure addresses a specific workload bottleneck. Through fine-grained ablation studies, the study quantified the impacts of various modules on representation fidelity, retrieval precision, update correctness, and long-horizon stability. The findings highlight significant cost-performance trade-offs, demonstrating that localized maintenance strategies are more cost-efficient than global reorganization. The research aims to guide the development of future agent-native memory systems, with code and resources now publicly available via the MemoryData repository on GitHub.

A research team led by Wei Zhou from Shanghai Jiao Tong University released a study on June 23 proposing an analytical framework for evaluating memory systems in large language model (LLM) agents. The authors argue that current evaluation methods rely too heavily on end-to-end task success metrics like F1 and BLEU, which treat memory as a monolithic black box while ignoring system-level performance characteristics. The proposed framework decomposes these memory systems into four core modules: memory representation and storage, extraction, retrieval and routing, and maintenance.

Researchers evaluated 12 representative memory systems alongside two reference baselines using five benchmark workloads spanning 11 datasets. The results indicate that no single architecture excels across all scenarios, as effectiveness is largely determined by how well a memory structure addresses a specific workload bottleneck. Through fine-grained ablation studies, the study quantified the impacts of various modules on representation fidelity, retrieval precision, update correctness, and long-horizon stability. The findings highlight significant cost-performance trade-offs, demonstrating that localized maintenance strategies are more cost-efficient than global reorganization. The research aims to guide the development of future agent-native memory systems, with code and resources now publicly available via the MemoryData repository on GitHub.