Why Agent Memory Requires Calibrated Abstention Over Ranking
- •Agent memory retrieval fails when models lack a mechanism for calibrated abstention on missing data.
- •Traditional RAG metrics focus on ranking accuracy, which causes systems to hallucinate on unanswered agent queries.
- •Giulio D'Erme released RE-call, an open-source Postgres-based engine designed to flag memory gaps and stale data.
Retrieval-Augmented Generation (RAG) systems designed for document question-answering often struggle when applied to agent memory, a concept Giulio D'Erme refers to as self-recall. In standard document QA, a system assumes the correct answer exists within the corpus, focusing exclusively on ranking search results by relevance. Conversely, agent-memory systems frequently encounter queries with no internal answer, such as an automated trading agent asking if a specific strategy was previously tested. Standard retrieval models, when forced to provide answers for these gaps, return the most statistically similar documents, causing the agent to hallucinate conclusions based on irrelevant information.
Self-recall introduces three primary failure modes that deviate from traditional RAG patterns: hallucinating over gaps, re-litigating decisions that were already settled, and acting on stale information from past memos. These issues persist regardless of ranking model quality because traditional metrics like MRR (Mean Reciprocal Rank) and nDCG (normalized Discounted Cumulative Gain) prioritize ordering results rather than measuring calibrated abstention—the system's ability to admit it lacks a relevant answer. Relying on standard RAG stacks for agent memory often leads to production failures as the models lack mechanisms to signal uncertainty.
To address these limitations, D'Erme released RE-call, an open-source retrieval engine built specifically for agent memory. It uses PostgreSQL and pgvector to handle combined dense vector and sparse full-text searches. The system includes three specific honesty guards: a gap warning for weak matches, a freshness signal to identify outdated information, and an anti-re-litigation check to prevent the repetition of failed past decisions. Furthermore, the engine introduces a false-confident rate metric to evaluate how frequently the system fails to abstain when it should. This architecture operates as an MCP server, allowing agents to query their own operational history directly while maintaining a focus on honest, calibrated results rather than simple relevance optimization.