MIT researchers have unveiled a long-term memory framework designed to enable robots to recall detailed environments and interact with human spaces more intuitively. The system, called Describe Anything, Anywhere, Anytime, at Any Moment (DAAAM), combines 3D robotic mapping with semantic descriptions to create a language-accessible mental model of the environment. Unlike previous methods that processed single annotations, DAAAM aggregates nearby objects during robot traversal and utilizes an optimization method to select clear key frames for parallel annotation. This process improves computational speed tenfold, allowing the system to function in large-scale settings in real-time.

To effectively manage stored data, the framework organizes objects into spatially clustered regions within a 3D map. When a user issues a query, such as asking a robot to locate an item or describe surroundings, the system employs an LLM equipped with specific tools to retrieve information from its database. This architecture minimizes hallucinations (erroneous model outputs) and enables accurate responses within seconds. In performance evaluations, DAAAM demonstrated 21 percent to 53 percent higher accuracy than existing state-of-the-art methods across various question types.

The research team, led by MIT graduate student Nicolas Gorlo and associate professor Luca Carlone, presented the work at the Conference on Computer Vision and Pattern Recognition (CVPR). Beyond robotics, the framework holds potential applications in augmented reality for maintenance assistance and commuter wayfinding. The team intends to further refine the system by adding the ability to track significant events and incorporating confidence levels into robot responses. This work received funding from the U.S. Army Research Laboratory and the Office of Naval Research, building toward the development of generalist agents capable of executing complex human-requested tasks.

MIT researchers have unveiled a long-term memory framework designed to enable robots to recall detailed environments and interact with human spaces more intuitively. The system, called Describe Anything, Anywhere, Anytime, at Any Moment (DAAAM), combines 3D robotic mapping with semantic descriptions to create a language-accessible mental model of the environment. Unlike previous methods that processed single annotations, DAAAM aggregates nearby objects during robot traversal and utilizes an optimization method to select clear key frames for parallel annotation. This process improves computational speed tenfold, allowing the system to function in large-scale settings in real-time.

To effectively manage stored data, the framework organizes objects into spatially clustered regions within a 3D map. When a user issues a query, such as asking a robot to locate an item or describe surroundings, the system employs an LLM equipped with specific tools to retrieve information from its database. This architecture minimizes hallucinations (erroneous model outputs) and enables accurate responses within seconds. In performance evaluations, DAAAM demonstrated 21 percent to 53 percent higher accuracy than existing state-of-the-art methods across various question types.

The research team, led by MIT graduate student Nicolas Gorlo and associate professor Luca Carlone, presented the work at the Conference on Computer Vision and Pattern Recognition (CVPR). Beyond robotics, the framework holds potential applications in augmented reality for maintenance assistance and commuter wayfinding. The team intends to further refine the system by adding the ability to track significant events and incorporating confidence levels into robot responses. This work received funding from the U.S. Army Research Laboratory and the Office of Naval Research, building toward the development of generalist agents capable of executing complex human-requested tasks.