Researchers Sangyun Lee, Sean McLeish, Tom Goldstein, and Giulia Fanti introduced a sleep-like consolidation mechanism for large language models (LLMs) in a paper submitted on May 25, 2026. This method addresses the limitation where Transformer-based models scale poorly with long context lengths. The proposed system periodically converts recent input into persistent fast weights, then clears the model's key-value cache to manage memory efficiently.

During this sleep-like phase, the model performs N offline recurrent passes over the accumulated context, updating the fast weights within its state-space model (SSM) blocks using a learned local rule. This approach shifts intensive computational tasks to the offline phase, allowing the model to preserve low-latency performance during active wake-time prediction.

Testing on synthetic tasks such as cellular automata and multi-hop graph retrieval, as well as complex math reasoning, showed that the consolidation method outperforms both standard Transformer and SSM-attention hybrid models. Researchers observed that increasing the sleep duration directly improves model performance, with the most significant gains occurring in tasks that require deeper reasoning capabilities.

Researchers Sangyun Lee, Sean McLeish, Tom Goldstein, and Giulia Fanti introduced a sleep-like consolidation mechanism for large language models (LLMs) in a paper submitted on May 25, 2026. This method addresses the limitation where Transformer-based models scale poorly with long context lengths. The proposed system periodically converts recent input into persistent fast weights, then clears the model's key-value cache to manage memory efficiently.

During this sleep-like phase, the model performs N offline recurrent passes over the accumulated context, updating the fast weights within its state-space model (SSM) blocks using a learned local rule. This approach shifts intensive computational tasks to the offline phase, allowing the model to preserve low-latency performance during active wake-time prediction.

Testing on synthetic tasks such as cellular automata and multi-hop graph retrieval, as well as complex math reasoning, showed that the consolidation method outperforms both standard Transformer and SSM-attention hybrid models. Researchers observed that increasing the sleep duration directly improves model performance, with the most significant gains occurring in tasks that require deeper reasoning capabilities.