Researchers from ByteDance Seed introduced Cortex in a paper published on October 14, proposing a way for large language models to retain performance gains from context after that context is removed. The method consolidates those gains into parameter updates that can be merged, while aiming to preserve capabilities unrelated to the update. The authors frame this as memory consolidation, linking context-based editing, skill learning and behavioral steering under one objective.

The paper presents a testbed with five task families to measure how well models retain changes after single, batch and sequential updates. Cortex generates prospective queries to anticipate how supplied context may be used, then distills context-conditioned outputs into mergeable low-rank or structural updates to a feed-forward network. The evaluation covers dense and mixture-of-experts models from 4B to 120B parameters, including text-only and multimodal models.

Across most evaluated settings, Cortex was on the empirical frontier balancing consolidation and retention. In one reported test, Cortex-Structural applied 50 sequential factual edits to a 14B model, recovered 88.9% of the gain induced by context, and retained 99.8% of the model’s original performance on non-target tasks. The paper describes this approach as enabling models to internalize knowledge for continual learning while retaining existing capabilities.

Researchers from ByteDance Seed introduced Cortex in a paper published on October 14, proposing a way for large language models to retain performance gains from context after that context is removed. The method consolidates those gains into parameter updates that can be merged, while aiming to preserve capabilities unrelated to the update. The authors frame this as memory consolidation, linking context-based editing, skill learning and behavioral steering under one objective.

The paper presents a testbed with five task families to measure how well models retain changes after single, batch and sequential updates. Cortex generates prospective queries to anticipate how supplied context may be used, then distills context-conditioned outputs into mergeable low-rank or structural updates to a feed-forward network. The evaluation covers dense and mixture-of-experts models from 4B to 120B parameters, including text-only and multimodal models.

Across most evaluated settings, Cortex was on the empirical frontier balancing consolidation and retention. In one reported test, Cortex-Structural applied 50 sequential factual edits to a 14B model, recovered 88.9% of the gain induced by context, and retained 99.8% of the model’s original performance on non-target tasks. The paper describes this approach as enabling models to internalize knowledge for continual learning while retaining existing capabilities.