Nested Learning: The Illusion of Deep Learning Architectures

This paper introduces Nested Learning (NL), a new paradigm that addresses fundamental challenges in AI self-improvement, continual learning and memory for models like Large Language Models (LLMs). NL suggests that existing deep learning methods compress their "context flow" and explains how in-context learning emerges in large models. The authors propose the HOPE architecture, a self-referential learning module with a Continuum Memory System (CMS), which is built on the NL insights that traditional optimizers are fundamentally associative memory modules. Experiments demonstrate that HOPE, using this novel framework, shows promising results across language modeling and common-sense reasoning tasks, often outperforming modern recurrent neural networks and Transformers.