Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

2024 | Dennis Wu, Jerry Yao-Chieh Hu, Teng-Yun Hsiao, Han Liu
This paper proposes a two-stage memory retrieval method for modern Hopfield models, termed U-Hop, which enhances memory capacity by transforming the Hopfield energy function into kernel space using a learnable feature map. The key contribution is a learnable feature map Φ that maps the Hopfield energy function into kernel space, enabling convergence between local minima of energy and fixed points of retrieval dynamics within the kernel space. This transformation results in a novel similarity measure based on the kernel norm induced by Φ, which utilizes stored memory patterns as learning data to enhance memory capacity across all modern Hopfield models. The U-Hop memory retrieval process consists of two stages: (Stage I) minimizing separation loss to achieve a more uniform memory distribution, followed by (Stage II) standard Hopfield energy minimization for memory retrieval. This results in a significant reduction of possible metastable states in the Hopfield energy function, thus enhancing memory capacity by preventing memory confusion. Empirically, U-Hop outperforms all existing modern Hopfield models and SOTA similarity measures, achieving substantial improvements in both associative memory retrieval and deep learning tasks. The code is available at GitHub; future updates are on arXiv.This paper proposes a two-stage memory retrieval method for modern Hopfield models, termed U-Hop, which enhances memory capacity by transforming the Hopfield energy function into kernel space using a learnable feature map. The key contribution is a learnable feature map Φ that maps the Hopfield energy function into kernel space, enabling convergence between local minima of energy and fixed points of retrieval dynamics within the kernel space. This transformation results in a novel similarity measure based on the kernel norm induced by Φ, which utilizes stored memory patterns as learning data to enhance memory capacity across all modern Hopfield models. The U-Hop memory retrieval process consists of two stages: (Stage I) minimizing separation loss to achieve a more uniform memory distribution, followed by (Stage II) standard Hopfield energy minimization for memory retrieval. This results in a significant reduction of possible metastable states in the Hopfield energy function, thus enhancing memory capacity by preventing memory confusion. Empirically, U-Hop outperforms all existing modern Hopfield models and SOTA similarity measures, achieving substantial improvements in both associative memory retrieval and deep learning tasks. The code is available at GitHub; future updates are on arXiv.
Reach us at info@study.space