Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

2024 | Dennis Wu * 1 Jerry Yao-Chieh Hu * 1 Teng-Yun Hsiao 2 Han Liu 1 3
The paper introduces a two-stage memory retrieval dynamics for modern Hopfield models, termed U-Hop, which enhances memory capacity. The key contribution is a learnable feature map Φ that transforms the Hopfield energy function into kernel space, ensuring convergence between local minima of energy and fixed points of retrieval dynamics within the kernel space. The kernel norm induced by Φ serves as a novel similarity measure, improving memory capacity across all modern Hopfield models. The U-Hop process consists of two stages: (Stage I) minimizing a separation loss LΦ to distribute stored memory patterns more uniformly in kernel space, followed by (Stage II) standard Hopfield energy minimization for memory retrieval. This approach significantly reduces metastable states in the energy function, enhancing memory capacity and preventing memory confusion. Empirical results on real-world datasets demonstrate that U-Hop outperforms existing modern Hopfield models and similarity measures in both associative memory retrieval and deep learning tasks, achieving substantial improvements in retrieval accuracy and generalization.The paper introduces a two-stage memory retrieval dynamics for modern Hopfield models, termed U-Hop, which enhances memory capacity. The key contribution is a learnable feature map Φ that transforms the Hopfield energy function into kernel space, ensuring convergence between local minima of energy and fixed points of retrieval dynamics within the kernel space. The kernel norm induced by Φ serves as a novel similarity measure, improving memory capacity across all modern Hopfield models. The U-Hop process consists of two stages: (Stage I) minimizing a separation loss LΦ to distribute stored memory patterns more uniformly in kernel space, followed by (Stage II) standard Hopfield energy minimization for memory retrieval. This approach significantly reduces metastable states in the energy function, enhancing memory capacity and preventing memory confusion. Empirical results on real-world datasets demonstrate that U-Hop outperforms existing modern Hopfield models and similarity measures in both associative memory retrieval and deep learning tasks, achieving substantial improvements in retrieval accuracy and generalization.
Reach us at info@study.space
[slides] Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models | StudySpace