29 Jun 2024 | Chengkai Liu, Jianghao Lin, Jianling Wang, Hanzhou Liu, James Caverlee
Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models
Sequential recommendation aims to estimate dynamic user preferences and sequential dependencies in historical behavior. While Transformer-based models are effective, they suffer from inference inefficiency due to quadratic complexity of attention operators, especially for long sequences. Inspired by state space models (SSMs), Mamba4Rec is the first to explore selective SSMs for efficient sequential recommendation. Built on the Mamba block, a selective SSM with efficient hardware-aware parallel algorithm, Mamba4Rec incorporates techniques like position embedding, residual connection, and layer normalization to enhance performance while maintaining efficiency. Experiments on public datasets show Mamba4Rec outperforms RNN- and attention-based baselines in both effectiveness and efficiency.
State space models (SSMs) have been widely adopted as alternatives to RNNs, CNNs, and Transformers. They offer inference efficiency due to their recurrent nature and strong performance in handling long-range dependencies through structured state matrices. A recent SSM variant, Mamba, introduces an input-dependent selective mechanism with efficient hardware-aware design, allowing the model to selectively extract essential knowledge and filter out noise. These advancements position SSMs as a core operator for sequential recommendation.
This work introduces Mamba4Rec, the first model to leverage selective SSMs for efficient sequential recommendation. Built on the basic Mamba block, Mamba4Rec incorporates techniques like position embedding, residual connection, and layer normalization to enhance performance while maintaining efficiency. The main contributions include exploring the potential of selective SSMs for sequential recommendation, proposing Mamba4Rec to improve sequential modeling without sacrificing efficiency, and demonstrating its superiority over baselines in both effectiveness and efficiency.
Mamba4Rec uses an embedding layer to map item IDs to a high-dimensional space, followed by a selective state space model and a prediction layer. The Mamba block processes input through linear projections, 1D convolution, and SiLU activation, generating a state representation. A position-wise feed-forward network enhances modeling of user actions in the hidden dimension. Stacking Mamba layers with residual connections improves performance on long sequences.
Experiments on MovieLens-1M and Amazon datasets show Mamba4Rec outperforms baselines in effectiveness and efficiency. It achieves superior performance in GPU memory usage, training time, and inference time, demonstrating faster convergence and reduced memory cost. Ablation studies show the importance of position embeddings, layer normalization, and feed-forward networks in enhancing performance. Mamba4Rec achieves strong performance on datasets with varying sparsity and sequence lengths, making it effective and efficient for sequential recommendation tasks. Future work aims to design state space models tailored for recommendation systems and promote SSM-based models in this domain.Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models
Sequential recommendation aims to estimate dynamic user preferences and sequential dependencies in historical behavior. While Transformer-based models are effective, they suffer from inference inefficiency due to quadratic complexity of attention operators, especially for long sequences. Inspired by state space models (SSMs), Mamba4Rec is the first to explore selective SSMs for efficient sequential recommendation. Built on the Mamba block, a selective SSM with efficient hardware-aware parallel algorithm, Mamba4Rec incorporates techniques like position embedding, residual connection, and layer normalization to enhance performance while maintaining efficiency. Experiments on public datasets show Mamba4Rec outperforms RNN- and attention-based baselines in both effectiveness and efficiency.
State space models (SSMs) have been widely adopted as alternatives to RNNs, CNNs, and Transformers. They offer inference efficiency due to their recurrent nature and strong performance in handling long-range dependencies through structured state matrices. A recent SSM variant, Mamba, introduces an input-dependent selective mechanism with efficient hardware-aware design, allowing the model to selectively extract essential knowledge and filter out noise. These advancements position SSMs as a core operator for sequential recommendation.
This work introduces Mamba4Rec, the first model to leverage selective SSMs for efficient sequential recommendation. Built on the basic Mamba block, Mamba4Rec incorporates techniques like position embedding, residual connection, and layer normalization to enhance performance while maintaining efficiency. The main contributions include exploring the potential of selective SSMs for sequential recommendation, proposing Mamba4Rec to improve sequential modeling without sacrificing efficiency, and demonstrating its superiority over baselines in both effectiveness and efficiency.
Mamba4Rec uses an embedding layer to map item IDs to a high-dimensional space, followed by a selective state space model and a prediction layer. The Mamba block processes input through linear projections, 1D convolution, and SiLU activation, generating a state representation. A position-wise feed-forward network enhances modeling of user actions in the hidden dimension. Stacking Mamba layers with residual connections improves performance on long sequences.
Experiments on MovieLens-1M and Amazon datasets show Mamba4Rec outperforms baselines in effectiveness and efficiency. It achieves superior performance in GPU memory usage, training time, and inference time, demonstrating faster convergence and reduced memory cost. Ablation studies show the importance of position embeddings, layer normalization, and feed-forward networks in enhancing performance. Mamba4Rec achieves strong performance on datasets with varying sparsity and sequence lengths, making it effective and efficient for sequential recommendation tasks. Future work aims to design state space models tailored for recommendation systems and promote SSM-based models in this domain.