This systematic literature review (SLR) examines AI-based anomaly detection techniques over encrypted traffic. The review aims to identify and evaluate the current state of research in this area, addressing five key research questions (RQs) related to datasets, feature extraction, feature selection, preprocessing, anomaly detection algorithms, and performance indicators. The study includes 30 high-quality articles selected from a comprehensive literature search across multiple databases, covering the period from January 2019 to August 2023. The review highlights the increasing importance of encrypted traffic in cyber-attacks and the challenges posed by traditional deep packet inspection (DPI) methods. It also discusses the limitations of existing anomaly detection models for encrypted traffic and the potential of AI technologies, such as machine learning and deep learning, to address these challenges. The review provides a detailed analysis of various datasets used in the studies, feature extraction methods, preprocessing techniques, and the performance of different AI algorithms. The findings suggest that while some techniques are similar to those used for unencrypted traffic, others are specifically designed for encrypted environments, reflecting the unique characteristics of encrypted data. The review concludes with a discussion of the current landscape of AI-based anomaly detection over encrypted traffic and identifies gaps in the literature that warrant further research.This systematic literature review (SLR) examines AI-based anomaly detection techniques over encrypted traffic. The review aims to identify and evaluate the current state of research in this area, addressing five key research questions (RQs) related to datasets, feature extraction, feature selection, preprocessing, anomaly detection algorithms, and performance indicators. The study includes 30 high-quality articles selected from a comprehensive literature search across multiple databases, covering the period from January 2019 to August 2023. The review highlights the increasing importance of encrypted traffic in cyber-attacks and the challenges posed by traditional deep packet inspection (DPI) methods. It also discusses the limitations of existing anomaly detection models for encrypted traffic and the potential of AI technologies, such as machine learning and deep learning, to address these challenges. The review provides a detailed analysis of various datasets used in the studies, feature extraction methods, preprocessing techniques, and the performance of different AI algorithms. The findings suggest that while some techniques are similar to those used for unencrypted traffic, others are specifically designed for encrypted environments, reflecting the unique characteristics of encrypted data. The review concludes with a discussion of the current landscape of AI-based anomaly detection over encrypted traffic and identifies gaps in the literature that warrant further research.