The paper introduces a novel framework called UCAD (Unsupervised Continual Anomaly Detection) for addressing the challenge of unsupervised anomaly detection in industrial manufacturing, where obtaining labeled data is costly and unpredictable. UCAD incorporates continual learning capabilities through a Continual Prompting Module (CPM) and Structure-based Contrastive Learning (SCL). The CPM uses a key-prompt-knowledge memory bank to guide task-invariant 'anomaly' model predictions using task-specific 'normal' knowledge. SCL, based on the Segment Anything Model (SAM), improves prompt learning and anomaly segmentation by treating SAM's masks as structures, drawing features within the same mask closer and pushing others apart. Comprehensive experiments on the MVTec AD and VisA datasets demonstrate that UCAD outperforms existing methods by 15.6% in detection and 26.6% in segmentation, even with rehearsal training. The code for UCAD is available at https://github.com/shirowalker/UCAD.The paper introduces a novel framework called UCAD (Unsupervised Continual Anomaly Detection) for addressing the challenge of unsupervised anomaly detection in industrial manufacturing, where obtaining labeled data is costly and unpredictable. UCAD incorporates continual learning capabilities through a Continual Prompting Module (CPM) and Structure-based Contrastive Learning (SCL). The CPM uses a key-prompt-knowledge memory bank to guide task-invariant 'anomaly' model predictions using task-specific 'normal' knowledge. SCL, based on the Segment Anything Model (SAM), improves prompt learning and anomaly segmentation by treating SAM's masks as structures, drawing features within the same mask closer and pushing others apart. Comprehensive experiments on the MVTec AD and VisA datasets demonstrate that UCAD outperforms existing methods by 15.6% in detection and 26.6% in segmentation, even with rehearsal training. The code for UCAD is available at https://github.com/shirowalker/UCAD.