This paper proposes a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. The approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Experiments on medical anomaly detection benchmarks demonstrate that the method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, and 2.03% and 2.37% for anomaly segmentation under zero-shot and few-shot settings, respectively. The framework includes a multi-level feature adaptation and comparison architecture, which enables the model to effectively discern both global and local anomalies through visual-language feature alignment. The method is evaluated on a challenging medical AD benchmark, encompassing datasets from five distinct medical modalities and anatomical regions, and shows superior performance compared to existing methods. The results highlight the effectiveness of the proposed multi-level feature adaptation approach in enhancing the model's generalization capabilities for medical anomaly detection.This paper proposes a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. The approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Experiments on medical anomaly detection benchmarks demonstrate that the method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, and 2.03% and 2.37% for anomaly segmentation under zero-shot and few-shot settings, respectively. The framework includes a multi-level feature adaptation and comparison architecture, which enables the model to effectively discern both global and local anomalies through visual-language feature alignment. The method is evaluated on a challenging medical AD benchmark, encompassing datasets from five distinct medical modalities and anatomical regions, and shows superior performance compared to existing methods. The results highlight the effectiveness of the proposed multi-level feature adaptation approach in enhancing the model's generalization capabilities for medical anomaly detection.