This paper addresses the issue of feature drifting in compressed images, which significantly affects the performance of high-level vision models. The authors propose a novel lightweight adaptive feature de-drifting module (AFD-Module) to enhance the classification performance of pre-trained models on compressed images. The AFD-Module consists of two sub-networks: the Feature Drifting Estimation Network (FDE-Net) and the Feature Enhancement Network (FE-Net). FDE-Net estimates the Feature Drifting Map (FDM) in the Discrete Cosine Transform (DCT) domain, while FE-Net uses the FDM to guide the feature de-drifting. The RepConv block, equipped with structural re-parameterization, enriches feature representation without increasing computational costs. The AFD-Module is trained on limited compressed images and can be directly integrated into existing classification networks. Experiments on ImageNet-C and multiple compression scenarios demonstrate that the proposed method significantly improves the accuracy of pre-trained classification models compared to existing JPEG Artifact Removal (JAR) methods and other feature enhancement techniques. The AFD-Module is also shown to be effective in handling multiple compression conditions and other lossy compression formats like AVIF.This paper addresses the issue of feature drifting in compressed images, which significantly affects the performance of high-level vision models. The authors propose a novel lightweight adaptive feature de-drifting module (AFD-Module) to enhance the classification performance of pre-trained models on compressed images. The AFD-Module consists of two sub-networks: the Feature Drifting Estimation Network (FDE-Net) and the Feature Enhancement Network (FE-Net). FDE-Net estimates the Feature Drifting Map (FDM) in the Discrete Cosine Transform (DCT) domain, while FE-Net uses the FDM to guide the feature de-drifting. The RepConv block, equipped with structural re-parameterization, enriches feature representation without increasing computational costs. The AFD-Module is trained on limited compressed images and can be directly integrated into existing classification networks. Experiments on ImageNet-C and multiple compression scenarios demonstrate that the proposed method significantly improves the accuracy of pre-trained classification models compared to existing JPEG Artifact Removal (JAR) methods and other feature enhancement techniques. The AFD-Module is also shown to be effective in handling multiple compression conditions and other lossy compression formats like AVIF.