This paper proposes a lightweight adaptive feature de-drifting module (AFD-Module) to improve the performance of pre-trained image classification models on JPEG-compressed images. JPEG compression introduces artifacts that cause feature drifting, degrading the performance of high-level vision tasks. Existing JPEG artifact removal (JAR) methods are not suitable for pre-processing due to inefficiency and lack of adaptation to high-level vision models. The AFD-Module addresses these issues by estimating feature drifting using a Feature Drifting Estimation Network (FDE-Net) in the DCT domain and enhancing features with a Feature Enhancement Network (FE-Net). The FE-Net uses a RepConv block with structural reparameterization to achieve efficient feature enhancement. The AFD-Module is trained on limited compressed images and can be directly plugged into pre-trained models to improve their performance on compressed images. Experiments on ImageNet-C and multiple compression scenarios show that the AFD-Module significantly improves classification accuracy and outperforms existing methods. The module is also effective for multiple compressions and other lossy formats like AVIF. The AFD-Module is efficient, lightweight, and adaptable to various compression conditions, making it suitable for deployment on mobile devices. The results demonstrate that the AFD-Module can effectively correct spatially varied feature drifting and enhance feature representation for image classification.This paper proposes a lightweight adaptive feature de-drifting module (AFD-Module) to improve the performance of pre-trained image classification models on JPEG-compressed images. JPEG compression introduces artifacts that cause feature drifting, degrading the performance of high-level vision tasks. Existing JPEG artifact removal (JAR) methods are not suitable for pre-processing due to inefficiency and lack of adaptation to high-level vision models. The AFD-Module addresses these issues by estimating feature drifting using a Feature Drifting Estimation Network (FDE-Net) in the DCT domain and enhancing features with a Feature Enhancement Network (FE-Net). The FE-Net uses a RepConv block with structural reparameterization to achieve efficient feature enhancement. The AFD-Module is trained on limited compressed images and can be directly plugged into pre-trained models to improve their performance on compressed images. Experiments on ImageNet-C and multiple compression scenarios show that the AFD-Module significantly improves classification accuracy and outperforms existing methods. The module is also effective for multiple compressions and other lossy formats like AVIF. The AFD-Module is efficient, lightweight, and adaptable to various compression conditions, making it suitable for deployment on mobile devices. The results demonstrate that the AFD-Module can effectively correct spatially varied feature drifting and enhance feature representation for image classification.