MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution

MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution

15 Apr 2024 | Yuxuan Jiang, Chen Feng, Fan Zhang, and David Bull
The paper introduces a novel framework called Multi-Teacher Knowledge Distillation (MTKD) for image super-resolution (ISR). MTKD leverages multiple teacher models to enhance the output of a compact student network, improving the efficiency and diversity of transferred knowledge. The framework consists of two stages: knowledge aggregation and model distillation. In the knowledge aggregation stage, a Discrete Cosine Transform Swin Transformer (DCTSwin) network combines the outputs of multiple ISR teacher models to generate an enhanced high-resolution image. In the model distillation stage, the student model is trained using a new wavelet-based loss function that optimizes the training process by observing differences in both the spatial and frequency domains. The effectiveness of MTKD is evaluated through quantitative and qualitative comparisons with five existing KD methods on three popular network architectures. Results show that MTKD achieves significant improvements in super-resolution performance, up to 0.46dB (PSNR) over state-of-the-art KD approaches. The source code for MTKD is available for public evaluation.The paper introduces a novel framework called Multi-Teacher Knowledge Distillation (MTKD) for image super-resolution (ISR). MTKD leverages multiple teacher models to enhance the output of a compact student network, improving the efficiency and diversity of transferred knowledge. The framework consists of two stages: knowledge aggregation and model distillation. In the knowledge aggregation stage, a Discrete Cosine Transform Swin Transformer (DCTSwin) network combines the outputs of multiple ISR teacher models to generate an enhanced high-resolution image. In the model distillation stage, the student model is trained using a new wavelet-based loss function that optimizes the training process by observing differences in both the spatial and frequency domains. The effectiveness of MTKD is evaluated through quantitative and qualitative comparisons with five existing KD methods on three popular network architectures. Results show that MTKD achieves significant improvements in super-resolution performance, up to 0.46dB (PSNR) over state-of-the-art KD approaches. The source code for MTKD is available for public evaluation.
Reach us at info@study.space