Distilling Semantic Priors from SAM to Efficient Image Restoration Models

Distilling Semantic Priors from SAM to Efficient Image Restoration Models

2 Apr 2024 | Quan Zhang, Xiaoyu Liu, Wei Li, Hanting Chen, Junchao Liu, Jie Hu, Zhiwei Xiong, Chun Yuan, Yunhe Wang
The paper "Distilling Semantic Priors from SAM to Efficient Image Restoration Models" addresses the challenge of leveraging semantic priors from segmentation models, particularly the Segment Anything Model (SAM), to enhance image restoration (IR) tasks without increasing computational complexity. The authors propose a framework that distills SAM's semantic knowledge into existing IR models, ensuring efficient inference. The framework consists of two main components: the Semantic Priors Fusion (SPF) scheme and the Semantic Priors Distillation (SPD) scheme. SPF fuses information from the restored image predicted by the IR model and the semantic mask predicted by SAM, while SPD uses self-distillation to transfer the fused semantic priors to the IR model. Additionally, a Semantic-Guided Relation (SGR) module ensures consistency in the semantic feature representation space. The effectiveness of the proposed framework is demonstrated through experiments on multiple IR tasks, including deraining, deblurring, and denoising, showing significant performance improvements while maintaining inference efficiency.The paper "Distilling Semantic Priors from SAM to Efficient Image Restoration Models" addresses the challenge of leveraging semantic priors from segmentation models, particularly the Segment Anything Model (SAM), to enhance image restoration (IR) tasks without increasing computational complexity. The authors propose a framework that distills SAM's semantic knowledge into existing IR models, ensuring efficient inference. The framework consists of two main components: the Semantic Priors Fusion (SPF) scheme and the Semantic Priors Distillation (SPD) scheme. SPF fuses information from the restored image predicted by the IR model and the semantic mask predicted by SAM, while SPD uses self-distillation to transfer the fused semantic priors to the IR model. Additionally, a Semantic-Guided Relation (SGR) module ensures consistency in the semantic feature representation space. The effectiveness of the proposed framework is demonstrated through experiments on multiple IR tasks, including deraining, deblurring, and denoising, showing significant performance improvements while maintaining inference efficiency.
Reach us at info@study.space
Understanding Distilling Semantic Priors from SAM to Efficient Image Restoration Models