19 Jul 2024 | Yunpeng Qu, Kun Yuan, Kai Zhao, Qizhi Xie, Jinhua Hao, Ming Sun, Chao Zhou
The paper "XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution" addresses the challenge of accurately restoring semantic details in Image Super-Resolution (ISR) tasks. The authors propose a framework called *Cross-modal Priors for Super-Resolution (XPSR)*, which leverages Multimodal Large Language Models (MLLMs) to extract high-level and low-level semantic priors from low-resolution (LR) images. These priors are then used to guide diffusion models, enhancing the restoration quality and realism of high-resolution (HR) images.
Key contributions of XPSR include:
1. **Semantic Priors**: Utilizing MLLMs to obtain high-level and low-level semantic priors for LR images.
2. **Semantic-Fusion Attention (SFA)**: A module that fuses these priors with the diffusion model in a parallel cross-attention manner.
3. **Degradation-Free Constraint (DFC)**: A constraint applied between LR and HR images to extract semantic-preserved features while reducing the impact of degradations.
The authors conduct extensive experiments on both synthetic and real-world datasets, demonstrating that XPSR outperforms state-of-the-art (SOTA) methods in terms of image quality metrics and user studies. The results show that XPSR generates more realistic and detailed images, even under challenging degradation conditions. The paper also includes ablation studies to validate the effectiveness of each component of the XPSR framework.The paper "XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution" addresses the challenge of accurately restoring semantic details in Image Super-Resolution (ISR) tasks. The authors propose a framework called *Cross-modal Priors for Super-Resolution (XPSR)*, which leverages Multimodal Large Language Models (MLLMs) to extract high-level and low-level semantic priors from low-resolution (LR) images. These priors are then used to guide diffusion models, enhancing the restoration quality and realism of high-resolution (HR) images.
Key contributions of XPSR include:
1. **Semantic Priors**: Utilizing MLLMs to obtain high-level and low-level semantic priors for LR images.
2. **Semantic-Fusion Attention (SFA)**: A module that fuses these priors with the diffusion model in a parallel cross-attention manner.
3. **Degradation-Free Constraint (DFC)**: A constraint applied between LR and HR images to extract semantic-preserved features while reducing the impact of degradations.
The authors conduct extensive experiments on both synthetic and real-world datasets, demonstrating that XPSR outperforms state-of-the-art (SOTA) methods in terms of image quality metrics and user studies. The results show that XPSR generates more realistic and detailed images, even under challenging degradation conditions. The paper also includes ablation studies to validate the effectiveness of each component of the XPSR framework.