This paper introduces an efficient diffusion model for image restoration (IR) that significantly reduces the number of diffusion steps required during inference, thereby improving both speed and performance. The proposed method establishes a Markov chain that facilitates transitions between high-quality (HQ) and low-quality (LQ) images by shifting their residuals, avoiding the need for post-acceleration and the associated performance degradation. A carefully designed noise schedule controls the shifting speed and noise strength, enhancing the flexibility and efficiency of the model. Extensive experiments on four classical IR tasks—image super-resolution, image inpainting, blind face restoration, and image deblurring—show that the proposed method achieves superior or comparable performance with only four sampling steps, outperforming or matching current state-of-the-art methods. The code and model are publicly available at <https://github.com/zsyOAOA/ResShift>.This paper introduces an efficient diffusion model for image restoration (IR) that significantly reduces the number of diffusion steps required during inference, thereby improving both speed and performance. The proposed method establishes a Markov chain that facilitates transitions between high-quality (HQ) and low-quality (LQ) images by shifting their residuals, avoiding the need for post-acceleration and the associated performance degradation. A carefully designed noise schedule controls the shifting speed and noise strength, enhancing the flexibility and efficiency of the model. Extensive experiments on four classical IR tasks—image super-resolution, image inpainting, blind face restoration, and image deblurring—show that the proposed method achieves superior or comparable performance with only four sampling steps, outperforming or matching current state-of-the-art methods. The code and model are publicly available at <https://github.com/zsyOAOA/ResShift>.