This paper investigates the representation learning capabilities of Denoising Diffusion Models (DDM) originally designed for image generation. The authors deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE), to explore how various components of modern DDMs influence self-supervised representation learning. They find that only a few components are critical for learning good representations, while many others are nonessential. Their study ultimately arrives at a highly simplified approach that resembles a classical DAE. The key finding is that a low-dimensional latent space, rather than specific tokenizer details, enables a DAE to achieve good representations. The authors also discover that the representation capability of DDMs is mainly gained by the denoising-driven process, not a diffusion-driven process. They compare their results with previous baselines and find that their approach achieves competitive self-supervised learning performance. The study suggests that there is more room for further research along the direction of DAE and DDM. The authors also show that the representation learning ability of DAEs is largely independent of their reliance on data augmentation. The study concludes that l-DAE, which largely resembles the classical DAE, can perform competitively in self-supervised learning. The critical component is a low-dimensional latent space on which noise is added. The authors hope their study will reignite interest in denoising-based methods in the context of today's self-supervised learning research.This paper investigates the representation learning capabilities of Denoising Diffusion Models (DDM) originally designed for image generation. The authors deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE), to explore how various components of modern DDMs influence self-supervised representation learning. They find that only a few components are critical for learning good representations, while many others are nonessential. Their study ultimately arrives at a highly simplified approach that resembles a classical DAE. The key finding is that a low-dimensional latent space, rather than specific tokenizer details, enables a DAE to achieve good representations. The authors also discover that the representation capability of DDMs is mainly gained by the denoising-driven process, not a diffusion-driven process. They compare their results with previous baselines and find that their approach achieves competitive self-supervised learning performance. The study suggests that there is more room for further research along the direction of DAE and DDM. The authors also show that the representation learning ability of DAEs is largely independent of their reliance on data augmentation. The study concludes that l-DAE, which largely resembles the classical DAE, can perform competitively in self-supervised learning. The critical component is a low-dimensional latent space on which noise is added. The authors hope their study will reignite interest in denoising-based methods in the context of today's self-supervised learning research.