11 Jul 2024 | Brian B. Moser, Federico Raue, Sebastian Palacio, Stanislav Frolov, Andreas Dengel
This paper introduces Latent Dataset Distillation with Diffusion Models (LD3M), a novel method for generating high-quality synthetic images for dataset distillation. LD3M leverages diffusion models to distill large datasets into a smaller set of synthetic samples, improving the generalization of the distillation process across different architectures and high-resolution images. The key innovation of LD3M is its use of latent space representations, which are more efficient and effective than pixel-level representations for generating synthetic images. LD3M avoids the need for expensive GAN inversion techniques and instead uses a pre-trained diffusion model to generate synthetic images. The method also allows for fine-tuning the number of diffusion steps to balance the trade-off between distillation speed and dataset quality. LD3M outperforms state-of-the-art methods like GLaD by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively, on several ImageNet subsets and high resolutions (128x128 and 256x256). The method is also more computationally efficient and allows for easier extension to new diffusion models. LD3M is compatible with any distillation algorithm and can be used out of the box with existing dataset distillation methods. The paper evaluates LD3M on various image sizes and architectures, showing that it consistently outperforms GLaD in terms of accuracy and efficiency. The results demonstrate that LD3M is a promising approach for dataset distillation, offering a flexible and efficient alternative to traditional methods.This paper introduces Latent Dataset Distillation with Diffusion Models (LD3M), a novel method for generating high-quality synthetic images for dataset distillation. LD3M leverages diffusion models to distill large datasets into a smaller set of synthetic samples, improving the generalization of the distillation process across different architectures and high-resolution images. The key innovation of LD3M is its use of latent space representations, which are more efficient and effective than pixel-level representations for generating synthetic images. LD3M avoids the need for expensive GAN inversion techniques and instead uses a pre-trained diffusion model to generate synthetic images. The method also allows for fine-tuning the number of diffusion steps to balance the trade-off between distillation speed and dataset quality. LD3M outperforms state-of-the-art methods like GLaD by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively, on several ImageNet subsets and high resolutions (128x128 and 256x256). The method is also more computationally efficient and allows for easier extension to new diffusion models. LD3M is compatible with any distillation algorithm and can be used out of the box with existing dataset distillation methods. The paper evaluates LD3M on various image sizes and architectures, showing that it consistently outperforms GLaD in terms of accuracy and efficiency. The results demonstrate that LD3M is a promising approach for dataset distillation, offering a flexible and efficient alternative to traditional methods.