Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

2024-03-31 | Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, Yang Gao
The paper "Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation" addresses the limitations of existing distribution matching-based methods in dataset distillation, which often suffer from insufficient class discrimination and incomplete feature distribution matching. To improve these issues, the authors propose two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by clustering samples within the same class, while the covariance matching constraint focuses on achieving more accurate feature distribution matching through local feature covariance matrices, even with limited sample sizes. Experiments on various datasets (SVHN, CIFAR10, CIFAR100, and TinyImageNet) demonstrate significant performance improvements, with up to 6.6% accuracy boost on CIFAR10. The method also shows robust performance in cross-architecture settings, maintaining minimal performance drop across different architectures. The contributions of the work include the introduction of these constraints, which significantly enhance the performance of dataset distillation methods.The paper "Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation" addresses the limitations of existing distribution matching-based methods in dataset distillation, which often suffer from insufficient class discrimination and incomplete feature distribution matching. To improve these issues, the authors propose two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by clustering samples within the same class, while the covariance matching constraint focuses on achieving more accurate feature distribution matching through local feature covariance matrices, even with limited sample sizes. Experiments on various datasets (SVHN, CIFAR10, CIFAR100, and TinyImageNet) demonstrate significant performance improvements, with up to 6.6% accuracy boost on CIFAR10. The method also shows robust performance in cross-architecture settings, maintaining minimal performance drop across different architectures. The contributions of the work include the introduction of these constraints, which significantly enhance the performance of dataset distillation methods.
Reach us at info@study.space
Understanding Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation