Prompt Learning via Meta-Regularization

Prompt Learning via Meta-Regularization

1 Apr 2024 | Jinyoung Park, Juyeon Ko, Hyunwoo J. Kim*
This paper proposes ProMetaR, a prompt learning framework that improves the generalizability of prompt tuning methods for vision-language models (VLMs). The main challenge in prompt learning is task overfitting, where the pre-trained knowledge of VLMs is lost during fine-tuning on small datasets. ProMetaR addresses this by meta-learning both the regularizer and soft prompts, enabling the model to retain general knowledge while adapting to new tasks. Additionally, ProMetaR introduces task augmentation to generate diverse virtual tasks, reducing meta-overfitting. The framework also analyzes how ProMetaR enhances generalization through gradient alignment. ProMetaR is evaluated on 11 image recognition datasets and four variants of ImageNet. It outperforms existing prompt learning methods in both base-to-base and base-to-new generalization settings, as well as in domain generalization. The results show that ProMetaR significantly improves the performance of traditional prompt learning methods, especially in data-deficient settings. The framework is also shown to be effective in various prompting methods as a general training scheme. The key contributions of this work include: (1) proposing ProMetaR, a prompt learning framework that meta-learns both the regularizer and soft prompts, incorporating task augmentation for more effective meta-learning; (2) providing a theoretical analysis of how ProMetaR improves the generalizability of prompt learning approaches; and (3) demonstrating the effectiveness and robustness of ProMetaR under base-to-base/base-to-new settings and domain generalization. The code for ProMetaR is available at https://github.com/mlvlab/ProMetaR.This paper proposes ProMetaR, a prompt learning framework that improves the generalizability of prompt tuning methods for vision-language models (VLMs). The main challenge in prompt learning is task overfitting, where the pre-trained knowledge of VLMs is lost during fine-tuning on small datasets. ProMetaR addresses this by meta-learning both the regularizer and soft prompts, enabling the model to retain general knowledge while adapting to new tasks. Additionally, ProMetaR introduces task augmentation to generate diverse virtual tasks, reducing meta-overfitting. The framework also analyzes how ProMetaR enhances generalization through gradient alignment. ProMetaR is evaluated on 11 image recognition datasets and four variants of ImageNet. It outperforms existing prompt learning methods in both base-to-base and base-to-new generalization settings, as well as in domain generalization. The results show that ProMetaR significantly improves the performance of traditional prompt learning methods, especially in data-deficient settings. The framework is also shown to be effective in various prompting methods as a general training scheme. The key contributions of this work include: (1) proposing ProMetaR, a prompt learning framework that meta-learns both the regularizer and soft prompts, incorporating task augmentation for more effective meta-learning; (2) providing a theoretical analysis of how ProMetaR improves the generalizability of prompt learning approaches; and (3) demonstrating the effectiveness and robustness of ProMetaR under base-to-base/base-to-new settings and domain generalization. The code for ProMetaR is available at https://github.com/mlvlab/ProMetaR.
Reach us at info@study.space