23 Jun 2024 | Keqin Peng1, Liang Ding2*, Yancheng Yuan3*, Xuebo Liu1, Min Zhang1, Yuanxin Ouyang1, Dacheng Tao5
This paper revisits the factors influencing demonstration selection in in-context learning (ICL) from a model perspective, finding that the choice of demonstrations is both data- and model-dependent. The authors propose a conjecture that effective demonstrations positively correlate with their contribution to the model's understanding of test samples. Based on this conjecture, they develop a data- and model-dependent demonstration selection method called TopK + ConE. Empirical results show that TopK + ConE consistently improves performance in language understanding and generation tasks across different model scales. Further analyses confirm the method's universality and robustness, providing a unified explanation for the effectiveness of previous demonstration selection methods. The code for the method is publicly available.This paper revisits the factors influencing demonstration selection in in-context learning (ICL) from a model perspective, finding that the choice of demonstrations is both data- and model-dependent. The authors propose a conjecture that effective demonstrations positively correlate with their contribution to the model's understanding of test samples. Based on this conjecture, they develop a data- and model-dependent demonstration selection method called TopK + ConE. Empirical results show that TopK + ConE consistently improves performance in language understanding and generation tasks across different model scales. Further analyses confirm the method's universality and robustness, providing a unified explanation for the effectiveness of previous demonstration selection methods. The code for the method is publicly available.