This paper explores the role of demonstrations in in-context learning, a technique where large language models (LMs) perform new tasks by inferring from a few input-label pairs. The authors find that ground truth demonstrations are not essential for effective in-context learning, as replacing labels with random ones only slightly affects performance across 12 different models, including GPT-3. They identify key aspects of demonstrations that contribute to performance, such as the label space, input text distribution, and sequence format. The study reveals that models benefit from knowing the label space and input text distribution, but not necessarily the exact input-label mapping. Meta-training with an in-context learning objective further emphasizes the importance of these aspects. The findings suggest that LMs do not rely heavily on the ground truth input-label mapping and can still achieve good performance by exploiting other components of the demonstrations. The paper also discusses broader implications, including the capacity of LMs and the role of meta-training.This paper explores the role of demonstrations in in-context learning, a technique where large language models (LMs) perform new tasks by inferring from a few input-label pairs. The authors find that ground truth demonstrations are not essential for effective in-context learning, as replacing labels with random ones only slightly affects performance across 12 different models, including GPT-3. They identify key aspects of demonstrations that contribute to performance, such as the label space, input text distribution, and sequence format. The study reveals that models benefit from knowing the label space and input text distribution, but not necessarily the exact input-label mapping. Meta-training with an in-context learning objective further emphasizes the importance of these aspects. The findings suggest that LMs do not rely heavily on the ground truth input-label mapping and can still achieve good performance by exploiting other components of the demonstrations. The paper also discusses broader implications, including the capacity of LMs and the role of meta-training.