This paper explores in-context learning (ICL) with long-context models, examining how performance scales with the number of demonstrations provided. The study shows that for many datasets with large label spaces, performance continues to improve with hundreds or thousands of demonstrations. In contrast, example retrieval and fine-tuning show diminishing returns with more demonstrations, and fine-tuning can sometimes outperform long-context ICL with additional data. The research highlights that long-context ICL is less sensitive to random input shuffling and that grouping same-label examples can negatively impact performance. The performance gains are not due to cumulative encoding of many examples but rather from retrieving more relevant examples. The study concludes that long-context ICL is effective, but much of its gain comes from attending to similar examples rather than task learning. The paper compares ICL with retrieval and fine-tuning, showing that ICL can be a strong alternative in some data regimes. The experiments use various models and datasets, demonstrating that ICL can approach or exceed the performance of fine-tuned models. The study also finds that long-context ICL is less affected by example order and that performance can be improved by sorting examples by label. The paper suggests that long-context ICL is effective due to retrieval from relevant examples rather than task learning. The results show that ICL can be a viable alternative to fine-tuning, especially when efficiency is a priority. The study also highlights the importance of understanding the underlying mechanisms of ICL and the need for further research on its behavior at larger scales.This paper explores in-context learning (ICL) with long-context models, examining how performance scales with the number of demonstrations provided. The study shows that for many datasets with large label spaces, performance continues to improve with hundreds or thousands of demonstrations. In contrast, example retrieval and fine-tuning show diminishing returns with more demonstrations, and fine-tuning can sometimes outperform long-context ICL with additional data. The research highlights that long-context ICL is less sensitive to random input shuffling and that grouping same-label examples can negatively impact performance. The performance gains are not due to cumulative encoding of many examples but rather from retrieving more relevant examples. The study concludes that long-context ICL is effective, but much of its gain comes from attending to similar examples rather than task learning. The paper compares ICL with retrieval and fine-tuning, showing that ICL can be a strong alternative in some data regimes. The experiments use various models and datasets, demonstrating that ICL can approach or exceed the performance of fine-tuned models. The study also finds that long-context ICL is less affected by example order and that performance can be improved by sorting examples by label. The paper suggests that long-context ICL is effective due to retrieval from relevant examples rather than task learning. The results show that ICL can be a viable alternative to fine-tuning, especially when efficiency is a priority. The study also highlights the importance of understanding the underlying mechanisms of ICL and the need for further research on its behavior at larger scales.