This paper introduces Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for in-context learning (ICL) in large language models (LLMs). By treating ICL as a meta-optimization process, the authors explain why LLMs are sensitive to the order of ICL examples. Batch-ICL processes $N$ separate 1-shot forward computations and aggregates the resulting meta-gradients, which are then applied to a zero-shot query to generate predictions. This approach reduces the randomness of meta-optimization and improves performance compared to standard ICL, often outperforming the best order of examples. The paper also introduces a multi-epoch variant of Batch-ICL, which further enhances performance by implicitly exploring permutations of ICL examples. Extensive experiments demonstrate that Batch-ICL consistently outperforms standard ICL in various tasks, reducing computational resources while maintaining or improving accuracy. The method is robust to the number of examples and the order of ICL examples, making it a promising solution for improving ICL performance.This paper introduces Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for in-context learning (ICL) in large language models (LLMs). By treating ICL as a meta-optimization process, the authors explain why LLMs are sensitive to the order of ICL examples. Batch-ICL processes $N$ separate 1-shot forward computations and aggregates the resulting meta-gradients, which are then applied to a zero-shot query to generate predictions. This approach reduces the randomness of meta-optimization and improves performance compared to standard ICL, often outperforming the best order of examples. The paper also introduces a multi-epoch variant of Batch-ICL, which further enhances performance by implicitly exploring permutations of ICL examples. Extensive experiments demonstrate that Batch-ICL consistently outperforms standard ICL in various tasks, reducing computational resources while maintaining or improving accuracy. The method is robust to the number of examples and the order of ICL examples, making it a promising solution for improving ICL performance.