This paper presents an information-theoretic analysis of in-context learning (ICL) in transformers. The authors introduce new tools that decompose error into three components: irreducible error, meta-learning error, and intra-task error. These tools provide a general framework for analyzing meta-learning from sequences, unifying various theoretical results. The decomposition allows for a clear understanding of how error decays with the number of training sequences and sequence lengths. The authors apply these tools to establish new results about ICL in transformers, showing that error decays linearly with both the number of sequences and their lengths. This is a significant improvement over previous results that relied on restrictive assumptions. The analysis also provides a theoretical foundation for understanding how ICL is possible with only a small amount of in-context data. The results are applied to a sparse mixture of transformers, demonstrating how the model can learn from a small number of examples. The paper also connects ICL to meta-learning from sequences, showing that the error in ICL can be decomposed into meta-estimation and intra-document estimation errors. The results are general and do not rely on assumptions about mixing times or stability. The authors conclude that their framework provides a novel and effective way to analyze ICL and other meta-learning tasks.This paper presents an information-theoretic analysis of in-context learning (ICL) in transformers. The authors introduce new tools that decompose error into three components: irreducible error, meta-learning error, and intra-task error. These tools provide a general framework for analyzing meta-learning from sequences, unifying various theoretical results. The decomposition allows for a clear understanding of how error decays with the number of training sequences and sequence lengths. The authors apply these tools to establish new results about ICL in transformers, showing that error decays linearly with both the number of sequences and their lengths. This is a significant improvement over previous results that relied on restrictive assumptions. The analysis also provides a theoretical foundation for understanding how ICL is possible with only a small amount of in-context data. The results are applied to a sparse mixture of transformers, demonstrating how the model can learn from a small number of examples. The paper also connects ICL to meta-learning from sequences, showing that the error in ICL can be decomposed into meta-estimation and intra-document estimation errors. The results are general and do not rely on assumptions about mixing times or stability. The authors conclude that their framework provides a novel and effective way to analyze ICL and other meta-learning tasks.