This paper presents a theoretical analysis of the performance of transformers with softmax attention in in-context learning (ICL) with linear regression tasks. The study compares the performance of single-head and multi-head attention mechanisms. The results show that multi-head attention with a large embedding dimension outperforms single-head attention. As the number of in-context examples D increases, the prediction loss for both types of attention is O(1/D), but the multiplicative constant is smaller for multi-head attention. The analysis considers various scenarios, including noisy labels, local examples, correlated features, and prior knowledge. It is found that multi-head attention is generally preferred over single-head attention. The study also shows that multi-head attention improves the flexibility of the transformer and provides a better kernel for linear regression tasks. The results provide a comprehensive understanding of the impact of single- and multi-head attention on ICL performance and offer practical guidance for selecting efficient attention mechanisms in real-world applications. The paper concludes that multi-head attention is preferred over single-head attention, and the total number of embedding dimensions should be much larger than the number of heads. The analysis is based on a data generation model where examples and queries are i.i.d. samples from a noiseless regression model. The study uses a simplified neural network architecture and derives the optimal solution for single-head attention under certain assumptions. The results show that the optimal solution for single-head attention has a worse ICL performance than multi-head attention. The paper also discusses the implications of the findings for the design of transformer architectures.This paper presents a theoretical analysis of the performance of transformers with softmax attention in in-context learning (ICL) with linear regression tasks. The study compares the performance of single-head and multi-head attention mechanisms. The results show that multi-head attention with a large embedding dimension outperforms single-head attention. As the number of in-context examples D increases, the prediction loss for both types of attention is O(1/D), but the multiplicative constant is smaller for multi-head attention. The analysis considers various scenarios, including noisy labels, local examples, correlated features, and prior knowledge. It is found that multi-head attention is generally preferred over single-head attention. The study also shows that multi-head attention improves the flexibility of the transformer and provides a better kernel for linear regression tasks. The results provide a comprehensive understanding of the impact of single- and multi-head attention on ICL performance and offer practical guidance for selecting efficient attention mechanisms in real-world applications. The paper concludes that multi-head attention is preferred over single-head attention, and the total number of embedding dimensions should be much larger than the number of heads. The analysis is based on a data generation model where examples and queries are i.i.d. samples from a noiseless regression model. The study uses a simplified neural network architecture and derives the optimal solution for single-head attention under certain assumptions. The results show that the optimal solution for single-head attention has a worse ICL performance than multi-head attention. The paper also discusses the implications of the findings for the design of transformer architectures.