This paper investigates and addresses the issue of compositional reasoning failures in large language models (LLMs). Compositional reasoning is crucial for learning systems to break down complex tasks into manageable subtasks and solve them step-by-step. However, LLMs struggle with basic compositional reasoning tasks, as highlighted by the "compositionality gap" in question-answering tasks. The authors identify that these failures are often due to improper generation or use of implicit reasoning results. Using Logit Lens and intervention experiments, they uncover that implicit reasoning results emerge in middle layers of LLMs and play a causal role in generating explicit reasoning results. They further locate multi-head self-attention (MHSA) modules in these layers, which are critical for accurate generation and use of implicit reasoning results. Based on these findings, the authors develop CREME, a lightweight method to correct compositional reasoning errors by editing the located MHSA modules. CREME is effective in correcting errors in compositional reasoning, not only for the query used for editing but also for paraphrased queries and other compositional queries sharing the first-hop knowledge. The method is evaluated on a dataset of compositional two-hop knowledge queries and shows significant improvements in correction, paraphrasing, generalization, and specificity. CREME is compared with other methods such as Memory Injection and CoT-PatchScopes, and it outperforms them in most metrics. The results demonstrate that CREME effectively corrects compositional reasoning failures by properly generating and leveraging implicit reasoning results in the middle layers of LLMs. The study also highlights the importance of understanding the inner workings of LLMs to improve their compositional reasoning capabilities.This paper investigates and addresses the issue of compositional reasoning failures in large language models (LLMs). Compositional reasoning is crucial for learning systems to break down complex tasks into manageable subtasks and solve them step-by-step. However, LLMs struggle with basic compositional reasoning tasks, as highlighted by the "compositionality gap" in question-answering tasks. The authors identify that these failures are often due to improper generation or use of implicit reasoning results. Using Logit Lens and intervention experiments, they uncover that implicit reasoning results emerge in middle layers of LLMs and play a causal role in generating explicit reasoning results. They further locate multi-head self-attention (MHSA) modules in these layers, which are critical for accurate generation and use of implicit reasoning results. Based on these findings, the authors develop CREME, a lightweight method to correct compositional reasoning errors by editing the located MHSA modules. CREME is effective in correcting errors in compositional reasoning, not only for the query used for editing but also for paraphrased queries and other compositional queries sharing the first-hop knowledge. The method is evaluated on a dataset of compositional two-hop knowledge queries and shows significant improvements in correction, paraphrasing, generalization, and specificity. CREME is compared with other methods such as Memory Injection and CoT-PatchScopes, and it outperforms them in most metrics. The results demonstrate that CREME effectively corrects compositional reasoning failures by properly generating and leveraging implicit reasoning results in the middle layers of LLMs. The study also highlights the importance of understanding the inner workings of LLMs to improve their compositional reasoning capabilities.