This paper investigates the limitations of large language models (LLMs) in handling multi-hop queries, focusing on how they internally process information. The study reveals that LLMs resolve the first hop of a two-hop query in early layers, while the second hop is resolved in later layers. This sequential processing can lead to failures in answering the full query, as the later layers may lack the necessary knowledge to correctly predict the answer. To address this, the authors propose a novel "back-patching" method, where a hidden representation from a later layer is patched back into an earlier layer. This method successfully corrects up to 57% of previously incorrect cases, demonstrating that later layers may indeed lack the required functionality for certain tasks. The study introduces a dataset of 82,020 two-hop queries based on Wikidata, and analyzes how LLMs process these queries. The results show that the first hop is resolved in early layers, and the information propagates to the last token to resolve the second hop. However, if the first hop is resolved too late, the second hop may fail. The back-patching method allows the model to access earlier layers, enabling it to correctly answer the query. The findings suggest that the transformer architecture has inherent limitations, where information may not be fully resolved before it is used for subsequent steps. This could explain why some multi-hop queries are answered incorrectly by LLMs. The back-patching method provides a promising approach to improve the performance of LLMs on multi-hop question answering by allowing the model to access earlier layers and correct its computations. The study also highlights the importance of understanding the internal mechanisms of LLMs to improve their reasoning capabilities.This paper investigates the limitations of large language models (LLMs) in handling multi-hop queries, focusing on how they internally process information. The study reveals that LLMs resolve the first hop of a two-hop query in early layers, while the second hop is resolved in later layers. This sequential processing can lead to failures in answering the full query, as the later layers may lack the necessary knowledge to correctly predict the answer. To address this, the authors propose a novel "back-patching" method, where a hidden representation from a later layer is patched back into an earlier layer. This method successfully corrects up to 57% of previously incorrect cases, demonstrating that later layers may indeed lack the required functionality for certain tasks. The study introduces a dataset of 82,020 two-hop queries based on Wikidata, and analyzes how LLMs process these queries. The results show that the first hop is resolved in early layers, and the information propagates to the last token to resolve the second hop. However, if the first hop is resolved too late, the second hop may fail. The back-patching method allows the model to access earlier layers, enabling it to correctly answer the query. The findings suggest that the transformer architecture has inherent limitations, where information may not be fully resolved before it is used for subsequent steps. This could explain why some multi-hop queries are answered incorrectly by LLMs. The back-patching method provides a promising approach to improve the performance of LLMs on multi-hop question answering by allowing the model to access earlier layers and correct its computations. The study also highlights the importance of understanding the internal mechanisms of LLMs to improve their reasoning capabilities.