This paper investigates how large language models (LLMs) encode context knowledge layer by layer through probing tasks. The study uses ChatGPT to generate diverse and coherent evidence for various facts and employs ν-usable information as a validation metric to assess the encoding capability across different layers. The experiments reveal that LLMs prefer to encode more context knowledge in upper layers, primarily within knowledge-related entity tokens at lower layers, and progressively expand knowledge in other tokens at upper layers. Additionally, LLMs gradually forget earlier context knowledge when provided with irrelevant evidence. The study also shows that LLMs encode newly acquired knowledge in knowledge-related entity tokens and transfer it to other tokens through the attention mechanism. However, the intermediate layers show performance degradation when faced with irrelevant evidence, indicating that LLMs do not encode irrelevant information orthogonally, leading to interference with previously encoded knowledge. The findings suggest that LLMs encode context knowledge more effectively in upper layers and that the layer-wise encoding capability varies depending on the type of knowledge. The study also highlights the importance of using ν-usable information as a more effective metric than test set accuracy for evaluating the encoding of context knowledge across layers. The results provide insights into the internal mechanisms of LLMs and their ability to encode and retain context knowledge. The study contributes to the understanding of how LLMs process and retain knowledge, and it highlights the need for further research into the long-term memory capabilities of LLMs. The paper also discusses the ethical considerations of using LLMs for encoding context knowledge, including the potential risks of generating misleading evidence and the need for stricter scrutiny of LLM applications. The study provides a comprehensive analysis of how LLMs encode context knowledge layer by layer and offers valuable insights into the inner workings of these models.This paper investigates how large language models (LLMs) encode context knowledge layer by layer through probing tasks. The study uses ChatGPT to generate diverse and coherent evidence for various facts and employs ν-usable information as a validation metric to assess the encoding capability across different layers. The experiments reveal that LLMs prefer to encode more context knowledge in upper layers, primarily within knowledge-related entity tokens at lower layers, and progressively expand knowledge in other tokens at upper layers. Additionally, LLMs gradually forget earlier context knowledge when provided with irrelevant evidence. The study also shows that LLMs encode newly acquired knowledge in knowledge-related entity tokens and transfer it to other tokens through the attention mechanism. However, the intermediate layers show performance degradation when faced with irrelevant evidence, indicating that LLMs do not encode irrelevant information orthogonally, leading to interference with previously encoded knowledge. The findings suggest that LLMs encode context knowledge more effectively in upper layers and that the layer-wise encoding capability varies depending on the type of knowledge. The study also highlights the importance of using ν-usable information as a more effective metric than test set accuracy for evaluating the encoding of context knowledge across layers. The results provide insights into the internal mechanisms of LLMs and their ability to encode and retain context knowledge. The study contributes to the understanding of how LLMs process and retain knowledge, and it highlights the need for further research into the long-term memory capabilities of LLMs. The paper also discusses the ethical considerations of using LLMs for encoding context knowledge, including the potential risks of generating misleading evidence and the need for stricter scrutiny of LLM applications. The study provides a comprehensive analysis of how LLMs encode context knowledge layer by layer and offers valuable insights into the inner workings of these models.