This paper investigates the layer-wise capability of large language models (LLMs) in encoding context knowledge through probing tasks. The authors leverage the generative capabilities of ChatGPT to construct diverse and coherent evidence corresponding to various facts, which are then used to probe the LLMs. The $\forall$-usable information metric is employed to evaluate the LLMs' ability to encode context knowledge across different layers. Experiments on conflicting and newly acquired knowledge show that LLMs prefer to encode more context knowledge in upper layers, primarily within knowledge-related entity tokens at lower layers, and gradually forget earlier context knowledge when provided with irrelevant evidence. The findings provide insights into the internal mechanisms of LLMs and highlight the importance of understanding how they handle context knowledge at different layers.This paper investigates the layer-wise capability of large language models (LLMs) in encoding context knowledge through probing tasks. The authors leverage the generative capabilities of ChatGPT to construct diverse and coherent evidence corresponding to various facts, which are then used to probe the LLMs. The $\forall$-usable information metric is employed to evaluate the LLMs' ability to encode context knowledge across different layers. Experiments on conflicting and newly acquired knowledge show that LLMs prefer to encode more context knowledge in upper layers, primarily within knowledge-related entity tokens at lower layers, and gradually forget earlier context knowledge when provided with irrelevant evidence. The findings provide insights into the internal mechanisms of LLMs and highlight the importance of understanding how they handle context knowledge at different layers.