This paper explores the feasibility of distinguishing between epistemic uncertainty (lack of knowledge) and aleatoric uncertainty (inherent randomness in data) in the outputs of large language models (LLMs). The study focuses on identifying tokens where the model's uncertainty is primarily epistemic, which could help improve the accuracy of LLM-generated text by reducing hallucinations. The researchers propose a method to approximate the uncertainty of a smaller model by using a larger model as a proxy for the true distribution. They show that small linear probes trained on the embeddings of frozen, pretrained models can accurately predict when larger models will be more confident at the token level. Additionally, they propose a fully unsupervised method, called the In-Context Learning Test (ICLT), which leverages the in-context learning capabilities of LLMs to estimate uncertainty without requiring labeled data. The results suggest that LLMs naturally contain internal representations of different types of uncertainty that could be used to develop more informative indicators of model confidence in various practical settings. The study also highlights the importance of distinguishing between epistemic and aleatoric uncertainty in improving the reliability of LLM outputs.This paper explores the feasibility of distinguishing between epistemic uncertainty (lack of knowledge) and aleatoric uncertainty (inherent randomness in data) in the outputs of large language models (LLMs). The study focuses on identifying tokens where the model's uncertainty is primarily epistemic, which could help improve the accuracy of LLM-generated text by reducing hallucinations. The researchers propose a method to approximate the uncertainty of a smaller model by using a larger model as a proxy for the true distribution. They show that small linear probes trained on the embeddings of frozen, pretrained models can accurately predict when larger models will be more confident at the token level. Additionally, they propose a fully unsupervised method, called the In-Context Learning Test (ICLT), which leverages the in-context learning capabilities of LLMs to estimate uncertainty without requiring labeled data. The results suggest that LLMs naturally contain internal representations of different types of uncertainty that could be used to develop more informative indicators of model confidence in various practical settings. The study also highlights the importance of distinguishing between epistemic and aleatoric uncertainty in improving the reliability of LLM outputs.