This paper proposes using multicalibration to generate interpretable and reliable confidence scores for outputs from large language models (LLMs). Multicalibration ensures calibration not just marginally but across various intersecting groupings of data. The authors develop techniques to form groupings for prompt/completion pairs that correlate with the probability of correctness, using clustering in an embedding space and self-annotation via yes-or-no questions. They also introduce novel multicalibration algorithms that improve performance by reducing overfitting. Systematic benchmarking across various question answering datasets and LLMs shows that their techniques yield confidence scores that significantly improve calibration and accuracy compared to existing methods. The paper discusses the challenges of hallucination detection in LLMs, where models often fabricate information. Multicalibration is used to produce calibrated probabilities indicating whether a generated response is a hallucination. Unlike conventional calibration, multicalibrated probabilities are self-consistent conditionally on various instance properties, making them more refined risk measures. The authors show how to apply multicalibration in the context of hallucination detection, including generating useful features for multicalibration through clustering and self-annotation. The paper introduces a new algorithm, Iterative Grouped Linear Binning (IGLB), which improves upon existing multicalibration methods by reducing overfitting. The algorithm uses a combination of binning strategies and linear scaling to enhance model performance. The authors also propose early stopping strategies to prevent overfitting and evaluate the effectiveness of their methods across various datasets and LLMs. Experiments show that multicalibration methods, particularly IGLB and GCULR, significantly outperform other calibration methods in terms of calibration and accuracy. The results demonstrate that multicalibration improves the reliability of confidence scores, making them more trustworthy for use in applications involving LLMs. The paper concludes that multicalibration is a promising approach for enhancing the reliability and interpretability of confidence scores in LLMs.This paper proposes using multicalibration to generate interpretable and reliable confidence scores for outputs from large language models (LLMs). Multicalibration ensures calibration not just marginally but across various intersecting groupings of data. The authors develop techniques to form groupings for prompt/completion pairs that correlate with the probability of correctness, using clustering in an embedding space and self-annotation via yes-or-no questions. They also introduce novel multicalibration algorithms that improve performance by reducing overfitting. Systematic benchmarking across various question answering datasets and LLMs shows that their techniques yield confidence scores that significantly improve calibration and accuracy compared to existing methods. The paper discusses the challenges of hallucination detection in LLMs, where models often fabricate information. Multicalibration is used to produce calibrated probabilities indicating whether a generated response is a hallucination. Unlike conventional calibration, multicalibrated probabilities are self-consistent conditionally on various instance properties, making them more refined risk measures. The authors show how to apply multicalibration in the context of hallucination detection, including generating useful features for multicalibration through clustering and self-annotation. The paper introduces a new algorithm, Iterative Grouped Linear Binning (IGLB), which improves upon existing multicalibration methods by reducing overfitting. The algorithm uses a combination of binning strategies and linear scaling to enhance model performance. The authors also propose early stopping strategies to prevent overfitting and evaluate the effectiveness of their methods across various datasets and LLMs. Experiments show that multicalibration methods, particularly IGLB and GCULR, significantly outperform other calibration methods in terms of calibration and accuracy. The results demonstrate that multicalibration improves the reliability of confidence scores, making them more trustworthy for use in applications involving LLMs. The paper concludes that multicalibration is a promising approach for enhancing the reliability and interpretability of confidence scores in LLMs.