This paper addresses the challenge of assessing uncertainty in Language Models (LMs), particularly in natural language generation tasks. Existing methods for quantifying uncertainty, such as semantic entropy and affinity-graph-based measures, often differ in their output ranges and are difficult to compare. The authors propose a novel framework called Rank-Calibration to evaluate the quality of uncertainty and confidence measures for LMs. The key idea is that lower uncertainty should correspond to higher generation quality, and the framework quantifies deviations from this ideal relationship without requiring arbitrary binary thresholds. The paper introduces the Rank-Calibration Error (RCE) as a metric to assess the deviation of uncertainty measures from the desired monotonic relationship between uncertainty and correctness. The RCE is calculated by comparing the probability of correctness given an uncertainty level with the probability of the uncertainty level being less than or equal to a given value. The authors demonstrate that RCE provides a more principled and flexible assessment compared to existing metrics like Expected Calibration Error (ECE), which are often limited to binary correctness values. The authors also introduce indication diagrams to visualize the deviation of uncertainty measures from the ideal rank-calibration. These diagrams show the relationship between the relative percentile of uncertainty and the expected correctness level. A well-calibrated measure should lie along the anti-diagonal line, indicating that lower uncertainty corresponds to higher correctness. The paper evaluates the proposed framework on various datasets, including TriviaQA and Meadow, and compares the performance of different uncertainty measures. The results show that the RCE provides a more accurate and reliable assessment of uncertainty measures compared to traditional metrics. The framework is also shown to be applicable to a wide range of uncertainty measures, including those with different output ranges. The authors conclude that the Rank-Calibration framework provides a more principled and flexible approach to assessing uncertainty in LMs. The framework is practical, interpretable, and applicable to a wide range of uncertainty measures, making it a valuable tool for evaluating the reliability of LMs in natural language generation tasks.This paper addresses the challenge of assessing uncertainty in Language Models (LMs), particularly in natural language generation tasks. Existing methods for quantifying uncertainty, such as semantic entropy and affinity-graph-based measures, often differ in their output ranges and are difficult to compare. The authors propose a novel framework called Rank-Calibration to evaluate the quality of uncertainty and confidence measures for LMs. The key idea is that lower uncertainty should correspond to higher generation quality, and the framework quantifies deviations from this ideal relationship without requiring arbitrary binary thresholds. The paper introduces the Rank-Calibration Error (RCE) as a metric to assess the deviation of uncertainty measures from the desired monotonic relationship between uncertainty and correctness. The RCE is calculated by comparing the probability of correctness given an uncertainty level with the probability of the uncertainty level being less than or equal to a given value. The authors demonstrate that RCE provides a more principled and flexible assessment compared to existing metrics like Expected Calibration Error (ECE), which are often limited to binary correctness values. The authors also introduce indication diagrams to visualize the deviation of uncertainty measures from the ideal rank-calibration. These diagrams show the relationship between the relative percentile of uncertainty and the expected correctness level. A well-calibrated measure should lie along the anti-diagonal line, indicating that lower uncertainty corresponds to higher correctness. The paper evaluates the proposed framework on various datasets, including TriviaQA and Meadow, and compares the performance of different uncertainty measures. The results show that the RCE provides a more accurate and reliable assessment of uncertainty measures compared to traditional metrics. The framework is also shown to be applicable to a wide range of uncertainty measures, including those with different output ranges. The authors conclude that the Rank-Calibration framework provides a more principled and flexible approach to assessing uncertainty in LMs. The framework is practical, interpretable, and applicable to a wide range of uncertainty measures, making it a valuable tool for evaluating the reliability of LMs in natural language generation tasks.