This paper addresses the issue of quantifying uncertainty in language models (LMs) for natural language generation (NLG). While LMs have shown promising performance, they often generate incorrect or hallucinated responses, necessitating a reliable method to assess their uncertainty. Traditional confidence measures, such as ROUGE and METEOR, are binary and may not capture the nuanced nature of uncertainty in NLG tasks. To overcome these limitations, the authors propose a novel framework called Rank-Calibration (RC). RC quantifies the deviation from the ideal relationship where lower uncertainty implies higher generation quality. The key contributions of the paper include: 1. **Mathematical Formalization**: The authors formally define the assessment of uncertainty/confidence measures for NLG tasks, extending beyond binary correctness. 2. **Empirical Limitations**: They demonstrate that existing assessment metrics like AUROC, ECE, and others have limitations, including dependence on LM performance, instability due to ad hoc thresholding, and incompatibility with diverse output ranges. 3. **Rank-Calibration Error (RCE)**: They introduce RCE as a principled metric to assess the quality of uncertainty measures, which is not affected by thresholding and is applicable to measures with different output ranges. 4. **Empirical RCE and Indication Diagrams**: They propose methods to estimate RCE and visualize rank-miscalibration using indication diagrams, which help in understanding the performance of uncertainty measures. 5. **Experiments**: Comprehensive experiments on various LMs and datasets show the broad applicability and granular interpretability of the proposed method. The results also highlight the robustness of RCE to key hyperparameters and the effectiveness of rank-calibrated measures. The paper concludes by discussing the advantages of the rank-calibration framework and suggesting future directions, including developing uncertainty measures with guaranteed rank-calibration and enhancing LM generative pipelines with rank-calibrated measures.This paper addresses the issue of quantifying uncertainty in language models (LMs) for natural language generation (NLG). While LMs have shown promising performance, they often generate incorrect or hallucinated responses, necessitating a reliable method to assess their uncertainty. Traditional confidence measures, such as ROUGE and METEOR, are binary and may not capture the nuanced nature of uncertainty in NLG tasks. To overcome these limitations, the authors propose a novel framework called Rank-Calibration (RC). RC quantifies the deviation from the ideal relationship where lower uncertainty implies higher generation quality. The key contributions of the paper include: 1. **Mathematical Formalization**: The authors formally define the assessment of uncertainty/confidence measures for NLG tasks, extending beyond binary correctness. 2. **Empirical Limitations**: They demonstrate that existing assessment metrics like AUROC, ECE, and others have limitations, including dependence on LM performance, instability due to ad hoc thresholding, and incompatibility with diverse output ranges. 3. **Rank-Calibration Error (RCE)**: They introduce RCE as a principled metric to assess the quality of uncertainty measures, which is not affected by thresholding and is applicable to measures with different output ranges. 4. **Empirical RCE and Indication Diagrams**: They propose methods to estimate RCE and visualize rank-miscalibration using indication diagrams, which help in understanding the performance of uncertainty measures. 5. **Experiments**: Comprehensive experiments on various LMs and datasets show the broad applicability and granular interpretability of the proposed method. The results also highlight the robustness of RCE to key hyperparameters and the effectiveness of rank-calibrated measures. The paper concludes by discussing the advantages of the rank-calibration framework and suggesting future directions, including developing uncertainty measures with guaranteed rank-calibration and enhancing LM generative pipelines with rank-calibrated measures.