This paper investigates the calibration of confidence measures in code-generating large language models (LLMs), focusing on their reliability in software engineering tasks such as code completion, function synthesis, and program repair. The study evaluates how well the confidence scores generated by LLMs align with the actual correctness of the code they produce. The results show that, in general, LLMs are not well-calibrated, with high Expected Calibration Error (ECE) values across various tasks and datasets. This indicates that the confidence scores provided by LLMs are not reliable indicators of code correctness. The paper explores methods to improve calibration, including Platt scaling, which can enhance the alignment between confidence scores and actual correctness. However, the effectiveness of these methods varies depending on the task and dataset. For instance, while Platt scaling improves calibration for some tasks, it may not be effective for others. The study also examines reflective approaches, where the model is instructed to estimate its own confidence in the generated code, and finds that these methods can yield better calibration in some cases. The research highlights the importance of calibration in software engineering, as it enables developers to make more informed decisions about the reliability of generated code. A well-calibrated confidence measure allows for rational risk management and quality control, helping developers decide how much review and care are needed when using code generated by LLMs. The study also discusses the limitations of current calibration methods and the need for further research to improve their effectiveness. Overall, the findings suggest that while LLMs can be improved in terms of calibration, there is still a need for better methods to ensure that confidence scores accurately reflect the correctness of generated code. The paper provides a framework for evaluating and improving the calibration of code-generating models, which can contribute to more reliable and trustworthy use of LLMs in software engineering.This paper investigates the calibration of confidence measures in code-generating large language models (LLMs), focusing on their reliability in software engineering tasks such as code completion, function synthesis, and program repair. The study evaluates how well the confidence scores generated by LLMs align with the actual correctness of the code they produce. The results show that, in general, LLMs are not well-calibrated, with high Expected Calibration Error (ECE) values across various tasks and datasets. This indicates that the confidence scores provided by LLMs are not reliable indicators of code correctness. The paper explores methods to improve calibration, including Platt scaling, which can enhance the alignment between confidence scores and actual correctness. However, the effectiveness of these methods varies depending on the task and dataset. For instance, while Platt scaling improves calibration for some tasks, it may not be effective for others. The study also examines reflective approaches, where the model is instructed to estimate its own confidence in the generated code, and finds that these methods can yield better calibration in some cases. The research highlights the importance of calibration in software engineering, as it enables developers to make more informed decisions about the reliability of generated code. A well-calibrated confidence measure allows for rational risk management and quality control, helping developers decide how much review and care are needed when using code generated by LLMs. The study also discusses the limitations of current calibration methods and the need for further research to improve their effectiveness. Overall, the findings suggest that while LLMs can be improved in terms of calibration, there is still a need for better methods to ensure that confidence scores accurately reflect the correctness of generated code. The paper provides a framework for evaluating and improving the calibration of code-generating models, which can contribute to more reliable and trustworthy use of LLMs in software engineering.