The article by Jean Carletta discusses the challenges in assessing reliability in subjective judgments made by computational linguists and cognitive scientists working on discourse and dialogue. Current methods, such as pairwise agreement percentages and ratios of observed agreements over possible agreements, are criticized for being difficult to interpret and comparable. Carletta argues that these measures do not account for expected chance agreement, making them unreliable and uninterpretable. He proposes the use of the kappa statistic, a widely accepted measure in content analysis, as a solution. The kappa statistic corrects for expected chance agreement and provides a standardized measure of reliability that can be compared across different coding schemes and experiments. Carletta suggests that adopting the kappa statistic would improve the interpretability and comparability of reliability measures in the field.The article by Jean Carletta discusses the challenges in assessing reliability in subjective judgments made by computational linguists and cognitive scientists working on discourse and dialogue. Current methods, such as pairwise agreement percentages and ratios of observed agreements over possible agreements, are criticized for being difficult to interpret and comparable. Carletta argues that these measures do not account for expected chance agreement, making them unreliable and uninterpretable. He proposes the use of the kappa statistic, a widely accepted measure in content analysis, as a solution. The kappa statistic corrects for expected chance agreement and provides a standardized measure of reliability that can be compared across different coding schemes and experiments. Carletta suggests that adopting the kappa statistic would improve the interpretability and comparability of reliability measures in the field.