The article discusses the challenges of assessing reliability in subjective judgments in computational linguistics and cognitive science, particularly in discourse and dialogue research. Current methods for measuring reliability are inconsistent and not easily comparable. The author argues that techniques from content analysis, such as the kappa statistic, should be adopted to improve reliability assessment. The kappa statistic is a measure of agreement between coders that corrects for chance agreement. It is widely used in content analysis and is more interpretable than other measures. The article critiques several existing reliability measures, noting that they do not account for chance agreement and are therefore not reliable. For example, measures based on pairwise agreement or overall agreement do not consider the expected chance agreement, making them difficult to interpret. The article also discusses the role of expert and naive coders. While some studies designate one coder as an expert, the author argues that there are no true experts in subjective coding. Instead, the reliability of coding should be assessed based on how well coders follow instructions, rather than on the status of the coder. The author concludes that the kappa statistic is a better measure of reliability than current methods and should be adopted more widely in discourse and dialogue research. It allows for the comparison of results across different coding schemes and experiments, and provides a more accurate assessment of reliability. The article also notes that while the kappa statistic is widely used in content analysis, its application in discourse and dialogue research is still underdeveloped.The article discusses the challenges of assessing reliability in subjective judgments in computational linguistics and cognitive science, particularly in discourse and dialogue research. Current methods for measuring reliability are inconsistent and not easily comparable. The author argues that techniques from content analysis, such as the kappa statistic, should be adopted to improve reliability assessment. The kappa statistic is a measure of agreement between coders that corrects for chance agreement. It is widely used in content analysis and is more interpretable than other measures. The article critiques several existing reliability measures, noting that they do not account for chance agreement and are therefore not reliable. For example, measures based on pairwise agreement or overall agreement do not consider the expected chance agreement, making them difficult to interpret. The article also discusses the role of expert and naive coders. While some studies designate one coder as an expert, the author argues that there are no true experts in subjective coding. Instead, the reliability of coding should be assessed based on how well coders follow instructions, rather than on the status of the coder. The author concludes that the kappa statistic is a better measure of reliability than current methods and should be adopted more widely in discourse and dialogue research. It allows for the comparison of results across different coding schemes and experiments, and provides a more accurate assessment of reliability. The article also notes that while the kappa statistic is widely used in content analysis, its application in discourse and dialogue research is still underdeveloped.