This paper provides an overview of inter-rater reliability (IRR) assessment, focusing on study design, selection of appropriate statistics, and computation, interpretation, and reporting of IRR statistics. It discusses common mistakes in IRR analysis, such as using percentages of agreement, not reporting the statistic used, using the wrong statistic for the study design, not analyzing variables in their final form, and not interpreting the effect of IRR on statistical power. The paper also provides computational examples using SPSS and R for computing Cohen's kappa and intra-class correlations (ICCs) for nominal and ordinal, interval, and ratio variables. It highlights the importance of selecting appropriate IRR statistics based on study design and data type, and emphasizes the need for thorough analysis and reporting of IRR results to ensure accurate interpretation and appropriate use in hypothesis testing. The paper also discusses the implications of IRR estimates on statistical power and the importance of considering factors such as restricted range, poor psychometric properties, and coder training when assessing IRR. It concludes by emphasizing the need for researchers to carefully select appropriate IRR statistics that fit their study design and goals, and to avoid using percentages of agreement or other indicators that do not account for chance agreement or provide information about statistical power.This paper provides an overview of inter-rater reliability (IRR) assessment, focusing on study design, selection of appropriate statistics, and computation, interpretation, and reporting of IRR statistics. It discusses common mistakes in IRR analysis, such as using percentages of agreement, not reporting the statistic used, using the wrong statistic for the study design, not analyzing variables in their final form, and not interpreting the effect of IRR on statistical power. The paper also provides computational examples using SPSS and R for computing Cohen's kappa and intra-class correlations (ICCs) for nominal and ordinal, interval, and ratio variables. It highlights the importance of selecting appropriate IRR statistics based on study design and data type, and emphasizes the need for thorough analysis and reporting of IRR results to ensure accurate interpretation and appropriate use in hypothesis testing. The paper also discusses the implications of IRR estimates on statistical power and the importance of considering factors such as restricted range, poor psychometric properties, and coder training when assessing IRR. It concludes by emphasizing the need for researchers to carefully select appropriate IRR statistics that fit their study design and goals, and to avoid using percentages of agreement or other indicators that do not account for chance agreement or provide information about statistical power.