This article provides a comprehensive survey of methods for measuring agreement among corpus annotators in computational linguistics (CL). It covers the mathematics and underlying assumptions of various agreement coefficients, including Krippendorff’s alpha, Scott’s pi, and Cohen’s kappa. The authors discuss the use of these coefficients in different annotation tasks and argue that weighted, alpha-like coefficients, which are less commonly used in CL, may be more appropriate for many corpus annotation tasks. However, their use complicates the interpretation of the coefficient values. The article also reviews past experiences with agreement coefficients in CL over the past decade, addressing issues such as the effect of skewed distributions on agreement values and the choice between individual coder marginals and pooled distributions for calculating chance agreement. The authors conclude by emphasizing the importance of selecting the appropriate coefficient based on the desired interpretation of chance agreement and the specific application.This article provides a comprehensive survey of methods for measuring agreement among corpus annotators in computational linguistics (CL). It covers the mathematics and underlying assumptions of various agreement coefficients, including Krippendorff’s alpha, Scott’s pi, and Cohen’s kappa. The authors discuss the use of these coefficients in different annotation tasks and argue that weighted, alpha-like coefficients, which are less commonly used in CL, may be more appropriate for many corpus annotation tasks. However, their use complicates the interpretation of the coefficient values. The article also reviews past experiences with agreement coefficients in CL over the past decade, addressing issues such as the effect of skewed distributions on agreement values and the choice between individual coder marginals and pooled distributions for calculating chance agreement. The authors conclude by emphasizing the importance of selecting the appropriate coefficient based on the desired interpretation of chance agreement and the specific application.