Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

2014 | Eyal Peer, Joachim Vosgerau, Alessandro Acquisti
Reputation on Amazon Mechanical Turk: A Sufficient Condition for Data Quality Researchers investigated whether high reputation workers on Amazon Mechanical Turk (MTurk) provide higher quality data than low reputation workers, and whether attention check questions (ACQs) are effective in improving data quality. The study compared two methods for ensuring data quality on MTurk: restricting participation to high reputation workers (those with above 95% approval ratings) and using ACQs to screen out inattentive respondents or increase their attention. In Experiment 1, high reputation workers rarely failed ACQs and provided higher quality data than low reputation workers. ACQs improved data quality only for low reputation workers in some cases. Experiment 2 confirmed these findings and suggested that more productive high reputation workers produce the highest quality data. The study concluded that sampling high reputation workers can ensure high quality data without using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc. The study found that high reputation workers provided higher reliability scores on established measures, showed lower rates of socially desirable responding, and exhibited the classic anchoring effect. ACQs improved data quality for low reputation workers but did not affect high reputation workers. The results suggest that sampling high reputation workers is a sufficient condition for obtaining high quality data on MTurk. The study also found that highly productive high reputation workers were less prone to fail ACQs and produced slightly higher data quality than less experienced workers. However, ACQs increased data quality among low productivity workers but not among high productivity workers. The study concluded that sampling high reputation workers is not only a necessary but also a sufficient condition for obtaining high quality data. Using ACQs does not seem to help researchers obtain higher quality data, despite previous emphasis on this approach. Sampling high reputation workers provides two advantages: it avoids selection bias and reduces the risk of reactance and disruption of the study's natural flow. The study also found that the number of low reputation workers is low and possibly decreasing, which may make MTurk an even more attractive pool for researchers. However, if requesters are approving HITs more than they should, reputation levels may become less indicative of high-quality workers, and ACQs may be needed again to differentiate 'good' from 'bad' workers.Reputation on Amazon Mechanical Turk: A Sufficient Condition for Data Quality Researchers investigated whether high reputation workers on Amazon Mechanical Turk (MTurk) provide higher quality data than low reputation workers, and whether attention check questions (ACQs) are effective in improving data quality. The study compared two methods for ensuring data quality on MTurk: restricting participation to high reputation workers (those with above 95% approval ratings) and using ACQs to screen out inattentive respondents or increase their attention. In Experiment 1, high reputation workers rarely failed ACQs and provided higher quality data than low reputation workers. ACQs improved data quality only for low reputation workers in some cases. Experiment 2 confirmed these findings and suggested that more productive high reputation workers produce the highest quality data. The study concluded that sampling high reputation workers can ensure high quality data without using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc. The study found that high reputation workers provided higher reliability scores on established measures, showed lower rates of socially desirable responding, and exhibited the classic anchoring effect. ACQs improved data quality for low reputation workers but did not affect high reputation workers. The results suggest that sampling high reputation workers is a sufficient condition for obtaining high quality data on MTurk. The study also found that highly productive high reputation workers were less prone to fail ACQs and produced slightly higher data quality than less experienced workers. However, ACQs increased data quality among low productivity workers but not among high productivity workers. The study concluded that sampling high reputation workers is not only a necessary but also a sufficient condition for obtaining high quality data. Using ACQs does not seem to help researchers obtain higher quality data, despite previous emphasis on this approach. Sampling high reputation workers provides two advantages: it avoids selection bias and reduces the risk of reactance and disruption of the study's natural flow. The study also found that the number of low reputation workers is low and possibly decreasing, which may make MTurk an even more attractive pool for researchers. However, if requesters are approving HITs more than they should, reputation levels may become less indicative of high-quality workers, and ACQs may be needed again to differentiate 'good' from 'bad' workers.
Reach us at info@study.space
[slides and audio] Reputation as a sufficient condition for data quality on Amazon Mechanical Turk