Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk

Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk

March 2, 2012 | Adam J. Berinsky, Gregory A. Huber, Gabriel S. Lenz
The article evaluates the use of Amazon.com’s Mechanical Turk (MTurk) for subject recruitment in experimental research, particularly in political science. It assesses the internal and external validity of experiments conducted on MTurk, comparing it to other subject pools such as convenience samples, Internet panels, and national probability samples. The authors find that MTurk respondents are more representative of the U.S. population than convenience samples but less so than Internet panels or national probability samples. They replicate three classic experiments using MTurk, showing that the results are similar to those obtained in other samples. The article also addresses concerns about habitual participants and subject attentiveness, concluding that these issues are not significant in practice. Overall, the authors argue that MTurk is a valuable tool for conducting low-cost, diverse subject recruitment in experimental research.The article evaluates the use of Amazon.com’s Mechanical Turk (MTurk) for subject recruitment in experimental research, particularly in political science. It assesses the internal and external validity of experiments conducted on MTurk, comparing it to other subject pools such as convenience samples, Internet panels, and national probability samples. The authors find that MTurk respondents are more representative of the U.S. population than convenience samples but less so than Internet panels or national probability samples. They replicate three classic experiments using MTurk, showing that the results are similar to those obtained in other samples. The article also addresses concerns about habitual participants and subject attentiveness, concluding that these issues are not significant in practice. Overall, the authors argue that MTurk is a valuable tool for conducting low-cost, diverse subject recruitment in experimental research.
Reach us at info@study.space
[slides and audio] Evaluating Online Labor Markets for Experimental Research%3A Amazon.com's Mechanical Turk