Data quality of platforms and panels for online behavioral research

Data quality of platforms and panels for online behavioral research

Accepted: 23 August 2021 / Published online: 29 September 2021 | Eyal Peer, David Rothschild, Andrew Gordon, Zak Everdend, Ekaterina Damer
The paper examines the data quality of online platforms and panels used for behavioral research, focusing on Amazon Mechanical Turk (MTurk), CloudResearch, Prolific, Qualtrics, and Dynata. The authors identify key aspects of data quality—attention, comprehension, honesty, and reliability—and explore differences in these aspects across the selected platforms and panels through two studies with approximately 4000 participants. Study 1, conducted without data quality filters, found that only Prolific provided high data quality on all measures. Study 2, with data quality filters, showed high data quality among CloudResearch and Prolific, while MTurk had alarmingly low data quality. The study also found that frequency and purpose of usage, rather than reputation, predicted data quality, particularly for MTurk participants who reported using the site as their main source of income but spent few hours on it per week. The authors provide a framework for future research on data quality in online research and highlight the evolving nature of data quality across different platforms and panels.The paper examines the data quality of online platforms and panels used for behavioral research, focusing on Amazon Mechanical Turk (MTurk), CloudResearch, Prolific, Qualtrics, and Dynata. The authors identify key aspects of data quality—attention, comprehension, honesty, and reliability—and explore differences in these aspects across the selected platforms and panels through two studies with approximately 4000 participants. Study 1, conducted without data quality filters, found that only Prolific provided high data quality on all measures. Study 2, with data quality filters, showed high data quality among CloudResearch and Prolific, while MTurk had alarmingly low data quality. The study also found that frequency and purpose of usage, rather than reputation, predicted data quality, particularly for MTurk participants who reported using the site as their main source of income but spent few hours on it per week. The authors provide a framework for future research on data quality in online research and highlight the evolving nature of data quality across different platforms and panels.
Reach us at info@study.space
[slides and audio] Data quality of platforms and panels for online behavioral research