Abstract
AbstractCrowdsourced psychological and other biobehavioral research using platforms like Amazon’s Mechanical Turk (MTurk) is increasingly common – but has proliferated more rapidly than studies to establish data quality best practices. Thus, this study investigated whether outcome scores for three common screening tools would be significantly different among MTurk workers who were subject to different sets of quality control checks. We conducted a single-stage, randomized controlled trial with equal allocation to each of four study arms: Arm 1 (Control Arm), Arm 2 (Bot/VPN Check), Arm 3 (Truthfulness/Attention Check), and Arm 4 (Stringent Arm – All Checks). Data collection was completed in Qualtrics, to which participants were referred from MTurk. Subjects (n = 1100) were recruited on November 20–21, 2020. Eligible workers were required to claim U.S. residency, have a successful task completion rate > 95%, have completed a minimum of 100 tasks, and have completed a maximum of 10,000 tasks. Participants completed the US-Alcohol Use Disorders Identification Test (USAUDIT), the Patient Health Questionnaire (PHQ-9), and a screener for Generalized Anxiety Disorder (GAD-7). We found that differing quality control approaches significantly, meaningfully, and directionally affected outcome scores on each of the screening tools. Most notably, workers in Arm 1 (Control) reported higher scores than those in Arms 3 and 4 for all tools, and a higher score than workers in Arm 2 for the PHQ-9. These data suggest that the use, or lack thereof, of quality control questions in crowdsourced research may substantively affect findings, as might the types of quality control items.
Publisher
Springer Science and Business Media LLC
Subject
General Psychology,Psychology (miscellaneous),Arts and Humanities (miscellaneous),Developmental and Educational Psychology,Experimental and Cognitive Psychology
Reference50 articles.
1. Adesida, P. O. (2020). Suicide capability scale development: PPCS and CSAS (Publication Number 28148178) [Azusa Pacific University]. ProQuest.
2. Agley, J., & Xiao, Y. (2020). Misinformation about COVID-19: evidence for differential latent profiles and a strong association with trust in science. BMC Public Health, 21, 89. https://doi.org/10.1186/s12889-020-10103-x
3. Agley, J., Xiao, Y., & Nolan, R. (2020). Amazon MTurk for research: Improving data quality. 10.17605/OSF.IO/SV9EA
4. Aguinis, H., Villamor, I., & Ramani, R. S. (2020). MTurk research: Review and recommendations. Journal of Management, 46(4), 823–837.
5. Amazon.com. (2020). Amazon Mechanical Turk: Access a global, on-demand, 24x7 workforce. Amazon. Retrieved November 25, 2020 from https://MTurk.com
Cited by
52 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献