My research team and I collected a batch of data this month(~150 workers) from Mturk. Despite having many quality checks embedded in the survey (e.g., multiple attention checks, reverse coded items) we still feel that the data are suspicious. We ran a similar study one year ago and one of our measures assesses sexual assault perpetration rates. We used the same measure in our current study and the perpetration rates are unusually high this time. Is anyone else having trouble finding quality participant responses in Mturk? Does anyone have suggestions for how we could target better participants? Are there any forums or blog posts that we should be aware of that will help us better understand what is going on? Any information would help and be greatly appreciated!
Thanks in advance!
Ms Breanne R. Helmers
Unfortunately, I found another recent study on the quality decrease of MTurk data: An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results by Michael Chmielewski, Sarah C. Kucker.
https://journals.sagepub.com/doi/abs/10.1177/1948550619875149?journalCode=sppa
Hello Breanne,
What an interesting query!
I strongly suspect that respondents/participants-for-hire portals (like Mturk) are shaky to begin with, and are quite possibly even less trustworthy in the covid-19 era. Worse, I would suspect that (like Kan & Drummey's 2018 study, which looks at only one aspect of such problems) many published studies are even worse off (especially with respect to external validity/generalizability) than since the days when Robert Rosenthal began systematically compiling research on the "volunteer effect" in research.
Kan, I. P., & Drummey, A. B. (2018). Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce. Computers in Human Behavior, 83, 243-253. doi: 10.1016/j.chb.2018.02.005
Good luck with your work!
David Morse thank you for sharing that paper and for your thoughts!
Ms Breanne R. Helmers
Unfortunately, I found another recent study on the quality decrease of MTurk data: An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results by Michael Chmielewski, Sarah C. Kucker.
https://journals.sagepub.com/doi/abs/10.1177/1948550619875149?journalCode=sppa
- Badges
- Science topic