I am interested in any personal experience you may have using this combination (MTurk and Qualtrics) for survey research, and any advice you may have.
Specifically ease of use, ability to get a representative sample, the number of responses and the cost. Beyond that, for what type of research would you feel comfortable using survey data obtained through this method? Would it be best for only testing survey ideas or is the data quality sufficient for more than that?
Finally has anyone had experience with data obtained this way and IRBs?
Thank you!
many studies based on Mturk "samples" have been published. However I think the answer to your question depends on the goals of your research. Mturk respondents are NOT a random or a population sample. So they cannot be considered as representative of anything. There are many application for which Mturk services can be at hand though, as preliminary tests on questionnaires and item designs
It also depends on the theory driving your research regarding MTurk. Yes, they are not necessarily a population sample in many respects but certain questions afford you to regard them as a population sample. In my work in evolutionary psych, I'm operated under the assumption that humans have certain universal preferences that are malleable based on a situation. I am very comfortable in saying that may data on preferences for faces possessing attractive features from MTurk are representative, as they are humans with greater adhesion to good genes.
Where MTurk could be an issue is if you're looking at questions that have certain cross-cultural variability (e.g., self-construal), which could preclude overall generalizability.
For our IRB process, all we just mention is that we are using MTurk and describe the procedure we use to compensate participants for their time and what measures we take to ensure their protection. I caution you, however, not to use manipulations or measures that may particularly volatile to emotional wellbeing, as there is no way to get hold of participants ethically if something negative were to happen. Thus, it's best to keep non-deception studies without distressing stimuli on MTurk for the appeasement of IRB.
Thank you Mitch--that was great information. We are developing a policy research project that includes some behavioral elements and we were trying to figure out if using this combo would be good to first test our ideas and maybe use those results to get funding to do a random sample survey so this was very helpful. Thank you very much!
- Badges
- Science topic
- Similar topics
- Statistics
- Survey