The growth of the crowd sourced solutioneering phenomenon shows how people will engage and support knowledge development. However, how much confidence can we have in a crowd sourced answer, where we don't know participants, and also some people may be trying to obstruct answers?
This is a good question especially since interest in CS as a research method has seen some notoriety in recent years.
I've experimented with 2 pilots on Mturk and am just wrapping up my data collection on Crowd Flower for my dissertation. My second pilot may be of interest to you since I completed my pilot in both online and in-person settings with comparable results:
http://untitledartist.com/image-based-research/
Look at articles by Lefever, Griffiths, Granello and Beaudoin regarding online research (not specific to CS) and there is every indication that anonymity can actually increase trust in survey answers (in particular those about SES) and there is also the added benefit of a decrease in survey bias.
Of course there are always those who will wish to provide obstructionist replies and subvert your research BUT the geographical and cultural diversity of your participant pool counters many of the detractors.
Although I've not had a chance to look at my latest data exclusively for this issue, there are a few things I've noticed. Since Crowd Flower has no quality control (as Mturk does) I was only able to achieve about 50% 'good' data meaning that the participants followed my instructions, filled out the survey and provided me with drawings. The main issue I had was many providing photos that were not theirs, however even those that did not follow instructions and provided photos still appear to have answered the survey questions truthfully (some of this I garnered by matching IPs to the location they provided in the survey). Within this half of participants roughly 10%+ seemed to complete the survey without any concern for the questions. It's also important to note that this fluctuates from region to region. For example larger more tech-savvy populations tended to have more of these obstructionist users who know the crowdsourcing system well and how to use/abuse it.
There is still very little literature on this emerging research niche and I'm looking forward to pursuing it further in the future.
Have you used any of the CS platforms for research?
I liked the question. I think the information, so received will be evaluated by the existing knowledge base of the individual, existing schema/ model the individual already has about the information (information seeker is not completely open), and the pattern individual has got after getting similar information elsewhere.
Thank you
By comparing with the other answers, you can establish a trust on certain workers. The following "ZenCrowd" technique requires:
- Run multiple repetitions for each task, ideally >=3.
- Have a bit of training data, ~5%.
- note: It was only applied to Multiple choice questions.
Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux: ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking. 21st International World Wide Web Conference (WWW2012), Lyon (France), April 16-20, 2012.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs