Folks, I have used Qualtrics panels - not sure if you are familiar with this, Qualtrics have panels and charge a certain amount per survey depending on how scarce the panel respondents are - for example CEOs may be $50 a survey, whereas a middle manager might be $8-10 depending on criterion. It is a little difficult to get response rate data in this case.
Another method some of my colleagues are using is Amazon Mechanical Turk. I have not tried this and am not familiar with the quality. It is definitely a less expensive option - I have heard it could be $1 per survey response.
What are your views and experience on some good ways to get quality survey data?
You can trust MTurk, there are studies investigating the validity of MTurk and they reveal similar results with online studies provided by other websites. However, I do not think you can find CEOs on MTurk but maybe other middle level job owners.
Actually, I clicked on this question to see what you had to say about Qualtics samples. I'm always skeptical of specialized samples from panels.
MTurk is a great way of getting responses, but I think MTurkers are not anything like a random sample of the population. For example, their ages seem bimodal with most being fairly young but with an excessive number of retired individuals.
You can target a special population by screening them. Usually you want to put your critical questions in a short set of filler questions. For example, if you had one question ("Are you a manager, Yes/No") then many non-managers would end up in your survey. If you had a drop down where they indicated their last two jobs and manager was one choice, I'd feel more confident that they were managers. It's ideal if your survey system can branch based on the screening questions to either say "I'm sorry. You don't fit the study requirements. Please return this HIT." or else go on to the survey. You may also want to start by asking them their MTurk WorkerID and prevent multiple attempts by the same worker (you set up the batch so workers can only complete a HIT once, but if they try your screening questions and fail and return the HIT, they can re-attempt often unless you prevent that).
MTurkers are motivated to finish the survey and get paid. I strongly recommend that you include inattention checks and that you tell respondents that there are inattention checks and that only attentive responses will be paid. I find that "Please pick 'Agree'" type attention checks stand out too much and even inattentive MTurkers will stop and answer those correctly. I prefer items that blend in. For a personality/attitude survey, I might have an item like "I strongly believe that 1 plus 2 equals 12" On an ability survey, I just put a couple really easy items towards the end.
Maybe an MTurk strategy for getting managers to complete your survey is to screen your MTurk sample to ensure that they have a manager. Then ask them to forward your survey to their manager and possibly offer some incentive for the managers who complete the survey. You run the risk that the MTurker will simply pretend to be a manager to receive the incentive.
Another strategy would be snowball sampling... Do the same thing but don't use MTurk but rather begin with your circle of acquaintances and their acquaintances, etc.
Definitely i do recommend Qualtrics over MTurk for academic research;
Qualtrics will provide a dedicated project manager to help conduct, manage the survey access, distribute questionnaires and collect data from diverse demographic of the targeted sample, which will increase the quality of the collected data.
In contrast, MTurk is an academic research method being done online. However; this method may be leading to flawed or unreliable data.
Dr. Kelly check the following video:
http://www.pbs.org/newshour/bb/professional-survey-takers-shaping-scientific-research/
- Badges
- Science method
- Similar topics
- Mathematical Sciences
- Statistics
- Applied Statistics
- Survey Design