The first bit is specific to the incomplete data question. Of course, you can't ethically require that they answer every question, but there are a couple of things you can do. How long does your survey take to complete? If it's long, try a planned-missing design (e.g., John Graham's XABC blocks; I can find the cite if you need it). That way no one respondent has to go through the whole thing. Counterbalancing instruments can also help with getting more complete data.

For both recruitment and completion, do you offer any incentives, preferably financial? Would your IRB allow a drawing, for example, even if you can't afford to reimburse all respondents. It gets tricky if it's completely anonymous, but I don't think it's impossible.

I'm not sure exactly what you mean by "distribute," but I know a lot of people use tools such as surveymonkey.com to administer the surveys and have good luck with it. A premium membership gives a lot of benefits, and isn't horribly expensive as research costs go.

Hope some this helped!

Dear Colleagues,

in my research involving university students in Italian Country I advertised the online questionnaire by means of facebook and web forum used by students. Hovewer, my study concerned the use of social network sites, so online participants were normal, while normally online methods could have the limits of pre-selecting a type of participants (those using the web and familiar with online test).

But I have a participation of 853 students ... one could say very high but, in my opinion is very low if I consider the total of potential participants (i.e. Italian university students).

For my last survey adressing the luxembourgish architects on the whole sample the response rate was 3.5%. and including the respondents with less than 5% of missing data the response rate was 3.7%.

The survey was send three time by mails.

As I ams adressing professionnal and regarding the distribution the output was not too bad. But in my opinion was not efficient if I consider the total of potential participants, so we decided to complete data collection by phone to ensure the quality of the results.

I collected data using web survey from Finnish managers in social administration. Response rate was pretty good (52 %). Another data collection concerned university students and the functionality of the Student Union Services, response rate around 30%. Third example comes from university personnel questionnaire, response rate again around 30 %. I think a succesful data gathering necessitates a motivation by respondents. You should pay attention on the letter of invitation in order to motivating the respondents.

It really differs depending on factors such as: time of sending invitation letter, wether you know respondents or not, problem you use etc. Usually response rate in web based research is low. There are trials of implementing multimode methods: traditional surveys for gathering reposndents agreements (sometimes CATI sometimes PAPI) and then CSAQ - check out WebDataNet materials - this is European initiative for improving web based gathering of data.

Some use incentives but I'm not in favour of them - they teach respondents that if they don't get paid they don't take part in any research (and there are some researchers that can't afford paying the participants). You must remember that incentives work only if paid before or after the research - you can't promise prises for only some participants).

CSAQ is fantastic if:

- you know the respondents (you aren't anonnymous to them)

- you already have their e-mail adresses (as above)

- you have proved agreement for taking part in research (as above)

- subject is important to the respondents.

In such case you will have quite high response rate (doesn't mean 100%). In any other case you'll have to send thousands of invitations to gather not more than hundreds of responses (of course these full not partial).

But as I mentioned - check the materials of WebDataNet (European COST action): www.webdatanet.cbs.dk.

@ Tyler Burleigh

On the other hand, paying people to participate in research does not mean an unbiased sample; you risk that those people will participate for whom these couple of bugs mean something. Someone with a full wallet will not care whether or not one is paid. Thus - paying can itself be a source of bias!

Mateusz,

A good comprehensive answer on factors to consider. I just want to add in response to this:

"you can't promise prises for only some participants"

In the U.S., the federal guidelines are not explicit on this issue. Many studies where the researcher cannot afford financial incentives to all participants use something like a prize drawing at the end of the study (for starting participation, not contingent on completion). But because the guidelines are not explicit, local institutional review boards vary on their tolerance of this practice.

A separate issue is that the Internal Revenue Service's guidelines can be read as requiring a full accounting of all money disbursed, including the Social Security Numbers of recipients. Again, institutions' business offices may differ on the strictness of application (for example, a small gift card may or may not be interpreted as payment), but the institutions that do require SSNs for everyone result in greater difficulty in researching some populations, particularly undocumented immigrants.

It's a set of dilemmas.

I would certainly agree that in the case of using incentives and prize drawings it would be important to get guidance from your IRB or ethics committee - I have also found that our internal audit department has had opinions related to their concerns about the potential for IRS and funder concerns. That said we have been able to proceed - but cash values have to be small so as not to hit thresholds of concern.

There is also an issue online that we have run into that too obvious incentives got picked up by search engines and we got a lot of phishing responses for the incentive. However they were easy to spot.

For several studies I have used related listservs as venues for snowball accumulation of respondents and have found that this does increase numbers and range of respondents but does mean that it is harder to gauge non-response rates. However, to the extent that there are already studies of the population I look at how similar or different my sample is to the reported samples to determine usefulness.

In one case for a doctoral student I was supervising there were dramatic differences using this approach and she was clearly reaching both numbers of respondents and a range of respondents which no one in the literature had reached before.

In a way I expected that mentioning incentives and prizes will trigger this discussion. There are quite fine solutions mentioned by eg. Philip McCallion. However, snowball sample is quite qualitative. But I use different solutions in sampling.

I found few texts on paying for participation and had really great talk with Michael Bosniak (responsible for German national panel research). Incentives work but they bias the sample - people take part in it not because they have opinion (at least not all of them) but are interested in "cheap/free money" (not all of them). What about "farming" - the situation when people take part in one research more than once to rise their chances to get money (or more money).

And now really serious part - we do assume that people who take part in the research:

- read carefully everything,

- have their own opinion,

- choose carefully the answer that is closest to their opinion,

- are doing it openly (not affraid of any consequances, even imagined).

Don't you guys think, that these assumptions are a little to much idealistic? I mean, anyone, who conducts research, knows pretty well, that such situation just doesn't exist. What is more - people change their opinion during or after the research.

In this context paying for taking part in the research is getting really serious methodological problem. Imagine, that you pay people for voting - who they vote for if they get paid. If I get paid by my employer I work for him/her, because they pay me. Respondent, especially the one who gets paid, starts to think in these categories - wondering what the researcher expects from them. Of course, if they don't know, they can always choose answers that are mostly socially expected (but not theirs). Of course (once again) there is no proof that they don't do this when not beeing paid, but the situation may strengthen this "syndrome".

Mateusz,

Your points are all well taken, but I'm not sure (except for farming) that they are any more issues for online studies than for any other format of questionnaire study.

And there is certainly a consent issue (in either case) for overly large payments resulting in a sense of coercion. It has to be finely tuned.

It's a tough balance, though I'm less concerned about demand characteristics (experimenter expectations) than you might be, especially with small payments -- and perhaps even more so for not being in the experimenter's presence. I'm thinking of the old insufficient justification research.

Good discussion!

@ Patrick Malone

It is true that all methodological problems noticed in other, more traditional techniques, occur also in CSAQ. And of course this fact doesn't impair CSAQ as a technique. Let's consider coverage for example - some people have the access to the Internet and some don't (Poland - it's ap. 60%).

PAPI seems so effective, especially with well designed sample. But it is so expensive and takes so much time.

@Tyler Burleigh

I do not suggest that if you don't pay you have more interested people and if you pay respondents have "milder beliefs". In my opinion paying also affects the sample (as Marcin Kozak mentioned). I presented some possible ways. I don't appreciate incentives because they "spoil" respondents who might not take part in research that is not paid. I had such situations when conducting traditional PAPI - some people refused after they knew that we didn't pay because few monthes earlier they had been paid for participation. They refused not because they were not interested in the topic (but with "milder beliefs"), they refused because they wanted money as in the previous cases.

And now, if they refuse with milder beliefs because are not paid, is it possible that they agree (because of the incentives) and strengthen their beliefs to milder level in situation when they have none?

Patrick is right - the same problem exists in traditional modes. I think that we still don't know why people take part in the research or refuse. We can count response rate but cannot explain it. Incentives are extemporary solution - we need higher response rate so can pay (few more thousands are economically acceptable). Is there any experiment showing the influence of incentives?

I too had a poor response rate of around 20% from participants in a pilot survey. The primary reasons were due to their busyness and they did not have time to read and complete the entire survey online eventhough some of them were friends. Some may say that on line surveys might be easier to implement but really it is about choosing the right method to get a high response with minimal bias. By setting compulsory questions which are to answered one after another, you could either get a complete response or no response at all. In developing countries, there are more challenges in completing online surveys especially those who do not have internet access or are internet illiterate. So far, I had better response from hard copy circulation with a cover letter under the letterhead of my university and employer in order to break the ice. I have also given a pen as a token. Overall, this process is more difficult and logistically challenging but heck, I got a good response rate of close to 90%. But if you are still adamant with online survey, perhaps, meeting, explaining the objectives, answering questions from respondents and basically humanising the entire experience, will help to make the answering process a bit easier.

Results from my online experiment justified the conclusions of my Master’s thesis, “How Conformity Shapes the World Wide Web.” My n = 72. That’s a small sample, especially considering that my references included data from de Argaez (2012), which put the number of Internet users at one-third the human population, and 1 billion Facebook users.

I spent some $250 on a Facebook ad, soliciting participation. I had high expectations, considering I could reach a global sample frame cheaply. The number of visits to my landing page was 1,578, over a data collection period of seven months.

Where were those billions of visits to my online experiment? Where were those thousands of qualified participants from a then-global population of 6 billion?

I will take the blame for a rather sexless experiment, which involved reading a dry article (the participants could not know how dry until they actually signed up and started to read it). I offered no material incentives, because I was and remain poor and underfunded.

References

de Argaez, E. (2012, October 24). Internet users for mid-year 2012. Internet World Stats

News, (062).

I think a basic rule is keep the survey short - better to find out a lot of information about important issues than a lot of vague information about not so important issues. No one wants to fill out lengthy questionnaires on or off line even if they are paid. Know your target-population and word questions accordingly. Another point is design - make the experience as pleasant as possible, snappy colours and a salient title and logo. Another good idea is for participants to have a clear indication of how long the survey will take (try and limit it to 10 minutes max). Some good designs include a time bar at the bottom of the survey which tracks the time left to completion..

According to time bars - I've met some opinions that they work contrary - they sometimes discourages showing how much is still left to do - drop-outs. We have some new experoences right now - quite fresh - sending about 300 invitations to carefully selected scientists with two-questions survey (one "yes/no", one open-ended). Only 32 people answered. It is 10%! No specific on-line survey system has been used - just e-mails (no aditional links or redirections).

Maybe it is a sign of weariness - so many surveys, everyone asking to take part in something.

@Meinland Thielsch - thanks a lot for the literature. Its great.

Last experiences from making commercial order - the firm that took it decided to turn off all staff like tmie bars and number of questions. They explained that they were afraid of drop-outs. We plan to make 6000 interviews.

First step: questionnaire must be divided into sections that filter respondents well (not making them to answer to question that aren't meant for them - quite obvious but sometimes hard).

Second step: whole staff must be really short.

Third step: questions must be simple (matrix isn't good).

Fourth step: find a firm that offers panel ;).

I'll give more information when I get the results.

In my opinion the best incentive is to give participants a feedback on some scales that are measured within the study once they complete the survey. Giving feedback increases not only the chance of good compliance but also increases data quality as participants are interested to have "true" feedback. This feedback can be automated. There are technical solutions for it. For instance www.cloud-solutions.net provides survey solutions with automated feedback.

In a study I conducted online where I invited participants through social media and e-mail the response rate was around 7/10. It took however some time to reach this point which is another difficulty of online research as people may keep postponing the process.

In conjunction with a book I am writing on Objective Beauty, I am conducting an on-line survey of women's views on beauty.

ENGLISH: https://www.surveymonkey.com/s/womenarebeautiful

HUNGARIAN: https://www.surveymonkey.com/s/szepseg

I don't have money to buy email addresses from SurveyMonkey, and neither do I have a good solution for how to increase my numbers. I thought of getting the attention of a women's magazine or online publication, but so far that has not created the number of respondents that I am looking for.

While you are here, please fill out my survey. It is very short. Thank you!

Well response rate is good and accepted to a level that any one expect