Dear Lasse,

I am not surprised by your findings. People are generally becoming busier, although I don't have any statistics on this. I think 20-30 minutes is quite a long way beyond the level of goodwill of time the average citizen would give to a stranger. My advice is therefore:

• Try to make some sort of social contact with your sample, e.g. telephone them or meet them in public with a clipboard
• Keep the questionnaire as short as possible - we recommend one sheet of paper (double sided) and a target time of under 10 minutes
• Try to engage your sample more with the purpose of your research so that they feel more socially connected with your request

Dear Lasse,

I am not surprised by your findings. People are generally becoming busier, although I don't have any statistics on this. I think 20-30 minutes is quite a long way beyond the level of goodwill of time the average citizen would give to a stranger. My advice is therefore:

• Try to make some sort of social contact with your sample, e.g. telephone them or meet them in public with a clipboard
• Keep the questionnaire as short as possible - we recommend one sheet of paper (double sided) and a target time of under 10 minutes
• Try to engage your sample more with the purpose of your research so that they feel more socially connected with your request

Hi Lasse

Agree with Peter regarding the time, also regarding his point in being more connected(part of) the research project, would include them, being sent some of the results (electronic or by post) which again would back Peter's first point- social contract?

All the Best

Rob

Are you aware of Don Dillman's work on how to maximize response rates in mail-out surveys?

Hi,

Here is a link to David's suggestion: http://faculty.washington.edu/jelmore/articles_online/Dillman-Des%26Admin_Ma.pdf

Rob

It is not strange To have a low response rate in surveys by mail. Time is a main factor, but other one is the relative Importance of the thematic for the interviews. Mystify the method is not necessarily the best, the purest statistically speaking, well, why want to raise the response rate if you can extend the original base of respondents?

Dear Lasse,

Everyone is on point... the world is changing, population orientation, with increase feeling of no-gain attending to the request,  time available to spare for others shrinking except under obligation, definitely enhanced social contact and summary caption of the study benefits could stimulate increase response. I'm not sure if this is ethical...any form of incentive could equally help.

Have you seen this Cochrane review? It has a lot of suggestions. 

http://www.ncbi.nlm.nih.gov/pubmed/19588449

Thanks all for your valuable comments!

I agree with you all, people are becoming more and more busy. I see how social contact can be important. However, the original purpose of using a Population register to acquire data was of course that it would be relatively cheap, and you would get a high n (in theory at least).

David and Robert, I was unaware of the work by Don Dillman - thanks! And thanks Per for the Cochrane review, looks like the largest boost to response rate is achieved through monetary incentives..

I'll think twice the next time we plan on doing a similar study, and try to maximize response rates through different means. Or even attempt to collect data in an entirely different way :)

Best,

-Lasse

Dear Lasse,

a sample size of n=400 is what you finally need in your data set (under the typical assumptions). Otherwise your estimators could be biased.

Since a response rate of almost 100% is unrealistic, the Initial base could be extended, as Ricardo pointed out. Given a typical response rate of 25%, one can send out 1600 mails (initial sample times four), or with a response rate of only 20% you send out 2000 mails, just to make sure you get eventually your 400 people. Anyway it raises the costs, unfortunately, so you could try to mix this approach with the phone calls Peter mentioned above.

All the best for your study,

Jörg

Thanks Jörg!

This sample is actually a comparison group in a study of patients. Power analyses showed that we need approx. 100 comparison participants, which is why we "only" extracted 400 (we were assuming a 25% responserate) population register. However, as our current response-rate is about 15%, we won't get enough participants.

We could of course assume a 15% response-rate and contact more people from the population register; but it raises the question of what sort of sample we are left with, and to what extent our estimators are biased.

An easier and cheaper way would be to contact Universities etc., and deliver the questionnaires in a class-room setting, but then again we would end up with a comparison sample that is biased - but would it be more biased (or biased in a different way) compared to our population-based recruitment with horrible response-rates..

-Lasse

All good discussion points.  However, with  low response rates it is hard to extrapolate your findings to any population base which was your reason for using the population register.  No matter how broad you cast your net by increasing your sample size you still have the problem that the responses you get could be biased.  In surey studies we have done we look to get a 50% response rate.  Below this it is hard to interpret findings and get your results published. 

One question; when you calculated your response rates did you account for mailings that were deliverable?   I am not sure about the mobility of the Norwegian population and the age of those you are trying to reach but it could be great and a fair number of the questionnaires mailed might  not have made it to their destination.  Thus, your true response rate would be higher if these are removed.  With low response rates it is frequently of use to do an non response bias evaluation by trying to find out a limited amount of information from non responders to see if the responders are representative of the population.  We have done this by sending a very short questionnaire or even a post card with a few easy to answer questions that would shed light on the representativeness of the responders.  You might find they do not mirror the population.  They could be older, more or less educated, different socioeconomic class, etc. You know the age range of the women you mailed to is 18-40, did you get a different response rate by age group.  I suspect younger women might not respond as readily.

For increasing response rates we have found two things to be effective; monetary incentive and alternative method of responding.  Adding a small bill to the mailing has a positive effect and might increase response by 10% say.  It is worth a try and you could pilot it on a small sample to see what impact it would have in Norway.  We have used one or two dollar bills in the mailing as a "reward " for returning the questionnaire.  Perhaps a 10 kroner note would work.  Also offer an online route to respond.  This helps especially for younger populations.  This could include a mobile phone app.  Any method could be tested on a smaller sample of the target population to see yield.  It would even be the basis of a good paper on how to increase response to questionnaire surveys. In addition, we have found that repeated mailings increase the yield. Did you do more than one mailing? 

As others have indicated, your questionnaire is long.  This could decrease response rates. One way around this is to send shorter versions of the questionnaire that would contains subsets of the questions if possible.  You would have a core to allow for comparison on certain measures but you might be able to be able to construct it such that you have a good response for the shorter versions that you could then compile.  It eliminates the possability for making all the comparisons you might wish, but you can still derive population estimates for the essential parameters.

One other thing we have found helpful.  A postal address that people fell it is important to respond to.  Often when doing surveys for government agencies we will use an address of the agency.  While Oslo Hospital is well respected in Oslo are all your survey subjects in the geographic area of Oslo Hospital?  In the US we get solicitations for contributions from many sources including research hospitals across the country.  People tend to ignore these mailings and just toss them. If they want to do a survey there has to be some thing that would entice people to even open the envelope.  Without knowing your procedures it is difficult to make constructive comments, but I am giving you things that worked in studies I have conducted.   

George

Dear George,

Thank you for your commentes and suggestions! First of all, yes our response rate was calculated based on all letters that were delivered (many were returned due to unknown address etc., and these were not included in the response rate calulcation). In this particular study we are unable to recontact participants. In later studies we will definitively ensure we have the opportunity to do this. We do know something about the non-responders already, but not a lot. We can look into which age groups, and potentially which geographic locations were more likely to respond. This is obviously an important point, and I liked your suggestion of mailing a short questionnaire to non-responders to shed more light on this.

We do have the opportunity to have participants respond online. I also suspect this will increase response rates, and we will consider adding this option in later studies.

As for monetary incentives, we could try this, but it quickly becomes an issue of cost/benefit (depending on sample size and response rate). An alternative would be to have a lottery of sorts, where 3-5 participants won a gift-card, an iPad etc. I am unsure if such lotteries really do impact response rates, or if it is better to give every participant a smaller compensation...

Thank you for all your suggestions!

Lasse

Dear Lasse,

I believe you will find that a lottery of sorts will not have much impact on a person responding.  The opening of the envelope with the incentive right there has more of an effect.  Perhaps we instill some guilt if they take the money and do not respond.  It is of  not that in the US the March of Dimes charity when they are seeking donations include a dime , 10 cents in the mailing.  They must find that it stimulates people opening their wallets to donate.  I agree with you that you have to see if the incentive is cost effective.  If you do a large mailing the incentive could take considerable resources. If I understand correctly, you were looking for 100  respondents.  You were expecting a 25% response rate and got only 15%,  If you could gain another 10% with a small monetary incentive you would achieve your target. If you increased your sample size to 676 with 15% you would achieve your goal and if you included an incentive the additional cost would be relatively minor, no?  In our studies we request an address correction and we get back the undelivered mailings so these have no additional costs. We typically follow-up with an additional mailing if we get a new address.  This helps increase response.  From all the work you have done, I am sure I am not telling you anything you have not already considered.  Getting good responses has gotten more difficult over time and more concerns are raised about the representativeness of our respondents.

Good luck.  It will be interesting to hear how your next survey does.

George

Thanks,

Yes true I will probably benefit greatly from including a monetary compensation in the present study.

I guess these are hard-learned lessons. Only 10 years ago a group at my hospital did something similar and ended up with a response rate of about 30%. And that was with quite lengthy questionnaires, no compensation, and overall not much effort put into making the study relevant or appealing in some way. I guess we have to work harder to get people to participate! Thanks for all your advice!

Lasse

Nowadays, what sort of response-rates can one expect in population-based studies? How to improve it?

My past experience on offline / online survey questionnaire only received ranging 5-50% response rates.  Finding some of the following ways can attract higher response rate: