Vinay gives a good answer with regards to surveys. In regards to drawing causal inference from intervention studies, my response is "it depends". It depends on whether you are estimating effects on "as treated" versus "intent to treat". It depends on the treatment effect estimator you will use (average treatment effects, or average treatment effects on the treated). It depends on the outcome variable, and whether censoring is an option.

So there are many angles here and the answer is not straight-forward. If you can provide more specifics about your study, I am sure we can provide more specific responses.

In fact, I'm working in a hospital-based HTA unit.(http://www.chuq.qc.ca/fr/evaluation/uetmis/)  We are currently revising our tools for assessing the methodological quality of evidence. Criteria used for evaluating observational studies have to be specific for this kind of study but also, they need to be applicable to every kind of observational study (i.e. before-after study, cohort study (prospective/retrospective), case-control, time series, case series)

I agree. Check out the checklist that the Cochrane Collaborative uses. It may be of help to you.

Cochrane Effective Practice and Organisation of Care (EPOC). 2013. EPOC Resources

for review authors. Oslo: Norwegian Knowledge Centre for the Health Services.

Available at: http://epocoslo.cochrane.org/epoc-specic-resources-review-authors.

These are all excellent answers.  For surveys you get anxious when the participation rate falls below 50%.  It is increasingly difficult to get high participation rates on random studies.  For cohort studies it is less difficult.  The key is numerous contacts with different techniques.  Frequently we do studies of the non responders to see how they compare to the responders on key variables.  This can sometimes be accomplished by sending return mail postcards or VERY short questionaires to the non responders so we can see how different they are from those who selected to answer or participate in your study. This affords you the opportunity to see whether the sample you is representative of the population you are studying. In a recent study of blood donors for example we found that the responders were older and made more frequent donations than those who did not return our questionnaires.  Thus we knew that we had a potential response bias since we were looking at knowledge, attitudes and practices and we were getting responses from more committed donors. . Whatever you can do to help characterize your population is invaluable to interpreting your findings.

In observational studies, what you are evaluating, how you are recording the information, and how you intended on analyzing that information will determine what is appropriate. The same basis hold for qualitative or quantitative studies. For small groups or numbers of people, you are really doing a case study. For this, you will want at least a 50% participation rate. However, how you intend on using the findings will determine if that is acceptable. For example, if you are publishing, your participation rate needs to be above 80% or you are going to have issues with reviewers. But for internal needs, when you get saturation of information (qualitative) or your standard deviation reduces and/or stabilizes (quantitative) or you have gotten 95-100% participation, then you have your finding. This might be at 70%. For larger studies with broader populations, you have the same consideration as small studies. You just have less coverage/participant rate (5-50%), and therefore, sample representativeness of the overall population becomes a consideration.

In short, for observational studies, you will need to plan every aspect before execution, understanding the what, how, who, where, and most importantly why. This will give you the best chance of success. Good luck with your study.

Cher Sylvain,

pourquoi pas une réponse en Français de temps en temps? Ma réponse serait que cela dépend de la question posée, du schéma d'étude, et donc avant tout du pourquoi vous faites cette étude. Dans la plupart des essais thérapeutiques - car je crois que vous vous intéressez surtout à ça - le taux de participation est considéré comme sans importance. En comparant deux groupes randomisés, l'hypothèse sous jacente est que si on met en évidence l'efficacité d'un traitement, il sera ensuite dans toute population pas trop éloignée des critères de constitution de celle de l'essai. C'est une hypothèse très forte et probablement assez fausse dans un grand nombre de cas. Cela explique en partie les variations d'efficacité observées dans la population générale. Votre question est donc tout à fait pertinente mais rajouterai un grand degré de complexité aux essais thérapeutiques - et peut-être une impossibilité à les faire.

bien cordialement

Christophe

Christophe,

Agreed for therapeutic trials. Participation rates are not critical, more important is how close the various arms are in the study.  Unfortunately, I am not a French speaker so might not fully appreciate your comments.  I do know that in the RCT often we have fairly low participation rates, often due to the stringent inclusion/ exclusion criteria.  True too, as you point out, frequently the results of these trials are not extrapolable to the general population as we often see in a phase III or IV study.  You made some very important points. Thanks.

Thanks for all your valuable answers. It will be very helpful for updating our own tool for evaluating observational study quality.

Merci beaucoup Christophe pour votre réponse en français! C'est très apprécié.

You need to be a bit careful here. An observational study means (more or less) that the patients get to select the treatment they are getting rather than having their choice be dictated by the flip of a coin. Any time the patients get to choose, you have the risk of selection bias, even if 100% percent of the patients participate in the observational study.

Now I know there are some observational studies where patients don't get to choose, such as a study comparing BRCA1 to BRCA2 breast cancer patients, but the overall point is still valid.

Problems when patients fail to respond to a call to participate in a survey, for example, might be better characterized as nonresponse bias. There is no consensus in the research community as to how low a participation rate would cause concerns about nonresponse bias. In some settings, you have basic demographic information about the nonresponders, and you can mitigate concerns about nonresponse if you can show that the demographic profile of the nonresponders is comparable to the demographic profile of the responders.

I personally do not worry about nonreponse if the response rate is greater than 90%,I start to worry if the response rate drops below 70% and I get very nervous if it drops below 50%. But this is a very arbitrary choice on my part. You can argue some of these numbers based on a sensitivity analysis, perhaps.

I have not seen any published cutpoints, arbitrary or not, in the peer-reviewed literature. If there are any, I'd love to see them.

By the way, selection bias and nonresponse bias are both important issues for randomized studies as well.

As someone pointed out, it depends on the type and setting of your study. No single size fits all.

In social research not only HIV, gender issues like spousal violence, contraceptive use etc. are very sensitive too specially for Indian subcontinent. So it can be missed more in that case we can go for qualitative or mixed method approach too. Vinay explained nicely too.