It is a good idea to work out the possible causes of bias on your own, before looking for checklists of various types. Different types of studies have different inherent weaknesses and different research cultures deal with the methodology differently. Once you have worked out what YOU want, go and look at what others did, at which time you are more alert to what they present that you do not need and what they lack that you need. It is very discouraging to see a quality assessment made on the basis of partly irrelevant questions. Gives you a false feeling of security, a little bit like looking at the results of a multivariate analysis with the wrong variables in the model.

'Correlation' in which regards? Correlation between two tests? I'm working a systematic review looking at correlation and accuracy between an index text and a standard test. I'm actually using QUADAS to assess the methodology quality of my included primary studies.

Yes, I will compare the correlations between two types of tests (e.g. MRI/CT/X-ray findings and self-reported questionnaires). However, QUADAS assesses the quality of diagnostic studies for a given examination rather than comparing two sets of tests.

Assessing the risk of bias and assessing the quality are two different things.

If its for comparing two measures, the COSMIN checklist is useful

This guide might be useful "Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions"http://effectivehealthcare.ahrq.gov/ehc/products/322/998/MethodsGuideforCERs_Viswanathan_IndividualStudies.pdf

Hi Arnold

We used QUADAS for our analysis in our paper ''Correlation between Ki-67 immunohistochemistry and 18F-fluorothymidine uptake in patients with cancer: A systematic review and meta-analysis'' and we modified it in order to fit our specifications. In our publication we discuss this process in length and we also have a table with the modifications so you might find this useful. Please let me know if you don't have access to it at [email protected] and I can forward a copy to you.

http://www.ncbi.nlm.nih.gov/pubmed/22658807

The first step for using the QUADAS tool is to tailor the guidelines for scoring each individual item so that they can be applied to the particular study in question. This is provided as an instruction from the QUADAS developers as they acknowledge that it is not possible to provide a generic description of each item applicable to all reviews.

Hi Arnold. I used the Quadas tools. You can find: Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25.

It is a good idea to work out the possible causes of bias on your own, before looking for checklists of various types. Different types of studies have different inherent weaknesses and different research cultures deal with the methodology differently. Once you have worked out what YOU want, go and look at what others did, at which time you are more alert to what they present that you do not need and what they lack that you need. It is very discouraging to see a quality assessment made on the basis of partly irrelevant questions. Gives you a false feeling of security, a little bit like looking at the results of a multivariate analysis with the wrong variables in the model.

Hi Arnold,

Here is a website that has collected a wide variety of quality assessment tools. You may be able to find one that best fits your needs:

http://www.unisa.edu.au/Research/Sansom-Institute-for-Health-Research/Research-at-the-Sansom/Research-Concentrations/Allied-Health-Evidence/Resources/CAT/#Non-Randomised

Hi Arnold,

I agree with Charlotte, that it depends on your question(s) what kind of bias definition might be of interest for you. When taking two or more correlated variables (or test results) into the statistical analysis, most of the algorithms in multivariate analyses or even multiple regression models and their results you receive tend to be "biased" or misleading because dependent information (i.e. the correlated variables) put into a data base more than once causes trouble in calculating model parameter estimates or loadings(=weights) in principal components. (see: collinearity problems and measures, matrix inversion/matrix determinants near zero).

Best wishes

Iris

I am so grateful for all your answers. They are really helpful. I will do my research and modify my assessment tool.

You may find the following 2 references of value. Their relevance is not restricted to correlational studies: Terwee CB, Bot SDM, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007; 60: 34-42. Seida JC, Schouten JR, Mousavi SS, et al. Comparative Efectiveness of Nonoperative and Operative Treatments for Rotator Cuff Tears. Comparative Effectiveness Reviews, No 22. Rockville (MD): Agency for Healthcare Research and Quality, 2010.

Depending on what types of studies you wish to analyze, you will use different tools for assessment of methodological quality. Diagnostic analyses and interventional analyses require different toos. Also, if you are looking at interventional analyses, it depends if you are looking and randomized studies only or you are looking at observational studies. They all have different potential risks of bias. The QUADAS tool mentioned above is for assessing methodological quality of diagnostic tests. First, define what type of analysis you are/will be performing, and they research what kind of methodological assessment is adequate for your research.

Regards,

Vanja

My answer is - do not use simple correlations - But us more sofisticated analysis for expample logistic regression models or linier regression to control for possible bias. I wil recommend the Cochrane Handbook of systematic reviews. Your can find it on the internet.

Yours Jorgen Lous

Sorry, but this is n o t what multiple regression analyses - either linear or logistic - can really do for you! It is a wide-spread belief that just putting all the information into (sophisticated) muliple regression models solves all the problems of collinearity, which is not the case. In case of collinearity the results of multiple regression analyses are misleading and not necessarily showing the structure and associations of variables in the data.

effect of confounders can be controlled by doing regression analysis and not bias. make sure to take care of bias during planning stage. you ca find that study results are affected by bias but not ratify them during analysis

An article addressing bias in a research studies of different designs is: Terwee CB, Bot SDM, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007; 60: 34-42.

Hi Arnold,

It is always prudent do look for the known confounders mentioned in the literature of a specific field of interest but the unknown confounders are tough cookies. A method for detectin their presence and possible impact is outlined in the paper attached.

Hi Arnold,

The attached link is for a systematic review that has used a previously validated tool for quality assessment of prognostic studies. Hope this is useful.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2923776/

Hi Everyone, Thank you so much for your answers. I have modified a risk of bias assessment tool based on your suggestions.

Bias creeps in due to methodological lapses or inadequate study design (lack of sample size/power; clustering etc.), such as convenience sampling in the case of cross-sectional surveys or intervention in a particular hospital or work area. Randomization, stratification and cross validation of correlation/association with the help of reliability analysis, stratified factor analysis and its adjustment at analysis phase with hierarchical, multilevel or mixed model regression analysis are the few tools to name.

STROBE by Vandenbroucke et al. is an instrument for better reporting of several observational studydesigns including correlation studies and can - like most instruments for better reporting - be used as an assessment tool for risk of bias, furthermore most principles formulated by GRADE can be used independent of studydesign, Deeks et al. provide a summary of tools in "Evaluating non-randomised intervention studies" just as Sanderson et al. in "Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography"

There is the " assess the risk of bias (RoB) of non-randomized studies (NRS)".

http://2011.colloquium.cochrane.org/es/abstracts/b8o3-risk-bias-assessment-tool-non-randomized-studies-robans-development-and-validation-ne

If as a result of your systematic review you are proceeding to a meta-analysis of extracted non-randomised studies, then I suggest you consider assessing risk of bias using the Newcastle-Ottawa Scale. I am working with a group which has recently used it . It was easy to use and seems to have acceptable psychometric credentials. To obtain initial information about the scale, Google its name, but this is a good start:

http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

PS I would be interested to hear about the experiences/comments of others on the use of this scale.

Dear Arnold

Please visit the BA Superhelp Desk http://ssc.bibalex.org/helpdesk/introduction.jsf.

It is a global initiative by The Library of Alexandria, and the WHO Collaborating Centre at the University of Pittsburgh are launching a project to improve productivity in developing countries by establishing a Research Methods Help Desk.

Regards