I would recommend looking at the checklists from STROBE. See link below:

http://www.strobe-statement.org/index.php?id=available-checklists

I would suggest the Newcastle-Ottawa scale to assess the quality of the retrospective studies. In particular if you are going to enroll studies for systematic reviews and/or meta-analyses.

Many thanks for all your valuable replies. My personal view on the STROBE and Cochrane checklists is the former helps to assess completeness of reporting an observational study rather than appraise quality while the latter is only applicable to intervention type studies, more specifically RCTs. Shall check out the Newcastle-Ottawa scale.

Best wishes,

Nisa

We have developed Liverpool Quality Assessment Tools (LQATs) that have now been quite extensively used in a variety of systematic reviews of observational studies - publication later this year. The tools were to address a number of limitations in other assessments (including NOS). They cover all study designs (including cross-sectional), allow comparison of quality across designs, measure all main sources of bias and misclassification for a given research question (eg adaptable) and use a reference score for sensitivity analysis in MA. Happy to share if you are interested (please email). Dan.  

Dear Dr. Souza,

You are right. There are easier tools intended to critical appraisal, Please, try checklists from Critical Appraisal Skills Programme (CASP).

With best wishes.

http://www.casp-uk.net/#!casp-tools-checklists/c18f8

The Methodological Index for NonRandomized Studies (MINORS) may be a good option. http://www.ncbi.nlm.nih.gov/pubmed/12956787  I used it in one of my previous meta-analysis where I pooled cohort studies. http://www.ncbi.nlm.nih.gov/pubmed/26215715

You should try the Newcastle-Ottawa scale. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

I am currently using this one for observational studies, I do not think it is valid but it is widely used:

https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort

Attached also is a systematic review that I recently conducted. The quality appraisal tool was adopted from previous reviews and amended to fit the aims of the attached review.

Article Testicular Cancer Awareness and Screening Practices: A Syste...

This tool seems easier to use than the Newcastle-Otawa. The checklist is very complete.

Several tools for assessing the methodological quality of observational studies can be identified. Deeks et al. identified 182 in 2003. Of these the Cochrane Handbook chapter 13.5.2.3 Tools for assessing methodological quality or risk of bias in non-randomized studies recommended to use the Downs and Black instrument and the Newcastle-Ottawa Scale (NOS). However recent studies (Hartling et al 2013; Stang 2010) have questioned the reliability and validity of the NOS scale.

The Center of research dissemination (CRD) has set some criteria’s for evaluating observational studies in CRD report 4 – Study quality assessment (separated in case-control, cohort and case series) but their criteria’s for cohort studies are missing the evaluation of the validity and reliability of the outcome. One way of dealing with that is the method in Øiestad et al. 2015 where criteria’s from CRD are added one criteria from the Black and Down about the outcome measure. Another possibility is to use the Scottish International Guideline Network (SIGN) checklist. The STROBE checklist is very useful tool for evaluating the reporting of observational studies.

It is important to differentiate between tools that:

1) Assess the level of evidence at the study level

2) Assess the level of evidence at the outcome level

3) Assess the methodological quality of each study

There is a huge difference between these 3 points

Absolutely in agreement... and this has been an issue for sometime even in Cochrane.. but the tools mentioned above are a good starting point... 

Thank you all for your helpful answers!

An update: Cochrane recently released a risk of bias tools for non-randomized controlled interventional studies called ROBINS-I. You can find it here www.riskofbias.info It is actually an updated version of the ACROBAT-NRSI. I find that ACROBAT is still useful when appraising case-control studies. Welcome any thoughts.

Hi

I am conducting a systematic review exploring the association and I will include cross sectional studies what quality assessment tool you would recommend I thought about NOS

Any advice

good information

Interesting discussion, I will like to know how helpful it was to use ROBINS-I to assess the risk of bias for your study since you stated at the beggining that you need a tool for observational studies. I have always seen the ROBINS-I tool as being only appropriate for Non randomised studies of intervention as indicated by the name of the tool.

I am currently conducting a review on a set of selected observational NRS (cohot and case control) and I am finding it very difficult to settle on an appropriate RoB assessment tool to use.

Please any assistance will be very much appreciated.

Thanks a lot in advance.

Hi Evans,

You are not alone. I too have difficulty using ROBINS-I for observational studies as this tool is meant for non-randomized clinical trials on intervention. I resort to using the Newcastle-Ottawa scale but thanks to Sifan for updating us about ROBINS-E! It's definitely worth checking out.

Cheers,

Nisa

Thanks a lot Nisa,

I have actually already used the NOS for study quality assessment but I need a RoB tool for the assessment Risk of Bias.

I will check it out.

I am very grateful for your prompt response.

Regards

Evans

Hi Evans and Nisa,

My search for established valid tools for assessing risk of bias in observational studies brought me here.

I have in the past used appraisal tools from the Joanna Briggs Institute (JBI). (Aromataris E, Munn Z. Joanna Briggs Institute Reviewer's Manual. The Joanna Briggs Institute, 2017. Available from https://reviewersmanual.joannabriggs.org/.) They have tools for all kinds of studies. The goal of the checklists is to determine the extent to which a study has addressed possibilities of bias in its design, conduct and analysis.

I had concerns earlier on when I was not finding many articles in my area of study using the JBI guidelines for assessing methodological quality in reviews. Keep in mind also, that the journals sometimes dictate what type of tools to use.

I have recently come across a protocol of an on going study to update the PRISMA reporting guidelines for systematic reviews and meta-analyses.

(Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T., Mulrow, C. D., ... & Moher10, D. (2018). Updating the PRISMA reporting guideline for systematic reviews and meta-analyses: study.) The group intends to include the manual from JBI (one of the over 30 guidelines listed recommended) in their review for the PRISMA update. Seeing that protocol, I am confident of the rigor of assessment methods provided by the tools from JBI and I hope this update will provide an endorsement for widespread use (if it's not already popular).

Regards

Nana

Hello Nana,

Thanks a lot for adding your voice to this discussion. I will take a look at the JBI reviewere's manual as well.

Thats a great discussion thread. I am just wondering whether there is any criticism on using the NIH tools for Observational Case-control and cross-sectional studies. I mean is there any reason for not using it?

Hi, i am conducting a review which is looking into "relationship between two diseases" regardless of exposure and doesn't involve any intervention. So, i am having difficulty in deciding the ROB tool for it. What do you guys think about ROBINS-I or ROBINS-E tool utility in my case? or if somebody can recommend any other tool which will be more suitable

I would also recommend that you take a look at the RoBANS Tool for non-randomized studies. Article Testing a tool for assessing the risk of bias for nonrandomi...

Hello!

About ROBINS-E tool, online, I can just download a pdf or doc file? Is there any excel file like RoB tool ? I am new to this subject, so I dont really understand , for ROBINS-E , how to calculate/define bias because there is no score....(just questions)

If someone can clarify that for me...

Thanks in advance !

I see that the final tool is still under construction and the online tool is preliminary only.https://www.bristol.ac.uk/population-health-sciences/centres/cresyda/barr/riskofbias/robins-e/

Hello! Does anybody know what tool can I use for assessing the risk of bias for a case series studies without control group? Thanks!

For case series I used this one: Development of a quality appraisal tool for case series studies using a modified Delphi technique (MOGAS´S Tool)

Mam u can use MINORS criteria(https://onlinelibrary.wiley.com/doi/full/10.1111/jep.12889) and its feasible to use. Or the new ROBINS-1 by Stearns et.al. criteria(https://www.bmj.com/content/355/bmj.i4919)

Well, for me is difficult to find a tool to appraise the risk of bias in systematic reviews!!!

Neida Neto Vicente Ramos I have used the ROBIS: Tool to assess risk of bias in systematic reviews. I am not very experienced with it, however. It is recommended by NICE.

Hi, I think that the best tool to assess the risk of bias of observational studies is the NOS checklist.