Cochrane recommend 

For cross-sectional/survey studies: the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies: http://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort

obviously, it is  Newcastle-Ottawa Scale to assess the quality of observational studies in systematic reviews.

https://www.editorialmanager.com/jognn/account/MOOSE.pdf

http://guides.library.harvard.edu/meta-analysis/guides

As I know for critical appraisal of nonrandomized interventsion studies Cochrane uses "The Risk Of Bias In Non-randomized Studies – of Interventions (ROBINS-I) assessment tool" See http://methods.cochrane.org/bias/risk-bias-non-randomized-studies-interventions  and http://www.bmj.com/content/355/bmj.i4919

Regads,

Ludwig

As a follow-on from the posts above, various checklists, ratings and scales for observational studies have been proposed including the Newcastle–Ottawa Scale, the Downs and Black checklist, Critical Appraisal Skills Programme checklists and the Scottish Intercollegiate Guidelines Network checklists. None of these use a contemporary domain-based approach to the assessment of bias and are dated (e.g. the current NOS version was published in 2000) and rating scales have been shown to be flawed.

As Ludwig points out, the Cochrane Collaboration has developed a comprehensive tool for non-randomized studies called the ROBINS-I tool (Risk Of Bias In Non-randomized Studies - of Interventions). It has been developed from the Cochrane Risk of Bias tool for RCTs (ROB 2.0) and assesses internal validity through a series of RoB judgments in seven chronologically organised domains to provide an overall RoB assessment for each study report. it can be found at https://sites.google.com/site/riskofbiastool/welcome/home

I hope this information is helpful.

Thank you all for the helpful suggestions.

If it helps, there's an existing discussion on this topic at:

https://www.researchgate.net/post/What_is_a_valid_tool_to_appraise_risk_of_bias_for_observational_studies 

I would recommend the Newcastle-Ottawa Scale rating the Quality of studies. See publication.

Article Exposure to benzene at work and the risk of leukemia: A syst...

Hi Carina,

If you can't find a standard checklist that fits your systematic review, you can create your own which is informed by existing checklists. As long as you can justify this in your methods it will be fine.

Dear Carina and others, 

I some time see people confused to critically understand the difference between quality assessment tools used during the (1) planning and carrying out of systematic review and (2) assessing the quality of individual articles selected for the review. Following your question, it would be also good to add the the question "WHEN"? For example the well known tools like PRISMA, STROBE and MOOSE, provide a standardized guide for carrying out systematic reviews, including constructing a protocol, testing for bias and heterogeneity, and other aspects of the review process. How ever, Newcastle-Ottawa Scale (NOS) checklist (cohort and case-control studies), Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies, etc are tools that are used to assess and rate the quality of individual articles that are included in a systematic review or meta analysis. 

More information, use and misuse of STROBE, http://bmjopen.bmj.com/content/1/1/e000048

List of methodological assessment tools,  https://www.ncbi.nlm.nih.gov/pubmed/25594108

I am using NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies to assess and rate the quality of individual articles that are included in a systematic review.

However, I am not sure as how to rate the 14-criteria tools for as good, fair or poor?

Example:

Is 10/14 Yes is considered Good?

Please advice.

Thanks in advance.

Dear Mastura Ahmed,

First, please check if all 14 check lists are suitable for your planned research topic. As far as I know, there is no black-and-white rule for that, but I can suggest you how we did.

You can categorize them as: >80% yes= High, 60-80% yes- Medium, and

Hi,

I was wondering whether anyone could provide a reference for where cochrane recommend using the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies? Just I am struggling to find anything that supports this.

Thanks,

Lucy

Thank you very much for your suggestions.

Finally, we decided to assess the methodological quality of the studies independently by two authors, working first separately and then together (any final disagreement was solved by a third author), as recommended by PRISMA. We used the National Heart, Lung, and Blood Institute (NHLBI) Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies. We adapted question number five to better capture the strengths and weaknesses of the studies (as detailed in the supplementary material). To further increase the reliability of the process the inter-rater reliability (Kappa coefficient of Cohen) was measured. I hope this will help you.

For a detailed description, please see:

Carbia, C., López-Caneda, E., Corral, M., & Cadaveira, F. (2018). A systematic review of neuropsychological studies involving young binge drinkers. Neuroscience & Biobehavioral Reviews. DOI 10.1016/j.neubiorev.2018.04.013

Thats a great discussion thread. I am just wondering whether there is any criticism on using the NIH tools for Observational Case-control and cross-sectional studies. I mean is there any reason for not using it?

"The questions on the form are designed to help you focus on the key concepts for evaluating the internal validity of a study. They are not intended to create a list that you simply tally up to arrive at a summary judgment of quality"

This is what I have found in every section of the NIH quality assessment tool. I wonder if its still ok to use it as a scoring tool?

There is no version for cross-sectional study of Newcastle-Ottawa study and I found some studies modified it? Which version do you use? May I also know your experience with the NIH tools for Observational Case-control and cross-sectional studies?

Kind Regards

@ Tafzila Akter Mouly

https://journals.plos.org/plosone/article/file?type=supplementary&id=info:doi/10.1371/journal.pone.0147601.s001

You can find an interesting paper on uses and misuses of the STROBE guidelines here. Using STROBE for quality assessment, even for systematic reviews is a widely spread misconception.

https://bmjopen.bmj.com/content/bmjopen/1/1/e000048.full.pdf

Best wishes

Dear All, are there any specific criteria to select the quality assessment tool for observational studies in a systematic review?

For systematic reviews, I suggest using the Newcastle-Ottawa Scale (NOS).

here is a link:

http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

Great discussion. I wonder if there is any objective criteria to define the study quality as poor, fair or good on NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies? Are there any cutoff points?

Newcastle-Ottawa Scale (NOS) is a good option for case-control and cohort studies. I found the staring system easy to conduct and time saving. It is also easier for the readers to interpret.

For observational studies, the most preferred tool according to the Cochrane handbook is the New castle Ottawa Scale (NOS).

Thank you, I am working on a systematic review of the literature looking at the prediction of in-hospital cardiac arrest using neural networks. The studies I have found are all observational cohort studies. I plan on using NOS to assess the risk of bias, is this appropriate in the setting of machine learning?

NiHR and NOS both can be used, but if you have decide between these two tools, is there any criteria or guideline to choose one over another?

I used the results from the Farrah et al 2019 article: Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Risk of bias tools in systematic reviews of health intervention...

NOS and ROBINS-I were the most commonly reported tools for non-randomized studies.

For cohort and case-control studies, we recommend the use of the Newcastle-Ottawa Scale

Agency for Healthcare Research and Quality (ARHQ) methodology checklist is applicable for cross-sectional studies

Interesting discussion. Thank you all for your suggestions.

Is there any tool specifically designed to assess quality of case series? I am preparing a systematic review of a topic in which there are no RCT, no cohort studies... which tool would you recommend?

You can refer the tool developed by Moga C et al.

Moga C, Guo B, Schopflocher D, Harstall C. Development of a Quality Appraisal Tool for Case Series Studies using a Modified Delphi Technique. 2012.

Where can I find the scoring criteria for the NIH quality assessment tool?

please check the following links

https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools

https://www.nihlibrary.nih.gov/services/systematic-review-service/tools-resources

Thank you so much Fayaz Khan !

The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses, it is available in different versions ( case-control studies or cohort studies)

http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

How do you categorise the included studies into "good" or "fair" or "poor" using NIH quality assessment tools' checklist for controlled intervention studies? Can someone help please!?

@Arghya Kundu Choudry

I have seen some published articles that set a number threshold to the yes/no answers to classify it. Though not always the way to do it. I have also seen that the two raters (or more) subjectively decide dependent on the yes/no answers and then discuss any conflicts to reach concensus.

Amalia Gil

do you have any references for these thresholds, for quantifying the quality assessments?

Also what are the exact thresholds, if you have any idea, will be really grateful!

thanks

I would agree with Andrew, they are not meant to be numerical thresholds, though I have read some articles that do it numerically. These articles don't have a basis, they randomly assign numerical thresholds. I think the move of most systematic reviews are to now do it subjectively, being guided by the questions.

Thank you so much for the help Andrew and Amalia.

I didn't refer to any numerical criteria actually. Was just looking forward to get some exact threshold values for categorising articles into "good", or fair or poor quality.

May be like if yes to >50 % questions then good, at 50 then fair or < 50% then poor, something like, which I had read somewhere.

Though I agree with Andrew too, that the analysis should be based subjective parameters.

Thanks for updating me on this, though if you can provide me with some informations, I will be more than happy!

Thanks

is NIH considered a standardized tool as CASP &Joanna briggs

?

Yes NIH tool is a standard tool for assessing the quality of studies

Thanks

Is it suitable to specific kind of research or any type?

Quality assessment is necessary for systematic reviews and meta-analysis

You can use EPHPP quality assessment tool.