Hi Mohammad,

It is important to take into consideration the overall aim of your systematic review, what you are hoping to achieve in the review. This will play an important role in the questions that you should use in the Methodological Quality Appraisal tool. A good site for guidelines on checklists for reviews is the Critical Appraisal Skills Programme (CASP): http://www.caspinternational.org/?o=1012

Once you have identified you aims of the review, you can choose questions that would yield the most desirable results that would add to the field of knowledge from the various checklists and quality appraisal tools (something which I have done in a recently published systematic review of mine, you could have a look at it: https://www.researchgate.net/publication/268505840_A_systematic_review_of_the_relationship_between_parenting_styles_and_childrens_physical_activity)

All the best,

Eugene

Article A systematic review of the relationship between parenting st...

I would suggest Strobe statement and to use its checklist. You can find it on: http://www.strobe-statement.org/

STROBE stands for an international, collaborative initiative of epidemiologists, methodologists, statisticians, researchers and journal editors involved in the conduct and dissemination of observational studies, with the common aim of STrengthening the Reporting of OBservational studies in Epidemiology.

The STROBE Statement is being endorsed by a growing number of biomedical journals.

Hello Eugene,

I'm reviewing articles on testicular cancer awareness and testicular self-exam practices among men. It is going to being a mixed method systematic review so I am already using the CASP for my qualitative studies. Is there a CASP tool for descriptive cross-sectional studies?

I have actually seen reviews that pick some items of certain quality assessment tools instead of using the tool as a whole(they pick items that apply basically). I will take a look at your  review, thanks for sharing.

Dear Giedrius,

Is the STROBE statement applicable for descriptive cross-sectional surveys. I found this http://www.strobe-statement.org/fileadmin/Strobe/uploads/checklists/STROBE_checklist_v4_cross-sectional.pdf on the STROBE website but I guess it assesses the availability of content rather than quality of studies.

Thank you for your valuable feedback

M.

It include questions for both descriptive and analytic design CS, so you have to be selective.

Dear mates, 

We developed in 2008 a tool for cross-sectional studies because we didn'f find any tool for assess thes kind of studies. It is in Spanish, a I was very sorry, i do not have the article if you are interested in.

[Aroot for the critical appraisal of epidemiological cross-sectional studies.]

Silvina Berra, Josep Maria Elorza-Ricart, Maria-Dolors Estrada, Emília Sánchez

[Hide abstract]

ABSTRACT: The aim was to develop a tool for the critical appraisal of epidemiological cross-sectional studies. Several recommendations or guidelines for assessing the strength of scientific evidence provided by observational studies were reviewed, like those from the Agency for Healthcare Research and Quality, the Scottish Intercollegiate Guidelines Group, the Osteba (Basque Office for Health Technology Assessment), and the STROBE Initiative. The tool has 27 items to assess: study question or objective, participants, comparability between groups, definition and measure of main variables; analysis and confusion, results, conclusions, external validity and applicability, and conflict of interest. This tool can be used to critically appraise research papers or to rate evidence during the elaboration of systematic reviews.

Gaceta Sanitaria 09/2008; 22(5):492-497.

Hi Mohammad,

you may find this paper somehow useful.

"The Development of a Critical Appraisal Tool for Use in Systematic Reviews: Addressing Questions of Prevalence" (DOI: 10.15171/ijhpm.2014.71). Here is the link: www.ijhpm.com/article_2870_607.html 

Thank you all for your valuable feedback. I took a look at all your links and adapted one tool for my quantitative study that I will share with you once I hopefully get my review published. I also used the CASP tool for the  qualitative studies and another tool by CASP to review systematic reviews.

Best

M.

According to the authors, using STROBE to assess study quality is inappropriate

http://bmjopen.bmj.com/content/1/1/e000048.full

Thank you Cristiano. 

 Hi Mohammed, I am currently undergoing my first systematic review so I am very new at this. I did look into a few appraising tools and found the CASP tool to my liking.

Mohammed you asked the question "Is there a CASP tool for descriptive cross-sectional studies?" and that is exactly what I am trying to find out. Did you find an answer to your own question? 

Hello Sarah,

I am after submitting my third systematic review and found several tools to assess the quality of cross-sectional studies. However, I used the tool that was used in this publication:

https://www.researchgate.net/publication/268505840_A_systematic_review_of_the_relationship_between_parenting_styles_and_childrens_physical_activity

It is pretty straightforward and user friendly. I did however make a few changes on it.

Please find attached a file with the table that I used. The references for the tool are in the table.

PS if you are reviewing intervention studies, go for the QAT

http://www.nccmt.ca/registry/view/eng/14.html

If you are reviewing qualitative studies or systematic reviews go for CASP

http://www.casp-uk.net/#!casp-tools-checklists/c18f8

All the best and let me know whether I can be of further assistance. 

M.

Article A systematic review of the relationship between parenting st...

Hello Mohamad,

I'm writting my second systematic review. I was looking for some checklist for cross-sectional studies. I found STROBE is inappropiate to assess study quality. I'm thinking about using the Quality Assessment Tool from NHLBI

Hello Davinia,

I did not find it appropriate either. I used a tool designed in 4 previous reviews and my systematic review got published. The tool appears in the full text systematic review that I posted earlier among my publications. Feel free to take a look at it. Alternatively, the NHLBI tool looks quite appropriate. The choice of the tool depends on tbe design of the studies sought. I used the CASP tool to review qualitative studies, another CASP tool to review systematic reviews, and the QAT tool to review intervention studies (recommended by Cochrane)

Best

Mohamad  

One more thing. If you are reviewing intervention studies, in addition to the the QAT tool, you must assess the level of evidence per review outcome using the GRADE tool.

Thank you for your answer Mohamad.

Actually I'm reviewing prospective studies so I'm going to use NHLBI to evaluate the quality.

Best,

Dear Mihretu,

You are right it is widely used in Observational studies. However, Da Costa et al (see attached) has confirmed, the use of this tool for quality assessment is inappropriate.

Thanks you guys we put your remarks above. QAT and CASP Tool respectively for Quant (interventional) and Qual studies assessment.

Thanks! This discussion has been very helpful as I'm doing a systematic review on cross-sectional studies. I was advised to use the "RTI item bank on risk of bias and precision of observational studies". Does anyone have experience with this quality assessment tool? I am now wondering whether to use the QAT instead. Any comments on this?

Hi all,

Around the same question can one use more different critical appraisal tools for an academic literature review? Most of the studies for my review are descriptive type which aim to use CASP. I have a few surveys and CASP don't seem to have a tool specifically for this. Any thoughts?

Thanks

Below are tools that I have previously used, are easy to use, and some of them are recommended by Cochrane:

For qualitative studies: the CASP tool http://media.wix.com/ugd/dded87_29c5b002d99342f788c6ac670e49f274.pdf

For systematic reviews: the CASP tool http://media.wix.com/ugd/dded87_a02ff2e3445f4952992d5a96ca562576.pdf

For cross-sectional/survey studies: the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies: http://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort

For intervention studies: the EPHPP tool http://www.ephpp.ca/tools.html (first two files)

For risk of bias assessment for interventions: the EPOC criteria https://www.biomedcentral.com/content/supplementary/2046-4053-3-103-S2.pdf) 

For level of evidence assessment per outcome: the GRADE tool http://cccrg.cochrane.org/sites/cccrg.cochrane.org/files/public/uploads/how_to_grade_revising_1_december_2016.pdf

Thanks Mohammed. I think I will go with the NIH Quality assessment tool.

Dear Mohamad,

Previously, you stated that the NIH QAT for Observational Cohort and Cross-Sectional Studies is recommended by Cochrane. I have looked vigorously to find this recommendation by cochrane, but could not find it.

Could you please include the information with which I can validate my use of the NIH QAT?

Thank you,

Rik

Dear Rik

Only the CASP tools and the EPHPP tool are recommended by Cochrane but the NIH tool is being increasingly used and its use in a publication mustn't be problematic as long as you address all the items in it.

I hope this helps

M. 

Dear Mohamad,

I first reach the paper regarding the Misuse of STROBE, then keep searching the correct way to do the systematic LR and arrive NIH one. And at last, I found this discussion. This discussion is very helpful and clear all my doubt. 

I just started to do a sys. LR. Can I further ask a question that whether there are any guidelines mentioned the included literature's quality level? Will we include/exclude a literature based on a cut-off score from the QA tool or based on my judgement of how much internal validity is affected by the bias to decide include/exclude? 

Regards

Terry

I would support the use of the STROBE tool and I would suggest you to develop your own quality tool. You could use sample size, appropriate or inappropiate sample selection (to assess the risk of selection bias), appropriate diagnostic test (or not) to diagnose the disease or condition you want to know, and so on.

Standard tolos are sometimes useful but are not very useful if you want to discriminate studies with a similar design or sample size but there are other aspects related to the disease of interest that could be relevant to assess the study quality.

Dear Alberto,

The main problem is STROBE is not a tool for quality assessment

Thank you all for your feedback. I did develop my own tool based on previous works. I am attaching my article FYI.

Hi Mohamad M. Saab,

You mention in this thread that the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies has been recommended by Cochrane.  I was just wondering if you have a reference for this, or if you can point me towards where you attained this information from?

Many thanks,

Lucy

 Dear Lucy,

Your question is similar to mine in this thread. I suspect that you misread it just like I did. Indeed, Mr. Saab does not state that the NIH QAT was recommended by Cochrane. Instead, he says that only the CASP tool and the EPHPP are recommended. He continues with mentioning that the NIH is increasing in popularity and one should not have problems when using it if used correctly.

I asked the question at Sep 27, 2016, if you wish to scroll up to find it.

Indeed, 

The NIH tool was not recommended by Cochrane but is widely used. 

M

ROBINS-I is used for interventional non-randomized studies and for this reason it is not recommended for cross-sectional studies. Actually I'm using the Joanna Briggs Institute Checklist for analytical cross-sectional studies. 

You can find it here: http://joannabriggs.org/research/critical-appraisal-tools.html

Check the tool for prevalence studies of Roy et al. (2012)

Hoy D, Brooks P, Woolf A, Blyth F, March L, Bain C, Baker P, Smith E, Buchbinder R. Assessing risk of bias in prevalence studies: modification of an existing tool and evidence of interrater agreement. Journal of clinical epidemiology. 2012 Sep 30;65(9):934-9.

STROBE Statement-checklist of items:

http://www.strobe-statement.org/index.php?id=available-checklists

Dear Vesna

The STROBE statement helps STrengthen the Reporting of OBservational studies in Epidemiology. In other words, it is a checklist just like PRISMA and cannot be used as a quality appraisal tool

M

Dear Mohamed and all

Do you have examples of how to assess confounders in question 2 of EPHPP?

I wonder how to indicate the percentage of relevant confounders. If I could get an example of the descriptive cross-sectional survey studies assessment will be very helpful.   And can we assess item G, intervention integrity, if the study is the survey? Thank you in advanced.

Hello Jutamanee

Please find below text from the EPHPP explaining this item:

By definition, a confounder is a variable that is associated with the intervention or exposure and causally related to the outcome of interest. Even in a robust study design, groups may not be balanced with respect to important variables prior to the intervention. The authors should indicate if confounders were controlled in the design (by stratification or matching) or in the analysis. If the allocation to intervention and control groups is randomized, the authors must report that the groups were balanced at baseline with respect to confounders (either in the text or a table)

As none of my studies reported on con-founders, I chose the option 'Can't Tell' which gave this items a Weak rating

I hope this helps

M

Dear Mohamad,

Thank you.

PJ

Hi Dr. Saab,

Thank you for the great discussion!

I am working on my first systematic review and it is on the prevalence of OA in COPD patients and I have got both cross-sectional and retrospective cohort studies. I can not find a suitable quality assessment tool but i am thinking to use the critical appraisal tool suggested in this article:

Munn Z, Moola S, Riitano D, Lisy K. The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. Int J Health Policy Manag 2014; 3: 123–128. doi: 10.15171/ijhpm.2014.71

I would be more than thankful if you you provide me with your thoughts/suggestions.

Best  

Hello Adnan,

I recommend looking at commonly used tools such as JBI: http://joannabriggs.org/research/critical-appraisal-tools.html 

and CASP:  http://www.casp-uk.net/casp-tools-checklists

I hope this helps

M

CASP tool is high;y recommended by many researchers 

I want to use NIH QAT assessment tool for my review. Can you suggests me some references or citations of this Tool?

Please follow the referencing guidelines as appropriate. Below is the reference using APA:

National Institute of Health (2014). Quality assessment tool for observational cohort and cross-sectional studies. Retrieved from https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort 

Hi Prof. Mohamad.

What do u suggest for using a good quality tool to assess observational descriptive studies for a systematic review? Im a psychology phd student and this is my first SR.

Greetings!

Hello Cristobal

See whether either of the below tools answers your question:

https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort

http://docs.wixstatic.com/ugd/dded87_5ad0ece77a3f4fc9bcd3665a7d1fa91f.pdf

Best of luck

M

Thanks you.

Hi every body,

I'm writing a systematic review about cross-sectional studies (prevalence), and from above advises, I'm intent to use NIH QAT, but I can't find in the guidelines an "overall score"; which I can use to evaluate the studies: good, poor,.....

Can anyone help me to get the scoring system, or give me an advice how to evaluate my studies?

Thanks..

Hello Sajed

QA instruments nowadays are moving away from providing an overall score. Instead, you can comment on the limitations of the studies based on your appraisal.

Hello All, I had planned to use the NIH QAT for a review I'm working on. Now, for some reason, I cannot seem to find it anywhere on their website (none of the above links are working). Does anyone have a pdf of the document they could share? I would appreciate it greatly!

Hello Lean,

This is the PDF..

Yes it seems like they removed it for some reason. I found another instrument called AXIS which more or less uses the same criteria:

http://bmjopen.bmj.com/content/6/12/e011458

Hi Mohamad!

There is no validated quality assessment tool out there, still you may use some of them by doing modified for yourself. Note that many researchers use STROBE for observational studies and CONSORT for RCT to assess the studies quality, this is completely misused of them. Here I send you a review on quality assessment based on six potential biases that may use to assess quality of the studies.

There is a movement away from quality assessment towards risk of bias. This movement is been driven by the Cochrane Collaboration. A risk of bias tool is strongly recommended over the use of quality assessment scales because the latter do not provide reliable measures of bias, and the summary scores they produce are difficult to interpret due to uncertainty over how each scale item should be weighted (Higgins and Green, 2011)

My collagues and I undertook a SR and used criteria based on guidelines for ‘STrengthening the Reporting of OBservational Epidemiological’ (STROBE) studies which Sanderson et al. (2007) reported as incorporating the key principal sources of bias. These criteria are: selection bias; measurement bias; design specific bias; confounding bias; statistical method bias; and conflict of interest or funding sources.

Here the link to the article:

https://www.researchgate.net/publication/319984547_Stress_in_Fathers_in_the_Perinatal_Period_A_Systematic_Review

I used a validated reference that was published by Viswanathan et al. (2012) to evaluate the risk of bias and precision of observational studies. The RTI Item Bank on Risk of Bias and Precision of Observational Studies includes a range of different study designs, and the authors have provided instructions regarding what items to use depending on the studies under assessment.

Viswanathan M, Berkman ND (2012) Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol 65, 163-178.

I will be using the National Heart, Lung, and Blood Institute (NHLBI) Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies - what are the cut-off scores to be allocated to Good, Fair, Poor? Appreciate your replies.

Bonnie, there are no cut off scores there's only an overall rating (which is a limitation I guess). Check the description per item for more details

https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools

M

Thank you :)

Dear Bonnie,

You may use cut-off points, but you will have to explain it- how and why? Here I attach an article where you will find the answer.

Dear Mohamad,

I am currently undergoing a systematic review based on cross sectional studies and some systematic reviews. This conversation was helpful me a lot to found out a good tools for mine. [ NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies & for systematic review and meta analysis].

So, thank you so much!

Best,

Uvini

Hello Mohaned

The tool that you should use depends on what you are doing . if it is a systematic review then you are assessing the risk . In that case it is better to use robins I took which a Cochran’s tool used for nonrandonized clinical trials and observational studies. If it is needed just for appraisal purpose or writing a commentary then you can use a quality assessment tool like Casp. Good luck

Iman Radi

Professor of prosthodontics

Member iof Evidence based dentistrt centre

Casp egypt trainee

Does anyone have a reference that supports the use of CASP tools ( Either QUAL / COHORT ) ? To be clear , I know CASP is recommended by COchrane but I can not find this evidence ? Any help please thanks in advance

Hoda

You can use JBI critical appraisal tool for descriptive cross sectional studies.

I will be using NIH for cross sectional quality assess. in my SR

thank you Mohamad

BW

Munirah

Dear mohd,

could you tell me which quality appraisal tool is best for cross - sectional and quantitative studies?

Dear Ambili Markose

See attached two options.

M

Thank you sir

Thank you all for the eye opening.

could somehow help me assess the quality of the attached tool and also give the the reference.

Franklin Ngwu see full text below, this is a good tool:

https://bmjopen.bmj.com/content/bmjopen/6/12/e011458.full.pdf

Wonderful discussion! It is very helpful who are trying to find a tool for quality assessment of cross-sectional studies for a systematic review. No tool is the best one, rather we have to decide as per our need for specific systematic review and nature of studies available. From above one I see that one of the given can be used for the purpose according to your need:

1. Appraisal tool for Cross-Sectional Studies (AXIS tool) - developed specifically for cross-sectional studies

2. NHLBI/ Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies (Some questions are not applicable for cross-sectional studies)

3. JBI Critical Appraisal Checklist for Studies Reporting Prevalence Data

4. JBI Critical Appraisal Checklist for Analytical Cross Sectional Studies (Good for analytical studies)

In addition to these, NewCastle Ottava also is a simple tool to use, many systematic review have used it but now I will prefer the AXIS tool.

Cochrane's Robins-I (ROB) is more appropriate for the studies involving assessing the interventions

Hi.

All your comments are really helpful.

I´m a PhD student and I´m doing my first systetic review. I have cohort and cross-sectional studies. So which quality appraisal tool is best for them?

Thanks in advance.

For systematic reviews, I suggest using the Newcastle-Ottawa Scale (NOS).

here is a link:

http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

Although not the most comprehensive tool and as with any it has it's limitations, the Newcastle-Ottawa Scale (NOS) can be used for all cohort, cross-sectional and case-control studies. An adapted version of the tool can be used for cross-sectional studies. Here is a link: https://journals.plos.org/plosone/article/file?type=supplementary&id=info:doi/10.1371/journal.pone.0147601.s001.

I just wanted to to say how much I learned from this and appreciate Dr. Saab babysitting this thread for 4 years! Mohamad M. Saab thank you!

Hoda K. Hassan I don't think Cochrane recommend the CASP tools, as cochrane systematic are conducted using RevMan. RevMan only has ROB tool. You may use other tools available for your systematic review such as JBI tools, CEBM or NHLBI tools which are equally useful and cited in many systematic reviews.

Indeed, a number of instruments "sprouted" since I created this thread 5 years ago. I highly recommend JBI tools (for now!). My students and I find them easy to use. JBI quality appraisal tools are available for almost every study design.

If the study is observational then the closest tool is the Cohort checklist (check the items to see if they apply to the study that you are reviewing). If the study involves a cross sectional survey then the tool is the cross sectional analytical checklist.

Hi everyone, I am looking for a CAT for cross sectional studies. I am completing a systematic review and need to assess individual study quality. I was looking at a recent study comparing the NOS with AXIS. Axis found to have low inter-rater reliability due to the amount of interpretive questions, however they say NOS is more suited to expert users. I would be a novice. I like the look of the JBI CS analytical checklist but is it validated? I'm so confused! Thanks in advance for any help!

Claire

Claire McRedmond I am personally using JBI tools and published a few papers with same.

Mohamad M. Saab thank you kindly for this.

No problem Claire McRedmond also check http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf

This tool can be used for multiple designs, you just 'mix and match' items based on study design. I have used it as well.

Mohamad M. Saab Dear Dr. Saab,

Thank you so much for this thread, I have learned a lot already.

I am embarking upon my first SR and the papers I am appraising include qualitative, descriptive cross-sectional and mixed methods papers, as well as a systematic review.

I was thinking of using the following tools:

For the SR: CASP

For the mixed-methods papers: MMAT

For the descriptive cross-sectional papers: JBI (would you recommend this over AXIS? I am unsure of which to use)

For the qualitative: CASP

Do you think this is the best selection of tools, or would I be better off trying to use JBI for as many of them as possible? (I cannot seem to find a descriptive cross-sectional tool from JBI but I wonder whether the analytical tool could be used).

In addition, I read an earlier post in this thread regarding the use of Risk of Bias tools. Is this something you use in your papers, and would you recommend doing so? Which tool is best for this? I looked at the RTI Item Bank for Risk of Bias but it focusses on interventions and exposures whereas my paper looks at attitudes of healthcare professionals so it doesn't seem to be the best fit.

Again, thank you so much for your help and dedication seeing this post through nearly 6 years!

All the best,

Rhianna

Hi, can anyone tell me how to cite the following NHLBI quality assessment tool?

For cross-sectional/survey studies: the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies: http://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort

I think, 1. Stratification. Instead of looking at all factors together, stratification separates data so you can identify patterns and specific problem areas

• 2. Control chart. This helps you see how processes historically change using controls. The chart helps you find and correct problems as they happen, predict a range of outcomes and analyze variations.

Hi, can anyone help me know how can i cite the JBI tool for critical appraisal.

https://joannabriggs.org/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Analytical_Cross_Sectional_Studies2017_0.pdf

Thank you in advance.

Hiba, the reference is on p. 4/7 of the PDF you linked:

How to cite: Moola S, Munn Z, Tufanaru C, Aromataris E, Sears K, Sfetcu R, Currie M, Qureshi R, Mattis P, Lisy K, Mu P-F. Chapter 7: Systematic reviews of etiology and risk . In: Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual. The Joanna Briggs Institute, 2017. Available from https://reviewersmanual.joannabriggs.org/

Thank you Nassim Tabri appreciate it.

Casp checklist is one of the best for quality assessment ,, fir risk assessment u can use robins-i