You set the rules for the types of studies that can be included in your systematic review/meta-analysis. However, you have to be able to justify those decisions to future reviewers, editors, etc. To do a meta-analysis, you have to be able to extract all the numbers from a study that are necessary to calculate an effect size. For a standardized mean difference, that means you normally extract the means, standard deviations, and sample sizes for two groups or conditions. For a risk ratio or odds ratio, you extract the numbers of events and non-events.

I think you need to be clear about the impact of potential biases such as selection and maturation (of which case studies and uncontrolled studies are particularly prone) on the validity of your eventual findings. If your review is one concerned with the "effectiveness" or impact of a treatment or intervention then (for me at least) that risk is probably too great and the "value" in reviewing the (limited) RCT or experimental evidence lies in highlighting the gaps in the evidence base and highlighting what decent studies in your area might look like to generate valid findings. The caveats you would need to insert into your interpretation of any synthesis/meta analysis would likely be so great in number and content that they would render interpretation on the part of the reader nigh on impossible - i.e. increase not decrease any uncertainty that led them to your review in the first place. 

Dear Lloyd,

your question includes the three terms "systematic review", "meta-analyses" and "case series". These three terms should be explained carefully. There are several websites with helpful explanations, for example see:

http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036068

There you can find the following explanations:

Case series and Case reports consist of collections of reports on the treatment of individual patients or a report on a single patient. Because they are reports of cases and use no control groups to compare outcomes, they have little statistical validity.

Systematic Reviews focus on a clinical topic and answer a specific question. An extensive literature search is conducted to identify studies with sound methodology. The studies are reviewed, assessed for quality, and the results summarized according to the predetermined criteria of the review question.

A meta-analysis will thoroughly examine a number of valid studies on a topic and mathematically combine the results using accepted statistical methodology to report the results as if it were one large study.

Related to these definitions, you can carry out a systematic review. At first you have to define the kinds of studies you include / exclude and you shoud justify your decisions. Continuing this idea, you can carry out a systematic review with all kinds of studies (case series, case reports, case control studies, cohort studies, randomized controlled clinical trials and even already published systematic reviews), but you have to represent it seperately, you can't pool the results.

Referring to the Cochrane Collaboration (see the link: http://www.cochrane.org/about-us/evidence-based-health-care/webliography/books/sysrev), I'd like to recommend:

Centre for Reviews and Dissemination (2008) Systematic Reviews. CRD’s guidance for undertaking reviews in health care. (see file below)

There, at first case series, case reports and partly case–controlled studies are excluded from the review, because of their high potential bias (see page 7). However certain research questions require case series, in order to collect information on adverse events (see page 9). Finally, the use of case series for observational studies is described (see page 11). The methodogy of observational studies is also described (see page 40 und 43). There's a useful checklist for observational studies in:

Agency for Healthcare Research and Quality (AHRQ) (2002): Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment: number 47. (see file below)

Finally, I've added the following articles for your insight:

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, et al. (2009): The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. PLoS Med 6(7): e1000100. doi:10.1371/journal.pmed.1000100

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009): Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

Panic N, Leoncini E, de Belvis G, Ricciardi W, Boccia S (2013): Evaluation of the Endorsement of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Statement on the Quality of Published Systematic Review and Meta-Analyses. PLoS ONE 8(12): e83138. doi:10.1371/journal.pone.0083138

Schmucker et al. (2013): A protocol for a systematic review on the impact of unpublished studies and studies published in the gray literature in meta-analyses. In: Systematic Reviews, 2,:24.  doi:10.1186/2046-4053-2-24

Mark Crowther, Wendy Lim and Mark A. Crowther (2010): Systematic review and meta-analysis methodology. In: Blood. 2010; 116(17): 3140-3146.

Davey et al.(2011): Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive

analysis. http://www.biomedcentral.com/1471-2288/11/160

Garg et al (2008): Systematic Review and Meta-analysis. When One Study Is Just not Enough. In: Clin J Am Soc Nephrol 3: 253–260. doi: 10.2215/CJN.01430307

Reeves et al (2008): Including Non-Randomized Studies. Chapter 13. In: Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series. DOI: 10.1002/9780470712184.ch13

Good luck with your research project and kind regards! Detlef

Thank you all for the valuable knowledge and input.

Gyanendra, thanks for the attachments. I read the MOOSE guidelines- they appear to be a checklist to assess the studies quality / risk of bias, is this correct? I have been considering a few for this purpose and am not sure which to use: The Cochrane guidelines, NIH, NICE and now MOOSE?

I am somewhat confused about classifying some of the studies I plan to use: I don't feel they are observational, as the treatment intervention is provided as part of the study, so they would be experimental? Let me elaborate, I want to look at the effects on airway volumes and breathing indices of maxillary expansion in children. The predominant study design is to obtain a group of children whom require maxillary expansion (treatment allocation is rarely random), and then take measurements before the expansion and the same ones again after the expansion. Would this be categorised as: case series, uncontrolled clinical trial, pre & post study, time series...??

Detlef, I appreciate all the information you provided. I know that the strength of evidence of a case-series and other weaker study designs being used in a systematic review is not ideal, but the topic does not have many controlled trials existing and I want to provide the best possible 'summary' or collection of 'best-available' evidence that I can manage on this treatment effect if nothing else for consideration by practitioners on what level of evidence there is or isn't to support it, obviously part of my review will be to acknowledge the limitations in the levels of evidence available and make this understood. I feel like a broader scope including more slightly lower evidence level studies may provide a better insight then a very small number of higher level studies on the subject? 

Please keep the advise coming!

Thanks Gyan,

If I come to include a combination of say RCTs, controlled trials, uncontrolled trials (observational studies) and even prospective & retrospective ones - do I need to use the same appraisal guidelines for study quality/risk of bias for all studies for consistency? For example do I use one that includes assessing the control group, which would be necessary in a controlled trial and then mark this component as 'poor' for the studies without a control group. Or do I use different assessment scale for each different type of study design I have included? One for RCT, another for case-control, etc.?

Also Gyan do you prefer the MOOSE guidelines over PRISMA?

Dear Lloyd,

there are few dissertations of your topic "maxillary expansion" ( in alphabetical order), but they are German-speaking. Sometimes you can find an English-speaking summary:

Steffi Bössner (2006): Die Wirkung der Gaumennahterweiterung auf die Nasenatmung

im Verlauf der aktiven Dehnung bei Patienten mit transversaler Enge im Oberkiefer.

Eine rhinomanometrisch - röntgenologische Studie. Dissertation, Ludwig-Maximilians-Universität München.

Daniel Martin Haßdenteufel (2008): Die forcierte Gaumennahterweiterung - klinischer Vergleich zweier Systeme. Eine kontrollierte kephalometrische Longitudinalstudie. Dissertation, Universität des Saarlands. (summary, see page 6)

Gloria Hoppe-Walter (2006): Prospektive klinische Studie zur Therapie des unilateral posterioren Kreuzbisses mittels Gaumennahterweiterung. Inaugural-Dissertation, Westfälische Wilhelms-Universität Münster.

Dominika Scherkowski (2003): Die frühe Gaumennahterweiterung. Effektivität und Rezidiv. Dissertation, Heinrich-Heine-Universität Düsseldorf.

Julia Spindler (2012): Auswirkung festsitzender Gaumennahterweiterungsapparaturen

auf den extrathorakalen Luftraum, Epiglottis und Hyoid. Eine retrospektive kephalometrische Studie. Dissertation, Universität des Saarlandes. (summary, see page 3)

Katja Wittenberger (2006): Die Effizienz der kieferorthopädischen Frühbehandlung.

Eine Literaturübersicht über die letzten 25 Jahre. Inaugural - Dissertation, Eberhard Karls Universität Tübingen.

Hope, there are a few hints for your research project. (The bibliograhies of these dissertations seem to me very interesting, there are many English-speaking articles!)

Kind regards, Detlef

PS: To be continued, in order to answer your statistical questions.

http://nbn-resolving.de/urn:nbn:de:hbz:6-50699562455

http://docserv.uni-duesseldorf.de/servlets/DocumentServlet?id=2478

http://scidok.sulb.uni-saarland.de/volltexte/2013/5210/

http://nbn-resolving.de/urn:nbn:de:bsz:21-opus-24313

the best evidence remains mata-analysis of randomised trials. If you have only few RCT, you might present meta-analysis of observational studies.

However, if you include observation studies, you should extract the adjusted OR and perform a inverse variance meta-analysis using LogOR and SE. This is particularly important as groups might differ regarding many baselines characteristics and crude comparison of outcomes might reflect differences in baselines rather than treatment effect. 

You might present pooled OR separately for RCT and observational studies, then pooled all together and analyse heterogeneity. Always use a random model if you are dealing with observational studies due to underlying heterogeneity. 

RavMan and Comprehensive Meta-analysis allow you inverse variance LOG and SE meta-analysis. 

Thanks again everyone, apologies for all the basic questions but this area is new to me!

Gyan,

OK, so MOOSE is more targeted at observational studies and thus is more appropriate for my review. Does MOOSE have a quality/risk of bias tool that is recommended to accompany it (is this STROBE)? Similar to how PRISMA recommends the Cochrane risk of bias checklist? The Cochrane list is predominantly focussed on random assignment, concealment of allocation, blinding, etc that are areas my studies will mostly not fulfill.

And secondly, I am still getting my head around the definitions: I read the major difference between the 'experimental' and 'observational' designs is that in observational studies inferences are made about the effects of a treatment/intervention but the researchers have no manipulation over the assignment to the treatment group (i.e. patients are not randomised). I have found 3 types of observational studies (cross-sectional, cohort & case-control). So the nature of my studies fit the design of: find patient's who have narrow upper jaw -> initial measurements -> administer maxillary expansion device (some studies may have an untreated control group also, mostly these are un-randomized) -> measurements again (time to follow-up or number of follow-up measurements can vary slightly between studies) -> calculate the change in measurements over period, comparison to control group changes if control used (majority don't have a control group).

I know this is not a cross-sectional study, they also don't seem to fit as case-control studies which assess patients with a chosen outcome against controls that all received a past 'exposure'. Thus I am thinking they are cohort studies, in particular a 'quasi-experimental' study, would this be correct? But then do I classify those studies with and without controls differently - is an uncontrolled cohort (quasi-experimental) study then a 'case-series'??? And what if a study compares this treatment against another treatment, so there is a comparator but not an untreated control??? Because I can also by some definitions see this classified as an 'un-controlled experimental study'?

Yes inclusion of case series is not uncommon in systematic reviews of medical devices and surgical interventions. The terminology related to case series is confusing in the literature - it could be seen as a cohort study without a control group; sometimes people also refer to it as 'uncontrolled before-and-after study' (if a pre-intervention measure is available) or simply an uncontrolled study.

The key thing to remember is that because of lack of a concurrent control group, it is difficult to draw any confirm conclusion regarding effectiveness as it is difficult to ascertain any observed effect is a 'true' intervention effect (unless the effect is rather dramatic) - as we cannot rule out  the contribution of the natural course of the condition, placebo/Hawthorne effect, or the effect of other concurrent treatment. 

The following references might be useful:

Fitzpatrick-Lewis, D., Ciliska, D., & Thomas, H. (2009). The Methods for the Synthesis of Studies without Control Groups. Hamilton, ON: National Collaborating Centre for Methods and Tools. Available at: 

http://www.nccmt.ca/publications/10/view-eng.html

Chambers et al. Not only randomized controlled trials, but also case series should be considered in systematic reviews of rapidly developing technologies. J Clin Epidemiol. 2009 Dec;62(12):1253-1260.e4.

http://www.ncbi.nlm.nih.gov/pubmed/19349144

Dalziel et al. Do the findings of case series studies vary significantly according to methodological characteristics? Health Technol Assess. 2005 Jan;9(2):iii-iv, 1-146

http://www.ncbi.nlm.nih.gov/pubmed/15588556

Hope these help!

Dear Lloyd,

sometimes examples could be helpful in order to illustrate relations, here between systematic review and case series / case reports. (I assume, it's regularly enough to read the abstracts of the examples in order to understand the relation between systematic review and case series / case reports respectively the purpose to use case series / case reports for a systematic review.)

Posadzki et al. (2012): Adverse effects of homeopathy: a systematic review of published case reports and case series. International Journal of Clinical Practice, Volume 66, Issue 12, pages 1178–1188. http://onlinelibrary.wiley.com/doi/10.1111/ijcp.12026/full

Cramer et al. (2013): Adverse Events Associated with Yoga: A Systematic Review of Published Case Reports and Case Series. PLoS ONE 8(10): e75515. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0075515

Xu et al. (2013): Adverse Events of Acupuncture: A Systematic Review of Case Reports. Evidence-Based Complementary and Alternative Medicine, vol. 2013, Article ID 581203. http://www.hindawi.com/journals/ecam/2013/581203/

Huisstede et al. (2006): Interventions for treating the posterior interosseus nerve syndrome: a systematic review of observational studies. Journal of the Peripheral Nervous System, 11: 101–110. http://onlinelibrary.wiley.com/doi/10.1111/j.1085-9489.2006.00074.x/abstract

Cook et al. (2012): Efficacy and safety of the “mother’s kiss” technique: a systematic review of case reports and case series. CMAJ, November 20, 2012, 184(17) http://www.cmaj.ca/content/184/17/E904.full.pdf+html

Sheikh et al. (2012): Oral immunotherapy for the treatment of peanut allergy: systematic review of six case series studies. Prim Care Respir J 2012;21(1):41-49. http://www.thepcrj.org/journ/view_article.php?article_id=851

Concluding, I add two general articles, which might be interesting for you:

O’Neil et al. (2014): Observational evidence and strength of evidence domains: case examples. Systematic Reviews 2014, 3:35. http://www.systematicreviewsjournal.com/content/3/1/35

Peinemann et al. (2013): Using Multiple Types of Studies in Systematic Reviews of Health Care Interventions – A Systematic Review. PLoS ONE 8(12): e85035. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0085035

Good luck and kind regards! Detlef

http://onlinelibrary.wiley.com/doi/10.1111/ijcp.12026/full

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0075515

http://www.hindawi.com/journals/ecam/2013/581203/

http://onlinelibrary.wiley.com/doi/10.1111/j.1085-9489.2006.00074.x/abstract

http://www.cmaj.ca/content/184/17/E904.full.pdf+html

http://www.thepcrj.org/journ/view_article.php?article_id=851

http://www.systematicreviewsjournal.com/content/3/1/35

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0085035

Thanks Detlef, those last 2 articles are very relevant to my work.

I really appreciate the time, suggestions and input from you all

With regards to the quality assessment/risk of bias of my observational studies is there any current widely accepted framework? (such as the Cochrane risk of bias) I note the checklist from STROBE states: “…the checklist is not an instrument to evaluate the quality of observational research” I have gathered that this is ideally performed with a number of defined headings for various study design components and not combined into a single scale or summated value. If I develop my now based on the others around how will I know of it validity and appropriateness?

For a Systematic Review with Meta-analysis, you need studies that have two comparison groups. Case series are usually descriptive and has a single study group.

Recently, I have written a Systematic review (with meta-analysis) on a skeletal muscle relaxant for Low Back Pain. This review consists of 5 RCTs, and 2 Observational studies (100 patients in each study) without control group. For observational studies, I performed meta-analysis by comparing Pre Vs Post outcomes. For RCTs, I have incomplete data to perform meta-analysis; one study reported results as percentage of patients benefited from the intervention. The other two RCTs reported results without SEM or SD!

The manuscript is now reviewed by two reviewers;

Reviewer 1: Observational studies do not provide valid evidence on efficacy. We need randomized controlled trials for that question. Observational studies might provide information on side effects, if the sample size is large enough. But there should at least be a control group. Observational studies without a control group do not provide any information on efficacy and should be excluded.

Reviewer 2: Why did authors consider both RCTs and observational studies? This is unusual in systematic reviews where RCTs are available.

Authors should assess methodological quality using the CBRG risk of bias assessment tool for RCTs and another appropriate, validated tool for observational studies.

Should I exclude the observational studies!

I have the same  problem my review includes 2 RCTs and 2observational studies and one before and after ttt my question is to evaluate efficacy and i am so confused about excluding them and how to choise the risk of bias tool .Please can any one  helps with similar previous experience ?

Hi Basma Abdelalim,

For risk of bias, I used Moga tools for case series studies and Cochrane Back and Neck Group tools for RCTs.

You can include uncontrolled/observational studies in your systematic; uncontrolled/observational studies without meta-analysis posts fewer problems in systematic review without meta-analysis posts fewer problems in systematic review.

References:

And, the systematic review which I was talking in my previous comment (a skeletal muscle relaxant for low back pain; consists 5 RCTs, and 2 observational studies without control group) got accepted in Pharmacological Reports. I will forward the article, Clinical efficacy and safety of eperisone for low back pain: A systematic literature review, for your reference.

Good Luck

http://www.ncbi.nlm.nih.gov/pubmed/25594108

http://www.ihe.ca/download/development_of_a_quality_appraisal_tool_for_case_series_studies_using_a_modified_delphi_technique.pdf

http://www.ncbi.nlm.nih.gov/pubmed/19349144

http://www.ncbi.nlm.nih.gov/pubmed/17712019

http://www.sciencedirect.com/science/article/pii/S1734114016300433

 Sharanbasappa Durg thank you very much .

Excellent, Thank you

Is there any tool to assess the risk of bias in Case report, though?

Because I usually use a check list for critical appraisal, but I can't find a risk assessment tool for case reports or case series.

We have recently published guidelines for conducting systematic literature reviews. We used the example of SCM, but it can be adapted to other disciplines. You can find our article on my ResearchGate page. Christian F. Durach, Joakim Kembro & Andreas Wieland (2017), A new paradigm for systematic literature reviews in supply chain management, Journal of Supply Chain Management, Vol. 53, Issue 4.

Thank you very much! I will check your paper :)

This thread was highly informational.. Thank you all for sharing the knowledge..

Hello everyone !

Can you please tell me that what type of "Quantitative Analysis" can we performed in a systematic Review of Case report and Case series ? Is there any example/papers etc. related to this ?

Please guide me

Thanks for sharing all those informative messages, which are of quite help to sys rev writing.

This manual is one of the good resources for referring to all quality/bias related check-lists for cohort study, case control study, case series, case report, analytical cross sectional studies (for the check lists look at the appendix).

https://wiki.jbi.global/display/MANUAL/Chapter+7%3A+Systematic+reviews+of+etiology+and+risk

Ideally for a good quality systematic review, inclusion of prospective randomized clinical trials only will generate a strong evidence regarding the effects of a treatment modality. However, if for some reason, enough studies are not available, many systematic reviews do include non-randomized trials and retrospective studies. However, this should be the exception and not the rule. There are several bias related tools available. You can find some helpful information from our systematic review mentioned below.

Abu Arqub S, Mehta S, Iverson MG, Yadav S, Upadhyay M, Almuzian M. Does Mini Screw Assisted Rapid Palatal Expansion (MARPE) have an influence on airway and breathing in middle-aged children and adolescents? A systematic review. Int Orthod. 2021;19(1):37-50. doi:10.1016/j.ortho.2021.01.004