Systematic Reviews and Meta-analyses

Information Overload

The sheer volume of material published in medical journals each week is well beyond any of us to keep up with, and in order to save us from drowning in information the writers of systematic reviews aim to collect together and appraise all the evidence from appropriate studies addressing a focussed clinical question. The Cochrane Collaboration has been working at this task for the past nine years and now has 1837 completed reviews on the Cochrane Database of Systematic reviews and a further 1344 protocols that will become reviews in the future.

The File-Drawer Problem

So what was wrong with the traditional narrative review from an expert in the field? The previous emphasis has been on understanding the mechanisms of disease and combining this with clinical experience to guide practice.(1) The main problem with this approach is that we all have our preferred way of doing things, and there is a natural tendency to take note of articles that fit in with our view. We may cut these out and keep them in our filing cabinet, whilst articles that do not agree are filed in the rubbish bin. This means that when asked to review a topic it is natural for an expert to go the drawer and quote all the data that supports their favoured approach.

What is a Systematic Review?

So how is a systematic review different? Let’s start with a definition:

Systematic review (synonym: systematic overview): A review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies.

The difference here is that the way the papers were found and analysed is clearly stated. The reader still needs to be satisfied that the search for papers was wide enough to obtain all the relevant data. Searching Medline alone is rarely enough, and if only English language papers are included this may leave out potentially important evidence.

All Cochrane reviews start as a published protocol; this states in advance how the review will be carried out (searching for data, appraising and combining study data). There is therefore some protection against the danger of post-hoc analysis, in which reviewers find that by dividing up the trials in a particular way spurious statistical significance can be generated in sub-groups of patients or treatment types.

Is the Question focussed?

But we have moved on to thinking about how the review was carried out before checking whether the question being addressed is an important one. The PICO structure set out in the first article in this series(2) can be used here to check that the Patient groups, Interventions used, Comparator treatment and Outcomes are sensible. Watch out in particular for surrogate outcomes that may not relate well to the outcome that matters to the patient. One example of this can be found in trials relating influenza vaccine to the prevention of asthma exacerbations. Some trials measure antibody levels to the flu-vaccine given, but what really matters is whether asthmatics have fewer exacerbations or admissions to hospital, and there is precious little data from randomised controlled trials about this (3).

What was the quality of the trials found?

A further issue to think about in Systematic Reviews is whether the type of included studies is appropriate to the question being asked. In a previous article in this series(4) the problems of bias was discussed. In general in questions related to treatment I would expect the review to focus on randomised controlled trials, as this will minimise the bias present in the included studies. Whilst Meta-analysis can be used to combine the results of observational studies, this is unreliable because they may all suffer from the same bias, and this will be combined in the pooled result from all the trials.

When looking at randomised controlled trials the reviewers should report whether the allocation of patients to the treatment and control groups was adequately concealed (allocation concealment). Allocation is best decided remotely after the patient is entered into the trial; even opaque sealed envelopes can be held up to a bright light by trialists who want to check which treatment the next patient will receive. Poor allocation concealment, failure to blind and poor reporting quality in reviews have all been shown to be associated with overoptimistic results of randomised controlled trials.(5)

Publication bias remains a problem, in that studies that may happen to produce results that are statistically significant are more likely to be published than ones that do not, since editors of medical journals like to have a story to present. This will never be fully overcome until all trials are registered in advance and the publication of results becomes mandatory (whether they show significant differences or not).

Forest Plots

The results of a Systematic Review are often shown graphically as a Forest plot (6). An example from the recently updated Cochrane Review comparing Spacers with Nebulisers for delivery of Beta-agonists(7) is shown below.

Figure 1 Forest Plot of Hospital Admissions for Adults and Children with Acute Asthma when treated with Beta-agonist delivered by Holding Chamber (Spacer) compared to Nebuliser.

The left hand column lists the included studies, which have been sub-grouped into those relating to adults and children. The columns listed ‘Holding Chamber’ and ‘Nebuliser’ list the proportion of patients in each group admitted to hospital and the Relative Risk of admission is shown next to them as a graphical display. Admission is undesirable so the squares and diamonds to the left of the vertical line favour the spacer group. The size of the blue square relates to the weight given to each study in the analysis; this is listed in the next column and generally increases for larger studies. The width of the horizontal line is the 95% confidence interval for each study and this is reported in text in the final column.

The pooled results from adults are shown in the top diamond, for children in the middle diamond and for all patients together in the bottom diamond. This shows that by combining all the studies we can be 95% sure that the true risk of admission when using a spacer lies between 0.55 and 1.07 in comparison with using a nebuliser. There is no significant difference between the two methods and the confidence interval suggests that nebulisers are at best no more than 7% better than spacers and may be up to 45% worse.

So how can these results be translated into clinical practice? This question will be the focus of the next article in this series.

References:

1. Haynes RB. What kind of evidence is it that Evidence-Based Medicine advocates want health care providers and consumers to pay attention to? BMC Health Serv Res 2002;2(1):3 http://www.biomedcentral.com/1472-6963/2/3

2. Cates C. Evidence-based medicine: asking the right question. Prescriber 2002;13(6):105-9.

3. Cates CJ, Jefferson TO, Bara AI, Rowe BH. Vaccines for preventing influenza in people with asthma (Cochrane Review). In: Cochrane Library: Update Software (Oxford); 2000.

4. Po A. Hierarchy of evidence: data from different trials. Prescriber 2002;13(12):18-23.

5. Bandolier. Bias. Bandolier 2000;80-2:1-5 http://www.jr2.ox.ac.uk/bandolier/band80/b80-2.html

6. Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ 2001;322(7300):1479-1480.

7. Cates CJ, Rowe BH. Holding chambers versus nebulisers for beta-agonist treatment of acute asthma (Cochrane Review). In: Cochrane Library: Update Software (Oxford); 2002.

I have sent that piece of work from http://www.nntonline.net/ebm/newsletter/2003/10/Systematic_Reviews_and_Meta-analyses.asp

There are others such as article from cochrane website

The Cochrane Collaboration

What is a Systematic Review?

Systematic reviews are like scientific investigations in themselves, using pre-planned methods and an assembly of original studies that meet their criteria as 'subjects'. They synthesize the results of an assembly of primary investigations using strategies that limit bias and random error.

One of the first steps in this type of review is a comprehensive search of all potentially relevant articles with a pre-defined search strategy. A quantitative systematic review (like the type found in Cochrane) often summarizes results using a statistical technique called meta-analysis when studies are comparable. This technique allows researchers to combine the results of several studies into a single estimate of their combined result. It is a key element of many systematic reviews or overviews. It is also what is meant when people talk about 'pooling data'.

Quantitative systematic reviews that use meta-analysis and search for all relevant articles differ greatly from narrative reviews, which are summaries of research that lack explicit descriptions of systematic methods. The evidence in narrative reviews can be tenuous, incomplete, or based on a biased citation of studies (Antman 1992). In addition, the quantitative techniques incorporated in systematic reviews are more likely to detect small, but clinically meaningful treatment effects (Cooper 1980).

Learn more about the Principles of Meta-analysis:

View a PowerPoint presentation on 'Meta-analysis for beginners ', written by CMSG editor and biostatistician George Wells. Or click here for PowerPoint slides from the Cochrane Collaboration Training Workshops on the Conduct of Meta-analyses Using Individual Patient Data

Cochrane reviews are acknowledged as the best single source of evidence about the effects of healthcare interventions. Rigorous methods are employed that provide a powerful tool for deriving meaningful conclusions from the data and help prevent errors in interpretation.

Conclusions from Cochrane reviews can be invaluable to many groups: health policy makers; health professionals who are preparing clinical practice guidelines; clinical practitioners trying to keep up to date with advances in health care knowledge; members of the public who wish to be informed about the evidence concerning health care practices; and health care researchers who need to define what is already known before starting new research.

Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268:240-8.

Cooper HM, Rosenthal R. Statistical versus traditional procedures for summarizing research findings. Psychol Bull. 1980;87:442-9.

Special thanks for your comprehensive answer. Are there any other comments?