EBM notebook

6. Woods DW, Wetterneck CT, Flessner CA. Behav Res Ther 2006;4:639–56. 11. Christenson GA, Mackenzie TB, Mitchell JE, et al. Am J Psychiatry 1991;148:
7. Van Minnen A, Hoogduin KA, Keijsers GP, et al. Arch Gen Psychiatry 2003;60: 1566–71.
517–22. 12. Lochner C, Seedat S, Niehaus DJ, et al. Int Clin Psychopharmacol 2006;21:255–9.
8. Dougherty DD, Loh R, Jenike MA, et al. J Clin Psychiatry 2006;67:1086–92. 13. Keijsers GP, Van Minnen A, Hoogduin CA, et al. Behav Res Ther 2006;44:359–70.
9. Ninan PT, Rothbaum BO, Marsteller FA, et al. J Clin Psychiatry 2000;61:47–50. 14. Bloch MH, Landeros-Weisenberger A, Dombrowski P, et al. Biol Psychiatry
10. Streichenwein SM, Thornby JI. Am J Psychiatry 1995;152:1192–6. 2007;62:839–46.

Interpreting meta-analysis in systematic reviews

A meta-analysis is a statistical method used to estimate an presents a Forest plot identifying its basic components. A vertical
average, or common effect, over several studies. With ther- ‘‘line of no effect’’ (in the centre of the graph) represents the result
apeutic interventions (whether drug or non-drug) the meta- if the intervention had no impact. Each included study is shown
analysis is usually based on randomised controlled trials. In this as a horizontal line with a square in the middle, which
reader’s guide we use the systematic review by Bravata et al1 of corresponds to the study’s effect and 95% CI. The size of the
the effects of pedometers to increase physical activity to central square is made equal to the size of the study (or more
illustrate these concepts. exactly the proportion of the weight that the study contributes to
A good systematic review should have done a thorough search the combined effect). Notice that, depending on the outcome
for all studies, appraised their quality, and selected the better being measured, a ‘‘positive’’ effect will mean that the square and
studies for answering the question. We won’t go over the the 95% CI lie on the left or the right of the line of no effect. In the
appraisal here but will focus on reading the combined results. pedometer review, an increase in steps taken is seen as a positive
Reading a meta-analyses can be broken down into 4 basic steps2: outcome, therefore we expect the positive studies to lie on the
1. What is the summary measure? right hand side of figure 1.
2. What does the Forest Plot show? The combined (average) effect appears as a diamond at the
3. What does the pooled effect (average effect) mean? bottom of the Forest plot. The centre of the diamond is equal to
4. Was it valid to combine studies? the average effect and the extremes of the diamond equal to the
95% CI of the average effect. In the pedometer review, the
average effect is positive with the intervention increasing the
1. WHAT IS A SUMMARY MEASURE?
number of steps by 2500 on average.
First read the horizontal scale of the Forest plot to check which
measure of effect has been used, and which side is ‘‘good’’ and
‘‘bad’’ for the treatment. The effect in each study can be 3. INTERPRETING THE COMBINED EFFECT
summarised in different ways. For binary outcomes (2 6 2 The diamond or average effect is obtained from combining the
table), the main options are relative risk (RR), odds ratios (OR), summary measures collected for all included studies. This gives
or risk differences (RD). For meta-analysis, usually relative a single (more precise) value of the effect of the intervention.
measures (RR and OR) are preferred as they are more likely to The idea is simple: each study provides some information about
remain similar from study to study (see the discussion on the real effect, but by combining all of them we are more likely
heterogeneity below). For continuous outcomes, the options are to be near the ‘‘truth.’’
the difference in means or standardised means between the As the information provided by the trials differs (some studies
interventions tested. In the pedometer review, the outcome of are larger than others and—if all were carried to the same
interest is the mean difference in the number of steps (post- standard—more likely to approximate to the truth), pooling
intervention steps per day 2 pre-intervention steps per day). All information usually involves obtaining some form of weighted
the measurement units are the same (step counts) so they do average. Larger weights are given to those studies that are
not need to use the standardised mean difference, which lacks believed to give more information (usually larger studies);
units and is therefore less clinically interpretable. however, all studies are taken into account for the final average.
There are 2 basic models for combining studies: the fixed-effect
model and the random-effects model. The fixed-effect model
2. WHAT DOES THE ‘‘FOREST’’ PLOT SHOW?
considers the variability between the studies as exclusively
Meta-analyses are usually displayed in graphical form by using
due to random variation. In other words, if the studies were
Forest plots, which present the findings for all studies plus
infinitely large they would all produce the same result. The
(usually) the combined results. This allows the reader to visualise
random-effects model assumes a different underlying effect for
how much uncertainty there is around the results. Figure 1
each study and takes this into consideration as an additional
source of variation. Under this assumption, if the studies were
Types of data in meta-analysis* infinitely large, they would still give different results of the
Type of data Definition effect of the intervention. The effects of the studies are assumed
to be randomly distributed and the central point of this
Binary data Something that either happens or not, such as numbers of patients
improved or not. Summary measures: OR, RR, or RD. distribution is the focus of the combined (pooled) effect
Continuous data Something that in theory can be any number in a range of values estimate (figure 2).
(eg, numbers of days to recovery, peak expiratory flow rate). The choice of model is a cause of great debate with exponents of
Summary measure: differences in mean values for treatment and either presenting examples when the other gives a clearly
control groups. These can be combined using weighted mean
differences (WMD) when units of measurement are the same, or ‘‘wrong’’ answer. In practice, the choice between these 2 is
standardised mean differences (SMD) when units of measurement important only in the case when the studies being combined do
differ. not provide coherent information (see the discussion on hetero-
*OR = odds ratio; RR = relative risk; RD = risk difference. geneity below). In this case, using a random-effects model leads to

EBM June 2008 Vol 13 No 3 67

 EBM notebook

Figure 1. Basic components of a Forest plot: Bravata et al. JAMA 2007;298:2296–2304.

relatively more weight being given to smaller studies and to wider CIs do not touch the line of no effect) and 3 non-significant
CIs (more conservative) for the pooled estimate of the effect. In studies (Eastep et al,3 de Blok et al,4 and Talbot et al5 [their CIs
the pedometer review, both random- and fixed-effect models were cross the line of no effect]). Vote counting is generally a
applied to the meta-analysis of step count. dangerous option, as all studies may be non-significant but the
(A short note on vote counting) pooled estimate can still be significant.
In some systematic reviews, meta-analysis is not feasible and
results are sometimes presented as a simple count of the number 4. WAS IT OK TO COMBINE STUDIES?
of studies supporting an intervention and the number not Obtaining a result from the combination of several studies raises
supporting it. This assumes equal weight being given to each an obvious question: was combining these studies valid? No 2
study, regardless of size. In figure 2 of the pedometer review, a studies are exactly alike: they will vary in their ‘‘PICOs’’—the
vote count reveals that there are 5 significant studies (their 95% population, the intervention, the comparator, or the outcome in

Figure 2. Graphical representation of the theoretical models used to combine data together.

68 EBM June 2008 Vol 13 No 3

 EBM notebook
some minor or major way. For this reason, this question has to to 3885), i-squared 91%. To account for this heterogeneity, the
be addressed in 3 steps: are the studies (a) clinically similar and study investigators removed the study by Moreau et al,7 which
(b) methodologically similar, and (c) are the findings statisti- reports a much higher increase in physical activity in postmeno-
cally similar (homogeneity)? For the first step, it is important to pausal women than the other studies (on the figure 2 plot this is the
have some degree of content expertise, while for the second study furthest to the right). This reduces the effect size to 2004
step, it is necessary to have critical appraisal skills. steps (CI 878 to 3129); however, a limitation of this approach is
The final step of assessing statistical homogeneity commonly that they do not provide the i-squared statistic for this result nor
uses a statistic to test if the variation is larger than what we would give reasons (clinical or methodological) for its exclusion.
have expected by chance. The 2 commonly used statistics are the
chi-squared (which tests the ‘‘null hypothesis’’ of homogeneity) Rafael Perera, DPhil, Carl Heneghan BA, MRCGP
and the i-squared6 (which measures the amount of variability due Centre for Evidence Based Medicine, University of Oxford; Oxford, UK
to heterogeneity). The i-squared is now the most commonly used
measure as it is a measure of the degree of heterogeneity and is less 1. Bravata DM, Smith-Spangler C, Sundaram V, et al. JAMA 2007;298:2296–304.
2. Perera R, Heneghan C, Badenoch D. Statistics toolkit. UK: Blackwell Publishing, 2008.
reliant on the number of studies included in the meta-analysis. An 3. Eastep E, Beveridge S, Eisenman P, et al. Percept Mot Skills 2004;99:392–402.
i-squared higher than 50% is deemed to be large enough to question 4. de Blok BMJ, de Greef MHG, ten Hacken NHT, et al. Patient Educ Couns
whether combining studies is valid. In the pedometer review, the 2006;61:48–55.
5. Talbot LA, Gaines JM, Huynh TN, et al. J Am Geriatr Soc 2003;51:387–392.
heterogeneity for the main outcome increase in step counts was 6. Higgins JP, Thompson SG. Stat Med 2002;21:1539–58.
high: increase in steps in the intervention group 2491 (95% CI 1098 7. Moreau KL, Degarmo R, Langley J, et al. Med Sci Sports Exerc 2001;33:1825–31.

Letter
foraging resources.3 4 While their criteria
Proactive monitoring for newly emerging evidence: make good sense, limited research exists to
assess whether these are the best or only
The lost step of EBP? important characteristics of a literature-
monitoring resource. In addition, no stu-
The 5 steps of evidence-based practice (EBP) such a proactive system is not an easy task. dies have investigated best practices in
were described in 1992.1 The steps are: In fact, personal experience of the authors teaching these skills to young practitioners.
1. Translation of uncertainty to an identifies that many healthcare practi- We believe that the importance of this
answerable question; tioners attempt to use more resources than proactive monitoring merits that it be
2. Systematic retrieval of best evidence they can effectively review during the time identified as an additional step of EBP. It
available; they dedicate to this task. Unsuccessful is important to note that this process
3. Critical appraisal of evidence for valid- attempts to manage many resources often occurs before the 5 steps begin. Therefore,
ity, clinical relevance, and applicability; lead practitioners to feel guilty and dis- we suggest that ‘‘Proactive monitoring for
4. Application of results in practice; and courage them from maintaining any habits newly emerging evidence’’ be considered a
5. Evaluation of performance. in this arena. Therefore, many practitioners parallel but equally important process to
According to the Sicily Statement on develop gaps in their ability to identify new the 5 steps. Accordingly, we recommend
Evidence-based Practice,2 the steps describe evidence that might change their practice. that thought leaders in the field of EBP
a codified set of behaviours that form the While no system will detect all new design studies to investigate the key
basis for evidence-based clinical practice. The relevant knowledge, it is desirable to components of monitoring systems, and
steps have also been the primary focus of identify as much as possible in a proactive the ideal way to teach new practitioners
teaching efforts in EBP. As the Sicily state- manner. Efficient and effective monitoring how to develop their own systems.
ment identifies, all of the 5 steps have been of emerging literature will plant the seeds
the focus of trials of teaching effectiveness. of knowledge to generate important Drew Keister, MD, Julie Tilson, DPT, NCS
The 5 steps constitute a reactive response answerable questions that might not have
to a question that arises from a clinical been recognised without such a system. 1. Cook DJ, Jaeschke R, Guyatt GH. J Intensive Care
Med 1992;7:275—82.
scenario. Successful evidence-based practi- Thus, the creation of a proactive system 2. Dawes M, Summerskill W, Glasziou P, et al. BMC
tioners also establish systems to proactively to monitor the emerging literature is Med Educ 2005;5:1.
monitor emerging literature to identify critical to successful EBP. Slawson and 3. Slawson DC, Shaughnessy AF. Acad Med
2005;80:685–9.
new information that would change their Shaughnessy identify this skill as ‘‘fora- 4. Rosser WW, Slawson DC, Shaughnessy AF.
practice. In short, they design ways to ging,’’ and they have several publications Information mastery: evidence-based family medicine.
‘‘keep up with the literature.’’ Creating that identify the facets of high-quality 2nd ed. Hamilton, ON: BC Decker Inc, 2004:8–14.

Reply from the editors: bmj.com, respectively), and BMJUpdates+ proactive monitoring is "just-in-case" learn-
We are delighted with Tilson and Keister’s (www.bmjupdates.com). ing. The efficient practice of EBM requires
suggestion of an additional step for EBM We recently described the journal pro- skills at both and a balance between the 2.
of "Proactive monitoring for newly emer- cesses and how they improve clinicians’ Paul Glasziou MBBS, PhD, R Brian Haynes,
ging evidence." This is exactly the role the "Number Needed to Read" to find 1 valid and MD PhD
ACP Journal Club and EBM journals were relevant article.1 We would see this alert
designed to fulfil, along with their website process as parallel to the "5-step" EBM. The 1. Eady A, Glasziou P, Haynes B. Evid Based Med
alerts (www.acpjc.org and http://ebm. latter is "just-in-time" learning, whereas the 2008;13:3.

EBM June 2008 Vol 13 No 3 69