Health Policy, 11 (1989) 79-85

Elsevier

HPE 00230

19

An introduction to Meta-Analysis

Mahesh S. Pate1

University of Lausanne, lnsrirure of Social and Preventive Medicine, Lausanne, Switzerland

Accepted 70 April 7988

Summary

The current rapid pace of social and technological change requires health policy makers to be up to date in their knowledge of health policy research. The “infor- mation explosion” has created a need for techniques that synthesize and summa- rize available information. This paper reviews the use of Meta-Analysis as a data pooling technique in a non-technical manner. Treatment of chronic pain by acu- puncture, provided as an example, illustrates the type of information that can be ob- tained from a Meta-Analysis, that is not conventionally available from individual trials.

Meta-Analysis; Health policy; Acupuncture

Introduction

For health policy makers, it is essential to formulate decisions according to the very best knowledge available. This is no easy task. The information explosion has dramatically increased the quantity of information in existence. But it has brought us no nearer to the frontier of the unknown. The information required to answer the questions at hand often seems sadly inadequate.

Earlier, only one study of an issue might have been performed. Now, the num- ber of studies available may be so great that the task of selecting relevant infor- mation of adequate quality can itself be almost overwhelming. Perhaps worst of all, results of different studies may be mutually contradictory.

While there may be a wealth of information on some subjects, adequate infor- mation on technologies perceived as new is often unavailable, almost by defini- tion. If information was available, the development would not be considered “new”. Conversely, information that is available as a basis for decisions may be out of date. It may be of poor quality.

Address for correspondence: M.S. Patel, IUMSP, 17 rue Bugnon, CH-1005 Lausanne, Switzerland.

0168-8510/89/$03.50 0 1989 Elsevier Science Publishers B.V. (Biomedical Division)

80

Clearly, it is essential to have some means of comparing and contrasting the re- sults of a range of studies. For these reasons, literature reviews, reports synthes- izing available information, and conferences of experts aimed at obtaining a con- sensus have been of great interest to policy makers.

Meta-Analysis (MA) is a relatively recent set of methodological techniques that is increasingly being used to synthesize available knowledge. The actual formulae used and calculations involved depend on the nature of the data available. While these technical issues are beyond the scope of this review, detailed descriptions of statistical methodology are available elsewhere [ 1,2]. Most MA equations can eas- ily be programmed onto standard microcomputer spreadsheets. This paper aims to describe Meta-Analysis in non-technical terms.

What is Meta-Analysis?

Meta-Analysis includes a range of result pooling techniques. These are used to combine and analyze the results of a number of studies in a given subject area, in order to obtain more definite, or even definitive, results. The function served by Meta-Analysis can be contrasted to those of literature reviews, information syntheses, and consensus conferences.

A literature review, while covering a range of different sources, is essentially a descriptive exercise. Classically, as a preface to a piece of original research, its purpose could be to show that previous work in that subject area was inadequate! Frequently, it is also used to predict the evolution of a field of research. There is no requirement that any consensus or definitive judgement on a subject emerges.

Information synthesis [3] explicitly emphasizes the process of summarizing in- formation from different sources. In order that a systematic summary be possible, the subject must be relatively well defined, and more limited in scope than that of a literature review. Typically, characteristics and perhaps some numerical results of a range of studies could be presented as a table. Actual synthesis of the infor- mation presented would usually be limited to frequency counts for characteristics, and simple arithmetic or weighted averages for numerical data.

A consensus conference could be described as an information synthesis in which the summarizing method is that of the group, rather than the individual intellect. Originally developed in the U.S.A., use of consensus conferences is increasing in Europe, particularly in countries having publicly financed and controlled health systems [4]. Previously, a range of “expert opinions” were often obtained by ques- tionnaire. More recently, consensus is formed by open discussion of available evi- dence at some forum. The purpose is the resolution of an issue [5]. Advantages of “intellect pooling”, particularly through debate, are that non-quantifiable aspects of the problem may be more adequately treated, and that recent and unpublished information may be available. The disadvantages are the potential for demagogic influence, and the lack of a formal objective criterion for defining the conclusion. (Consensus is procedurally formal in that discussion has a defined structure, but not logically formal in that there is no basis for choosing between contradictory evidence other than informed judgement.)

81

Meta-analysis is a logically formal, mathematical, information synthesis tech- nique. It aims at a synthesis of numerically quantified information that is extracted from a comprehensive list of studies. While apples and pears can be compared and contrasted (as in a literature review), or preferred one to the other (as in a con- sensus conference) they cannot be added; or if added, may be added only on the basis of those characteristics they have in common in order to obtain, for example, the total number of pieces of fruit. Similarly, the component studies of a Meta- Analysis must have something quantifiable in common, and it is typically this com- mon factor, rather than their differences, that is the core problem addressed by a Meta-Analysis. The MA addresses a hypothesis that has a numerical solution [6]. One drawback of such a formal analysis is that the effects of non-quantifiable in- formation (flavor - to stretch an analogy), that might in certain cases influence conclusions, may be lost.

Naturally, these synthetic techniques may be themselves combined; Meta-Anal- yses, information syntheses, and literature reviews might all provide input to a consensus conference, and literature reviews to a Meta-Analysis. The goals of an MA could be defined as: (1) to obtain the “golden mean”, or ideal central estimate, from a series of quan-

titative estimates; (2) to improve the precision with which a quantity is estimated; (3) to resolve uncertainty when a series of reports differ in their conclusions; (4) to provide answers to questions that are not dealt with in any individual re-

port, but that can be examined, for example, in the context of systematic dif- ferences between subgroups of reports.

Meta-Analysis is increasingly used to define the “state of the art” by cumulating knowledge in a given subject area. In the years 19761979, the MEDLINE data base noted the publication of only one or zero articles containing the keyword “Meta-Analysis” each year. In 1985 there were 26 such publications, and in 1986 there were 28 [7].

When compared to the number of MA studies published, the influence of MA is disproportionately large. This is probably due to the relatively large amount of information that is treated in an MA study. A single MA may synthesize the in- formation from half a dozen studies, or from over a hundred. The subsequent sec- tion describes Meta-Analyses that have generally been acknowledged as “suc- cesses” in the domain of health policy.

Successes of MA

The application of Meta-Analysis could be seen as a problem solving exercise for quantifiable problems upon which a certain amount of information is already available. In this context it has had a number of notable successes in the medical [8] and public health [9] domains, where a number of difficult issues that had been repeatedly studied, were either resolved or clarified by the application of MA techniques.

82

In terms of specific MA studies, Cuzick et al. [lo] studied the possible effect on survival of postoperative radiotherapy in breast cancer by pooling the effects of 10 trials. While it was generally accepted that radiotherapy reduced local recurrence, it was considered possible that radiotherapy had a negative effect on survival. It was confirmed, by MA, that long-term survival (after 10 years), was adversely af- fected.

Yusuf and co-workers [ll] showed, in an MA reviewing some 65 trials in that subject area, that long-term (approximately one year) p blockade after myocardial infarction could yield mortality reductions of about 25%.

Wortman and Yeaton [12] evaluated the effects of coronary artery bypass sur- gery against medical treatment. A small mortality benefit of 6.8% was noted for randomized trials, and 12.8% for non-random assignment. A similar difference, between randomized and non-randomized trials, was noted for quality of life out- come (46% versus 29.8% angina free).

Mumford and colleagues [ 131 found that psycho-educational intervention re- duced the length of hospital stay in a Meta-Analysis of 34 experimental (con- trolled) studies. Mazzuca [14] showed an effect of patient education on a range of outcome indicators, including health status.

Lam et al. [15] found that nicotine chewing gum obtained higher success rates than placebo gum in specialized smoking cessation clinics, but that this difference was not significant for general practices (14 RCTs).

Interpreting Meta-Analyses

Meta-Analysis is a means of cumulating quantified research findings. The sta- tistic obtained by pooling results is a weighted average of the findings pooled. Weighting of results of Randomized Controlled Trials (RCTs), for example, are generally defined according to numbers of subjects and variance of results of each trial. Naturally, the measure of effect (outcome) pooled must be similar in all studies. If the measure used is effect difference (the difference between treatment and control groups in terms of the percentages of patients improved), then MA will predict an overall pooled (or weighted average) effect difference.

As with the effect difference of individual component studies, an MA should also present some indicator of the precision of the result obtained. This could be in terms of the standard error, or the confidence interval (CI), of the estimate.

Fig. 1 shows the effect difference, and the 95% CI, for 14 RCTs of acupuncture treatment for chronic pain published in English since 1970. In most of these stud- ies acupuncture was better “on average” than the control procedure. Only in the studies numbered 1, 7, 8, and 10 was this difference statistically significant in a manner favorable to acupuncture.

The pooled effect difference and its CI are also illustrated for “All RCTs”. This shows that, over all 14 studies as a whole, acupuncture obtained better results than the control treatment in a statistically significant manner.

83

Individual Studtis

Validity and bias in Meta-Analysis

Naturally, the results of any Meta-Analysis must be considered in terms of the context in which they were produced. Pooling the results of any number of ex- periments is pointless if the original experiments were not well done.

It may be necessary to distinguish between sub-groups of studies with significant differences in method. If patients know which group they are placed in, experi- mental or control, then that may influence the results. Patients sympathetic to one type of treatment might pretend improvement.

In the example described above, concerning randomized controlled trials of acu- puncture treatment for the relief of chronic pain, the sub-group of trials that has “No Blindness” achieved a better result than the sub-group of trials that had “Some Blindness” (Fig. 1). In the latter, some agents, generally patients, were blind, and did not know whether real acupuncture or “pseudo-acupuncture” was used. This sub-group of randomized controlled trials that was at least partially blind, did not attain statistical significance in favor of acupuncture. This illustrative example is reported in greater detail elsewhere [16].

It is, in other words, insufficient simply to pool results using a mathematically appropriate technique. methodological issues are at least as important as statistical ones. Therefore careful examination of published study plans is essential in order to assure that the results obtained are meaningful.

The most commonly cited source of bias in Meta-Analysis is publication bias. It is assumed that publishers are more inclined to publish studies with positive results than with negative or inconclusive results. It is also probable that there is author self-selection bias. Authors too may put more effort into getting positive results published. It is therefore possible that pooling the results of a series of studies will simply put on display these biases.

This issue has been mathematically formalized as the “file drawer problem”. It is possible to calculate the number of unpublished similar trials with, on average,

84

zero results, that would have to be lying in the drawers of filing cabinets, in order to influence the significance of a MA finding (67 unpublished randomized con- trolled trials at P= 0.05 in this case).

The issue of publication bias is most often used to criticize Meta-Analyses. It should perhaps be added, in defence of Meta-Analysis, that these issues are at least made explicit by the use of MA. Individual studies that obtain positive results do not conventionally state that there may be large numbers of unpublished studies that obtained the opposite conclusion. In Meta-Analysis it has become conven- tional to note that reservation.

A scoring system has been proposed to measure the quality of a Meta-Analysis [17]. This system defines 6 major areas of quality within which 23 quality items were defined. The 6 major areas included: (1) Study Design. A Meta-Analysis should commence with a well-defined study

plan, conduct a comprehensive literature search, list both papers included and excluded, and depict the original results of the included studies.

(2) Combinability. The studies used must be similar enough, in terms of published study methodology and measure of results, to justify pooling of data.

(3) Control of bias. Unconscious bias should be controlled for, when possible. Bias can arise in selecting studies to be included, in extracting data from those stud- ies, and as a result of financial support from an interested agency. (It is sug- gested that the latter is declared, rather than refused!)

(4) Statistical analysis. Adding together, or simple averaging of results is rarely adequate. The formulae to be applied to correctly pool data can be easily im- plemented on any micro-computer spreadsheet package. Such packages have the advantage that they will often also provide a graphic illustration of confi- dence intervals. The calculations and graphics in the example illustrated above were programmed on an IBM PC, with the spreadsheet package SuperCalc 3.

(5) Sensitivity analysis. The quality of the studies included, and the influence of different assumptions, tests, and criteria on the results should be assessed. The potential effects of publication bias should be calculated [18] (the “file drawer” problem).

(6) Application of results. The results obtained should be put in perspective. Are they definitive evidence to settle an issue, or simply a guide to further re- search? It is also helpful to estimate the economic impact of results obtained, if possible.

Conclusions

Meta-Analysis is extremely useful to the health policy maker for several rea- sons. It is a succinct means of defining the state of the art of knowledge in a given subject area. It may resolve conflicts between studies, or yield a conclusive result when individual studies were all inconclusive. It yields quantified results that can be input to decision making processes.

For these reasons it is essential that health policy makers are aware of Meta-

85

Analysis, are able to interpret and judge the quality of results obtained, and know when such an analysis might appropriately be requested or commissioned as an aid to decisions.

References

1 DerSimonian, R. and Land, N., Meta-analysis in clinical trials, Controlled Clinical Trials, 7 (1986) 177-188.

2 Hunter, J.E., Schmidt, F.L. and Jackson, G.B., Meta-analysis: cumulating research findings across studies, Sage U.S.A., 1982.

3 Marcus, S.H., Grover, P.L. and Revicki, D.A., The method of information synthesis and its use in the assessment of health care technology. Int of Technology Assessment of Health Care, 3 (1987) 497-508.

4 Vang, J., The consensus development conference and the European experience, International Journal Technology Assessment in Health Care, 2 (1986) 65-76.

5 Mullan, F. and Jacoby, I., The town meeting for technology: the maturation of consensus confer- ences, Journal of the American Medical Association, 254 (1985) 1068-1072.

6 Greenland, S., Quantitative methods in the review of epidemiologic literature. Epidemiologic Re- views, 9 (1987) l-30.

7 Wolf, F.M., Letter, New England Journal of Medicine, 317 (1987) 576. 8 The Lancet, Whither Meta-analysis? Lancet, i (1987) 897-898. 9 Louis, T.A., Fineberg, H.V. and Mosteller, F., Findings for public health from meta-analysis, An-

nual Review of Public Health, 6 (1985) l-20. 10 Cuzick, J., Stewart, H., Peto, R. et al., Overview of randomized trials of postoperative adjuvant

radiotherapy in breast cancer, Cancer Treatment Reports, 71 (1987) 15-29. 11 Yusuf, S., Peto, R., Lewis, J., Collins, R. and Sleight, P., Beta blockade during and after my-

ocardial infarction: an overview of the randomized trials, Progress in Cardiovascular Disease, 27 (1985) 335-371.

12 Wortman, P.M. and Yeaton, W.H., Using research synthesis in medical technology assessment. In- ternational Journal of Technology Assessment in Health Care, 3 (1987) 509522.

13 Mumford, E., Schlesinger, H.J. and Glass, G.V., The effects of psychological intervention on re- covery from surgery and heart attacks: An analysis of the literature, American Journal of Public Health, 72 (1982) 141-151.

14 Mazzuca, S.A., Does patient education in chronic disease have therapeutic value? Journal of Chronic Disease, 35 (1982) 521-529.

15 Lam, W., Sze, P.C., Sacks, H.S. and Chalmers, T.C., Meta-analysis of randomised controlled trials of nicotine chewing-gum, Lancet, (1987) 27-30.

16 Patel, M., Gutzwiller, F., Paccaud, F. and Marazzi, A., A meta-analysis of acupuncture for chronic pain, International Journal of Epidemiology, in press.

17 Sacks, H.S., Berrier, J., Reitman, D., Ancona-Berk, V.A. and Chalmers, T.C., Meta-analyses of randomized controlled trials. Special Article, New England Journal of Medicine, 316 (1987) 450-455.

18 Rosenthal, R., The ‘file drawer problem’ and tolerance for null results, Psychological Bulletin, 86 (1979) 638-641.