Meta-analysis in alcohol treatment research: does it help us to know what works?

EFFECTIVENESS OF BRIEFINTERVENTIONS PROVED BEYONDREASONABLE DOUBT

All three reviews (Miller & Wilbourne 2002; Moyer,Finney & Vergun 2002a; Moyer, Finney & Swearingen2002b) are enormously valuable additions to the lit-erature on the treatment of alcohol problems but, in thelimited space available, I will restrict comment to thepaper by Moyer and colleagues on brief interventions foralcohol problems (Moyer et al. 2002a).

There is no question that the meta-analysis of briefinterventions reported by these authors is the most com-prehensive and methodologically sound of any to haveappeared on this topic so far. Doubts have been expressedrecently concerning the effectiveness of brief interven-tions but the main impact of the review by Moyer et al.,to my mind, is that it has shown those doubts to be unreasonable; in other words, the effectiveness of briefinterventions has been proved, to borrow the legalconcept, beyond reasonable doubt. This applies both to the absolute effectiveness of brief interventions by generalists in the non-treatment-seeking population ofexcessive drinkers and to the relative effectiveness of briefinterventions compared with more intensive treatmentby specialists in a segment of the treatment-seeking population.

This is not to say that no questions remain about theeffectiveness of brief interventions, questions that couldperhaps be the focus of future meta-analytical reviewsand should certainly be addressed in future research.Perhaps the main issue here concerns the distinctionbetween ‘efficacy’ and ‘effectiveness’ trials (Holder et al.1999). In the trials of brief interventions in primaryhealth care in which I have been involved (Heather et al.1987; Richmond et al. 1995), screening for excessivedrinkers has been incorporated into routine practice asopposed to recruiting research subjects by some specialprocedure, and brief intervention has been offered imme-diately upon a positive case being identified. Thus, inthese terms, these studies qualify as effectiveness trialscarried out under real-world conditions of generalmedical practice, rather than in special conditionsdesigned to optimise internal validity. In both cases, wefailed to find an effect of intervention on level of alcohol

consumption. In one case (Heather et al. 1987) the studywas under-powered but in the other case (Richmond et al.1995) we found a significant reduction in alcohol problems which we interpreted as a genuine effect ofintervention.

The point, however, is that brief interventions deliv-ered in naturalistic conditions are probably less effectivethan estimates of effect size based on efficacy trials,which make up the majority of trials reported in the literature, would suggest. This may even account for the fact that some brief interventions trials, particularlythose in non-Anglo-Saxon cultures, have reported nega-tive findings (Aalto et al. 2001). The evidence reviewedby Moyer and her colleagues shows that brief interven-tions among excessive drinkers can work but whetherthey do actually work in practice will depend on adjust-ing the means of implementation and delivery to the par-ticular characteristics of the primary health care systemand possibly to wider cultural factors in each society. Thisis precisely the aim of the ongoing Phase IV of the long-standing World Health Organization collaborative projecton the detection and management of alcohol-relatedproblems in primary health care (Monteiro & Gomel1998).

Apart from the obvious question of the longer-termeffects of opportunistic brief interventions, probably themost important substantive issue concerns the optimallength and associated contents of these interventions.The work of Moyer et al. and of others shows clearly thatthe widespread implementation of brief, structured andpersonalized advice, taking no more than 5–10 minutesto deliver, would be effective in reducing alcohol-relatedharm in a population and would potentially be a hugelycost-effective response to that harm. But could somewhatlonger interventions be even more effective? If so, to whattypes of excessive drinker should they be offered and whoshould offer them? An intriguing but neglected findingfrom the WHO Phase II trial (Babor & Grant 1992) wasthat ‘simple advice worked best for male patients who hadexperienced a recent alcohol-related problem, while briefcounselling (15 minutes following 20 minutes’ assess-ment) worked better for those who did not have a recentproblem’ (p. 3, parentheses added). This suggests that alonger type of brief intervention based on principles ofmotivational enhancement could be offered usefully to

© 2002 Society for the Study of Addiction to Alcohol and Other Drugs Addiction, 97, 293–299

Commentaries

294 Commentaries


excessive drinkers who do not recognize a problem withtheir alcohol consumption, assuming that certain classesof health professionals have the time and inclination todeliver it. There is some evidence to support this hypoth-esis from a trial of brief interventions among hospital in-patients in Sydney (Heather et al. 1996), but moreresearch is clearly needed on this key issue.

Turning lastly to the class of brief interventions inspecialist centres among the treatment-seeking popula-tion, I note the finding of the review that brief interven-tions ‘were not more effective that control conditions instudies where more severely impaired persons were notexcluded’, suggesting that such interventions ‘are usefulonly for patients with less severe drinking problems’ (pp. 279–292). Against this, however, findings from theProject Match Research Group (1997, 1998) showedthat a briefer treatment (motivational enhancementtherapy given in four sessions over 12 weeks) was no lesseffective than two kinds of more intensive treatmentacross all patients taking part in the trial; the only spe-cific contraindication found was that out-patients with asocial network supportive of drinking did better withTwelve-Step facilitation than with motivational enhance-ment therapy. While I concur fully with the warnings inthe last paragraph of the review against ‘wholesalereplacement of specialist, extended approaches to treat-ment with considerably brief interventions’, I believe it ispossible that research will confirm an applicability ofbriefer treatments of the kind studied in Project MATCHto a broader range of problem drinkers than indicated bythe prudent policy Moyer et al. quite rightly recommendon the basis of present evidence. Findings from the UKAlcohol Treatment Trial (UKATT Research Team 2001),which will be published over the next couple of years, willbe directly relevant to this possibility.

NICK HEATHER

Centre for Alcohol and Drug StudiesNewcastle, North Tyneside and Northumberland MentalHealth NHS TrustUniversity of Northumbria at NewcastleNewcastle upon TyneUK

References

Aalto, M., Seppa, K., Mattila, P., Mustonen, H., Ruuth, K.,Hyvarinen, H., Pulkkinen, H., Alho, H. & Sillanaukee, P.(2001) Brief intervention for male heavy drinkers in routinegeneral practice: a three-year randomized controlled study.Alcohol and Alcoholism, 36, 224–230.

Babor, T. F. & Grant, M. (1992) Project on Identification andManagement of Alcohol-Related Problems. Report on Phase II: aRandomized Clinical Trial of Brief Interventions in the PrimaryHealth Care Setting. Geneva: World Health Organization.

Heather, N., Campion, P., Neville, R. & Maccabe, D. (1987)Evaluation of a controlled drinking minimal intervention forproblem drinkers in general practice (the DRAMS Scheme).Journal of the Royal College of General Practitioners, 37,358–363.

Heather, N., Rollnick, S., Bell, A. & Richmond, R. (1996) Effectsof brief counselling among male heavy drinkers identified ongeneral hospital wards. Drug and Alcohol Review, 15, 29–38.

Holder, H., Flay, B., Howard, J., Boyd, G., Voas, R. & Grossman,M. (1999) Phases of alcohol problem prevention research.Alcoholism: Clinical and Experimental Research, 23, 183–194.

Miller, W. R. & Wilbourne, P. L. (2002) Mesa Grande: a method-ological analysis of clinical trials of treatments for alcohol usedisorders. Addiction, 97, 265–277.

Monteiro, M. G. & Gomel, M. (1998) World Health Organizationproject on brief interventions for alcohol-related problems inprimary health care settings. Journal of Substance Abuse, 3,5–9.

Moyer, A., Finney, J. W., Swearingen, C. E. & Vergun, P. (2002a)Brief interventions for alcohol problems: a meta-analyticreview of controlled investigations in treatment-seeking andnon-treatment-seeking populations. Addiction, 97, 279–292.

Moyer, A., Finney, J. W. & Swearingen, C. E. (2002b)Methodological characteristics and quality of alcohol treat-ment outcome studies, 1970–98: an expanded evaluation.Addiction, 97, 253–263.

Project Match Research Group (1997) Matching alcoholismtreatments to client heterogeneity: Project MATCH posttreat-ment drinking outcomes. Journal of Studies on Alcohol, 58,7–29.

Project Match Research Group (1998) Matching alcoholismtreatments to client heterogeneity: Project MATCH three-yeardrinking outcomes. Alcoholism: Experimental and ClinicalResearch, 22, 1300–1311.

Richmond, R., Heather, N., Wodak, A., Kehoe, L. & Webster, I.(1995) Controlled evaluation of a general practice-based briefintervention for excessive drinking. Addiction, 90, 119–132.

UKATT Research Team (2001) United Kingdom AlcoholTreatment Trial: hypotheses, design and methods. Alcohol andAlcoholism, 36, 11–21.

RESEARCHING THE TREATMENT OFDRINKING PROBLEMS: A CALL FOREXTERNAL AS WELL AS INTERNALVALIDITY

‘Continuous multiple experimentation is more typical ofscience than once and for all definitive experiments.Experiments will need replication and cross-validation atother times under other conditions before they canbecome an established part of science, before they can betheoretically interpreted with confidence.’ Those words,written by Campbell & Stanley (1963) in the early 1960s,resonate as clearly today as they did when they were firstpublished four decades ago. However, while replicationand cross-validation are the hallmark of science, theexpanding literature that accompanies the developmentof a scientific field challenges investigators and readersalike to evaluate and synthesize the knowledge base

Commentaries 295


accurately into coherent and meaningful statements.This, depending on the field, is quite often easier said than done. To this end, in a series of papers (Miller &Wilbourne 2002; Moyer et al. 2002a; Moyer et al. 2002b)contained in the current issue, researchers in the field ofalcohol use disorders treatment grapple with this chal-lenge; and in what is perhaps the most impressive featureof this work, the results generated and the conclusionsdrawn almost pale in comparison to the effort that wasneeded to make sense of it all.

As a field develops, so should the quality of its sup-porting research. Otherwise the discipline would cease torefine itself and academic inquiry would rapidly come toan end. It is perhaps not surprising, then, that two of thethree papers in this series provide empirical support forthis notion. In the Mesa Grande work, Miller & Wilbourne(2002) assess the methodological rigor of clinical trials of treatment for alcohol use disorders and, in so doing,provide an intuitively appealing means for differentiallyweighting study results based on the soundness of themethodology that produced them. Moyer, Finney &Swearington (2002) similarly assessed methodologicalcharacteristics/quality of alcohol treatment outcomestudies in the last three decades. While different criteriawere employed in these two large-scale evaluations, convergent results were obtained as each project foundthat the overall quality of alcohol use disorders treat-ment research has improved significantly in the past 30 years. However, that said, it must be noted that overall indices of methodological quality still remain low, with many studies lacking sufficient statistical power to detect an effect; this despite the fact that the authors restricted their reviews to top-of-the-linecontrolled trials.

While controlled trials of treatment that feature ran-domization should be the goal of researchers studyingaddiction treatment and outcome (allowing us to dimin-ish the influence of uncontrolled factors when compar-ing treatment groups so that differences in outcome may be more accurately attributed to treatments) theenhanced internal validity that accompanies suchresearch is inevitably gained at the expense of externalvalidity (i.e. study conditions become so removed fromcustomary clinical practices as to raise serious questionsabout generalizability.) Thus, when designing studiesresearchers are forced to weigh the advantages of experi-mental rigor against the disadvantages of less control,hopefully remembering that we do not always learnabout causation in strictly controlled environments(Gottheil et al. 1981; Gottheil et al. 1995).

In recognizing this schism the National Institute ofDrug Abuse in the United States has begun the clinicaltrials network for the purpose of transferring university-based research findings to clinical settings with the goal

of evaluating their applicability in real practice sites.Thus, it seems appropriate to close by emphasizing theneed for methodologies and research settings thatenhance internal validity as well as clinical applicability,so that the benefits of our technological achievementsmight be maximized.

ROBERT C. STERLING

Division of Substance Abuse ProgramsJefferson Medical CollegeThomas Jefferson UniversityPhiladelphia, PAUSA

References

Campbell, D. & Stanley, J. (1963) Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally,College Publishing Co.

Gottheil, E., McLellan, A. T. & Druley, K. (1981) Reasonable andunreasonable methodological standards for the evaluation of alcoholism treatment. In: Gottheil, E., McLellan, A. T. &Druley, K., eds. Matching Patient Needs and Treatment Methodsin Alcoholism and Drug Abuse, pp. 371–389. Springfield:Charles Thomas.

Gottheil, E., Sterling, R. & Weinstein, S. (1995) Generalizingfrom controlled treatment outcome studies: sample data froma cocaine treatment program. American Journal on Addictions,4, 331–338.


Moyer, A., Finney, J. W. & Swearingen, C. E. (2002a)Methodological characteristics and quality of alcohol treat-ment outcome studies, 1970–98: an expanded evaluation.Addiction, 97, 253–263.

Moyer, A., Finney, J., Swearingen, C. & Vergun, P. (2002b) Brief interventions for alcohol problems: a meta-analyticreview of controlled investigations in treatment-seeking and non-treatment-seeking populations. Addiction, 97,279–292.

A NEED FOR COMPARABILITY INREVIEW DESIGN: COMMENT ON THREEREVIEWS OF STUDIES FOR ALCOHOLUSE DISORDER TREATMENT

Reviews of treatment outcome studies concerningmethodological quality and/or effective interventionsprovide additional information compared to the analysisof single studies. Researchers, health-service providersand therapists have an easy and time-saving opportunityto acquire a general idea of recent developments in treat-ment research as well as in effective interventions. Theseadvantages might explain the ‘boom’ of reviews (e.g.meta-analyses) in the past years. From the three reviewsin this issue (Miller & Wilbourne 2002; Moyer, Finney &

296 Commentaries


Vergun 2002a; Moyer, Finney & Swearingen 2002b) wemight learn, among other things:∑ That we have to be careful with the target group

for brief interventions. They seem to be best for non-treatment-seeking samples, after excluding individualswith severe alcohol problems. The intervention effects waste away in about 12 months (Moyer et al.2002a).

∑ That not all interventions are equally effective (as practitioners often state); instead a group of behav-ioural orientated interventions and two pharma-cotherapies are in the top 10 list (Miller & Wilbourne2002).

∑ That, alas, the quality of methodological treatmentstudies has not increased impressively in the last30 years and still reaches only a poor 40% of amaximum possible score (Moyer et al.2002b).

The co-publishing of three reviews in one issue obviouslydemonstrates the need of concern for some important‘basics’ of this type of investigation.

Classification of interventions: a brief intervention is abrief intervention?

Moyer et al. (2002a) have defined interventions as ‘brief ’if they provide no more than four sessions. This definitionstill allows a grey area, e.g. studies which have additionalfollow-up (intervention) contacts. Miller & Wilbourne’s(2002) definition of ‘brief ’ is one to two sessions. But, toincrease confusion, if a brief intervention was describedas a specific treatment modality, this category was pre-ferred. There is no easy solution for this exemplarilydemonstrated mess. Probably one step towards a betterclassification of interventions would be to implement atwo-dimensional system: the first dimension for the typeof intervention, the second for the intensity of exposure.

What characterizes methodological quality?

When comparing methodological quality criteria be-tween reviews, almost nothing seems to match: in themethodological analysis of alcohol treatment outcomestudies, Moyer et al. (2002b) use a quality index based on19 items, each specifically weighted, leading from zero to28.5 points. They also cite several other indices, scoringa maximum of between 1.0 and 100 points. Moyer et al.¢s(2002a) review on brief interventions is based on effectsizes, and they do not care for any other methodologicalaspects. Miller & Wilbourne (2002) use several methodo-logical standards as inclusion criteria (e.g. randomizationof groups). In addition, they rated these further 12methodological quality criteria on outcome logic studies,which were weighted. Again, some consensus based har-

monization of evaluation standards would be helpful. In addition, related to the poor methodological quality of studies, I would suggest—as a preventive measure—developing some guidelines for basic aspects of (1) study design, (2) the carrying-out of studies and (3) nec-essary information for publications. The standards ofthe German Society for Addiction Research and Therapy are one example (Deutsche Gesellschaft fürSuchtforschung und Suchttherapie 1992; see also Plattet al. 1996).

Reviews enable us to gain added value compared tothe specific analysis of single studies. It is now time to improve the comparability of review designs andmethodologies.

As a final comment, I am happy that the ‘friendly disagreement’ about the influence of methodologicalstudy strength on the probability of significant out-come differences between Bill Miller (positive relation)and Griffith Edwards (inverse relation) (Miller &Wilbourne 2002, p. 3) could not be settled: the datashowed only a very modest positive correlation. I have theold-fashioned opinion that progress in bona fide treat-ment also counts for the progress of science and—not toforget—the improvement of the patient’s alcohol-relateddisorders.

GERHARD BÜHRINGER

DirectorIFT Institut fur TherapieforschungParzivalstrasse 25MunichD-80304Germany E-mail: [email protected]

References

Deutsche Gesellschaft für Suchtforschung und Suchttherapie,ed. (1992) Documentation Standards 2 for the Treatment ofSubstance Abuse. Freiburg: Lambertus.

Miller, W. & Wilbourne, P. (2002) Mesa Grande: a methodologi-cal analysis of clinical trials of treatments for alcohol dis-orders. Addiction, 97, 265–277.

Moyer, A., Finney, J. W., Swearingen, C. E. & Vergun, P. (2002a) Brief interventions for alcohol problems: a meta-analyticreview of controlled investigations in treatment-seeking and non-treatment-seeking populations. Addiction, 97, 279–292.

Moyer, A., Finney, J. W. & Swearingen, C. E. (2002b) Method-ological characteristics and quality of alcohol treatmentoutcome studies: an expanded evaluation. Addiction, 97, 253–263.

Platt, J., Bühringer, G., Widman, M., Künzel, J. & Lidz, V. (1996)Uniform standards for substance user treatment research: an example from Germany for the United States and othercountries. Substance Use and Misuse, 31, 479–492.

Commentaries 297


META-ANALYSIS IN ALCOHOLTREATMENT RESEARCH:DOES IT HELP US TO KNOW WHAT WORKS?

Every so often a paper on alcohol treatment comes alongthat either cracks an old chestnut, challenges orthodoxthinking or points to a new research direction in an area that has become stuck. This was true of Edwardset al.’s (1977) study of treatment versus advice for alcoholics, Wallace, Cutler & Haines’ (1988) study ofbrief alcohol intervention in primary care, and it is also true in relation to Project MATCH (1997). All three represented the state of the art in terms of method-ological quality of their time. In the first example theorthodoxy of ‘intensive treatment works’ was chal-lenged and the research field spent the next 20 yearstrying to disprove the main assertions of the study. In the case of Wallace, the potential of targeted briefalcohol interventions in primary care was identifiedclearly, and this has spawned a wide range of research inthis setting. While Project MATCH did not find much inthe way of matching effects it cranked up the method-ological quality standards for the treatment outcomeresearch field, particularly in relation to evaluation ofpsychotherapies.

Here we have not one but three papers that help tomove the field forward into a new era of research endeav-our (Miller & Wilbourne 2002; Moyer, Finney & Vergun2002a; Moyer, Finney & Swearingen 2002b). All threedeal with long-standing and fundamental questions inthe alcohol treatment outcome research field, and allthree represent the state of the art in reviews of this area.All three differ greatly in methodology from the threeexamples of landmark research given above. Edwards,Wallace and Project MATCH are obviously randomizedcontrolled trials, each addressing a key question of theday. The three papers that are the subject of this com-mentary are all reviews of a large number of publishedtrials. It is perhaps a measure of the growing influencethat the evidence-based practice (EBP) movement hashad across the board of health-care research, with itsemphasis on systematic reviews and meta-analysis.Many of us who have worked in the pre-EBP era,however, do not necessarily regard meta-analysis as apanacea to answer the burning questions of the day.Meta-analysis is only as good as the studies upon whichit is based, and Moyer’s and Miller’s papers warn us about the methodological limitations of some alcoholtreatment studies under consideration. So to what extentdo these three papers crack old chestnuts, or chal-lenge orthodox thinking, or point to new research directions?

Miller’s (2002) latest version of Mesa Grande has theambitious remit of studying the entire gamut of alcoholtreatment outcome trials, to provide an update of earlierreviews on the question of ‘what works?’ The startingpoint of Mesa Grande was, according to Miller, a friendlydisagreement with Professor Edwards that many of usremember well at a meeting in London in 1988. Edwardset al.’s contention was that studies of higher method-ological quality were less likely to find significant differ-ences between treatment and control conditions, withechoes of his 1977 treatment versus advice study. Millerbelieved the opposite. Thirteen years later, after review-ing, rating and tabulating 361 trials, and develop-ing such indices as methodological quality scores, cumulative evidence scores and outcome logic scores,Miller provides us with not so much a meta-analysis as a ‘mega-analysis’. Miller’s conclusion is that essen-tially he was right: methodological quality is positivelycorrelated with treatment effects (Project MATCHnotwithstanding).

So it appears that Miller’s review has shed some lighton an old argument. However, before we get too carriedaway with Mesa Grande’s implications for the field, it isworth noting a few limitations in the evidence base thathis review highlights. Caution needs to be exercised ininterpreting the results of the analysis in terms their gen-eralizability. Miller notes that few studies have been repli-cated in more than one clinical site (or indeed more thanone country or health-care system). Even in multi-centretrials, site differences are common. Since 73% of thestudies reviewed were conducted in North America those of us living on this side of the Atlantic, or in the rest of the world, should be careful not to extrapolatefrom the results in a simple-minded way. We need replication studies outside of North America. Careshould also be taken in extrapolating the research evi-dence to women with alcohol problems: even in trialspublished after 1995, women still represent only 27% of the samples but at least we have some pointers as tothe type of intervention trials we should be seeking toreplicate.

Moyer et al.¢s (2002a) meta-analysis of brief interven-tions for alcohol problems helps to chisel at, if not crack,one of my favourite old chestnuts. I wanted to know if, interms of alcohol interventions, ‘do the best things come in small packages?’ (Drummond 1997). My concern was that the evidence regarding brief interventions had often been interpreted in a simple-minded way by health-care commissioners as indicating that all we needed were non-specialists delivering brief inter-ventions: ‘scrap these costly intensive specialist pro-grammes and put the money into primary care’, theycried; and in many parts of the United Kingdom specialist

298 Commentaries


alcohol treatment facilities that had taken 30 years to develop were closed down. (Whether or not the funding then went into primary-care interventions is ofcourse another question for another time.) Further,Heather (1995) has cautioned against extrapolating from research on brief interventions research with non-treatment-seeking samples identified by opportunisticscreening to interventions with treatment seeking populations.

Moyer’s meta-analysis shows essentially that briefinterventions are more effective in less severe cases identified by opportunistic screening. In more severetreatment-seeking populations there is little differencebetween brief and extended interventions. Twenty-fiveyears on Edwards et al.’s (1977) conclusions about treat-ment intensity still have currency. So the best things docome in small packages. But, hold on a minute, how doMoyer’s findings fit alongside Miller’s review? Did he not find a whole host of treatments that seemed toperform much better than control treatments in terms of cumulative evidence scores (CES) in people with more severe alcohol problems? It appears to be the case that it is not so much the intensity of treatment as the specific type or modality of treatment that is important. Edwards compared the prevailing intensivetreatment of the day to brief advice. But could it be the case that the prevailing treatment was not particu-larly effective, no matter how intensive it was? This could also be true of the intensive treatments reviewed by Moyer. In Edwards et al.’s study the intensive treat-ment comprised counselling and an introduction toAlcoholics Anonymous, calcium carbimide, in-patientadmission including group/milieu therapy with an AAbias, occupational therapy, marital therapy and socialwork advice. This is an oversimplification, but essentiallymany of the treatments employed at that time are notnow supported by a positive CES in Miller’s review. So thenext question may be ‘what type of intensive interventionis better than brief intervention in treatment seekingpopulations?’

Moyer’s (2002b) review of the methodological qualityof alcohol treatment outcome studies published since1970 is in itself a model of methodological quality in thisgenre of study. While methodological quality is clearlycentral to the design and evaluation of treatmentresearch, it is only recently that this has been examinedin a systematic way. As in previous methodological analy-ses Moyer found a significant improvement in researchquality over time. However, Moyer’s study goes furtherthan previous analyses in terms of the degree of scrutinyof research design and reporting than previous reviewsand assesses a larger number of studies. The conclusionsare nevertheless congruent with earlier smaller method-ological studies; namely that improving methodological

quality need not necessarily be expensive to implement.Also one cannot argue with the conclusion that researchshould be of high quality. It is less clear, however, whatare the priorities for improving methodological quality inthis area.

In an earlier review (Moncrieff & Drummond 1998)we pointed out that measures to maximize internal valid-ity (e.g. highly selected samples, tight control of inter-ventions, employing few centres of clinical excellence,frequent and assiduous follow-up) might serve to compromise the external validity (or generalizability) ofa trial to the more typical patient or clinical setting.Similarly, a trial that maximizes external validity (e.g.multi-centre, pragmatic design, broad entry criteria for subjects, choosing centres and clinicians that are not primarily research orientated, cost-effectivenessanalysis) is likely to provide less understanding of theprecise mechanisms of effect or the key active ingredi-ents. The latter type of study also tends to be more costly to conduct. It is difficult to achieve a high degree of both internal and external validity in the onetrial and so a conscious choice rather than a compromiseneeds to be made depending on the primary researchquestion.

Stepping back from these meta-analytical reviews and the treatment studies on which they are based onehas a sense of the power and the weakness of both theindividual trial of a particular treatment approach con-ducted in one country in one era, and the mega-analysisof a huge number and variety of trials that span a widerange of countries, eras and patient types. While themega-analysis provides a helpful overview of a wholerange of treatment approaches and suggests pointers tothe types of intervention that are most likely to be effec-tive and generalizable, it obscures a multitude of method-ological weaknesses and individual differences betweentrials. For example, identifying the similarities in whatexactly comprised the six studies of ‘case management’reviewed by Miller is much more difficult than identify-ing the differences between them. Further, there aremany types of ‘brief intervention’ among the 31 studiesof this modality reviewed in Mesa Grande. By the sametoken, it is difficult to know what is the generalizability ofone trial of advice versus extended treatment conductedin London in the 1970s to, for instance, a programme inthe rural Midwest of America in 2001, even if in the1970s the trial was of the highest methodological qualityand validity in London.

However, it should not be a case of choosing betweenthe mega-analysis and the individual trial, or choosingbetween trials of either high internal or external validity.As Heather (1998) has pointed out (citing Flay’s (1986)classification of trials), research on a particular treatmentapproach needs to progress through distinct phases from

Commentaries 299


efficacy research to effectiveness research. In other words,we need to establish whether and how a treatment per-forms under tightly controlled conditions, and if it does,we then need to establish whether it is effective and cost-effective in the usual clinical setting and in a variety of cul-tures and health-care settings using pragmatic trialdesigns. The latter type of study needs to be much largerand more complex, and hence expensive, than the former.Evidence of cost-effectiveness is important in terms ofdeveloping treatment systems, as our EBP gurus keeptelling us. The next stage after that is to carry out meta-analysis of the effectiveness trials. It seems to me that thealcohol field is still more at the efficacy phase than at eitherthe effectiveness or meta-analysis phase (with a few excep-tions such as brief interventions in screened samples, buteven this area needs cost-effectiveness studies).

This suggests that what we need now are large multi-centre effectiveness and cost-effectiveness trials of thetreatments emerging from the type of analysis of efficacystudies that Miller has conducted. A more planned andphased strategy to the development of treatmentresearch should be ultimately of more benefit to the treat-ment community and health-care purchasers. Studyingtreatment in a more systematic way will be more able totell us what works.

D. COLIN DRUMMOND

Department of Addictive Behaviour and PsychologicalMedicineSt George’s Hospital Medical SchoolUniversity of LondonCranmer TerraceLondon SW17 0REUKE-mail: [email protected]

References

Drummond, D. C. (1997) Alcohol interventions: do the bestthings come in small packages? Addiction, 92, 375–379.

Edwards, G., Orford, J., Egert, S., Guthrie, S., Hawker, A.,Hensman, C., Mitcheson, M., Oppenheimer, E. & Taylor, C.(1977) Alcoholism: a controlled trial of ‘treatment’ and‘advice’. Journal of Studies on Alcohol, 38, 1004–1031.

Flay, B. R. (1986) Efficacy and effectiveness trials (and otherphases of research) in the development of health promotionprograms. Preventive Medicine, 15, 451–474.

Heather, N. (1995) Interpreting the evidence on brief interven-tions for excessive drinkers: the need for caution. Alcohol andAlcoholism, 30, 287–296.

Heather, N. (1998) Brief opportunities for change in medical settings. In: Miller, W. R. & Heather, N., eds. Treating AddictiveBehaviours, pp. 133–147. New York: Plenum.


Moncrieff, J. & Drummond, D. C. (1998) The quality of alcoholtreatment research: an examination of influential controlledtrials and development of a quality rating system. Addiction,93, 811–823.

Moyer, A. M., Finney, J. W., Swearingen, C. E. & Vergun, P.(2002a) Brief interventions for alcohol problems: a meta-analytic review of controlled investigations in treatment-seeking and non-treatment-seeking populations. Addiction,97, 279–292.

Moyer, A., Finney, J. W. & Swearingen, C. E. (2002b)Methodological characteristics and quality of alcohol treat-ment outcome studies, 1970–98: an expanded evaluation.Addiction, 97, 253–263.

Project MATCH, Research Group (1997) Matching alcoholismtreatments to client heterogeneity: Project MATCH posttreat-ment drinking outcomes. Journal of Studies on Alcohol, 58,7–29.

Wallace, P., Cutler, S. & Haines, A. (1988) Randomised con-trolled trial of general practitioner intervention in patientswith excessive alcohol consumption. British Medical Journal,297, 663–668.

Documents

Meta-analysis in alcohol treatment research: does it help us to know what works?