PerspectiveonDecisionAnalysis Applications,1990–2001€¦ · (1997), Goodwin and Wright (1998), Marshall and Oliver(1995),McNameeandCelona(1990,2001),and Skinner(1999).Kirkwood(1992,1999)providesbrief

Perspective on Decision AnalysisApplications, 1990–2001

Donald L. Keefer • Craig W. Kirkwood • James L. CornerDepartment of Supply Chain Management, Arizona State University, Tempe, Arizona 85287-4706Department of Supply Chain Management, Arizona State University, Tempe, Arizona 85287-4706

Department of Management Systems, University of Waikato, Hamilton, New [email protected] • [email protected] • [email protected]

This article identifies, and provides perspective on, trends and developments in decisionanalysis applications, based primarily on an exhaustive survey of decision analysis appli-

cations published in the period 1990–2001 in major English-language operations researchand closely related journals. It serves as a guide to those interested in recent applicationsin specific areas or in applications that illustrate the use of particular methods. We com-pare the characteristics of the applications articles surveyed here with those of applicationsarticles appearing in a similar set of journals between 1970 and 1989 and conclude that theoverall rate of publication of decision analysis applications has increased. In addition, wefind that both the mix of application areas and the specific aspects of decision analysis thatare emphasized in applications publications have shifted somewhat. We also identify anddiscuss noteworthy trends in, and developments affecting, published applications, includ-ing those in computer software and software-related tools, decision conferencing, stochastictrees, value-focused thinking, normative systems, organizational processes, and real options.We highlight several award-winning decision analysis applications and discuss formationof a new practitioner-oriented professional group. Finally, we present some concerns andthoughts on future needs for advancing decision analysis practice.(Decision Analysis, Applications: Survey and Perspective on Applications; Utility/Preferences, Appli-cations: Survey and Perspective on Applications)

1. IntroductionThis article provides our perspective on the state ofdecision analysis applications, based primarily on asurvey of applications articles published from 1990 to2001 in major English-language operations researchjournals and other closely related journals. We com-pare the results of this survey with an earlier surveyby Corner and Kirkwood (1991) that covered a similarset of journals for the period 1970–1989 and identifynoteworthy trends in, and developments affecting,decision analysis applications. A companion techni-cal report by Keefer et al. (2002) provides short sum-maries of the individual applications articles.

Based on our study, we believe that the state ofdecision analysis applications is healthy, and thatthere was a substantial increase in the rate of publica-tion of decision analysis applications over the period1990–2001 relative to 1970–1989. Furthermore, therewas also an expansion in the use of new methods,particularly those related to problem formulation,implementation, and computation. Thus, while cer-tain decision analysis application areas have maturedand applications therein have become somewhat rou-tine, there continue to be new types of applicationsand use of new methods.This article also provides a guide to published

applications of decision analysis for practitioners and

1545-8490/04/0101/00051545-8504 electronic ISSN

Decision Analysis © 2004 INFORMSVol. 1, No. 1, March 2004, pp. 5–24

KEEFER, KIRKWOOD, AND CORNERPerspective on Decision Analysis Applications

instructors who are interested in specific areas ofapplication or in applications illustrating specific deci-sion analysis methods. In addition, it provides use-ful information for researchers interested in learningmore about which research topics have had an impacton decision analysis applications.We use the term decision analysis to refer to a

set of quantitative methods for analyzing decisionsbased on the axioms of consistent choice (Clemen1996, Chapter 14; Kirkwood 1997, §9.9). Decisionanalysis is normative, rather than descriptive. Thatis, it provides a systematic quantitative approachto making better decisions, rather than a descrip-tion of how unaided decisions are made. There issome subjectivity in deciding whether a particularapplication qualifies as an application of decisionanalysis. To be included, an application generallyhad to explicitly analyze alternatives for a decisionproblem using judgmental probabilities and/or sub-jectively assessed utility/value functions. Ambiguouscases were resolved by including the article if, on bal-ance, it took a decision analysis approach. There isalso some subjectivity in deciding whether an articlereports an application. Many of the surveyed articlesreport case histories of the use of decision analysis toaddress a specific decision problem, while other arti-cles report on analysis that provided background orinsights for policy making.This article does not address the analytic hierar-

chy process or multicriteria decision making, twoapproaches that are related to multiattribute deci-sion analysis. The interested reader is referred toMollaghasemi and Pet-Edwards (1997) or Yoon andHwang (1995) for more information about thoseapproaches. Brown (1992) and Howard (1992), whoare both prominent decision analysts, present dif-fering philosophical views from a decision analysisperspective about the relationship between decisionanalysis and other decision-oriented methods, such asfuzzy logic and the analytic hierarchy process.This article is organized as follows: Section 2 con-

siders background decision analysis references forapplications work. Section 3 describes the applica-tions articles that we surveyed for the period 1990–2001 by providing tables that show the applicationarea and methods used for each application, as well

as the overall characteristics of the set of applications,including a comparison with applications from theperiod 1970–1989 surveyed by Corner and Kirkwood(1991). Section 4 provides our perspective relative totrends and developments in decision analysis appli-cations that we observed. Section 5 presents ourviews on needs and concerns for the future. Section 6presents concluding remarks.

2. Background ReferencesThe vigor and continued development of decisionanalysis is demonstrated by the substantial num-ber of textbooks and other references on decisionanalysis applications that were published between1990 and 2001. Decision analysis textbooks includeBell and Schleifer (1995), Clemen (1996), Golub(1997), Goodwin and Wright (1998), Marshall andOliver (1995), McNamee and Celona (1990, 2001), andSkinner (1999). Kirkwood (1992, 1999) provides briefintroductions to decision analysis methods, whileHammond et al. (1999) provide a relatively nonquan-titative introduction to systematic decision analysisprocedures. Keeney (1992) presents a value-focusedapproach to formulating decision problems, andKirkwood (1997) reviews methods for analyzing deci-sions with multiple conflicting objectives, includingspreadsheet procedures to implement these methods.Oliver and Smith (1990) address influence diagrams,which became widely used during the 1990s.In addition to these publications of general decision

analysis methods, several publications specificallyaddress probability assessment, approximation proce-dures, or utility/value function assessment. Shephardand Kirkwood (1994) provide an annotated transcriptof a probability elicitation interview that illustratesstandard probability elicitation procedures. Morganand Henrion (1990) address specifics of probabil-ity assessment and communication about uncertaintyin the context of policy analysis. Clemen et al.(2000) examine methods for assessing probabilisticdependence among pairs of random variables. Keefer(1994) discusses three-point approximations to rep-resent continuous probability distributions in deci-sion analysis problems, including those where riskpreferences are important. Poland (1999) reviews

6 Decision Analysis/Vol. 1, No. 1, March 2004


an approximate probabilistic analysis procedure toreduce the assessment and computational complex-ity of decision analyses with many uncertainties,some of which may be dependent. This work drawsupon “moment methods” described by Smith (1993),whereby an approximation to the distribution of anoutput variable is obtained from its moments, whichare calculated from moments of the (assessed) inputdistributions. Keefer (1991) presents a framework foraddressing resource allocation decisions with riskaversion and probabilistic dependence. Borcherdinget al. (1991) and Lai (2001) provide empirical compar-isons of different methods for assessing weights formultiattribute utility and value functions. Edwards(1992) reviews theory and applications issues associ-ated with using expected utility analysis.A number of other sources from 1990 to 2001 pro-

vide additional reference material relevant for appli-cations. Corner and Corner (1995) summarize thecharacteristics of decision analysis applications from1970 to 1989 that were surveyed by Corner andKirkwood (1991). Zeckhauser et al. (1996) includesome papers of interest for decision analysis practice,

Table 1 Number of Applications Articles, by Journal, with Trends

Number of articles Average number of articles per year

1970–1989 1990–2001 1970–1989 1990–2001 Percent change

Number of years covered 20 12Number of journals covered 10 16Total number of articles 85 86 4�25 7�17 69Total number of articles (common journals) 85 63 4�25 5�25 24

Decision Sciences 4 1 0�20 0�08 −58European Journal of OR 5 0 0�25 0�00 −100IEEE Trans. on Engineering Mgt. not covered 4 0�33IEEE Trans. on SMC 7 1 0�35 0�08 −76Interfaces 18 40 0�90 3�33 270Journal of MCDA not published 6 0�50Journal of the OR Society 18 1 0�90 0�08 −91Management Science 10 4 0�50 0�33 −33Military OR not published 8 0�67Omega 3 1 0�15 0�08 −44Operations Research 15 11 0�75 0�92 22OR Letters 0 0 0�00 0�00 0Reliability Engrg. and System Safety not covered 2 0�17Research • Tech. Mgt. not covered 2 0�17Risk Analysis 6 4 0�30 0�33 11Theory and Decision not covered 1 0�08

including applications to public policy and medicaldecision making. Magat et al. (1996) review a gen-eral approach for establishing a death-equivalent met-ric for valuing long-term health effects. Noonan andVidich (1992) present a decision analysis frameworkfor utilizing hazardous waste site assessment in realestate acquisition. Two websites were established thatprovide additional information about decision analy-sis and its applications, one by the Decision AnalysisSociety of INFORMS (www.informs.org/Society/DA)and one by the Decision Analysis Affinity Group(www.daag.net).

3. Applications Articles andPublications Trends

This section summarizes characteristics of the decisionanalysis applications articles published in the oper-ations research literature for the period 1990–2001, andcompares these to characteristics of applications arti-cles published during the period 1970–1989, as sur-veyed by Corner and Kirkwood (1991). Table 1 shows

Decision Analysis/Vol. 1, No. 1, March 2004 7


the journals that were surveyed and the number ofdecision analysis applications articles in each journal.The first four rows of this table provide overall sum-mary data for each of the two time periods, includ-ing the number of years in each period, the numberof journals covered, the total number of applicationsarticles in all of the journals, and the total numberof articles in the journals that were covered for bothtime periods (the “common journals”). Note that thethird row of Table 1 lists 85 applications articles for theperiod 1970–1989 even though Corner and Kirkwood(1991) show 86 applications for that period. This isbecause Corner and Kirkwood (1991) include one arti-cle in the medical category that is a survey, rather thanan application, and that article is not included in thesecomparisons.The remaining rows of the table relate to the num-

ber of applications articles for each journal. For theserows, the second column shows the number of appli-cations articles identified in each journal by Cornerand Kirkwood (1991) for the period 1970–1989, thethird column shows the number of articles that weidentified in each journal for the period 1990–2001, thefourth column shows the average number of appli-cations articles published per year in each journalfor the period 1970–1989, and the fifth column showsthis average for the period 1990–2001. Finally, thesixth column shows the percent change in the aver-age number of articles published each year from theperiod 1970–1989 to the period 1990–2001. For exam-ple, Table 1 shows that there were 18 decision analysisapplications articles published in Interfaces between1970 and 2001, and 40 published in Interfaces between1990 and 2001. As the table also shows, this corre-sponds to an average of 18/20= 0�90 articles per yearbetween 1970 and 1989 and an average of 40/12 =3�33 articles per year between 1990 and 2001. Thus,as shown in the right-most column of the table, therewas a �3�33− 0�90�/0�90 = 270% increase in the aver-age number of decision analysis applications articlespublished per year in Interfaces from 1970–1989 to1990–2001.As row 3 shows, we identified 86 application arti-

cles in the 16 journals that we surveyed for the period1990–2001. The set of journals that we surveyed is thesame as the set covered for 1970–1989 by Corner and

Kirkwood (1991), except that we added the follow-ing six journals: (1) Two new journals, the Journal ofMulti-Criteria Decision Analysis and Military OperationsResearch, that began publication in the 1990s, (2) Reli-ability Engineering and System Safety, Research • Tech-nology Management, and Theory and Decision, whichwere added at the recommendation of knowledge-able individuals, and (3) IEEE Transactions on Engi-neering Management, which we feel is comparable tosome of the other journals added. The third row inTable 1 shows that there was a 69% increase in theaverage number of decision analysis applications arti-cles published per year from 1970–1989 to 1990–2001.This substantial increase in the average publicationrate may be somewhat overstated because of the sixjournals that were added for 1990–2001. However, thefourth row in Table 1 shows that even without con-sidering those six journals the average annual publi-cation rate increased by 24%.This overall positive trend masks what may be a

less positive trend, namely that applications publica-tions appear to be concentrating in a smaller numberof journals, most of which are published in the UnitedStates. In particular, 47% of the applications duringthe 1990–2001 period were published in Interfaces,while only 21% of the applications during the 1970–1989 period were published in that journal. Another13% of the 1990–2001 articles appeared in OperationsResearch, primarily in its OR Practice Section (9 of11 articles)—which, like Interfaces, explicitly targetsapplications. And the new journal Military OperationsResearch came in third with 9% of the articles. Therewas a substantial drop in the number of decisionanalysis applications articles published in the mainEuropean OR journals (European Journal of OperationalResearch, Journal of the Operational Research Society, andOmega), with decision analysis applications virtuallydisappearing from those journals. While the new Jour-nal of Multi-Criteria Decision Analysis, which is basedin the United Kingdom, is publishing decision analy-sis applications, there appears to be an overall trendfor a greater portion of the decision analysis applica-tions to be published in U.S. journals, and especiallyin the INFORMS practice journal Interfaces.The increase in decision analysis applications pub-

lished by Interfaces is partially explained by two



milestones. First, in 1995, the new regular column“Practice Abstracts,” edited by one of the currentauthors, began soliciting submissions. This produced6 of the 40 Interfaces applications articles that we sur-vey. Second, two special issues of Interfaces appearedbetween 1990 and 2001 (November–December 1992and November–December 1999) that focused ondecision analysis applications. These special issuesyielded an additional 12 of the 40 applications arti-cles published in Interfaces. Of course, some of thosearticles might have been published in Interfaces evenwithout the special issues. Also, even if the PracticeAbstracts and special issues are ignored, the annualrate of publication of decision analysis applicationsin Interfaces increased substantially from 1970–1989to 1990–2001. Note, also, that there were two specialdecision analysis issues in journals during the period

Table 2 Application Articles Listed by Application Area

ENERGYBidding and Pricing: Keefer (1995), Keefer et al. (1991), Kidd and Prabhu (1990).Environmental Risk: Balson et al. (1992), French (1996), Hämäläinen et al. (2000), Keeney and von Winterfeldt (1991), Procaccia et al. (1997).Product and Project Selection: Borison (1995), Burnett et al. (1993), Dyer et al. (1990), Keeney et al. (1995), Parnell (2001), Smith and McCardle (1999),

Walls et al. (1995).Strategy: Keeney and McDaniels (1992), Keeney and McDaniels (1999), Skaf (1999).Technology Choice: Dyer et al. (1998), Jackson et al. (1999), Perdue and Kumar (1999), Toland et al. (1998), von Winterfeldt and Schweitzer (1998).Miscellaneous: Dunning et al. (2001), Rios Insua and Salewicz (1995), Taha and Wolf (1996).

MANUFACTURING AND SERVICESFinance: Engemann and Miller (1992), Mulvey (1994).Product Planning: Beccue (2001), Dillon and Haimes (1996), Keeney (2000), Millet (1994), Yassine et al. (1999).R&D Project Selection: Bruggink (1997), Hess (1993), Islei et al. (1991), Perdue et al. (1999), Rzasa et al. (1990), Spradlin and Kutoloski (1999),

Stonebraker et al. (1997), Thurston (1990).Strategy: Bodily and Allen (1999), Clemen and Kwit (2001), Keeney (1999b), Krumm and Rolle (1992), Kusnic and Owen (1992), Matheson and Matheson

(1999), Quaddus et al. (1992).Miscellaneous: Chien and Sainfort (1998).

MEDICALBrown (1997), Feinstein (1990), Hazen et al. (1998), Smith and Winkler (1999), Winkler et al. (1995).

MILITARYBresnick et al. (1997), Buede and Bresnick (1992), Burk and Parnell (1997), Davis et al. (1999), Davis et al. (2000), Doyle et al. (2000), Griggs et al.

(1997), Jackson et al. (1997), Kerchner et al. (2001), Parnell et al. (1998), Parnell et al. (2001), Rayno et al. (1997), Stafira et al. (1997).

PUBLIC POLICYBana e Costa (2001), Hall et al. (1992), Heger and White (1997), Jones et al. (1990), Keeney (1997), Keeney and McDaniels (2001), Keeney and

von Winterfeldt (1994), Keeney et al. (1990), Lehmkuhl et al. (2001), McDaniels (1995), Reagan-Cirincione et al. (1991), Spector (1993), Taylor et al.(1993).

GENERALBaker et al. (2000), Hurley (1998), Keller and Kirkwood (1999), Matzkevich and Abramson (1995), Paté-Cornell and Fischbeck (1994), Vári and Vecsenyi

(1992).

1970–1989 (the January–February 1980 issue of Opera-tions Research and the April 1982 issue of the Journal ofthe Operational Research Society). Therefore, the overallcount of decision analysis applications articles shouldnot be significantly skewed by these special issues,and it is clear that there was a significant increase inthe rate of publication of applications articles from1970–1989 to 1990–2001.

Classification of Applications Areas, with TrendsIn Table 2, each applications article that we surveyedfor the period 1990–2001 is classified into exactly oneof the applications areas or subareas shown in thattable. The area/subarea selected for a particular arti-cle is the one that, on balance, is most emphasizedin the article. This table is presented as a guide forreaders who are interested in finding applications arti-



cles that address specific application areas. Brief sum-maries of all of these articles are included in thetechnical report by Keefer et al. (2002).Table 3 compares the application area for the arti-

cles published during the period 1970–1989 (Cornerand Kirkwood 1991) with the application area for thearticles that we surveyed. There are some differencesin our classification scheme relative to the schemeused by Corner and Kirkwood (1991) because of shiftsin decision analysis application areas between 1970–1989 and 1990–2001. (A classification category that isnot used in one of the two survey articles is indicatedin Table 3 with a “Not Applicable (NA)” entry.) How-ever, we used similar classification areas to those inCorner and Kirkwood (1991) to the extent possible tofacilitate comparisons between the two time periods.Because of the slightly different classification cat-

egories, as well as the addition of some new jour-

Table 3 Number of Applications Articles by Application Area,with Trends

Number of articles

1970–1989 1990–2001

ENERGY 24 26Bidding (and pricing) 3 3Environmental risk NA∗ 5Product and project selection 4 7Regulation 5 NASite selection 8 NAStrategy NA 3Technology choice 4 5Miscellaneous NA 3

MANUFACTURING AND SERVICES 16 23Budget allocation 3 NAFinance NA 2Product planning 4 5R&D project selection NA 8Strategy 5 7Miscellaneous 4 1

MEDICAL 16 5

MILITARY NA 13

PUBLIC POLICY 20 13Standard setting 8 NAMiscellaneous 12 13

GENERAL 9 6

∗Not applicable.

nals and the differing lengths of the two time periodscompared in Table 3, caution should be used in draw-ing conclusions about trends in applications. How-ever, there are some significant points. First, withinthe energy area, there appears to have been a shifttoward applications in environmental risk, productand project selection, and strategy, and away fromregulation and site selection. Overall, energy contin-ues to be a significant application area for decisionanalysis. Within manufacturing and services, R&Dproject selection applications increased during theperiod 1990–2001 to the extent that we include a sep-arate category for these applications, which was notneeded by Corner and Kirkwood (1991) for the period1970–1989.Medical applications published in the OR litera-

ture dropped substantially between 1970–1989 and1990–2001. However, we believe this is because deci-sion analysis methods are now so well establishedwithin the medical community that most medicaldecision analysis applications are published in spe-cialized medical journals. Published military appli-cations increased from virtually none in the period1970–1989 to the extent that the numbers now justifya separate military category. This appears to be duein part to the new journal Military Operations Research,which has published a substantial number of decisionanalysis applications since it was established in 1994.Finally, while public policy applications still continueat a substantial rate, there was not a sufficiently largenumber of these during the 1990–2001 period focusedon standard setting to justify a separate subcategoryfor these.

Methodological and Implementation Issues,with TrendsTable 4 lists those application articles that presentsignificant detail about a particular decision analysismethodological or implementation issue. Many arti-cles deal with nearly all of the methodological andimplementation issues shown in that table, but anarticle is included in Table 4 only if it providesdetailed information on a topic. Thus, this table canbe used to identify articles that emphasize a partic-ular methodological or implementation issue. Arti-cles are included in the strategy and/or objectives



Table 4 Application Articles Addressing Methodological and Implementation Issues

Strategy and/or objectives generationBaker et al. (2000), Bana e Costa (2001), Bodily and Allen (1999), Bresnick et al. (1997), Brown (1997), Buede and Bresnick (1992), Burk and Parnell

(1997), Burnett et al. (1993), Chien and Sainfort (1998), Davis et al. (1999), Davis et al. (2000), Doyle et al. (2000), Dyer et al. (1998), Dyer et al.(1990), French (1996), Hämäläinen et al. (2000), Islei et al. (1991), Jackson et al. (1997), Jackson et al. (1999), Jones et al. (1990), Keeney (1999b),Keeney and McDaniels (1992, 1999, 2001), Keeney et al. (1995), Keeney and von Winterfeldt (1994), Keeney et al. (1990), Keller and Kirkwood (1999),Kerchner et al. (2001), Krumm and Rolle (1992), Kusnic and Owen (1992), Lehmkuhl et al. (2001), McDaniels (1995), Parnell et al. (1998), Parnellet al. (2001), Perdue and Kumar (1999), Rayno et al. (1997), Reagan-Cirincione et al. (1991), Skaf (1999), Spector (1993), Spradlin and Kutoloski(1999), von Winterfeldt and Schweitzer (1998).

Problem structuring/formulation (via decision trees and influence diagrams)Balson et al. (1992), Bodily and Allen (1999), Borison (1995), Brown (1997), Dillon and Haimes (1996), Dunning et al. (2001), Dyer et al. (1998), Dyer

et al. (1990), Engemann and Miller (1992), Feinstein (1990), Griggs et al. (1997), Hazen et al. (1998), Heger and White (1997), Hess (1993), Jacksonet al. (1999), Keefer (1995), Keefer et al. (1991), Keeney (1997), Keeney et al. (1995), Keeney and von Winterfeldt (1994), Krumm and Rolle (1992),Matheson and Matheson (1999), Matzkevich and Abramson (1995), Millet (1994), Perdue et al. (1999), Quaddus et al. (1992), Rzasa et al. (1990),Smith and McCardle (1999), Smith and Winkler (1999), Stafira et al. (1997), Stonebraker et al. (1997), Taylor et al. (1993), Walls et al. (1995), Yassineet al. (1999).

Probability assessmentBalson et al. (1992), Chien and Sainfort (1998), Dillon and Haimes (1996), Dunning et al. (2001), Dyer et al. (1990), Feinstein (1990), Keefer (1995),

Keeney et al. (1995), Keeney and von Winterfeldt (1991), Keeney and von Winterfeldt (1994), McDaniels (1995), Paté-Cornell and Fischbeck (1994),Perdue et al. (1999), Procaccia et al. (1997), Smith and McCardle (1999), Smith and Winkler (1999), Stafira et al. (1997), Taha and Wolf (1996), Tayloret al. (1993), von Winterfeldt and Schweitzer (1998), Winkler et al. (1995), Yassine et al. (1999).

Utility/value assessmentBaker et al. (2000), Bana e Costa (2001), Bresnick et al. (1997), Burk and Parnell (1997), Doyle et al. (2000), Dyer et al. (1998), Dyer et al. (1990), Hall

et al. (1992), Hämäläinen et al. (2000), Hazen et al. (1998), Jackson et al. (1997), Jackson et al. (1999), Keeney (2000), Keeney and McDaniels (1992),Keeney and McDaniels (1999), Keeney et al. (1995), Keeney and von Winterfeldt (1994), Keeney et al. (1990), Kerchner et al. (2001), Kidd and Prabhu(1990), Lehmkuhl et al. (2001), McDaniels (1995), Mulvey (1994), Parnell et al. (1998), Rayno et al. (1997), Rios Insua and Salewicz (1995), Thurston(1990), Walls et al. (1995).

Sensitivity analysisBaker et al. (2000), Bana e Costa (2001), Bodily and Allen (1999), Brown (1997), Doyle et al. (2000), Dyer et al. (1998), Hess (1993), Jackson et al.

(1999), Keeney and von Winterfeldt (1994), Kerchner et al. (2001), Lehmkuhl et al. (2001), McDaniels (1995), Millet (1994), Perdue et al. (1999),Quaddus et al. (1992), Reagan-Cirincione et al. (1991), Smith and Winkler (1999), Spradlin and Kutoloski (1999), Stafira et al. (1997), Taylor et al.(1993), Thurston (1990), Walls et al. (1995), Yassine et al. (1999).

Communication/facilitationBodily and Allen (1999), Borison (1995), Bresnick et al. (1997), Feinstein (1990), French (1996), Hämäläinen et al. (2000), Islei et al. (1991), Jones

et al. (1990), Keefer (1995), Keefer et al. (1991), Keeney (1999b), Keeney and McDaniels (1992), Keeney and McDaniels (1999), Keeney et al. (1995),Keeney et al. (1990), Keller and Kirkwood (1999), Kerchner et al. (2001), Krumm and Rolle (1992), Kusnic and Owen (1992), Lehmkuhl et al. (2001),McDaniels (1995), Quaddus et al. (1992), Reagan-Cirincione et al. (1991), Skaf (1999), Spector (1993), Spradlin and Kutoloski (1999), Vári andVecsenyi (1992), von Winterfeldt and Schweitzer (1998), Winkler et al. (1995).

Group issuesBaker et al. (2000), Bana e Costa (2001), Bresnick et al. (1997), Hämäläinen et al. (2000), Keeney and McDaniels (1999), Keeney et al. (1990), Keller and

Kirkwood (1999), Kusnic and Owen (1992), Quaddus et al. (1992), Reagan-Cirincione et al. (1991), Vári and Vecsenyi (1992), Winkler et al. (1995).

ImplementationBaker et al. (2000), Bodily and Allen (1999), Burk and Parnell (1997), Doyle et al. (2000), Dyer et al. (1990), Engemann and Miller (1992), Islei et al.

(1991), Jackson et al. (1997), Keefer et al. (1991), Keeney and McDaniels (1992), Keeney and McDaniels (1999), Keeney et al. (1995), Keeney andvon Winterfeldt (1994), Keeney et al. (1990), Keller and Kirkwood (1999), Kerchner et al. (2001), Kusnic and Owen (1992), Lehmkuhl et al. (2001),Parnell et al. (1998), Parnell et al. (2001), Paté-Cornell and Fischbeck (1994), Rzasa et al. (1990), Skaf (1999), Smith and Winkler (1999), Spradlinand Kutoloski (1999), Vári and Vecsenyi (1992), Walls et al. (1995).



generation category if they discuss overall decisionstrategy and/or present an objectives or value hier-archy, or discuss the decision structuring process indetail. Articles are included in the problem structur-ing/formulation category if they describe and presenta decision tree and/or influence diagram and discussits development and use. The probability assessmentcategory includes articles that discuss the elicitation ofsubjective probabilities, probabilistic dependence orindependence, and/or risk assessment. Similarly, arti-cles are listed in the utility assessment category if sub-jective utility/value functions or trade-offs betweenattributes are discussed in depth. Articles are listed inthe sensitivity analysis category if tornado or rainbowdiagrams, or something similar, are presented and/orstatistical or mathematical approaches to model sen-sitivity analysis are discussed.The communication/facilitation category includes

articles that discuss the role of the analyst, how deci-sion analysis facilitates the decision process, and/orhow communication channels are opened due to theuse of the approach. Articles are included in the groupissues category if there is discussion about aggregat-ing individual preferences into a group function, ordiscussion of the solicitation and treatment of mul-tiple individual inputs into the preference or prob-ability model. Finally, the implementation categoryincludes articles that discuss post-modeling issuesrelated to implementing chosen alternatives or thevalue of decision analysis techniques for the individ-uals or organization in their decision-making efforts.Table 5 compares the classifications of articles by

methodological and implementation issues in Cornerand Kirkwood’s (1991) survey of application dur-ing the 1970–1989 period with our classification forapplications during the 1990–2001 period. As withthe comparison of application areas in Table 3, cau-tion is warranted in interpreting Table 5. Specifically,the classification categories that we use for our sur-vey of 1990–2001 applications is somewhat differentfrom Corner and Kirkwood’s (1991) classification cat-egories because of changes in methodological andimplementation issues that are emphasized in thearticles. However, some general conclusions can bedrawn about trends in methodological and implemen-tation issues.

Table 5 Number of Applications Articles by Methodological and Imple-mentation Issue, with Trends

Number of articles

1970–1989 1990–2001

Strategy and/or objectives generation NA∗ 42Problem structuring/formulation 24 34Decision trees 36 NAProbability assessment 15 22Utility/value assessment 28 28Sensitivity analysis NA 23Communication/facilitation 23 29Group decision making (issues) 13 12Implementation NA 27

∗Not applicable.

First, Corner and Kirkwood’s (1991) classificationcategories “problem structuring/formulation” and“decision trees” for the 1970–1989 applications donot adequately represent the corresponding topics inthe 1990–2001 applications. A new classification cat-egory “strategy and/or objectives generation” wasadded primarily because of the expanded use ofvalue-focused thinking approaches in a variety ofapplications. Corner and Kirkwood’s (1991) category“decision trees” was expanded to “problem struc-turing/formulation (via decision trees and influencediagrams),” primarily because of the expended useof influence diagrams during the 1990–2001 period.A new category, “sensitivity analysis,” was addedbecause of the expanded discussion in the 1990–2001 articles of such sensitivity analysis methods astornado and rainbow diagrams. Of course, sensitiv-ity analysis was well recognized prior to 1990 as animportant part of decision analysis, but it receivedsubstantially increased emphasis in the articles pub-lished in the 1990–2001 period relative to its empha-sis in the 1970–1989 period. While we cannot say forcertain why this is true, it seems reasonable that theexpanded use of spreadsheets and personal computerdecision analysis packages may have facilitated con-ducting and reporting sensitivity analyses.Finally, a new “implementation” category was

added because of the expanded consideration of thistopic in published applications. Of course, imple-mentation has always been important in applica-tions. The increase in emphasis on implementation



in the 1990–2001 publications relative to the 1970–1989 publications may be due in part to the increasedconcentration of articles in Interfaces, as well as thesubstantial number of articles appearing in the newjournal, Military Operations Research, which appears toencourage emphasis on implementation issues.

4. Additional NoteworthyTrends and Developments

In the preceding section, we discuss trends based pri-marily on counts and classifications of the applica-tions articles that we surveyed. In this section, wedraw more broadly on the contents of the applicationsarticles to discuss a number of additional noteworthytrends and developments relevant to decision analysisapplications and practice between 1990 and 2001. Inaddition, we note developments in professional soci-eties relevant to decision analysis practice and citeaward-winning entries in practice competitions. Thespecific sources highlighted in this section, some ofwhich are not applications articles, are those that fitwithin these trends and developments. As indicatedabove, the companion technical report to this article(Keefer et al. 2002) contains brief summaries of allthe applications articles that we surveyed, includingadditional information about those highlighted in thissection.

General Trends and Developments

Computer Software and Related Tools. Duringthe period 1990–2001, increasingly powerful personalcomputer decision analysis software was developed,refined, and utilized in applications. This facilitatedmore widespread use of decision analysis tools devel-oped in the 1980s such as influence diagrams andstrategy tables, and tornado diagrams for sensitiv-ity analysis. It also facilitated structuring and analyz-ing larger decision analysis models. For example, inan analysis to help the New York Power Authoritydevelop a 10-year schedule for refueling its IndianPoint 3 Nuclear Power Plant, Dunning et al. (2001)utilized software in applying a spectrum of decisionanalysis tools including strategy tables, an influencediagram, and a decision tree with over 200 million

paths. Similarly, in a study to help Amgen select astrategy for developing and commercializing a newdrug, Beccue (2001) employed a variety of software-based decision analysis tools including an influencediagram, a tornado diagram, and a decision tree withapproximately 500,000 scenarios for each of eight keystrategies that were identified via a strategy table.Regarding overall software developments, Call and

Miller (1990) discuss computational approaches toautomating decision analysis calculation procedures.Stand-alone software packages using decision treesand/or influence diagrams that were developed orupdated during the period 1990–2001 include DATA,DPL, and Supertree, and spreadsheet decision analy-sis add-ins include PrecisionTree and TreePlan. Evenstand-alone packages are often used in conjunctionwith spreadsheet models, which are used pervasivelyin practice. Development also continued on the stand-alone software package Logical Decisions that specif-ically focuses on decisions with multiple objectives.Several of these packages are now available in con-junction with decision analysis or basic OR textbooks.Current information about these and other softwarepackages can be obtained from the Decision Analy-sis Society website or from the biennial surveys ofdecision analysis software in OR/MS Today (Maxwell2002).

Decision Conferencing. A decision conference isan intensive computer-assisted group meeting, orworkshop, focused on a specific decision problem andutilizing outside facilitators skilled in decision anal-ysis and group facilitation techniques. The idea isto generate and evaluate alternatives in a structuredfashion and use real-time quantitative models to helpthe group reach consensus on a preferred alterna-tive while avoiding “groupthink” pitfalls. Decisionconferences typically last about two days. Althoughoriginated prior to the period surveyed in this arti-cle, decision conferencing has become more promi-nent in applications in the OR literature during theperiod covered by this survey, perhaps in part due toimprovements in computer software and hardware.This approach offers a powerful synthesis of tech-niques from decision analysis, decision-support sys-tems, and group management.



Decision conferencing is most often applied in con-junction with multiattribute value models, and typi-cally in the public sector. Bresnick et al. (1997) andBuede and Bresnick (1992) describe military applica-tions utilizing decision conferences and multiattributemodels. French (1996) and Hämäläinen et al. (2000)discuss the use of decision conferences and multiat-tribute analysis for nuclear accident management inconjunction with the RODOS project, a European ini-tiative to build a decision-support system for emer-gency response. Reagan-Cirincione et al. (1991) andQuaddus et al. (1992) describe multiattribute applica-tions involving, respectively, strategic policy optionsfor medical malpractice insurance for the New YorkState Insurance Department and strategic planningin a volunteer organization providing services to thedisabled. Vári and Vecsenyi (1992) discuss the useof decision analysis methods in conjunction with26 decision conferences on a variety of decision prob-lems for manufacturing, services, and governmentorganizations in Hungary.

Stochastic Trees. Stochastic trees were developedduring the period covered by our survey to aidmedical decision making. They combine features ofcontinuous-time Markov chains with those of decisiontrees and, in particular, enable time to be modeled asa continuum where health state transitions can occurat any instant. They retain the familiar rollback proce-dure for decision trees and can also accommodate riskpreferences via utility functions. Hazen et al. (1998)and Pellissier et al. (1996) review stochastic tree anal-ysis methods and their application to hip replacementsurgery.This is an exciting development for decision anal-

ysis in an important application area. The basicassumption that health state transitions can occur atany moment in continuous time appears to be natu-ral in the medical context, and the case for applyingstochastic trees more widely to medical decision prob-lems seems persuasive. Understandably, additionalmedical applications are likely to appear primarily inthe medical literature rather than the OR literature.Methodological developments continue in this rela-tively new area (see, for example, Hazen 2000).

Value-Focused Thinking. Keeney (1992, 1994,1999a, 2001) makes the case for using values as theprimary driver for problem structuring and analysis,including the generation of alternatives, and providesmethods to aid in this process, as well as illustrativeexamples. This value-focused thinking expands uponearlier work on multiattribute utility and value mod-els, and has been a major force in increasing the num-ber and scope of multiattribute applications, as wellas the quantity and quality of alternatives generatedin decision analyses.In particular, the book by Keeney (1992) on value-

focused thinking, along with the spreadsheet-orientedtext on multiobjective decision analysis by Kirkwood(1997), appears to have been influential in militaryapplications during the survey period. For exam-ple, Burk and Parnell (1997), Davis et al. (2000),Doyle et al. (2000), Jackson et al. (1997), Kerchneret al. (2001), Parnell et al. (1998), and Parnell et al.(2001) all describe military applications that makeprominent use of value-focused thinking and multiat-tribute value models. All of these articles cite Keeney(1992), and all except for Burk and Parnell (1997) citeKirkwood (1997).Baker et al. (2000), Keeney (1999b), Keeney and

McDaniels (1992, 2001), and Lehmkuhl et al. (2001)describe applications of value-focused thinking inother areas, including strategy and public policy.Drawing upon both the descriptive and prescriptiveliterature on decision making, Corner et al. (2001) sug-gest a dynamic synthesis of value-focused thinkingwith alternative-focused thinking.

Interdisciplinary Trends and DevelopmentsEach of the following trends or developments linksdecision analysis methods with another disciplineand, thus, is a synthesis of traditional decision anal-ysis with another established discipline. Because ofthe interdisciplinary nature of these trends and devel-opments, we include several sources beyond the ORapplications literature that was considered in oursurvey.

Normative Systems. Connections between deci-sion analysis and artificial intelligence (AI) methodshave been increasingly recognized, and this has led to



significant research and applications, much of whichhas been published outside of the OR literature. Therewere a variety of creative applications involving com-binations of decision analysis and AI methods overthe period covered by our survey. Henrion et al.(1991) provide an extensive introduction to the con-nections between decision analysis and knowledge-based expert systems. Matzkevich and Abramson(1995) survey and synthesize research from the deci-sion analysis and AI communities involving influ-ence diagrams and belief networks. Their discussionof normative systems, which are AI systems basedon influence diagrams or belief nets and thus onBayesian principles, is of particular interest. They pro-vide brief descriptions of several implemented sys-tems in areas including medical diagnosis, energyprice and demand forecasting, and machine vision.Silverman (1994) focuses on approaches to unifyingexpert systems methods with mathematical modelingapproaches to decision making, including decisionanalysis. Hedbert (1998) reviews decision analysis-oriented work at Microsoft Research, including devel-opment of a Bayesian-based system for automatingmore responsive interfaces that was applied to theOffice Assistant in Microsoft Office.This area appears to have great potential for addi-

tional applications. Further information about deci-sion analysis methods in AI can be found at thewebsite of the Association for Uncertainty in ArtificialIntelligence (www.auai.org).

Organizational Processes. As decision analysis hasmatured, increasing attention has been devoted tospecifying procedures for conducting and implement-ing decision analysis successfully in organizations.In large-scale strategic decision analyses in particu-lar, a well-defined process typically is used for man-aging the efforts of, and the interactions between,carefully constructed teams composed of analysts,managers, and executives. Such a process typicallyis used first in structuring and analyzing the deci-sion problem at hand and then in following throughto manage and carry out recommended action plansand accompanying changes. Organizational processesfor decision analysis were developed within deci-sion analysis consulting and practitioner groups inthe 1980s, but were not widely known outside these

groups until the 1990s. Bodily and Allen (1999) reviewa dialogue process to manage the interaction betweendecision analysts and other stakeholders in a deci-sion. Kusnic and Owen (1992), Krumm and Rolle(1992), and Skaf (1999) provide guidance on conduct-ing and implementing large-scale decision analyseswithin major industrial firms, including the effectiveuse of teams and large databases, as well as methodsfor dealing with multiple decision makers. Mathesonand Matheson (1999) discuss the use of an outside-in approach to take better account of a company’sexternal environment during strategic decision anal-ysis. Clemen and Kwit (2001) review the history ofdecision and risk analysis at Eastman Kodak Com-pany from the early 1980s to 2001 and summarize thecharacteristics of 178 projects conducted between 1990and 1999.In related work, Horowitz (1990) provides several

different authors’ perspectives on organizational deci-sion making from a decision analysis point of view.Bordley (2001) compares and contrasts conventionaldecision analysis as commonly applied in industrialpractice with soft OR techniques and finds much incommon between decision analysis and classical inter-active planning. Kasanen et al. (2000) examine char-acteristics of six major real-world decision processesrelative to common assumptions or myths in the mul-ticriteria decision making and multiattribute utilityfields, and suggest changes in assumptions and prac-tices to make models from these fields more widelyuseful.Decision analysis frameworks for R&D organiza-

tional planning processes received particular atten-tion. Bordley (1998) considers organizational issuesrelated to using decision analysis for R&D projectselection and emphasizes the benefits of stimulatingresearchers to develop better projects by improvingcommunications. Matheson et al. (1994), Mathesonand Menke (1994), and Menke (1994) review decisionanalysis approaches to R&D planning. Matheson andMatheson (1998) present a framework for applyingdecision analysis methods to R&D strategy.As a result of the work cited above, consid-

erable additional guidance is now available con-cerning processes for successfully conducting andimplementing a major decision analysis projectwithin an organization.



Real Options. During the period covered by thissurvey, the importance of modeling sequential deci-sions in conjunction with the resolution of uncer-tainties over time became more widely recognized.Downstream decision alternatives provide real optionsthat can increase flexibility in managing real-worldprojects with evolving risks, and these real optionsare analogous to financial options. In R&D projects,for instance, the cost of conducting research can beviewed as the price of a call option to develop or com-mercialize a new product subsequently, and exercis-ing this option incurs an additional cost (the exerciseprice). Relevant methodologies for addressing deci-sions with real options include larger decision trees,dynamic programming, and financial options meth-ods (Amram and Kulatilaka 1999, Dixit and Pindyck1994, and Trigeorgis 1996). Smith (1999) provides aconcise nontechnical overview of the similarities anddifferences between conventional decision analysisand the real-options approach.Several authors have pointed out the benefits of

options thinking in providing more realistic evaluationsthan traditional analyses. For instance, Morris et al.(1991) use an options framework in conjunction witha simple decision tree to demonstrate that the riskierof two R&D projects having the same expected valueis typically the better choice when sequencing is prop-erly incorporated. Faulkner (1996) provides a brief,nontechnical introduction to real options, and basedon several years of experience at Eastman Kodak,describes the advantages of evaluating R&D projectsas sequential adaptive strategies. Pauwels et al. (2000)illustrate the advantages of applying options think-ing along with conventional decision analysis toolsin emergency response situations by analyzing a sim-ple two-period nuclear incident evacuation model.Benaroch (2001) presents methods for structuringand evaluating multiple, possibly interacting, operat-ing options in technology investments and includesan illustrative example involving establishment of aweb-based sales channel. Howard (1996) provides anoverview of various types of options in decision prob-lems and emphasizes the importance of recognizingand creating, as well as modeling, these options, andhe illustrates several evaluation methods.

Perdue et al. (1999) discuss a method using bothoptions-pricing techniques and decision analysis toolsin R&D planning. Smith and McCardle (1999) providea tutorial introduction to options-pricing methods andtheir integration with decision analysis methods, witha focus on evaluating oil and gas investments. Realoptions have also received attention outside the tra-ditional journal literature, such as in various confer-ences of the Society of Petroleum Engineers (SPE).(Abstracts, as well as ordering instructions, for SPEconference papers are on the web at www.spe.org.)For instance, Claeys and Walkup (1999) discuss fram-ing techniques used to ensure that all key options anduncertainties are included and illustrate use of theseon examples from several actual petroleum valuationefforts. Gallant et al. (1999) discuss learning modelsfor analyzing dynamic complexity, that is, changesin information over time, and illustrate their use inexploration and production examples based on actualprojects. Faiz (2000) discusses how real-options val-uation, a combination of options-pricing theory anddecision analysis, relates to other popular manage-ment tools, such as portfolio optimization, and pro-vides illustrative real-world case studies.As suggested by Smith (1999), both decision analy-

sis and finance professionals could benefit from learn-ing more about each other’s tools and incorporatingthem into models appropriately. Although a num-ber of applications in our survey utilized real-optionsthinking in generating downstream options, we foundonly two articles that provided in-depth descrip-tions of applications where real-options methods werecombined with conventional decision analysis meth-ods (Perdue et al. 1999 and Smith and McCardle1999). Hence, the potential synergism has not yet hada major impact on published applications.

Practice Competitions and Professional SocietiesIn this section, we highlight decision analysis applica-tions that achieved significant recognition in practicecompetitions held by professional societies during thesurvey period. In addition, we discuss the foundingand progress of a new professional group for decisionanalysis practitioners.



Edelman Award Competition. During the periodcovered by this survey, three applications of decisionanalysis became finalists in the Franz Edelman Awardcompetition held annually by INFORMS to recognizeand reward outstanding examples of managementscience and operations research practice worldwide.Burnett et al. (1993) describe the long-term use of aproject appraisal methodology (PAM) within the GasResearch Institute’s annual five-year R&D planningprocess. PAM calculates benefit-to-cost ratios for R&Dprojects at multiple levels of funding to aid in allocat-ing the R&D budget, obtaining expected benefits ateach level by using a multiattribute scoring functionand judgmental probabilities for technical and com-mercial successes. The authors estimate that benefitsfrom using PAM have been in the tens of billions ofdollars.Paté-Cornell and Fischbeck (1994) perform a prob-

abilistic risk analysis of failure of the exterior sur-face tiles on the United States’ space shuttle orbiter.Expert opinion and the experience of the first 30 shut-tle flights were used to build a decomposed modelof risk for various zones on the shuttle’s tile-bearingsurface. The analysis showed that roughly 15% of thetiles contribute 85% of the risk of failure. The studyfurther highlighted organizational factors that con-tribute to potential tile failure risks and led to variouspolicy changes in the management and maintenanceof the tiles.Von Winterfeldt and Schweitzer (1998) describe

an analysis to help the U.S. Department of Energychoose which tritium-supply alternatives to pursue toreplenish tritium for the U.S. nuclear weapons stock-pile. Ten alternatives were evaluated with respectto production assurance, cost, and environmentalimpacts based in part on multiple-expert probabilityassessments and results from a dynamic production-simulation model. The analysis was influential inshaping the final choice by the U.S. Secretary ofEnergy, and its defensibility helped avoid lawsuitsfrom vendors whose alternatives were not chosen.As a result of his experience with this entry in the

Edelman Award competition, von Winterfeldt workedto establish the Practice Award (see below) of theDecision Analysis Society of INFORMS (DAS), whileserving as the DAS Chair. We anticipate that the DAS

Practice Award competition will stimulate additionalentries from decision analysis in the Edelman Awardcompetition in the future.

Decision Analysis Society Practice Award. In1999, DAS inaugurated an annual Practice Award torecognize, promote, and publicize good decision anal-ysis practice. The competition for the award beginswith submission of a brief written summary of arecent decision analysis application and culminateswith presentations by the finalists at a DAS-sponsoredsession at the annual INFORMS meeting. The winnersof the first Practice Award in 1999 were Mazen A.Skaf of Navigant Consulting, Inc., and Donald W.Spillman of Shell (Oil) Offshore, Inc., for “A PortfolioManagement Process and System for an Upstream Oiland Gas Organization.” Described in Skaf (1999), thisproject produced a portfolio management process andsystem that was used to help manage a large port-folio of upstream oil and gas assets in the Gulf ofMexico through exploration, development, and pro-duction, and to provide analytical support for a vari-ety of portfolio, lease-bidding, drilling, development,and resource-requirements decisions. The system uti-lizes a variety of decision analysis tools and concepts,and its use has significantly impacted the client orga-nization, including value-added in the hundreds ofmillions of dollars.The winner in 2000 was David A. Mauney of Struc-

tural Integrity Associates, Inc., for “Best Practicesin the Application of Decision/Financial Analysis toRepair/Replacement Decisions of Plant Components.”This work developed a process that uses optimizationtechniques in conjunction with a risk-based decisionmodel to aid in planning the timing of major mainte-nance investments for fossil-fuel power plants. Wherenecessary, this process includes interviews tailored toobtain judgmental probability data from plant person-nel most familiar with the components. Its applicationin the power industry has resulted in savings in thetens of millions of dollars.The 2001 winner was Eric Johnson of Pharsight Cor-

poration for “Life Cycle Strategy Analysis,” which isdescribed in Johnson and Petty (2003). In this study,consultants from Pharsight helped a client firm reachconsensus on a development strategy for a cancerdrug. They used a variety of decision analysis meth-



ods to construct and evaluate a manageable num-ber of candidate strategies, and they subsequentlysucceeded in constructing a hybrid strategy with anexpected NPV $50 million greater than that of anyof the original candidate strategies and $100 milliongreater than that of the status quo strategy.

Decision Analysis Affinity Group. No discussionof decision analysis practice over the 1990–2001period would be complete without mentioning theDecision Analysis Affinity Group (DAAG), which wasfounded in 1995 to promote the use of decision anal-ysis in industry and to further the development andcareers of industrial practitioners. Since 1995, DAAGhas held annual conferences focusing on the use andimplementation of decision analysis within major cor-porations. This has successfully filled an importantniche for decision analysis practitioners, and recentconferences have typically attracted 75 to 100 partic-ipants. Historically, to create an open and collegialatmosphere among peers, attendance by consultantsand academics was discouraged and was limited tothose explicitly invited for some purpose such as apresentation. Beginning with the 2002 conference, thispolicy was liberalized (Spradlin and Skinner 2001).While some DAAG members are also members of

DAS and/or INFORMS, most are not and have nodesire to become members. Furthermore, most DAAGmembers have little or no interest in, or motiva-tion for, publishing. Despite the differences in focusbetween DAS and DAAG, we hope the change inattendance policy for DAAG conferences will lead tomore interaction and collaboration between these twogroups. Additional information about DAAG and itsconferences, including abstracts and presentations, isavailable at the DAAG website (www.daag.net).

5. Needs and ConcernsIn §§3 and 4, we identify a number of trends anddevelopments in decision analysis applications andpractice based primarily on our survey of the OR lit-erature. These are predominately positive, and thus,encouraging for the future of decision analysis. In thissection, we describe some needs and potential pitfallsfor the continued advancement of decision analysispractice. These reflect our personal perspectives, and

we hope what follows will stimulate discussion anddebate, as well as additional research and action.

Status in Companies and UniversitiesIt is well documented that the number of internal ORgroups within corporations has decreased in recentyears (for example, see Fildes and Ranyard 2000), andthere is evidence that decision analysis groups aresubject to the same factors that affect internal cor-porate OR groups. Recent examples where decisionanalysis groups have in effect been disbanded includeEastman Kodak (Clemen and Kwit 2001) and GeneralMotors (Lieberman 2002). Moreover, decision analy-sis itself, not just the corresponding internal group,has fallen in and out of favor in a number of corpo-rations over the years. Spradlin (2001) suggests thatmany internal decision analysis consulting groups arelikely to disappear unless they redefine themselvesto look like something else and that decision anal-yses, where done at all, will be done either by thebusiness units themselves or by external consultants.Additional research into factors that influence the riseand fall of internal decision analysis groups and ofdecision analysis itself within corporations could helpstrengthen the position of decision analysis withincorporations.Even among INFORMS members, “decision anal-

ysis” does not convey the unambiguous meaningthat “linear programming” or “queueing” does. Forinstance, Interfaces classifies many articles under deci-sion analysis, presumably because they involve somesort of analysis of decisions, that we would not rec-ognize as decision analysis. (In fact, we found dur-ing our survey of applications that key word searchesin indices and even article abstracts were of limitedvalue because of the differing definitions that vari-ous authors use for the term “decision analysis.”) Thecoverage of decision analysis in many OR/MS text-books largely focuses on mechanics and mathematicsand omits such important topics as problem structur-ing, probability assessment, cognitive biases, and dis-cretization of continuous distributions. And computerscientists often do not mean what we mean when theyuse the term “decision trees.” Hopefully, the contentsof the new journal Decision Analysis will help to delin-eate the field more clearly.



Since the founding of decision analysis, a smallgroup of institutions have been the primary sourcesof ideas, methodological tools, and well-trained grad-uate students. (See Raiffa 2002 for an interesting per-sonal account of the origin and evolution of the field,including the origin of the “decision analysis” name.)Our concern is that education for decision analysispractitioners and faculty appears to depend on spe-cific individuals; as retirements continue in academiaand as acquisitions, reorientations, and retirementscontinue at decision analysis consulting firms, well-trained decision analysts may become harder tofind—particularly those having both strong academicbackgrounds and practical experience with applica-tions. Development of tools, software, and innova-tions in organizational implementation, which haveoften come from the consulting firms, could also suf-fer. As an old country-music song asks, “Who’s gonnafill those shoes?” It is worth noting that the DecisionEducation Foundation (www.decisioneducation.org)is focusing on educating high school students, aswell as their teachers and parents, in better decision-making approaches. This is certainly laudable, but itis not a replacement for strong decision analysis pro-grams at the university level.

Better ToolsHere, we briefly highlight two additional areas wherewe think further developments could significantlyenhance the applicability of decision analysis in prac-tice. (Section 4 includes discussion of several notewor-thy methodological developments that had an impacton decision analysis applications during the 1990–2001 period. Of course, further progress in those areaswould also be welcome.)First, better methods are needed for modeling and

assessing probabilistic dependence among randomvariables. Whether the variables are discrete, contin-uous, or discrete representations of continuous vari-ables, the size and complexity of the assessment taskgrows rapidly with the number of dependent randomvariables unless simplifying assumptions are made.In practice, independence is often assumed, and thistypically is adequate if expected value is the sole cri-terion of choice and the output variable (for exam-ple, NPV) is a linear, or nearly linear, function of

the random variables. However, assuming indepen-dence can introduce significant errors if substantialnonlinearities are present—which can affect the accu-racy of the expected value—or if the entire distribu-tion of the output variable is of interest, for example,due to risk aversion. Keefer (1991) shows that con-ditional independence, perfect positive dependence,and combinations thereof can be useful in modelingdependence in the context of bidding for oil and gasleases. In the context of business portfolios, Poland(1999) addresses interbusiness dependencies by con-ditioning evaluations for individual businesses on“global” outcome scenarios for variables that impactmultiple businesses, while treating business-specificrandom variables as independent across businesses.Recently, constructing approximate joint probabilitydistributions via multivariate functions called copu-las, which utilize dependence information such ascorrelations to combine univariate marginal distri-butions, has received considerable attention (Clemenand Reilly 1999, Clemen et al. 2000, Reilly 2000, Yi andBier 1998). This approach appears promising, espe-cially for continuous variables, but we have not yetseen a decision analysis application using copulas inthe OR literature. Thus, despite some progress dur-ing the survey period in handling dependence, morework is needed in this area—especially work gearedtoward practitioners.Second, we continue to need more realistic meth-

ods for dealing with the time dynamics of many deci-sion problems. Sequential decision models have beenpart of the conceptual toolkit of decision analysissince its early days, and it is interesting to see thefinance profession advancing real-options methods asan alternative to decision analysis for these types ofdecisions. The difficulty with using decision analysismethods for such decisions seems to be the complex-ity and size of the models that are needed when con-ventional decision analysis methods are used. In §4,we cited the use of real-options methods, dynamicprogramming, and stochastic trees to address vari-ous types of time dynamics. Use of system dynam-ics methods has also been proposed (Howard et al.1998). Additional work on these approaches, com-bined with the increased capabilities that moderndecision analysis software provides for constructing



and analyzing large-scale decision analysis models,will hopefully facilitate better analyses of decisionswith time dynamics.

6. Concluding RemarksAlthough there are some institutional, educational,and methodological needs that merit further atten-tion, our analysis of the applications articles wesurveyed and of other sources, as presented above,shows that the state of decision analysis applicationsis healthy. Both the number of published applicationsand the rate of publication have increased. Further-more, these applications cover a broad range of deci-sions in both the public and private sectors, and theydemonstrate that decision analysis is used increas-ingly for a wide variety of strategic and tactical deci-sions. Readers interested in brief summaries for eachof the surveyed applications articles are referred tothe companion technical report by Keefer et al. (2002).

AcknowledgmentsThe authors thank Brian Bajuk, Daniel G. Brooks, Richard F.Deckro, Emilia Iovtcheva, Jeffrey M. Keisler, Jack M. Kloeber, Jr.,Gregory S. Parnell, Robert K. Perdue, and James E. Smith for theirassistance in providing material for this article. They thank the edi-tors and referees for their helpful comments and suggestions. Theyalso thank the Departments of Management and Supply ChainManagement at Arizona State University for their assistance.

ReferencesAmram, M., N. Kulatilaka. 1999. Real Options: Managing Strate-

gic Investment in an Uncertain World. Harvard Business SchoolPress, Boston, MA.

Baker, S. F., S. G. Green, J. K. Lowe, V. E. Francis. 2000. A value-focused approach for laboratory equipment purchases.MilitaryOper. Res. 5(4) 43–56.

Balson, W. E., J. L. Welsh, D. S. Wilson. 1992. Using decision anal-ysis and risk analysis to manage utility environmental risk.Interfaces 22(6) 126–139.

Bana e Costa, C. A. 2001. The uses of multi-criteria decision analysisto support the search for less conflicting policy options in amulti-actor context: Case study. J. Multi-Criteria Decision Anal.10 111–125.

Beccue, P. 2001. Choosing a development strategy for a new prod-uct at Amgen. D. L. Keefer, ed. Practice abstracts. Interfaces31(5) 62–64.

Bell, D. E., A. Schleifer, Jr. 1995. Decision Making Under Uncertainty.Course Technology, Inc., Cambridge, MA.

Benaroch, M. 2001. Option-based management of technologyinvestment risk. IEEE Trans. Engrg. Management 48(4) 428–444.

Bodily, S. E., M. S. Allen. 1999. A dialogue process for choosingvalue-creating strategies. Interfaces 29(6) 16–28.

Borcherding, K., T. Eppel, D. von Winterfeldt. 1991. Comparisonof weighting judgments in multiattribute utility measurement.Management Sci. 37 1603–1619.

Bordley, R. F. 1998. R&D project selection versus R&D project gen-eration. IEEE Trans. Engrg. Management 45 407–413.

Bordley, R. F. 2001. Relating value-focused thinking and interactiveplanning. J. Oper. Res. Soc. 52 1315–1326.

Borison, A. 1995. Oglethorpe Power Corporation decides aboutinvesting in a major transmission system. Interfaces 25(2) 25–36.

Bresnick, T. A., D. M. Buede, A. A. Pisani, L. L. Smith, B. B. Wood.1997. Airborne and space-borne reconnaissance force mixes: Adecision analysis approach. Military Oper. Res. 3(4) 65–78.

Brown, G. M. 1997. Evaluation of vision correction alternatives formyopic adults. Interfaces 27(2) 66–84.

Brown, R. V. 1992. The state of the art of decision analysis: A per-sonal perspective. Interfaces 22(6) 5–14.

Bruggink, P. R. 1997. The contribution of project analysis to an R&Dproject at an industrial R&D center. D. L. Keefer, ed. Practiceabstracts. Interfaces 27(2) 107–111.

Buede, D. M., T. A. Bresnick. 1992. Applications of decision analy-sis to the military systems acquisition process. Interfaces 22(6)110–125.

Burk, R. C., G. Parnell. 1997. Evaluating future military space tech-nologies. Interfaces 27(3) 60–73.

Burnett, W. M., D. J. Monetta, B. G. Silverman. 1993. How the GasResearch Institute (GRI) helped transform the US natural gasindustry. Interfaces 23(1) 44–58.

Call, H. J., W. A. Miller. 1990. A comparison of approaches andimplementations for automating decision analysis. ReliabilityEngrg. System Safety 30 115–162.

Chien, C.-F., F. Sainfort. 1998. Evaluating the desirability of meals:An illustrative multiattribute decision analysis procedure toassess portfolios with interdependent items. J. Multi-CriteriaDecision Anal. 7 230–238.

Claeys, J., G. Walkup, Jr. 1999. Discovering real options in oilfieldexploration and development. Paper 52956, Soc. Petroleum Engi-neers Hydrocarbon Econom. Evaluation Sympos., Dallas, TX.

Clemen, R. T. 1996. Making Hard Decisions: An Introduction to Deci-sion Analysis, 2nd ed. Duxbury Press, Belmont, CA.

Clemen, R. T., R. C. Kwit. 2001. The value of decision analysis atEastman Kodak Company, 1990–1999. Interfaces 31(5) 74–92.

Clemen, R. T., T. Reilly. 1999. Correlations and copulas for decisionand risk analysis. Management Sci. 45(2) 208–224.

Clemen, R. T., G. W. Fischer, R. L. Winkler. 2000. Assessing depen-dence: Some experimental results. Management Sci. 46(8) 1100–1115.

Corner, J. L., P. D. Corner. 1995. Characteristics of decisions in deci-sion analysis practice. J. Oper. Res. Soc. 46 304–314.

Corner, J. L., C. W. Kirkwood. 1991. Decision analysis applicationsin the operations research literature, 1970–1989. Oper. Res. 39206–219.



Corner, J. L., J. Buchanan, M. Henig. 2001. Dynamic decision prob-lem structuring. J. Multi-Criteria Decision Anal. 10 129–141.

Davis, C. C., R. F. Deckro, J. A. Jackson. 1999. A methodology forevaluating and enhancing C4 networks. Military Oper. Res. 4(2)45–60.

Davis, C. C., R. F. Deckro, J. A. Jackson. 2000. A value focusedmodel for a C4 network. J. Multi-Criteria Decision Anal. 9 138–162.

Dillon, R., Y. Y. Haimes. 1996. Risk of extreme events via multiob-jective decision trees: Application to telecommunications. IEEETrans. Systems, Man, Cybernetics—Part A: Systems Humans 26262–271.

Dixit, A. K., R. S. Pindyck. 1994. Investment Under Uncertainty.Princeton University Press, Princeton, NJ.

Doyle, M. P., R. F. Deckro, J. M. Kloeber, J. A. Jackson. 2000. Mea-sures of merit for offensive information operations courses ofaction. Military Oper. Res. 5(2) 5–18.

Dunning, D. J., S. Lockfort, Q. E. Ross, P. C. Beccue, J. S.Stonebraker. 2001. New York Power Authority uses decisionanalysis to schedule refueling of its Indian Point 3 nuclearpower plant. Interfaces 31(5) 121–135.

Dyer, J. S., T. Edmunds, J. C. Butler, J. Jia. 1998. A multiattributeutility analysis of alternatives for the disposition of surplusweapons-grade plutonium. Oper. Res. 46 749–762.

Dyer, J. S., R. N. Lund, J. B. Larsen, V. Kumar, R. P. Leone. 1990. Adecision support system for prioritizing oil and gas explorationactivities. Oper. Res. 38 386–396.

Edwards, W., ed. 1992. Utility Theories: Measurements and Applica-tions. Kluwer, Boston, MA.

Engemann, K. J., H. E. Miller. 1992. Operations risk management ata major bank. Interfaces 22(6) 140–149.

Faiz, S. 2000. Real options application: From successes in asset val-uation to challenges for an enterprise-wide approach. Paper62964, Soc. Petroleum Engineers Annual Tech. Conf. Exhibition,Dallas, TX.

Faulkner, T. W. 1996. Applying “options thinking” to R&D valua-tion. Res. Tech. Management 39(3) 50–56.

Feinstein, C. D. 1990. Deciding whether to test student athletes fordrug use. Interfaces 20(3) 80–87.

Fildes, R., J. Ranyard. 2000. Internal OR consulting: Effective prac-tice in a changing environment. Interfaces 30(5) 34–50.

French, S. 1996. Multi-attribute decision support in the event of anuclear accident. J. Multi-Criteria Decision Anal. 5 39–57.

Gallant L., H. Kieffel, R. Chatwin. 1999. Using learning models tocapture dynamic complexity in petroleum exploration. Paper52954, Soc. Petroleum Engineers Hydrocarbon Econom. EvaluationSympos., Dallas, TX.

Golub, A. L. 1997. Decision Analysis: An Integrated Approach. Wiley,New York.

Goodwin, P., G. Wright. 1998. Decision Analysis for Management Judg-ment, 2nd ed. Wiley, Chichester, England.

Griggs, B. J., G. S. Parnell, L. J. Lehmkuhl. 1997. An air missionplanning algorithm using decision analysis and mixed integerprogramming. Oper. Res. 45 662–676.

Hall, N. G., J. C. Hershey, L. G. Kessler, R. C. Stotts. 1992. A modelfor making project funding decisions at the National CancerInstitute. Oper. Res. 40 1040–1052.

Hämäläinen, R. P., M. R. K. Lindstedt, K. Sinkko. 2000. Multiat-tribute risk analysis in nuclear emergency management. RiskAnal. 20(4) 455–467.

Hammond, J. S., R. L. Keeney, H. Raiffa. 1999. Smart Choices: A Prac-tical Guide to Making Better Decisions. Harvard Business SchoolPress, Boston, MA.

Hazen, G. 2000. Preference factoring for stochastic trees. Manage-ment Sci. 46(3) 389–403.

Hazen, G., J. M. Pellissier, J. Sounderpandian. 1998. Stochastic-treemodels in medical decision making. Interfaces 28(4) 64–80.

Hedbert, S. R. 1998. Executive insight: Is AI going mainstream atlast? A look inside Microsoft Research. IEEE Intelligent Systems13(2) 21–25.

Heger, A. S., J. E. White. 1997. Using influence diagrams for dataworth analysis. Reliability Engrg. System Safety 55 195–202.

Henrion, M., J. S. Breese, E. J. Horvitz. 1991. Decision analysis andexpert systems. AI Magazine 12(4) 64–91.

Hess, S. W. 1993. Swinging on the branch of a tree: Project selectionapplications. Interfaces 23(6) 5–12.

Horowitz, I., ed. 1990. Organization and Decision Theory. KluwerAcademic Publishers, Boston, MA.

Howard, R. A. 1992. Heathens, heretics, and cults: The religiousspectrum of decision aiding. Interfaces 22(6) 15–27.

Howard, R. A. 1996. Options. In Zeckhauser et al. (1996) 81–101.Howard, R. A., M. B. Conn, M. Paich, J. E. Smith, J. M. Smith. 1998.

Panel discussion: Downstream decisions (options) & dynamicmodeling. INFORMS Annual Meeting, Seattle, WA. (Completetranscript is available at www.informs.org/Society/DA.)

Hurley, W. J. 1998. Optimal sequential decisions and the content ofthe fourth-and-goal conference. Interfaces 28(6) 19–22.

Islei, G., G. Lockett, B. Cox, S. Gisbourne, M. Stratford. 1991. Model-ing strategic decision making and performance measurementsat ICI Pharmaceuticals. Interfaces 21(6) 4–22.

Jackson, J. A., J. M. Kloeber, Jr., B. E. Ralston, R. F. Deckro. 1999.Selecting a portfolio of technologies: An application of decisionanalysis. Decision Sci. 30 217–238.

Jackson, J. A., G. S. Parnell, B. L. Jones, L. J. Lehmkuhl, H. W.Conley, J. M. Andrew. 1997. Air Force 2025 operational analysis.Military Oper. Res. 3(4) 5–21.

Johnson, E., N. Petty. 2003. Apimoxin development strategy analy-sis. D. L. Keefer, ed. Practice abstracts. Interfaces. 33(3) 57–59.

Jones, M., C. Hope, R. Hughes. 1990. A multi-attribute value modelfor the study of UK energy policy. J. Oper. Res. Soc. 41 919–929.

Kasanen, E., H. Wallenius, J. Wallenius, S. Zionts. 2000. A study ofhigh-level managerial decision processes, with implications forMCDM research. Eur. J. Oper. Res. 120 496–510.

Keefer, D. L. 1991. Resource allocation models with risk aversionand probabilistic dependence: Offshore oil and gas bidding.Management Sci. 37 377–395.

Keefer, D. L. 1994. Certainty equivalents for three-point discrete-distribution approximations. Management Sci. 40 760–773.



Keefer, D. L. 1995. Facilities evaluation under uncertainty: Pricinga refinery. Interfaces 25(6) 57–66.

Keefer, D. L. 1997. Practice abstracts. Interfaces 27(2) 107–111.Keefer, D. L. 1999. Practice abstracts. Interfaces 29(4) 96–98.Keefer, D. L. 2000. Practice abstracts. Interfaces 30(5) 31–33.Keefer, D. L. 2001a. Practice abstracts. Interfaces 31(4) 109–111.Keefer, D. L. 2001b. Practice abstracts. Interfaces 31(5) 62–64.Keefer, D. L. 2003. Practice abstracts. Interfaces. 33(3) 57–59.Keefer, D. L., C. W. Kirkwood, J. L. Corner. 2002. Decision analysis

applications in the operations research literature, 1990–2001.Technical report, Department of Supply Chain Management,Arizona State University, Tempe, AZ.

Keefer, D. L., F. B. Smith Jr., H. B. Back. 1991. Development and useof a modeling system to aid a major oil company in allocatingbidding capital. Oper. Res. 39 28–41.

Keeney, R. L. 1992. Value-Focused Thinking: A Path to Creative Deci-sionmaking. Harvard University Press, Cambridge, MA.

Keeney, R. L. 1994. Using values in operations research. Oper. Res.42 793–813.

Keeney, R. L. 1997. Evaluating electromagnetic field implicationsfor a transmission-line moratorium. IEEE Trans. Engrg. Man-agement 44 268–275.

Keeney, R. L. 1999a. The value of Internet commerce to the cus-tomer. Management Sci. 45 533–542.

Keeney, R. L. 1999b. Developing a foundation for strategy at Sea-gate Software. Interfaces 29(6) 4–15.

Keeney, R. L. 2000. Evaluating customer acquisition at AmericanExpress using multiple objectives. D. L. Keefer, ed. Practiceabstracts. Interfaces 30(5) 31–33.

Keeney, R. L. 2001. Modeling values for telecommunications man-agement. IEEE Trans. Engrg. Management 48(3) 370–379.

Keeney, R. L., T. L. McDaniels. 1992. Value-focused thinking aboutstrategic decisions at BC Hydro. Interfaces 22(6) 94–109.

Keeney, R. L., T. L. McDaniels. 1999. Identifying and structuringvalues to guide integrated resource planning at BC Gas. Oper.Res. 47 651–662.

Keeney, R. L., T. L. McDaniels. 2001. A framework to guide thinkingand analysis regarding climate change policies. Risk Anal. 21(6)989–1000.

Keeney, R. L., D. von Winterfeldt. 1991. Eliciting probabilities fromexperts in complex technical problems. IEEE Trans. Engrg. Man-agement 38 191–201.

Keeney, R. L., D. von Winterfeldt. 1994. Managing nuclear wastefrom power plants. Risk Anal. 14 107–130.

Keeney, R. L., T. L. McDaniels, C. Swoveland. 1995. Evaluatingimprovements in electric utility reliability at British ColumbiaHydro. Oper. Res. 43 933–947.

Keeney, R. L., D. von Winterfeldt, T. Eppel. 1990. Eliciting publicvalues for complex policy decisions. Management Sci. 36 1011–1030.

Keller, L. R., C. W. Kirkwood. 1999. The founding of INFORMS:A decision analysis perspective. Oper. Res. 47 16–28.

Kerchner, P. M., R. F. Deckro, J. M. Kloeber. 2001. Valuing psycho-logical operations. Military Oper. Res. 6(2) 45–65.

Kidd, J. B., S. P. Prabhu. 1990. A practical example of a multi-attribute decision aiding technique. Omega, Internat. J. Manage-ment Sci. 18 139–149.

Kirkwood, C. W. 1992. An overview of methods for applied deci-sion analysis. Interfaces 22(6) 28–39.

Kirkwood, C. W. 1997. Strategic Decision Making: Multiobjective Deci-sion Analysis with Spreadsheets. Duxbury Press, Belmont, CA.

Kirkwood, C. W. 1999. Chapter 28: Decision analysis. In Sage andRouse (1999) 1119–1145.

Krumm, F. V., C. F. Rolle. 1992. Management and application ofdecision and risk analysis in Du Pont. Interfaces 22(6) 84–93.

Kusnic, M. W., D. Owen. 1992. The unifying vision process: Valuebeyond traditional decision analysis in multiple-decision-maker-environments. Interfaces 22(6) 150–166.

Lai, S.-K. 2001. An empirical study of equivalence judgments vs.ratio judgments in decision analysis. Decision Sci. 32(2) 277–302.

Lehmkuhl, L., D. Lucia, J. Feldman. 2001. Signals from space: Thenext-generation global positioning system. Military Oper. Res.6(4) 5–18.

Lieberman, J. 2002. Marketing of DA at GM: Rise and fall. Pre-sented at Decision Analysis Affinity Group Conference, LasVegas, NV.

Magat, W. A., W. K. Viscusi, J. Huber. 1996. A reference lotterymetric for valuing health. Management Sci. 42 1118–1130.

Marshall, K. T., R. M. Oliver. 1995. Decision Making and Forecasting,with Emphasis on Model Building and Policy Analysis. McGraw-Hill, New York.

Matheson, D., J. Matheson. 1998. The Smart Organization: CreatingValue Through Strategic R&D. Harvard Business School Press,Boston, MA.

Matheson, D., J. Matheson. 1999. Outside-in strategic modeling.Interfaces 29(6) 29–41.

Matheson, D., J. Matheson, M. M. Menke. 1994. Making excellentR&D decisions. Res. Tech. Management 37(6) 21–24.

Matheson, J. E., M. M. Menke. 1994. Using decision quality prin-ciples to balance your R&D portfolio. Res. Tech. Management37(3) 38–43.

Matzkevich, I., B. Abramson. 1995. Decision analytic networks inartificial intelligence. Management Sci. 41 1–22.

Maxwell, D. T. 2002. Decision analysis: Aiding insight VI—It’s notyour grandfather’s decision analysis software. OR/MS Today29(3) 44–51.

McDaniels, T. L. 1995. Using judgment in resource management:A multiple objective analysis of a fisheries management deci-sion. Oper. Res. 43 415–426.

McNamee, P., J. Celona. 1990. Decision Analysis with Supertree, 2nded. Scientific Press, South San Francisco, CA.

McNamee, P., J. Celona. 2001. Decision Analysis for the Professional,3rd ed. SmartOrg Inc., Menlo Park, CA.

Menke, M. M. 1994. Improving R&D decisions and execution. Res.Tech. Management 37(5) 25–32.

Millet, I. 1994. A novena to Saint Anthony, or how to find inventoryby not looking. Interfaces 24(2) 69–75.



Mollaghasemi, M., J. Pet-Edwards. 1997. Technical Briefing: MakingMultiple-Objective Decisions. IEEE Computer Society Press, LosAlamitos, CA.

Morgan, M. G., M. Henrion. 1990. Uncertainty: A Guide to Dealingwith Uncertainty in Quantitative Risk and Policy Analysis. Cam-bridge University Press, Cambridge, England.

Morris, P. A., E. O. Teisberg, A. L. Kolbe. 1991. When choosing R&Dprojects, go with long shots. Res. Tech. Management 34(1) 35–40.

Mulvey, J. M. 1994. An asset-liability investment system. Interfaces24(3) 22–33.

Noonan, F., C. A. Vidich. 1992. Decision analysis for utilizing haz-ardous waste site assessments in real estate acquisition. RiskAnal. 12 245–251.

Oliver, R. M., J. Q. Smith, eds. 1990. Influence Diagrams, Belief Nets,and Decision Analysis. Wiley, Chichester, England.

Parnell, G. S. 2001. Work-package-ranking system for the Depart-ment of Energy’s Office of Science and Technology. D. L.Keefer, ed. Practice abstracts. Interfaces 31(4) 109–111.

Parnell, G. S., H. W. Conley, J. A. Jackson, L. J. Lehmkuhl, J. M.Andrew. 1998. Foundations 2025: A value model for evaluatingfuture air and space forces. Management Sci. 44 1336–1350.

Parnell, G. S., B. I. Gimeno, D. Westphal, J. A. Engelbrecht, R.Szafranski. 2001. Multiple perspective R&D portfolio analysisfor the National Reconnaissance Office’s technology enterprise.Military Oper. Res. 6(3) 19–34.

Paté-Cornell, M.-E., P. S. Fischbeck. 1994. Risk management for thetiles of the space shuttle. Interfaces 24(1) 64–86.

Pauwels, N., B. Van de Walle, F. Hardeman, K. Soudan. 2000. Theimplications of irreversibility in emergency response decisions.Theory Decision 49 25–51.

Pellissier, J. M., G. B. Hazen, R. W. Chang. 1996. A continuous-risk decision analysis of total hip replacement. J. Oper. Res. Soc.47(6) 776–793.

Perdue, R. K., S. Kumar. 1999. Decision analysis of high-levelradioactive waste cleanup end points at the West ValleyDemonstration Project waste tank farm. D. L. Keefer, ed. Prac-tice abstracts. Interfaces 29(4) 96–98.

Perdue, R. K., W. J. McAllister, P. V. King, B. G. Berkey. 1999. Val-uation of R and D projects using options pricing and decisionanalysis models. Interfaces 29(6) 57–74.

Poland, W. B. 1999. Simple probabilistic evaluation of portfoliostrategies. Interfaces 29(6) 75–83.

Procaccia, H., R. Cordier, S. Muller. 1997. Application of Bayesianstatistical decision theory for a maintenance optimization prob-lem. Reliability Engrg. System Safety 55 143–149.

Quaddus, M. A., D. J. Atkinson, M. Levy. 1992. An applicationof decision conferencing to strategic planning for a voluntaryorganization. Interfaces 22(6) 61–71.

Raiffa, H. 2002. Decision analysis: A personal account of how it gotstarted and evolved. Oper. Res. 50 179–185.

Rayno, B., G. S. Parnell, R. C. Burk, B. W. Woodruff. 1997.A methodology to assess the utility of future space systems.J. Multi-Criteria Decision Anal. 6 344–354.

Reagan-Cirincione, P., S. Schuman, G. P. Richardson, S. A. Dorf.1991. Decision modeling: Tools for strategic thinking. Interfaces21(6) 52–65.

Reilly, T. 2000. Sensitivity analysis for dependent variables. DecisionSci. 31(3) 551–572.

Rios Insua, D., K. A. Salewicz. 1995. The operation of Lake Kariba:A multiobjective decision analysis. J. Multi-Criteria DecisionAnal. 4 203–222.

Rzasa, P. V., T. W. Faulkner, N. L. Sousa. 1990. Analyzing R&Dportfolios at Eastman Kodak. Res. Tech. Management 33(1)27–32.

Sage, A. P., W. B. Rouse, eds. 1999. Handbook of Systems Engineeringand Management. Wiley, New York.

Shephard, G. G., C. W. Kirkwood. 1994. Managing the judg-mental probability elicitation process: A case study of ana-lyst/manager interaction. IEEE Trans. Engrg. Management 41414–425.

Silverman, B. G. 1994. Unifying expert systems and the decisionsciences. Oper. Res. 42 393–413.

Skaf, M. A. 1999. Portfolio management in an upstream oil and gasorganization. Interfaces 29(6) 84–104.

Skinner, D. C. 1999. Introduction to Decision Analysis, 2nd ed. Prob-abilistic Publishing, Gainesville, FL.

Smith, J. E. 1993. Moment methods for decision analysis. Manage-ment Sci. 39 340–358.

Smith, J. E. 1999. Much ado about options? Decision Anal. Newsletter18(2) 4–5, 8. Newsletter Archive, www.informs.org/Society/DA.

Smith, J. E., K. F. McCardle. 1999. Options in the real world: Lessonslearned in evaluating oil and gas investments. Oper. Res. 471–15.

Smith, J. E., R. L. Winkler. 1999. Casey’s problem: Interpreting andevaluating a new test. Interfaces 29(3) 63–76.

Spector, B. I. 1993. Decision analysis for practical negotiation appli-cation. Theory Decision 34 183–199.

Spradlin, T. 2001. Internal DA consultants; What’s going to hap-pen? Presented at Decision Analysis Affinity Group Confer-ence, Houston, TX.

Spradlin, T., D. M. Kutoloski. 1999. Action-oriented portfolio man-agement. Res. Tech. Management 42(2) 26–32.

Spradlin, T., D. C. Skinner. 2001. Decision analysis affinitygroup. Decision Anal. Newsletter 20(2–3) 5. Newsletter Archive,www.informs.org/Society/DA.

Stafira, S. Jr., G. S. Parnell, J. T. Moore. 1997. A methodologyfor evaluating military systems in a counterproliferation role.Management Sci. 43 1420–1430.

Stonebraker, J. S., J. J. Sage, B. L. Leak. 1997. Decision analysisprovides insight to Ford Microelectronics Incorporated. D. L.Keefer, ed. Practice abstracts. Interfaces 27(2) 107–111.

Taha, H. A., H. M. Wolf. 1996. Evaluation of generator mainte-nance schedules at Entergy Electric System. Interfaces 26(4)56–65.

Taylor, A. C., J. S. Evans, T. E. McKone. 1993. The value of animaltest information in environmental control decisions. Risk Anal.13 403–412.



Thurston, D. H. 1990. Multiattribute utility analysis in design man-agement. IEEE Trans. Engrg. Management 37(4) 296–301.

Toland, R. J., J. M. Kloeber Jr., J. A. Jackson. 1998. A comparativeanalysis of hazardous waste remediation alternatives. Interfaces28(5) 70–85.

Trigeorgis, L. 1996. Real Options: Managerial Flexibility and Strategyin Resource Allocation. MIT Press, Cambridge, MA.

Vári, A., J. Vecsenyi. 1992. Experiences with decision conferencingin Hungary. Interfaces 22(6) 72–83.

Von Winterfeldt, D., E. Schweitzer. 1998. An assessment of tri-tium supply alternatives in support of the US nuclear weaponsstockpile. Interfaces 28(1) 92–112.

Walls, M. R., G. T. Morahan, J. S. Dyer. 1995. Decision analy-sis of exploration opportunities in the onshore US at PhillipsPetroleum Company. Interfaces 25(6) 39–56.

Winkler, R. L., T. S. Wallsten, R. G. Whitfield, H. M. Richmond,S. R. Hayes, A. S. Rosenbaum. 1995. An assessment of the riskof chronic lung injury attributable to long-term ozone expo-sure. Oper. Res. 43 19–28.

Yassine, A. A., K. R. Chelst, D. R. Falkenburg. 1999. A decision ana-lytic framework for evaluating concurrent engineering. IEEETrans. Engrg. Management 46(2) 144–157.

Yi, W., V. M. Bier. 1998. An application of copulas to accident pre-cursor analysis. Management Sci. 44(12, Part 2) S257–S270.

Yoon, K. P., C. Hwang. 1995. Multiple Attribute Decision Making: AnIntroduction. Sage Publications, Thousand Oaks, CA.

Zeckhauser, R. J., R. L. Keeney, J. K. Sebenius, eds. 1996. WiseChoices: Decisions, Games, and Negotiations. Harvard BusinessSchool Press, Boston, MA.

Received on August 9, 2001. Accepted by Robert Clemen and Don Kleinmuntz on October 23, 2002, after 1 revision.


Clinical Applications in the DecisionAnalysis Literature

(Comment on Keefer et al. 2004)

Scott B. CantorDepartment of Biostatistics, The University of Texas M. D. Anderson Cancer Center, 1515 Holcombe Boulevard,

Houston, Texas [email protected]

The author provides a comment to the companion literature review, focusing on clinicalapplications of decision analysis. The theory and application of clinical decision analysis

has decreased in the management science and operations research literature, but has becomea field in and of itself, with contributions to the medical and analytical communities. Decisionscientists should be aware of methodological contributions in the health decision scienceliterature.(Decision Analysis; Medical Decision Making; Clinical Decision Analysis; Literature Review)

IntroductionThe article by Keefer, Kirkwood, and Corner (2004)(“Perspective on Decision Analysis Applications,1990–2001”) identified trends and developments indecision analysis applications, based primarily onarticles published in English-language operationsresearch and closely related journals. An unpub-lished companion paper (available on the DecisionAnalysis Society website, http://faculty.fuqua.duke.edu/daweb/) provides an annotated bibliographyfor the papers cited in the published paper.Like the authors of this article, I am a “card-

carrying” member of the Decision Analysis Soci-ety of the Institute for Operations Research and theManagement Sciences (INFORMS). However, I amone of a very few members who focus their work onclinical applications of decision analysis. The medi-cal community has a somewhat different approach toresearch and academic publication, and I would liketo focus my critique of the Keefer et al. (2004) paperon this basis. This is, perhaps, an unfair approach,

since the authors of the original article are not inthe medical community; however, it is the model thatI am most accustomed to these past several years asan academic.I have not done an extensive analysis of applica-

tions of decision analysis in the medical field. Perhapsthat is worthy of an updated study, as it has beendone previously (Fries 1976, Kassirer et al. 1987). Also,two medical decision-making texts provide an exten-sive bibliography of clinical decision analysis appli-cations in their respective appendices (Weinstein andFineberg 1980, Hunink et al. 2001).However, an important article that should have

been included in this review, though not necessar-ily an individual application of decision analysis, isthe study by Tengs et al. (1996). In this study, 587economic evaluations of medical and public policyinterventions were re-analyzed to demonstrate theenormous variability in cost effectiveness of currentlyapplied technologies. The underlying methodologyfor these re-analyses was decision analysis—and this

1545-8490/04/0101/00251545-8504 electronic ISSN


CANTORComment

study was published in Risk Analysis, included as oneof the “operations research and closely related jour-nals” under investigation by the authors of the Keeferet al. (2004) study—but it was not included in thisstudy.The comments made by the authors regarding med-

ical decision making are intriguing. They state thatthe number of published medical applications of deci-sion analysis have dropped substantially primarilybecause “� � �decision analysis methods are now sowell established within the medical community thatmost medical decision analysis applications are pub-lished in specialized medical journals” (Keefer et al.2004, p. 10). This is partially true. Clinical decisionanalyses have been published in the most prestigiousmedical journals: Journal of the American Medical Asso-ciation (JAMA), The New England Journal of Medicine,and Annals of Internal Medicine. In addition, clinicaldecision analyses have been published in the jour-nals of most medical specialties; such journals includeJournal of Clinical Oncology, Journal of General InternalMedicine, Journal of Family Practice, Obstetrics and Gyne-cology, Pediatrics, and Gastroenterology, to name a few.More importantly, however, there are several jour-

nals that have been established over the past 20 yearsthat focus on the methods of medical decisionmaking and related disciplines, including health eco-nomics, technology assessment, and pharmacoeco-nomics. Historically, these journals were created toidentify a place to submit manuscripts that did notseem to have an obvious “placement.” Many ofthe early papers concerning the methods of clini-cal decision analysis were, in fact, published in jour-nals such as Operations Research and ManagementScience (Krischer 1976, Pliskin et al. 1980, Torrance1976). However, the early members of the Societyfor Medical Decision Making recognized the need tocreate their own journal for such manuscripts, andthe journal Medical Decision Making published its firstissue in 1981. Medical Decision Making is consideredby clinical decision analysts to be the premier journalof the field, publishing primarily theoretical but alsoapplied developments in clinical decision analysis.Other journals have been sponsored by health ser-

vices research professional societies that publish clin-ical decision analyses. The International Society for

Pharmacoeconomics and Outcomes Research spon-sors the journal Value in Health, which focuses onpharmacoeconomics, i.e., the description and analy-sis of the cost and outcomes of pharmaceutical ther-apy. Until very recently, the International Society ofTechnology Assessment in Health Care sponsoredthe International Journal of Technology Assessment inHealth Care, which focuses, not surprisingly, on tech-nology assessment, with a particular emphasis onmultinational settings. Other journals, such as HealthEconomics, Journal of Health Economics, and MedicalCare, have a focus on economic methods, but alsopublish applied and theoretical developments in clin-ical decision analyses.Decision analysis is used in the medical field for

two primary sets of problems. The first set of prob-lems is for clinical decision analysis. This is wherean individual patient is the decision maker. Typi-cally, a decision-analytic model is constructed andthe strategy that maximizes a given objective, suchas life expectancy or quality-adjusted life expectancy.The second set of problems involves decision-analyticmodels with two attributes, typically cost and quality-adjusted life expectancy. However, unlike multiat-tribute utility analysis, these attributes are usuallykept distinct and are not combined into one util-ity function. These problems, generally referred to ascost-effectiveness analysis, ideally take a larger per-spective, e.g., a health-care or societal perspective.The fascinating problems of medical decision mak-ing occur when the optimal strategy for an individ-ual patient is different than an optimal strategy forsociety, i.e., when one strategy might be clinicallymore effective than another, but not cost effective. Ora clinical strategy may be more cost effective thananother, but not necessarily be more fair, i.e., the ben-efits may not be equally distributed across the popu-lation. (Cantor 1994). These are serious dilemmas forhealth-care decision making.However, the use of decision analysis in medicine

is still a hard sell. The majority of medical schoolsdo not include clinical decision analysis in their cur-riculum. Specialty medical journals publish decisionanalyses of clinical problems, but typically authorsrequire several revisions before a manuscript is appro-priate for a clinical audience, as the readers must


CANTORComment

understand not only the clinical problem but also thetechnical nature of a mathematical model. Organiza-tions such as the Society for Medical Decision Making(www.smdm.org) help educate clinicians and model-ers in methods useful for mathematical modeling ofclinical problems.The past few years have seen developments in

both the theoretical and applied foci of medical deci-sion making. Research in the discipline of clinicaldecision analysis has made significant methodologicalcontributions to the general field of decision analy-sis. For example, within the realm of clinical decisionanalysis, the area of utility assessment has seen sig-nificant methodological development. The results ofclinical decision analysis models are often dependenton the preferences of the decision maker (i.e., thepatient), and so the methods and mechanics of utilityassessment make an important difference. Throughthe purposes of enhancing utility assessment in clin-ical decision analysis, innovative research has beendone on the mechanics of utility assessment (Lenertet al. 1998), automation of utility assessment, alter-natives and approximations to the standard gamblemethod, and comparisons of normative and descrip-tive preferences.Another set of contributions from the discipline of

medical decision making to the field of decision anal-ysis are in the area of managing uncertainty. Theurgency of medical decisions and the costs and ethi-cal challenges of acquiring better information are typ-ically more challenging than in business problemsthat are analyzed using decision analysis. Medicaldecisions will be made, often at a substantial eco-nomic cost and with clinical consequences, regard-less of whether all the information exists to supportthem. This raises special challenges and may explainwhy clinical decision analysts are in the forefront ofthe development of new methodology in the areas ofvalue of information analysis (Briggs 1999, O’Brienand Briggs 2002, Claxton 1999), model-based cost-effectiveness analysis (Goldman et al. 2001, Freedberget al. 1998), and novel decision support (Kuntz et al.1999, Fryback et al. 2001).As the Keefer et al. (2004) paper shows, the appli-

cation of decision analysis to problems in clini-cal medicine no longer appears very often in the

operations research and management science litera-ture. Most of the clinical applications are, in fact,published in medical journals. However, significantmethodological contributions to decision analysis arepublished in the health decision-science literature.Decision analysts should take notice of them.

AcknowledgmentsThe author thanks A. David Paltiel for his comments and contribu-tions to this editorial.

ReferencesBriggs, A. H. 1999. A Bayesian approach to stochastic cost-

effectiveness analysis. Health Econom. 8 257–261.Cantor, S. B. 1994. Cost-effectiveness analysis, extended dominance

and ethics: A quantitative assessment. Med. Decision Making14 259–265.

Claxton, K. 1999. Bayesian approaches to the value of information:Implications for the regulation of new pharmaceuticals. HealthEconom. 8 269–274.

Freedberg, K. A., J. A. Scharfstein, G. R. Seage III, E. Losina,M. C. Weinstein, D. E. Craven, A. D. Paltiel. 1998. The cost-effectiveness of preventing AIDS-related opportunistic infec-tions. JAMA 279 130–136.

Fries, B. E. 1976. Bibliography of operations research in health-caresystems. Oper. Res. 24 801–814.

Fryback, D. G., N. K. Stout, M. A. Rosenberg. 2001. An elementaryintroduction to Bayesian computing using WinBUGS. Internat.J. Tech. Assessment Health Care 17 98–113.

Goldman, L., K. A. Phillips, P. Coxson, P. A. Goldman, L. Williams,M. G. Hunink, M. C. Weinstein. 2001. The effect of risk fac-tor reductions between 1981 and 1990 on coronary heart dis-ease incidence, prevalence, mortality and cost. J. Amer. CollegeCardiology 38 1012–1017.

Hunink, M. G. M., P. P. Glasziou, J. E. Siegel, J. C. Weeks,J. S. Pliskin, A. S. Elstein, M .C. Weinstein. 2001. Decision Mak-ing in Health and Medicine: Integrating Evidence and Values. Cam-bridge University Press, Cambridge, England.

Kassirer, J. P., A. J. Moskowitz, J. Lau, S. G. Pauker. 1987. Decisionanalysis: A progress report. Ann. Internal Med. 106 275–291.

Keefer, D. L., C. W. Kirkwood, J. L. Corner. 2004. Perspective ondecision analysis applications, 1990–2001. Decision Anal. 1(1).

Krischer, J. P. 1976. Utility structure of a medical decision-makingproblem. Oper. Res. 24 951–972.

Kuntz, K. M., J. Tsevat, M. C. Weinstein, L. Goldman. 1999.Expert panel vs. decision-analysis recommendations for post-discharge coronary angiography after myocardial infarction.JAMA 282 2246–2251.

Lenert, L. A., D. J. Cher, M. K. Goldstein, M. R. Bergen, A. Garber.1998. The effect of search procedures on utility elicitations.Med. Decision Making 18 76–83.


CANTORComment

O’Brien, B. J., A. H. Briggs. 2002. Analysis of uncertainty in healthcare cost-effectiveness studies: An introduction to statisticalissues and methods. Statist. Methods Med. Res. 11 455–468.

Pliskin, J. S., D. S. Shepard, M. C. Weinstein. 1980. Utility functionsfor life years and health status: Theory, assessment, applica-tion. Oper. Res. 28 206–224.

Sox, H. C., M. A. Blatt, Jr., M. C. Higgins, K. I. Marton. 1988.MedicalDecision Making. Butterworths, Boston, MA.

Tengs, T. O., M. E. Adams, J. S. Pliskin, D. G. Safran,J. E. Siegel, M. C. Weinstein, J. D. Graham. 1996. Five-hundred

life-saving interventions and their cost-effectiveness. Risk Anal.

15 369–390.

Torrance, G. W. 1976. Health status index models: A unified math-

ematical view. Management Sci. 22 990–1001.

Torrance, G. W., M. H. Boyle, S. P. Horwood. 1982. Application of

multiattribute utility theory to measure social preferences for

health states. Oper. Res. 30 1043–1069.

Weinstein, M. C., H. V. Fineberg. 1980. Clinical Decision Analysis.

W. B. Saunders, Philadelphia, PA.

Received on April 16, 2003. Accepted by Robert Clemen and Don Kleinmuntz on May 13, 2003, without revision. This comment was refereed.


Reversing the Perspective on theApplications of Decision Analysis

(Comment on Keefer et al. 2004)

Raimo P. HämäläinenHelsinki University of Technology, Systems Analysis Laboratory, P.O. Box 1100,

02150 HUT, Helsinki, [email protected]

This paper looks into the strengths, weaknesses, opportunities, and threats (SWOT) of thefield of the applications based on DA techniques; focusing especially on MAVT. The need

for best practices and bias-resistant analysis procedures is pointed out. In most applicationpapers there are no reports on the verification or testing of the procedures used. The Internetprovides great opportunities in the delivery of DA software and e-learning material. Publicsites such as Decisionarium (www.decisionarium.hut.fi) provide encouragement and a lowentry level to try value tree analysis in new applications. There are also great opportunities inreaching out the other MCDA communities such as AHP practitioners. Today the need anddemand for decision support is perhaps growing most rapidly in environmental problems. Aclear trend is also the integration of DA into GIS and other models. Many of the applicationsare now published in non-OR/MS subject area journals.(DA Applications; Software; Internet; Environment; Multiple Models)

IntroductionThe perspective taken by Keefer et al. (2004) focuseson specific decision analysis (DA) methods with a rel-atively high level of maturity. These include decisiontrees, influence diagrams, as well as multiattributevalue trees and utility theory. Since the methods arewell known it would be interesting for the readersto reverse the perspective. The point of view couldoriginate from the real problems in the solution ofwhich DA methods in general should potentially havea role. One could review the attempts made withany DA-like methods. Also, why a particular casewas not approached or why was it nonapproachablewith DA methods but approached with some others.Today, we may be in a situation where challengingapplications, such as dynamic decision problems (see,e.g., Virtanen et al. 1999, 2001), drive new theoretical

work, wherefore a reversed perspective would behelpful.From the practitioners’ perspective, the selection

criterion of Keefer et al. (2004) for applications seemstoo restrictive: “We use the term decision analysis torefer to a set of quantitative methods for analyzingdecisions based on the axioms of consistent choice”(p. 6). However, the application of DA techniquesdoes not guarantee that actual decisions are takenin keeping with the recommendations generated bymethods based on the axioms of rational choice. Inother words, one has to recognize that the applicationof a particular DA technique, on one hand, and theeventual decision, on the other hand, are two separatethings. Indeed, one could even speculate that the pop-ularity of some methods (e.g., the Analytic HierarchyProcess (AHP)) stems from their ability to generate

1545-8490/04/0101/00291545-8504 electronic ISSN


HÄMÄLÄINENComment

“workable” recommendations, even if their axiomaticfoundation does not concur with similar axioms ofrational choice.In my view, an applications’ review should be

both analytical and critical with respect to the tech-niques and procedures used in the papers covered.The reader would enjoy learning about best prac-tice procedures. From the practical applications’ pointof view, a review should also report what can, ordid, go wrong. The discussion and papers referred todo not report if and how possible behavioral biaseswere avoided (for references see, e.g., Pöyhönen andHämäläinen 1998, Hämäläinen and Alaja 2003). Onecan encourage the practitioners to find new ways toimprove their procedural skills by pointing out bothsuccessful cases and those with problems. This wouldalso present a challenge to revisit cases in which ear-lier analyses have not been complete enough. I seethis as an important way to develop the field.In industrial problems, the estimated savings can

perhaps be used as a measure of the success of anapplication as is done in the review. The effective-ness of multiattribute policy support can seldom bedirectly measured in terms of money, however. Inthese cases, success depends on the satisfaction ofthe public, for example, which typically is difficultto measure. In the case of nonrepeatable public pol-icy decisions there usually are important nonmone-tary and even ethical values at stake (Rauschmayer2001). However, in general, policy analyses andrecommendations should be based on solid and easy-to-understand transparent procedures (Hämäläinenand Salo 1997, Renn 1999, Gregory 2000). This couldbe one perspective to evaluate the applications in areview.The following comments are mainly related to the

application of multiattribute value models. I alsoemphasize the strong role DA has already establishedin environmental applications and the opportunitiesoffered by the Internet. I have included an illustrativeset of new references from a wider range of journalswithout repeating those already included in Keeferet al. (2004).I start by looking into the strengths, weaknesses,

opportunities, and threats (SWOT) of the field ofapplications based on DA techniques, especially

MAUT and MAVT, and including Value FocusedThinking. This helps to determine if the field is in ahealthy state as concluded by Keefer et al. (2004).

StrengthsClearly the main strength is the theoretical basisof MAUT and MAVT models, which justifies theprescriptive approach provided the problem own-ers accept the related rationality assumptions. Thereare comprehensive textbooks on DA and methodsare also briefly covered in many general OR texts.Now there also is literature on the structuring phase(e.g., Keeney 1992, French et al. 1998, and Belton andStewart 2002) as well as on environmental applica-tions (e.g., Jensen 1992, Cothern 1996, and Hobbs andMeier 2000).

WeaknessesThe recent book by Hobbs and Meier (2000) is a wel-come addition to the applications literature, whichhas almost completely ignored the possibility of theoccurrence of the well-demonstrated human biasesin modelling and elicitation procedures (Weber andEisenführ 1993). In most application papers there areno reports on the verification or testing of the proce-dures to guarantee that the respondents have under-stood the questions as intended. Many of the projectsseem to have been based on questionnaires withoutthe possibility of immediate feedback on the con-sequences of the replies. Computer-supported inter-views provide an alternative approach (Marttunenand Hämäläinen 1995). The effects of computer sup-port, if used, on the decision quality is not reported.In general, the role and importance of the interactionof the problem owner and decision analyst receivelittle attention in the value elicitation phase.One of the surprising weaknesses is the lack of

literature on bias-resistant analysis procedures. Howshould one use DA methods safely, i.e., so that theprocedures would not produce misunderstandingsand incorrect or biased results? These issues are, how-ever, finally receiving more attention (Keeney 2002,Belton and Stewart 2002). The risk of bias is a problemboth in value elicitation and in the model structuring.



The structure of the value tree can easily be the ori-gin of a number of undesired phenomena (see, e.g.,Pöyhönen et al. 2001). Our research with real stake-holders has revealed that attribute splitting can bea big problem. Members of the general public seemto have great difficulties in the consistent adjustmentof their responses when the number of attributes ischanged (Hämäläinen and Alaja 2003), while engi-neering students do, indeed, succeed (Pöyhönen andHämäläinen 2000). This raises questions on the pro-cedures used in the applications, in particular, whenprescriptive decision aiding is the objective.The review by Keefer et al. (2004) discusses deci-

sion conferencing too narrowly and even mislead-ingly. The original two-day format (Phillips andPhillips 1993) is rarely satisfactory. Currently thereare different ways to carry out such events rang-ing from spontaneous analysis sessions (Hämäläinenand Leikola 1996) to facilitated workshops of dif-ferent length often preceeded with advance struc-turing sessions (see, e.g., Hämäläinen et al. 1998,Salo et al. 2003). There are also consulting compa-nies offering support for group moderation processes(see, e.g., www.metaplan.com) as well as very simplebut efficient structuring tools such as the mindmap(see, e.g., www.mindmapper.de). Practitioners clearlywould be interested in comparative analyses on thepros and cons of these different approaches to groupfacilitation.The procedures of achieving successful real stake-

holder participation seldom follow the original deci-sion conference scheme. Environmental problems, inparticular, have shown that this, in fact, still remains achallenging research topic (Renn 1999, Gregory 2000).One of the main issues is to find ways to guar-antee that the stakeholders are interested in work-ing together for a value based decision. Here a newapproach, the decision structuring dialogue, has beensuccessfully introduced into environmental applica-tions (Slotte and Hämäläinen 2003).

OpportunitiesThe Internet provides great opportunities. Thefirst web-based MAVT software Web-HIPRE (www.hipre.hut.fi) was released in 1998 (Hämäläinen and

Mustajoki 1998, Mustajoki and Hämäläinen 2000). Thefact that a public site is widely accessible encouragesresearchers and practitioners to try value tree anal-ysis in new applications. This software allows youto use the MAVT and AHP approaches in parallel;they can be combined if desired. Now visitors canalso easily test the results with different procedures.Those used to work with AHP can find a bridgeto MAVT by observing the possibility of the con-vergence of the results (Salo and Hämäläinen 1997,Pöyhönen and Hämäläinen 2001). The Decisionariumsite (Hämäläinen 2000) also offers more advancedDA software that supports negotiations and decisionsunder incomplete information. The Smart-Swaps soft-ware (www.smart-swaps.hut.fi), which supports theEven-Swaps method, is also now available (Hämäläi-nen et al. 2003).The possibility to access web-based learning sites

will be a valuable way to support practitioners inthe future. The site developed in the Systems Analy-sis Laboratory, Helsinki, Finland (www.mcda.hut.fi),allows visitors to customize learning modules by theavailable interactive software, videoclips, slide pre-sentations, and text material (Hämäläinen 2002).The need and demand for decision support in envi-

ronmental problems is growing in a rapid pace. Evenif decision analysts have long been active in the fieldof energy, one should not ignore the other environ-mental application opportunities. Without going intodetail, it is possible to identify some areas where DAand models are already widely applied. These includeforestry (Rauscher et al. 2000, Kangas and Kangas2002), agriculture (Hayashi 2000), land use (Beinatand Nijkamp 1998), and water resources (Marttunenand Hämäläinen 1995, Håkanson et al. 2000, Soncini-Sessa et al. 2000, Hämäläinen et al. 2001). The impactof different processes and products producing green-house effects is analyzed in the important researchfield of environmental life-cycle assessment (LCA).Many of its indexes and tools seek to producescientifically justified multiattribute evaluations of theimpacts of products and processes. LCA researchershave actually been developing a parallel methodologyto what there already is in MAVT. It is only recentlythat the theoretical basis of value tree analysis hasbeen introduced into the LCA community (Miettinen



and Hämäläinen 1997, 1999; Geldermann et al. 1999;Seppälä and Hämäläinen 2001).It is quite likely that today there are many more

DA-oriented groups in environmental institutionsthan in corporations. There is also an increasing num-ber of DA courses offered in environmental depart-ments. One example is the course at Duke University(www.duke.edu/∼meb6).I think that the large AHP literature (see refer-

ences at www.expertchoice.com/hierarchon) and therelated research community should not be ignoredbut seen to offer an opportunity and challenge. Oneshould be interested in understanding the reasons forthe wide use of AHP. There is a lot to be gained fromthis. I would assume that most of the individual prac-titioners who have chosen to work with the method,have not made a deliberate choice between AHP andMAVT. The explanation is not likely that AHP prac-titioners would avoid MAVT models because of thedifferences in the axioms or because the latter can beused in a prescriptive way.My personal feeling is that the interest in AHP is

likely due to the attractiveness of its elicitation proce-dure, which includes redundancy, easy-to-read basicliterature, well-marketed computer support, as wellas the active promotion of the method by its propo-nents. With a growing emphasis on these same issuesit would not be difficult for MAVT to become as popu-lar as AHP. Learning about MAVT and how to use theAHP procedure correctly in the MAVT way (Salo andHämäläinen 1997) could easily become an encourag-ing experience for AHP practitioners and a steppingstone toward more advanced DA techniques.

ThreatsOne of the main threats is too narrow a definitionof what DA is and the narrow basis of the key peo-ple in the field. This latter concern is also pointedout in Keefer et al. (2004). We in the DA communitycan attract future researchers by growth and healthydevelopment based on openness. Young talented peo-ple, who are looking for challenges along new lines ofthought tend to avoid closed professional groupings.Based on this realization, we should remain open and

accessible and reach out for innovative, even experi-mental, applications including the possibility of inte-grating DA with other models and approaches.As noted above, the continuing AHP phobia can

also be seen as a threat to the growth of the appli-cations field. If there is no active bridge buildingbetween the DA and AHP communities, the lattermay continue to grow on its own without ever refer-ring to DA. AHP researchers have strong ties to prac-titioners, and the relationship has created a situationwhere many decision modelling practitioners con-sider AHP the norm choice for a multiattribute eval-uation method.The list of papers on applications based on AHP

and on other multicriteria methods is vast (seewww.expertchoice.com and, e.g., Belton and Stewart2002). Some of these applications seem purely aca-demic but clearly there are real ones as well. Isthis literature really uninteresting, or are the prac-tical problems addressed not real? Why is it thatthese researchers have chosen to work with a non-DA technique? Presumably, a value tree model wouldhave been appropriate in most cases where anothermodel was used. Why is it that practitioners—whodo not have a stake in the academic rivalry betweenthe schools of thought—select non-DA tools andapproaches?

Future DirectionsA general future trend not envisioned in the reviewis the integration of MAVT/MAUT with other eval-uation models into decision support systems (DSS),which include other modules like simulation andmultiobjective optimization. Reports of such systemscan be difficult to find as they are often publishedin specialized journals outside the OR field. Manysuch multimodel systems have already been used inthe every day practice of environmental planning andpolicy analysis (see, e.g., Hobbs and Meier 2000 andKangas and Kangas 2002).A growing trend is also that DA and geographi-

cal information systems (GIS) are coupled togetherin many environmental and logistics applications(Keisler and Sundell 1997, Beinat and Nijkamp 1998,Joerin and Musy 2000, Store and Kangas 2001). It is



interesting to note that a new electronic journal titledJournal of Geographic Information and Decision Analysis(www.geodec.org) is now available.Decision makers do not always behave according

to the rationality of the economic man. Evolution-ary processes have created other kinds of heuristics,which seem to have been successful in many situ-ations (Gigerenzer and Todd 1999, Gigerenzer andSelten 2001). It would be an interesting challenge tosee if some elements of these heuristics could be intro-duced into DA models to help decision makers. Isthere something to be learned from the viewpoint ofprescriptive decision support?The pioneers of DA have confirmed the need for a

reversed problem-driven way forward. The bestsellerbook Smart Choices by Hammond et al. (1999) does notexplicitly discuss DA. However, it gives a clear pro-cedure for approaching decision problems by valuestructuring and the Even Swaps trade-off method. Sofar, there have not been many academic papers onapplications of the method, but those published (see,e.g., Gregory and Wellman 2001 and Kajanus et al.2001) do strongly refer to the DA literature.

DA in European JournalsEven though Keefer et al. (2004) note that the num-ber of reported DA applications has diminished inEuropean journals, this is most likely not the com-plete picture. Many Europeans already seem to behave taken the reversed problem-driven perspective.If one adopts a broader view of DA that covers abroader set of quantitative decision models for nor-mative decision support, including models that mayviolate conventional axioms of consistent choice, theEuropean DA scene, indeed, seems to be well andalive. This is characterized by lively publication activ-ity within the context of several schools of thought.One should also note that AHP and other kinds ofdecision analytic evaluation models have been widelyapplied in Asia also, especially in China and Japan.Thus, even if compelling arguments can be presentedin support of the widely heralded axioms of con-sistent choice, these “other” methods may neverthe-less have appealing qualities from which other DAresearchers and practitioners might learn. Arguably,

the increasing fragmentation of the DA field may bedetrimental, particularly if the proponents of alterna-tive methods continue to disparage each other, thusconfusing prospective DA customers.Assuming that the DA field has truly matured over

the past few decades (even in the sense that research-ers in other fields have adopted DA methods), onewould expect that the number of reported DA appli-cations in non-DA journals has grown. In this regard,the narrow focus of this review on DA in OR/MSjournals seems restrictive. Exciting DA applicationsare often published in journals where the targetedreadership is more concerned with the potentialimpact of DA techniques than the methodologicaltechnicalities of the application of DA.

ReferencesBeinat, E., P. Nijkamp. 1998. Multicriteria Analysis for Land-Use

Management. Kluwer, Dordrecht, The Netherlands.Belton, V., T. J. Stewart. 2002.Multiple Criteria Decision Analysis—An

Integrated Approach. Kluwer Academic Publishers, Boston, MA.Cothern, C. R. 1996. Handbook for Environmental Risk Decision

Making—Value, Perceptions & Ethics. Lewis Publishers, CRCPress, Boca Raton, FL.

French, S., L. Simpson, E. Atherton, V. Belton, R. Dawes,W. Edwards, R. P. Hämäläinen, O. Larichev, F. Lootsma,A. Pearman, C. Vlek. 1998. Problem formulation for multi-criteria decision analysis: Report of a workshop. J. Multi-Criteria Decision Anal. 7 242–262.

Geldermann, J., T. Spengler, O. Rentz. 1999. Proposal for an inte-grated approach for the assessment of cross-media aspects rele-vant for the determination of “Best Available Techniques” BATin the European union. Internat. J. Life Cycle Assessment 4(2)94–105.

Gigerenzer, G., P. M. Todd. 1999. Simple Heuristics That Make UsSmart. Oxford University Press, New York.

Gigerenzer, G., R. Selten, eds. 2001. Bounded Rationality—The Adap-tive Toolbox. MIT Press, Cambridge, MA.

Gregory, R., K. Wellman. 2001. Bringing stakeholder values intoenvironmental policy choices: A community-based estuarycase study. Ecological Econom. 39 37–52.

Hammond, J. S., R. L. Keeney, H. Raiffa. 1999. Smart Choices—A Practical Guide to Making Better Decisions. Harvard BusinessSchool Press, Boston, MA.

Hayashi, K. 2000. Multicriteria analysis for agricultural resourcemanagement: A critical survey and future perspectives. Eur. J.Oper. Res. 122 486–500.

Hobbs, B. F., P. Meier. 2000. Energy Decisions and the Environment:A Guide to the Use of Multicriteria Methods. Kluwer AcademicPublishers, Boston, MA.



Håkanson, L., E. Gallego, S. Rios-Insua. 2000. The application ofthe lake ecosystem index in multi-attribute decision analysis inradiology. J. Environ. Radioactivity 49 319–344.

Hämäläinen, R. P. 2000. Decisionarium—global space for deci-sion support. Decisionarium, Helsinki University of Technol-ogy, Systems Analysis Laboratory, Helsinki, Finland, www.decisionarium.hut.fi.

Hämäläinen, R. P. 2002. Multiple Criteria Decision Analysis—eLearning Site. MCDA, Helsinki University of Technology, Sys-tems Analysis Laboratory, www.mcda.hut.fi.

Hämäläinen, R. P., S. Alaja. 2003. The threath of biases in environ-mental decision analysis. Research reports, E12, Systems Anal-ysis Laboratory, Helsinki, Finland, www.e-reports.sal.hut.fi.

Hämäläinen, R. P., R. Kalenius. 2002. Opinions-Online—Platformfor global participation, voting, surveys and group decisions,v. 2.0. Opinions-Online, Systems Analysis Laboratory, HelsinkiUniversity of Technology, Helsinki, Finland, www.opinions-online.com.

Hämäläinen, R. P., O. Leikola. 1996. Spontaneous decision confer-encing with top-level politicians. OR Insight 9(1) 24–28.

Hämäläinen, R. P., J. Mustajoki. 1998. Web-HIPRE—Java-applet forvalue tree and AHP analysis. Web-HIPRE, Systems AnalysisLaboratory, Helsinki University of Technology, Helsinki, Fin-land, www.hipre.hut.fi.

Hämäläinen, R. P., A. A. Salo. 1997. The issue is understanding theweights. J. Multi-Criteria Decision Anal. 6 340–343.

Hämäläinen, R. P., J. Mustajoki, P. Alanaatu. 2003. Smart swaps—Smart choices with the even swaps method. Computer soft-ware, Systems Analysis Laboratory, Helsinki University ofTechnology, Helsinki, Finland, www.smart-swaps.hut.fi.

Hämäläinen, R. P., E. Kettunen, M. Marttunen, H. Ehtamo. 2001.Evaluating a framework for multi-stakeholder decision sup-port in water resources management. Group Decision Negotia-tion 10(4) 331–353.

Hämäläinen, R. P., K. Sinkko, M. Lindstedt, M. Ammann, A. Salo.1998. RODOS and decision conferencing on early phase pro-tective actions in Finland. Finnish Centre for Radiation andNuclear Safety research report, STUK-A159, Helsinki, Finland,1–76, www.sal.hut.fi/Publications/pdf-files/rham98.pdf.

Janssen, R. 1992. Multiobjective Decision Support for EnvironmentalManagement. Kluwer Academic Publishers, Dordrecht, TheNetherlands.

Joerin, F., A. Musy. 2000. Land management with GIS and multi-criteria analysis. Internat. Trans. Oper. Res. 7 67–78.

Kajanus, M., J. Ahola, M. Kurttila, M. Pesonen. 2001. Applica-tion of even swaps for strategy selection in a rural enterprise.Management Decision 39(5) 394–402.

Kangas, J., A. Kangas. 2002. Multiple criteria decision supportmethods in forest management. An overview and compar-ative analyses. T. Pukkala, ed. Multi-Objective Forest Plan-ning 37-70. Kluwer Academic Publishers, Dordrecht, TheNetherlands.

Kangas, J., A. Kangas, P. Leskinen, J. Pykäläinen. 2001. MCDMmethods in strategic planning of forestry on state-owned lands

in Finland: Applications and experiences. J. Multi-Criteria Deci-sion Anal. 10 257–271.

Keefer, D. L., C. W. Kirkwood, J. L. Corner. 2004. Perspectiveon decision analysis applications 1990–2001. Decision Anal. 1(1).

Keeney, R. L. 1992. Value-Focused Thinking—A Path to Creative Deci-sionmaking. Harvard University Press, Cambridge, MA.

Keeney, R. L. 2002. Common mistakes in making value trade-offs.Oper. Res. 50(6) 935–945.

Keisler, J. F., R. C. Sundell. 1997. Combining multi-attribute utilityand geographic information for boundary decisions: An appli-cation to park planning. J. Geographic Inform. Decision Anal. 1(2)101–118.

Marttunen, M., R. P. Hämäläinen. 1995. Decision analysis inter-views in environmental impact assessment. Eur. J. Oper. Res.87(3) 551–563.

McDaniels, T. L., C. Roessler. 1998. Multiattribute elicitation ofwilderness preservation benefits: A constructive approach.Ecological Econom. 27 299–312.

Miettinen, P., R. P. Hämäläinen. 1997. How to benefit from decisionanalysis in environmental life cycle assessment. Eur. J. Oper.Res. 102(2) 279–294.

Miettinen, P., R. P. Hämäläinen. 1999. Indexes for fixed and flexibleenvironmental target setting—A decision analytical perspec-tive. Internat. J. Environ. Pollution 12(2/3) 147–164.

Mustajoki, J., R. P. Hämäläinen. 2000. Web-HIPRE: Global decisionsupport by value tree and AHP analysis. J. Inform. SystemsOper. Res. (INFOR) 38(3) 208–220.

Phillips, L., M. Phillips. 1993. Facilitated work groups. J. Oper. Res.Soc. 44(6) 533–549.

Pöyhönen, M., R. P. Hämäläinen. 1998. Notes on the weightingbiases in value trees. J. Behavioral Decision Making 11 139–150.

Pöyhönen, M., R. P. Hämäläinen. 2000. There is hope in attributeweighting. J. Inform. Systems Oper. Res. (INFOR) 38(3) 272–282.

Pöyhönen, M., R. P. Hämäläinen. 2001. On the convergence of mul-tiattribute weighting methods. Eur. J. Oper. Res. 129(3) 569–585.

Pöyhönen, M., H. C. J. Vrolijk, R. P. Hämäläinen. 2001. Behavioraland procedural consequences of structural variation in valuetrees. Eur. J. Oper. Res. 134(1) 218–227.

Rauscher, H. M., F. T. Loyd, D. L. Loftis, M. J. Twery. 2000. A practi-cal decision-analysis process for forest ecosystem management.Comput. Electronics Agriculture 27 195–226.

Rauschmayer, F. 2001. Reflections on ethics and MCA in environ-mental decisions. J. Multi-Criteria Decision Anal. 10 65–74.

Renn, O. 1999. A model for an analytic-deliberative process in riskmanagement. Environ. Sci. Tech. 33(18) 3049–3055.

Salo, A. A., R. P. Hämäläinen. 1997. On the measurement of prefer-ences in the analytic hierarchy process. J. Multi-Criteria DecisionAnal. 6 309–319.

Salo, A. A., T. Gustafsson, P. Mild. 2003. A prospective evaluationof a cluster program for Finnish forestry and forest industries.Internat. Trans. Oper. Res. Forthcoming.

Schmoldt, D. L., et al., eds. 2001. The analytic hierarchy process innatural resource and environmental decision making. KluwerAcademic Publishers, Dordrecht, The Netherlands.



Seppälä, J. 1999. Decision analysis as a tool for life cycle impactassessment. W. Klöpffer, O. Hutzinger, eds. LCA Documents,Vol. 4. Eco-Informa Press, Bayreuth, Germany.

Seppälä, J., R. P. Hämäläinen. 2001. On the meaning of the distance-to-target weighting method in life cycle impact assessment.Internat. J. LCA 6(4) 211–218.

Slotte, S., R. P. Hämäläinen. 2003. Decision structuring dia-logue. Research report, E13, Systems Analysis Laboratory,Helsinki University of Technology, Helsinki, Finland, www.e-reports.sal.hut.fi.

Soncini-Sessa, R., D. Canuti, A. Colorni, L. Villa, B. Vitali, E. Weber,F. B. Losa, E. Laniado, A. Rizzoli. 2000. Use of multi-criteriaanalysis to resolve conflicts in the operation of a transactional

multipurpose water system. Internat. Water Resources Assoc.25(3) 334–346.

Store, R., J. Kangas. 2001. Integrating spatial multi-criteria evalu-ation and expert knowledge for GIS-based habitat suitabilitymodelling. Landscape Urban Planning 55 79–93.

Virtanen, K., T. Raivio, R. P. Hämäläinen. 1998. Modeling pilot’ssequential manuevering decisions by a multistage influencediagram. Proc. AIAA (Amer. Inst. Aeronautics Astronautics) Guid-ance, Control, Navigation Conf. Montreal, Canada.

Virtanen, K., T. Raivio, R. P. Hämäläinen. 1999. Decision theoreticalapproach to pilot simulation. J. Aircraft 36(4) 632–641.

Weber, M., K. Eisenführ. 1993. Behavioural influence on weightjudgements in multiattribute decision making. Eur. J. Oper. Res.67 1–12.

Received on April 8, 2003. Accepted by Robert Clemen and Don Kleinmuntz on May 13, 2003, without revision. This comment was refereed.


Response to Comments onKeefer et al. (2004)

Donald L. Keefer • Craig W. Kirkwood • James L. CornerDepartment of Supply Chain Management, Arizona State University, Tempe, Arizona 85287-4706Department of Supply Chain Management, Arizona State University, Tempe, Arizona 85287-4706

Department of Management Systems, University of Waikato, Hamilton, New [email protected] • [email protected] • [email protected]

Our article “Perspective on Decision Analysis Applications, 1990–2001” provides a com-prehensive listing of decision analysis applications published between 1990 and 2001 in

the 16 journals that we surveyed and uses this together with results from a previous sur-vey to identify, and provide perspective on, trends and developments in decision analysisapplications. Cantor (2004) and Hämäläinen (2004) provide useful comments and additionalreferences that supplement the material in our article, and we agree with many of their com-ments. However, we do not agree with their view that our definition of “decision analysis” istoo restrictive. We believe that, after 35 years, the core of decision analysis is now well estab-lished and that we have used the generally understood definition in selecting the applicationsincluded in our article.(Decision Analysis, Applications: Survey and Perspective on Applications; Utility/Preferences, Appli-cations: Survey and Perspective on Applications)

1. IntroductionWe thank Scott B. Cantor and Raimo P. Hämäläinenfor useful comments on our article “Perspective onDecision Analysis Applications, 1990–2001.” We havesome specific responses to their individual comments,but we will begin with some general statements.The intent of our article is to provide a compre-hensive listing of decision analysis applications pub-lished between 1990 and 2001 in the 16 journalsthat we surveyed, and to use this survey to iden-tify, and provide perspective on, trends and devel-opments in decision analysis applications. By com-prehensive we mean that a reader of our article canbe assured that all decision analysis applications thatappeared in those journals over the specified periodof time are listed in our article. We defined what wemean by decision analysis applications in our article—see its fourth paragraph—but Cantor and Hämäläi-

nen both raise concerns about the scope of what wecover under the term decision analysis, so additionaldiscussion may be useful (Cantor 2004, Hämäläinen2004).We use decision analysis as this term is used in the

traditional early references by Raiffa (1968), Howard(1968), and Keeney and Raiffa (1976), or in morerecent books, such as Clemen (1996) or Kirkwood(1997). This definition is consistent with that usedin Corner and Kirkwood (1991) and facilitates com-parisons between that article and the current one inidentifying trends between the periods covered. Wedid not include applications of the Analytic HierarchyProcess (AHP), fuzzy sets, multiple-criteria decisionmaking, cost-benefit analysis, cost-effectiveness analy-sis, traditional mathematical programming, or any ofthe large number of other methods that are used toaid decision making. Each of these methods can be


1545-8490/04/0101/00361545-8504 electronic ISSN

KEEFER, KIRKWOOD, AND CORNERResponse

useful, but they are not within the scope of decisionanalysis as defined in our article.

Response to Comments byScott B. CantorCantor (2004) discusses the state of decision analysisin the medical field including journal outlets, problemtypes, models, education, applications, acceptancewithin the medical community, and methodologicalcontributions. This material supplements the lim-ited material on medical applications in operationsresearch journals and provides background and per-spective from inside the field that we cannot offer. Wewelcome this contribution and would like to see anupdated article in the operations research literaturedevoted to medical applications of decision analysisalong the lines of Krischer (1980).Cantor has pointed out an ambiguity in our use

of the term “specialized medical journals,” which weintended to include such journals as Medical DecisionMaking. A better term would have been “medicaljournals and other journals focusing on medical deci-sion making.” He appears to use a broader definitionof the term decision analysis that we do. In particular,he points out Tengs et al. (1995) as an article that weshould have referenced. That article retrospectivelysummarizes the cost-effectiveness of more than 500life-saving interventions, and while this is certainlyrelevant to decision making, it does not fit withinthe definition of decision analysis used in our article.(And indeed the term decision analysis is not usedanywhere in the title, abstract, keywords, or body ofthe Tengs et al. article.)

Response to Comments byRaimo P. HämäläinenHämäläinen (2004) comments that the selection crite-ria for our article were too restrictive, and he goes onto cite additional work related mainly to the appli-cation of multiattribute value models. These comefrom a variety of journals, books, and websites, mostof which were not within the scope of our article,and include applications of methodologies outsideour definition of decision analysis, such as AHP and

multiple-criteria optimization. These references sup-plement the material provided in our article andwill undoubtedly be of interest to some readers.We agree with his comments that decision analystsshould try to learn from the successes of AHP andrelated methodologies and that the fragmentation ofseemingly closely related disciplines is regrettable,but this is not the focus of our article. We alsoagree that in-depth studies that “reverse the perspec-tive” on applications of decision analysis and relatedapproaches would be interesting, although we areskeptical that the level of information needed to pro-vide this perspective could be obtained for most pub-lished applications.Regarding strengths of the decision analysis

approach in his SWOT analysis, he cites its firm the-oretical basis, thorough documentation, and growingliterature on structuring. We agree, but would also addthe large number and variety of successful applica-tions documented in the literature, as evidenced byour article, as an important additional indication ofstrength. Regarding weaknesses, he comments that“the applications literature � � �has almost completelyignored the possibility of the occurrence of the well-demonstrated human biases in modelling and elici-tation procedures used,” and “one of the surprisingweaknesses is the lack of literature on bias-resistantanalysis procedures” (p. 30). In fact, these topics havebeen addressed in a variety of research-oriented deci-sion analysis studies and are well known to manypractitioners. Although dealing with biases is not amajor topic of our article, a number of the applicationsarticles describe methods used to combat biases—see,for example, the articles in Table 4 under ProbabilityAssessment.He also states that “in industrial problems, the esti-

mated savings can perhaps be used as a measureof the success of an application as is done in thereview” (p. 30). This might lead some readers to con-clude that expected savings was the only evaluationmeasure used in the applications that we surveyed.Although we highlight impressive monetary savingswhen these were documented for an application, vari-ous other evaluation measures were also used in someof these applications.


KEEFER, KIRKWOOD, AND CORNERResponse

Concluding CommentsA criticism by Cantor and Hämäläinen is that ourscope was too narrow in terms of what we consideredto be “decision analysis.” In our judgment, the defini-tion of the term decision analysis is fairly settled nowafter more than 35 years of use. The fact that there aredistinct, recognized communities of researchers andpractitioners for related activities such as the AHPand multiple-criteria decision making indicates thatthey are at least somewhat different from decisionanalysis.This is not meant to imply that those other ap-

proaches to aiding decision making do not havevalue, but simply that they differ from decisionanalysis as conventionally defined in the same waythat physics and chemistry differ from each other.Of course, it has long been recognized and has beendemonstrated in practice that combining decisionanalysis with other analytical methods (for example,optimization) can be useful, just as it can be usefulto combine methods from physics and chemistry toattack certain problems. However, we believe that thecore of decision analysis is now well established, andthat we have used the generally understood definition

in selecting the applications that were included in ourarticle.

ReferencesCantor, S. B. 2004. Clinical applications in the decision analysis lit-

erature (Comment on Keefer et al. 2004). Decision Anal. 1(1).Clemen, R. T. 1996. Making Hard Decisions: An Introduction to Deci-

sion Analysis, 2nd ed. Duxbury Press, Belmont, CA.Corner, J. L., C. W. Kirkwood. 1991. Decision analysis applications

in the operations research literature, 1970–1989. Oper. Res. 39206–219.

Hämäläinen, R. P. 2004. Reversing the perspective on the appli-cations of decision analysis (Comment on Keefer et al. 2004).Decision Anal. 1(1).

Howard, R. A. 1968. The foundations of decision analysis. IEEETrans. Systems Sci. Cybernetics SSC-4 211–219.

Keeney, R. L., H. Raiffa. 1976. Decisions with Multiple Objectives:Preferences and Value Tradeoffs. Wiley, New York.

Kirkwood, C. W. 1997. Strategic Decision Making: Multiobjective Deci-sion Analysis with Spreadsheets. Duxbury Press, Belmont, CA.

Krischer, J. P. 1980. An annotated bibliography of decision analyticapplications to health care. Oper. Res. 28 97–113.

Raiffa, H. 1968. Decision Analysis. Addison-Wesley, Reading, MA.Tengs, T. O., M. E. Adams, J. S. Pliskin, D. G. Safran, J. E. Siegel,

M. C. Weinstein, J. D. Graham. 1995. Five-hundred life-saving interventions and their cost-effectiveness. Risk Anal. 15369–390.

Received on June 20, 2003. Accepted by Robert Clemen and Don Kleinmuntz on July 9, 2003, without revision. This response was refereed.
