The Two Faces of Decision Analysis Decision Analysis ...salserver.org.aalto.fi/.../EAPDA/presentations/Mat24194-2009-Corner… · • Does theory drive practice, or the reverse? –

The Two Faces of The Two Faces of

Decision Analysis Decision Analysis Practice (1)Practice (1)

James L. Corner

Department of Management Systems

University of Waikato,

Hamilton, New Zealand

[email protected]

Aalto University

Sept/Oct 2009

AgendaAgenda

1. Decision analysis is alive and well

(the first hour of the course)

2. Decision analysis is not so alive and well

(the last hour of the course)

The Nature of Decision AnalysisThe Nature of Decision Analysis

• For hard decision problems with many competing objectives, much uncertainty

• Process (called Value Focused Thinking)– Elicit an objective function (value vs utility)

– Identify alternatives (decision trees, IDs, etc)– Identify alternatives (decision trees, IDs, etc)

– Determine probability density function (pdf) over alternative outcomes

– Integrate utility function over pdf for each alternative

– Choose one with maximum utility, or

– Use sensitivity analysis to find out the value of more information

Perspective on Decision Analysis

Applications in the Operations

Research Literature

James L. Corner


University of Waikato, NZ University of Waikato, NZ

withDonald L. Keefer

Craig W. Kirkwood

Department of Supply Chain Management

Arizona State University

Tempe, AZ, USA

Purpose

•Present highlights from and stimulate interest in:

“Perspective on Decision Analysis Applications, 1990-2001”

(Decision Analysis, 2004)

•Companion technical report, “Summary of Decision Analysis Applications in

the Operations Research Literature, 1990-2001”Available at www.public.asu.edu/~kirkwood

•Accompanying comments on above article, with our responses•Accompanying comments on above article, with our responses

•Paper identifies and classifies DA applications from 16 major English

Language OR and closely related journals from 1990-2001

•Period-to-period comparisons with Corner and Kirkwood (Operations

Research, 1991) covering 1970-1989

•Perspective and opinions on trends and developments

•Practice competitions.

•Decision Analysis Affinity Group.

•Needs and concerns.

Ground Rules for Selecting DA Applications

•Is an application a “decision analysis” application?

•Include if it explicitly analyzes alternatives using judgmental

probabilities and/or subjectively assessed utility/value

functions.

•Include if it, on balance, uses a DA approach.

•Is it an “application”? •Is it an “application”?

- Include if it represents a case history of the use of DA to

address a specific decision problem.

- Include if it describes an analysis performed to provide

background for policy making.

- Include for reference if material is clearly of direct

interest for applications, even if an explicit application is

not included.

No. of Articles % Change in Ave.

No. per Year1970-1989 1990-2001

Total Number of Articles 85 86 69%

Total Articles in Common Journals 85 63 24%

Decision Sciences 4 1 -58%

European Journal of OR 5 0 -100%

IEEE Trans. on Engr. Mgmt. Not Cov 4 —

IEEE Trans. on SMC 7 1 -76%

Interfaces 18 40 270%

Number of DA Applications Articles, by Journal, with Trends

Journal of MCDA Not Pub 6 —

Journal of the OR Society 18 1 -91%

Management Science 10 4 -33%

Military OR Not Pub 8 —

Omega 3 1 -44%

Operations Research 15 11 22%

OR Letters 0 0 0%

Reliability Engr. & Sys. Safety Not Cov 2 —

Research• Tech. Mgmt. Not Cov 2 —

Risk Analysis 6 4 11%

Theory and Decision Not Cov 1 —

•Positive overall rate of change (1970-1989 to 1990-2001) over 32 years.

•69% increase in average rate per year if all 16 journals included.

•24% increase if confined to 10 journals included in both surveys.

•Applications articles concentrated in fewer journals for 1990-2001.

- Interfaces: 47% of total, versus 21% for 1970-1989.

Observations

- Interfaces: 47% of total, versus 21% for 1970-1989.

- Operations Research: 13%.

- Military OR (new): 9%.

•Substantial decreases in European OR journals, except for new Journal of MCDA.

•Implications.

- For DA field

- For new journal Decision Analysis

Classification by Area of Application

Each article classified into exactly one of six application areas and, where

appropriate, a subarea — numerous judgment calls required.

Area

Subarea

Increase Decrease

Energy (26) Envt Risk Regulation

Product Selection Site Selection

Project Selection

Strategy

Mfg Services (23) R&D Proj Selection

Medical (5) Down in OR lit from 16

Military (13) Large increase, new area

Public Policy (13) Standard Setting

General (6)

Classification by Methodological and Implementation Issues

Article included in (multiple) classification area(s) only if it provides

significant details about issue in question.

•Strategy and/or Objectives Generation (42) — substantial increase.

•Problem Structuring/Formulation (34) — substantial increase.

•Probability Assessment (22).

•Utility/Value Assessment (28).

•Sensitivity Analysis (23) — substantial increase.

•Communication/Facilitation (29).

•Group Decision Making (12).

•Implementation (27) — substantial increase.

General Trends and Developments

•Powerful PC software for DA.

- Wider use of IDs, strategy tables, tornado diagrams, etc.

- Solution of large problems.

•Decision conferencing — computer-assisted group meeting focused on

decision problem, with facilitator(s).

- Synthesis of techniques from DA, DSS, and group management. - Synthesis of techniques from DA, DSS, and group management.

- Typically apply MAV models within military or public sector.

•Stochastic trees — combine continuous-time Markov chains with DTs.

- Medical applications with time as continuum.

•Value-focused thinking — values as primary driver for structuring and

analysis, including generation of alternatives.

•Numerous military, strategic, and public policy applications.

Interdisciplinary Trends and Developments

•Normative systems — DA and AI.

- AI systems based on influence diagrams or belief nets (Bayesian

principles).

- Applications in medical diagnosis, energy price and demand

forecasting, and machine vision.

•Organizational processes — DA and OB.

- Well-defined procedures for conducting and implementing DA in large-

scale strategic analyses.

- Managing interactions between analysts and other stakeholders

•Real options — DA and Finance.

- Synthesis of methods from DA and finance proposed; few such

applications in OR literature so far.

- Increased emphasis on sequential (downstream) decisions.

Practice Competitions — Edelman Award Finalists

•Burnett, W. M., D. J. Monetta, B. G. Silverman. 1993. How the Gas Research Institute

(GRI) helped transform the US natural gas industry. Interfaces 23(1) 44−58.

•Long-term use of project appraisal methodology within GRI’s annual five-year R&D planning

process.

•Estimated benefits in tens of billions of dollars.

•Paté-Cornell, M.-E., P. S. Fischbeck. 1994. Risk management for the tiles of the space

shuttle. Interfaces 24(1) 64-86.

•PRA of failure of exterior surface tiles on US space shuttle orbiter.

•Highlighted high-risk zones of surface and organizational factors contributing to tile failure

risks.

•Von Winterfeldt, D., E. Schweitzer. 1998. An assessment of tritium supply alternatives

in support of the US nuclear weapons stockpile. Interfaces 28(1) 92−112.

•Analysis to help US DOE choose tritium-supply alternatives to replenish tritium for US nuclear

weapons stockpile.

•Influential in final choice by US Secretary of Energy.

Practice Competitions — DAS Practice Award Winners

1999. Mazen A. Skaff and Donald W. Spillman, “A Portfolio Management Process

and System for an Upstream Oil and Gas Organization.”

-Process and system to help manage large portfolio of upstream oil and gas

assets in GOM and support variety of related decisions.

2000. David A. Mauney, “Best Practices in the Application of Decision/Financial

Analysis to Repair/Replacement Decisions of Plant Components.”

-Process to aid in planning timing of major maintenance investments for fossil--Process to aid in planning timing of major maintenance investments for fossil-

fuel power plants.

2001. Eric Johnson, “Life Cycle Strategy Analysis.”

- Helped client pharmaceutical firm reach consensus on development strategy

for cancer drug.

2002. Jeff Stonebraker, “Commercial Evaluation of a Blood Clot-Busting Drug.”

- First decision analysis project done by Bayer Biological Products.

Decision Analysis Affinity Group (DAAG)

•Founded in 1995 to promote use of DA in industry and to further

development and careers of industrial practitioners.

- Fills important need for DA practitioners in industry.

•Annual conferences focusing on use and implementation of DA within major

corporations.

- Emphasis on mutual support and information-sharing among industrial

practitioners.

- Attendance by consultants and academics generally discouraged until

2002 meeting. 2002 meeting.

•DAS-DAAG relationships.

- Some joint members of DAS and DAAG.

•Most DAAG members not interested in publishing, DAS/INFORMS

technical sessions, etc.

- A number of DAAG-organized sessions at INFORMS meetings.

- Future relationships?

Needs and Concerns

•Status of DA in companies and universities.

- DA and DA groups in industry continue to fall in and out of favor.

- Ambiguity of “DA” even within INFORMS community.

•Contents in functional area for Interfaces or in keyword searches in

indices.

•Coverage in OR/MS textbooks.

•Key role of small group of institutions and individuals in DA education and

applications-oriented innovations.

- Effects of retirements, acquisitions, and reorientations?

- “Who’s gonna fill those shoes?”

•Better Tools.

- Modeling and assessing probabilistic dependence.

- Time-dynamics (sequential nature) of decision problems.

Concluding Remarks

1. DA alive and well as evidenced by applications published in the OR

literature over the last dozen years.

•Significant numbers of DA applications published outside the OR

literature.

2. Small fraction of real-world applications of DA written up and

published. published.

•Beneficial to DA field if more practitioners and consultants

published or presented applications.

3. How representative are publications (OR literature or otherwise) of

DA applications and practice in the real world?

Critique of the AboveCritique of the Above

Scott Cantor – Dept of Biostats, University of Texas

Raimo Hämäläinen – Aalto University

Some valid comments…Some valid comments…

• Don’t forget the specialized journals, eg in medicine:

– JAMA, NEJM, AIM, Medical DMg• Clinical DA, individual patient is DMer, vs

• Cost-effectiveness analysis – what’s best for society?

• Does theory drive practice, or the reverse?

– Medical field has advanced theory in:– Medical field has advanced theory in:• value-of-life,

• managing uncertainty

• value of information

• Just what is decision analysis anyway?

– AHP? Multicriteria Optimization?• If it helps, shouldn’t we use it and consider it DA?

• Problem first, solution technique second

Some valid comments (cont’d)…Some valid comments (cont’d)…

• Be more critical

– What worked, what didn’t, why?

– How do we measure decision success?• Monetary?

• Happiness?

• Transparency?

– How did the application deal with behavioral issues• Understanding of the technique

• Cognitive biases

• Effect of computer tool usage

• Group issues

• Use multiple methods (esp AHP) and heuristics

• Use online help

ConclusionConclusion

• Decision analysis is alive and well

• No. of publication outlets is increasing

• Some concerns but not fatal

• Challenges remain• Challenges remain

The Two Faces of The Two Faces of

Decision Analysis Decision Analysis Practice (2)Practice (2)

James L. Corner


University of Waikato,

Hamilton, New Zealand

[email protected]

Aalko University

Sept/Oct 2009

AgendaAgenda

1. Decision analysis is alive and well

(the first hour of the course)

2. Decision analysis is not so alive and well

(the last hour of the course)

We already know so much about...We already know so much about...

• Descriptive decision processes –

what we do, unaided

• Prescriptive decision processes –

how we can do betterhow we can do better

• Normative decision processes –

what we should do ideally

6 Lessons From Considering

Descriptive Decision Processes

Decision Making

Alternatives

Values

Politics

Constraints & ResourcesTradeoffs

Information

Preferences

Culture

Lesson 1. The Basic Descriptive Decision Process

Decision Making

StructureObjectives

Recognition

Uncertainty & Risk

Implementation

Outcomes

Ethics

Choice

Decision Making

2. Alternatives

7. Values

Politics

5. Constraints & Resources10. Tradeoffs

4. Information

8. Preferences

Culture

Alternative Focused Thinking

Decision Making

6. Structure9. Objectives

1. Recognition

3. Uncertainty & Risk

12. Implementation

13. Outcomes

Ethics

11. Choice

Lessons LearnedLessons Learned

1. People are alternative focused, not value focused.

Lesson 2. A Question of OntologyLesson 2. A Question of Ontology

• Realism vs nominalism in decision making

• Need for a dialectical approach

Intuition (Nominalism) Rationality (Realism)

- automatic - intentional

- holistic - analytic

- non-verbal - verbal

- associated with emotion & - emotion-free

Two Parallel Systems of Knowing

- associated with emotion &

feeling

- emotion-free

- encodes information as

examples, images, stories

- encodes information as

abstract symbols & concepts

- operates in the sub-conscious - operates in the conscious



2. People need to be both intuitive and rational.

Lesson 3. Two Basic Human DifferencesLesson 3. Two Basic Human Differences

• Holistic vs. Non-Holistic evaluation

• Compensatory behaviour

Schoemaker’s FrameworkSchoemaker’s FrameworkSubjective Probabilities

Subjective Weights

Decision

Holistic

EU Models with Probability Transformations

Expected Utility Theory

Mean-Risk Models

Additive Models

Satisficing

Garbage Can

Image Theory

Recognition Primed Decisions

Disjunctive Models

Conjunctive Models

Image Theory

Decision

Models

Non-Holistic

(Sequential

Elimination)

Recognition Primed Decisions

Comparisons against some Standard

Additive Difference Models

Dominance ModelsComparisons across Attributes

Lexicographic Model

Elimination By Aspects Model

Experience the Situation in a changing Context

Reassess

Situation

Seek More

Information

Are experiences

Is the situation

familiar?

Recognition has four aspects

Goals Cues

Are experiences

Violated? Expectancies Actions 1…n

Mental Simulation of Action (n)

Will it work?

Implement

Modify

Mapping of Descriptive ModelsMapping of Descriptive Models

Compensatory Non-CompensatoryHolistic

Recognition Printed Decisions

Image Theory

Garbage Can

SatisficingAdditive Difference

Additive

Non-Holistic E.B.A

Lexicographic

Conjunctive/Disjunctive

Additive Difference

Additive

Dillon, Buchanan, Corner 2005

Mapping of Empirical Results Mapping of Empirical Results

onto Descriptive Theoriesonto Descriptive Theories

Compensatory Non-Compensatory

Holistic

Recognition Printed Decisions

Image Theory

Garbage Can

XXXX XXXXXXXX XXXXXXXX XXXX

XXXXXXXX XXXX XXXX XXXX

Non-Holistic

Holistic

Garbage Can

Satisficing

E.B.A

Lexicographic


Additive Difference

Additive

XXXXXXXX

XXXX

XXXX

XXXX

XXXXXXXX XXXX XXXX XXXX

XXXXXXXX XXXX XXXX

XXXX

XXXXXXXX

XXXXXXXX

XXXX

Dillon, Buchanan, Corner 2005




3. People are not normally compensatory and non-holistic.

Lesson 4. A Question of Framing Lesson 4. A Question of Framing ((LipshitzLipshitz’ ’ Framework 1993)Framework 1993)

Consequential Choice Matching Reassessment

Additive

Additive difference

Satisficing

Garbage CanGarbage Can


Elimination By Aspects

Lexicographic

Image Theory (2) Image Theory (1) Image Theory (3)

Recognition Primed Decisions (1) Recognition Prime Decisions (2)





4. Framing

1. People wish not to always look toward the future1. People wish not to always look toward the future

Four Types of Formulation Process Four Types of Formulation Process Nutt 2003Nutt 2003

Issue Processes (26% of cases, least successful)An issue is analysed for the decision process to consider. Concerns and difficulties imply problems, which in turn suggest solutions (DMs become problem solvers).

Idea Processes (33%, third most successful)Direction is provided with an initial solution imposed by the DM. Development is concerned with certifying the idea’s virtues and suggesting refinements.suggesting refinements.

Objective-Directed Processes (29%, second)Direction setting with objectives, goals, or aims. Freedom to search for alternatives.

Reframing Processes (12%, most successful)Focusing on solutions, problems, or new practices to justify the need to act. (Avoids Raiffa’s 1968 “errors of the third kind.”)

How to Reframe: How to Reframe:

Dynamic Problem Structuring Dynamic Problem Structuring (Corner (Corner et al. et al. 2001)2001)

VFT AFT

Alternatives Criteria





4. Framing


2. Re-framing should be the goal

Lesson 5. Consider Individual PersonalityLesson 5. Consider Individual Personality

Extrovert

Sensor

Thinker

Introvert

Intuitor

FeelerThinker

Judgement

Feeler

Perception

Decision Making Aspects of Decision Making Aspects of

PersonalityPersonalityThinker (50% of population)

- likes many alternatives

- needs analysis time

- does not rush to choice

Feeler (50%)- more concerned with impact on others

- does not rush to choice- does not rush to choice

- tends to be holistic

- tends toward reassessment

Intuitor (25%)- tends to be consequential choice

- likes unique alternatives

- does not like detail

Sensor (75%)- likes few alternatives

- likes graphs and diagrams

- tends toward matching





4. Framing is very important



5. Decision behavior differs by personality

Lesson 6. Consider Decision Style

Instructions





4. Framing is very important



5. Decision behavior differs by personality

6. Decision behavior differs by individual style

Concluding Remarks

1. Our natural inclination does NOT follow the decision

analytic approach

2. People do not like to make and are not good at making

tradeoffs

3. People Match; they do not consider consequences

appropriately

4. Decision making is not a linear process4. Decision making is not a linear process

5. People do not know what they want

6. There is more than one way to structure a decision

problem

7. Framing is absolutely key in structuring and is often

overlooked

8. There is a need for model management systems

The question is…..

… How do we incorporate

this descriptive behavior

into our prescriptive

techniques?

Or, even better…

… What new prescriptive

techniques can we

envision that improve on

descriptive behavior?

Questions?

Comments?

Final thought – which way is the sun going?

Documents

The Two Faces of Decision Analysis Decision Analysis ...salserver.org.aalto.fi/.../EAPDA/presentations/Mat24194-2009-Corner… · • Does theory drive practice, or the reverse? –