Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Decision Making with Incomplete Information:Measures, Mediators, and Decision Support
Marc C. Canellas
Advisor: Dr. Karen M. Feigh
Ph.D. Thesis Proposal, School of Aerospace Engineering
Dr. Amy Pritchett, School of Aerospace Engineering, Georgia Tech
Dr. Brian German, School of Aerospace Engineering, Georgia Tech
Dr. Stephen Cross, School of Industrial and Systems Engineering, Georgia Tech
Dr. Juan Rogers, School of Public Policy, Georgia Tech
Sept. 3, 2015 1
• What is heuristic decision making?• Decision algorithms that only use a subset of “necessary” information
• Why is heuristic decision making important?• Heuristics are used by people in time-stressed, high stakes environments (e.g. aviation,
medical, emergency response, and military domains)• Can established decision support research help support heuristic decision makers?
• Previous decision support research has almost exclusively focused on supporting normative decision making strategies which do not apply to heuristic decision making environments.
• What recent methods for heuristic decision support have been developed?• Structuring the decision making process and environment
How should systems be designed to support heuristic decision makers?
Sept. 3, 2015
• What is an unstudied area of heuristic decision support with potential?• Heuristic decision support for structuring information (information acquisition)
2
Issues with normative decision making
Sept. 3, 2015
1Orasanu and Connolly, 1993; 2Green and Mehr, 1997;3Tversky and Kahneman, 1974; 4Katsikopoulos and Fasolo, 2006; 5Elwyn et al., 2001; 6This result is the output of the naturalistic decision making and fast-and-
frugal heuristics research programs3
Descriptive problem: • People often do not make decisions by gathering and processing all
information (due to time, cost, experience, etc.)1
• Necessary information often not available2
• People often cannot provide reliable assessments of probabilities, attributes weights, and value3
• Lack of transparency and understanding of underlying methods4,5
Normative decision making is not a good basis for building decision support tools for these environments4.
• People are often very well-adapted to their environment and use simple rules that enable them to perform just-as-good if not, better than normative methods6
Normative vs. Heuristic
Sept. 3, 20151Nelson, 2005; 2Nelson, 2008; 3Meder and Nelson, 2012; 4Katsikopoulos and Fasolo, 2006; 5Todd,
Gigerenzer, and the ABC Research Group, 2012 4
Normative HeuristicBenefits of Heuristic
Decision Making
Decision Making
Strategies
Gather and process all information about the
environment
Simple rules that process information deemed “necessary”
• Faster to perform4
• Easier to communicate4
• More accurate and robust in many environments5
Information Acquisition
Quantify the value of each potential piece of information1,2,3 ?
Heuristic information acquisition methods have the potential tobenefit heuristic decision making performance.
Simple rules for determining which
information to acquire
Sept. 3, 2015
Objectives and MethodDevelop methods of heuristic information acquisition
5
Research Objectives and Method
Sept. 3, 2015 6
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: EmpiricalEnvironment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
Q: What measures define distributions of incomplete information and affect decision making accuracy?
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?
Q2: What aspects of the environment affect when the methods work?
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&Gmethods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristicinformation R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress Future work Future work Future Work
Sept. 3, 2015
Task 0: Construct a Decision Making Simulation Engine Enables comprehensive study of all combinations of environments and incomplete information
7
Example Decision Task: UAV Path Selection
Sept. 3, 2015 8
Three attributes of the paths: time, danger, and control.
Preferences• Minimum time is preferred• Low danger is preferred to high danger• Autopilot control is preferred to manual control
Decision Task
BASE
TARGET
B30 min
Autopilot OptionTime to Target
Danger Control
A 10 High Manual
B 30 Low Autopilot
A10 minManual
Decision Making Simulation Engine
Sept. 3, 2015 9
Decision Task Decision Making Strategies
OptionTime to Target
Danger Control
A 10 High Manual
B 30 Low Autopilot
Incomplete information
Environment
OptionTime to Target
Danger Control
A K ? K
B ? K K
OptionTime to Target
Danger Control
A 10 ? Manual
B ? Low Autopilot
ChoiceEnvironment
Incomplete information
Decision Task
The state of the world
Missing information about theoptions’ attributes
Option-attribute informationpresented to the decision maker
Process information to choosea course of action
Weighted Additive (WADD)1
Equal Weighting (EW)1 Tallying2 Take-Two3,4 Take-the-Best (TTB)5
Linear model with weights
Linear model without weights
Linear modelcounting positive
attributes
Two-attribute reasoning by ranking
One-attribute reasoning by ranking
Multi-attribute decision making6;Novice decision
makers7
When weights cannot be determined or
agreed upon8
Criminal sentencing decisions9;
Medical decision making10
People often search for a second,
confirming attribute3,4
Consumer choice11;Expert decision
makers7
Decision Making Strategies
Sept. 3, 2015Operationalizations: 1Payne et al., 1990; 2Gigerenzer and Gaissmaier, 2011; 3Karelaia, 2006; 4Dieckmann and Rieskamp, 2007; 5Gigerenzer and Goldstein, 1996; 6Park, 2004; 7Garcia-Retamero and Dhami, 2009; 8Dawes, 1979; 9von Helversen
and Rieskamp, 2009; 10Kattah et al., 2009; 11Kohli and Jedidi10
Analytic:gather as much information as
possible to use in a linear model
Heuristic:Use only a subset of “necessary”
information
Decision Making Simulation Engine
Sept. 3, 2015 11
Decision Task Decision Making Strategies
OptionTime to Target
Danger Control
A 10 High Manual
B 30 Low Autopilot
Incomplete information
Environment
OptionTime to Target
Danger Control
A K ? K
B ? K K
OptionTime to Target
Danger Control
A 10 ? Manual
B ? Low Autopilot
ChoiceEnvironment
Incomplete information
Decision Task
The state of the world
Missing information about theoptions’ attributes
Option-attribute informationpresented to the decision maker
Process information to choosea course of action
WADD
EW
Tallying
Take Two
TTB
Sept. 3, 2015
Task 1. Define measures of incomplete information
12
Task 1: Identify important measures of distributions of incomplete information
Sept. 3, 2015 13
Define important measures of distributions of incomplete information.
Identify potential methods for decision support based on incomplete information.
Current measures of incomplete information are not sufficient
Sept. 3, 2015 14
Option Time to Target Danger Control
A ? ? ?
B 30 Low Autopilot
Option Time to Target Danger Control
A 10 ? ?
B 30 Low ?
Equivalent total information
3 pieces(50%)
Total information is not sufficient to understand the full effects of incomplete information.
1 2
Measuring distributions of incomplete information
Sept. 3, 20151Martignon and Hoffrage, 2002; 2Garcia-Retamero and Rieskamp, 2008; 3Canellas et al., 2014; 4Canellas
et al., 2015; 5Canellas and Feigh, 201415
Measure Definition Measures
Total Information (TI)1,2 Count (percent) of attribute scores known
- The amount of information known to the decision maker
Information Imbalance (II)3,4 Difference in total information for each option.
- Approximates the number of known-unknown attribute comparisons typically used by heuristics
- Situations where one option is well-known to the decision maker
Complete Attribute Pairs (CAP)5 Count (percent) of attributes in which attribute scores are known for both options.
- The number of known-known attribute comparisons
Measuring distributions of incomplete information
Sept. 3, 2015 16
OptionTime to Target
Danger Control
A ? ? ?
B 30 Low Autopilot
OptionTime to Target
Danger Control
A 10 ? ?
B 30 Low ?
Total Information 3 (50%)Know half the information
Information Imbalance 3 (100%)
Know all the information about
one option and nothing about the
other
Complete Attribute Pairs
0 (0%)No attributes have
information for both options.
Total Information 3 (50%)Know half the information
Information Imbalance 1 (33%)Know slightly more
information about one option than the other
Complete Attribute Pairs
1 (33%)For one attribute, there
is information about both options.
Task 1. Decision Making Simulation Engine (artificial environment)
Sept. 3, 2015 Simulation set-up used in: Canellas et al., 2014; Canellas and Feigh, 2014; Canellas et al., 2015 17
Decision Task Decision Making Strategies
Option A1 A2 A3 A4
O1 40 20 80 40
O2 40 60 40 60
Incomplete information
EnvironmentChoice
Environment (48)
Incomplete Information (28)
Decision Task (88: 16 million)
Artificial: randomly generated attribute scores and weights.
Option A1 A2 A3 A4
O1 ? ? K K
O2 K K K K
Option A1 A2 A3 A4
O1 ? ? 80 40
O2 40 60 40 60
Accuracy is determined by a linear model that is not representative and biased toward WADD.
WADD
EW
Tallying
TTB
Two analytic strategies and two heuristic strategies.
Accuracy: percent of decision tasks that the decision making strategy selects the correct option.
Decision Making Strategies
Choice
Task 1. Effect of incomplete information on decision making accuracy
Sept. 3, 2015 18
50%
60%
70%
80%
90%
100%
0% 25% 50% 75% 100%
Information Imbalance
50%
60%
70%
80%
90%
100%
0% 20% 40% 60% 80% 100%
Accuracy
Total Information
50%
60%
70%
80%
90%
100%
0% 25% 50% 75% 100%
Complete Attribute Pairs
Decision Support Recommendation:
Provide decision makers with more information.
Decision Support Recommendation:
For heuristic decision makers, provide equal information about both options.For analytic decision makers, there is no effect.
Decision Support Recommendation:
Provide decision makers with attribute information for both options.
4 883 84 822 79 77 771 75 72 72 720 64 65 67 68
1 2 3 4 5 6 7 8
4 1003 88 912 78 81 851 71 73 77 810 62 68 73 78
1 2 3 4 5 6 7 8
Tradeoff: Total Information – Complete Attribute Pairs
Sept. 3, 2015 19
Total Information
Complete Attribute
Pairs
Complete Attribute
Pairs
Take-the-Best (TTB)Weighted-Additive (WADD)
Total Information
Cells are empty when that combination cannot occur in a 2-option, 4-attribute task.
• Increasing total information:• Always increases accuracy
• Increasing complete attribute pairs while keeping total information constant:• Does not increase accuracy
• Increasing total information:• Decreases accuracy (if CAP ≥ 1)
• Increasing complete attribute pairs while keeping total information constant:• Increases accuracy
4 883 84 822 79 77 771 75 72 72 720 64 65 67 68
1 2 3 4 5 6 7 8
Task 1: Identify important measures of distributions of incomplete information
Sept. 3, 2015 20
Define important measures of distributions of incomplete information.
Within levels of total information, changing information imbalance or complete attribute pairs has a statistically significant effect on heuristic decision making performance. 50%
60%
70%
80%
90%
100%
0% 25% 50% 75% 100%
Information Imbalance Identify potential methods for decision support based on incomplete information.
Identified trade-offs between total information and the two measures of incomplete information (information imbalance and complete attribute pairs)
Complete Attribute
Pairs
Take-the-Best (TTB)
Total Information
Research Objectives and Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 21
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: EmpiricalEnvironment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Added new metrics: information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?
Q2: What aspects of the environment affect when the methods work?
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&Gmethods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristicinformation R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress Future work Future work Future Work
Sept. 3, 2015
Task 2. Develop methods of heuristic information acquisition
22
Task 2. Develop methods of heuristic information acquisition
Sept. 3, 2015 23
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?
Q2: What aspects of the environment affect when the methods work?
Choice
Decision Making Strategies
Task 2. Decision Making Simulation Engine (empirical environments)
Sept. 3, 2015Environments from 1Czerlinski et al., 1999, used in numerous simulations (e.g. 2Hogarth and Karelaia,
2006; and 3Katsikopoulos, 2013); 4Hogarth and Karelaia, 200724
Decision Task (36 million)
Incomplete Information
Environment
Artificial: randomly generated attribute scores and weights.
Accuracy is determined by a linear model that is not representative.
WADD
EW
Tallying
TTB
Two analytic strategies and two heuristic strategies.
Empirical: 15 standard benchmarking simulation environments that represent “real-world” non-linear environments1,2,3
Measure environmental parameters4 that have been shown to affect decision making accuracy.
Measure: total information, information imbalance, and complete attribute pairs
Examine 3, 4, and 5 attribute decision tasks.
Take Two
Two analytic strategies and three heuristic strategies.
4 Rules of Information Restriction and Guidance
Sept. 3, 2015 25
Restriction(Remove 1 piece of information)
Guidance(Add 1 piece of information)
Complete Attribute Pairs
Maintain the same number of complete attribute pairs
Increase the number of complete attribute pairs
Information Imbalance
Reduce information imbalance Reduce information imbalance
86.6 A1 A2 A3 A4
O1 K K K
O2 K K
89.674.6 83.2
67.9 77.6
Example: Reduce information while maintaining the number of complete attribute pairs
Sept. 3, 2015 26
86.6 A1 A2 A3 A4
O1 K K K ?
O2 K K ? ?
86.6 A1 A2 A3 A4
O1 -12.0 -3.4 +3.0 ?
O2 -18.7 -9.0 ? ?
Removing one piece of information in a way that keeps the same number of complete attribute pairs can increase accuracy.
Total information: 5Complete attribute pairs: 2
Total information: 4Complete attribute pairs: or 21
Total information: 4Complete attribute pairs: or 21
Initial decision task with incomplete information (initial accuracy: 86.6%)
Resulting accuracy of TTB(heuristic)
Change in accuracy of TTB(heuristic)
2
Sept. 3, 2015 27
If the decision maker is using Tallying, each instance
Results Summary: Reduce information while maintaining the number of complete attribute pairs
WADD EW Tallying Take Two TTB
AverageAccuracyChange
Complete Attribute Pairs – Restriction
73.0 A1 A2 A3 A4
O1 K K +3.0 ?
O2 K K ? ?
Total information: 4Complete attribute pairs: 2
Change in accuracy of TTB(heuristic)
Analytic Heuristic
Sept. 3, 2015 28
StrategyComplete
Attribute Pairs- Guidance
Complete Attribute Pairs
- Restriction
InformationImbalance- Guidance
InformationImbalance
- Restriction
WADD +3.9 +3.2 +1.4 -0.5
EW +4.1 +3.3 +2.0 -0.1
Tallying +4.6 +2.0 +9.8 +2.5
Take Two +5.0 +2.1 +7.3 +1.4
TTB +4.8 +2.7 +6.3 +1.4
If the decision maker is using Tallying, each instance
Task 2 Summary: Average accuracy change of all 4 information restriction and guidance methods
By reducing information or adding information in specific ways, the accuracy of decision making strategies can be increased – heuristic decision making strategies in particular.
Each value is average accuracy change that results from using the rule once when applicable(averaged across all 15 environments and most of the 36 million runs)
Analytic
Heuristic
Task 2. Develop methods of heuristic information acquisition
Sept. 3, 2015 29
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?• For heuristic strategies, the average accuracy change is generally positive when
using the methods.• For analytic strategies, the average accuracy change generally positive for the
complete attribute pairs methods.
Q2: What aspects of the environment affect when the methods work?The methods enable decision making strategies to achieve close to their full information accuracy. Therefore, in environments where the strategies perform well, the information restriction and guidance methods perform well.
Sept. 3, 2015
Current and Future Work
30
Research Objectives and Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015 31
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: EmpiricalEnvironment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Defined information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.
2. Work best in environments conducive to decision strategies performance
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&Gmethods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristicinformation R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress In-progress Future work Future Work
Task 3: Examine the effect of distributions of incomplete information
Sept. 3, 2015 32
Using the more comprehensive simulation environment developed for Task 2:Q1. Do the effects of complete attribute pairs and information imbalance generalize
to empirical simulation environments?Q2. Do any of the environmental parameters mediate the effect of incomplete
information?
50%
60%
70%
80%
0% 25% 50% 75% 100%
Information Imbalance
50%
60%
70%
80%
0% 25% 50% 75% 100%
Information Imbalance
Accuracy
Artificial Environment Empirical Environment
Task 4: Compare normative and heuristic information acquisition methods
Sept. 3, 2015 1Meder and Nelson, 2012; 2Canellas and Feigh, 2015; Table contains hypothetical results 33
NormativeInformation Acquisition
HeuristicInformation Acquisition
Quantify the value of each potential piece of
information1
Adapt the normative methods to my decision task, then use the simulation engine to study:
Q1. How often and in what cases do themethods agree or disagree as to what information should be acquired or not?
Q2. How do the methods differentially affect performance of decision making strategies?
Simple rules for determining which
information to acquire
Heuristic information
restriction and guidance methods2
A1 A2 A3
O1 K K ?
O2 K ? ?
?→K
?→K
WADD TTB
Normative +2% +3%
Heuristic +5% +9%
Research Objectives and Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015 34
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: EmpiricalEnvironment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Defined information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.
2. Work best in environments conducive to decision strategies performance
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&Gmethods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristicinformation R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress In-progress Future work Future Work
Task 5: Examine human decision making with incomplete information
Sept. 3, 2015 1Rieskamp and Hoffrage, 2008; 2Monti et al., 2012 35
Simulated Decision Making Strategies
Choice
Environment
Incomplete Information
Decision Task with Incomplete
Information
Human Decision Maker
Q. How do distributions of incomplete information on decision making accuracy extend to human subjects?
Method: • Present decision tasks with incomplete information
• Use varying time pressure to incentivize the use of heuristic or analytic decision making strategies1
• Design decision tasks with incomplete information such that decision strategies can be identified (process- and outcome-oriented methods)1,2
OptionTime to Target
Danger Control
A 10 ? ?
B 30 Low ?
Task 6: Implement the heuristic information acquisition methods in a decision support tool
Sept. 3, 2015 36
BASE
TARGET
B30 min
Autopilot
A10 minManual
Objective: Identify methods for implementing heuristic acquisition methods
Method: • Instead of tables of decision tasks being
presented to decision makers (Task 5), use context-relevant displays of information
• The planned task domain is a planning tool for controlling and operating a swarm of UAVs
Research Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015 37
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: EmpiricalEnvironment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Defined information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.
2. Work best in environments conducive to decision strategies performance
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&Gmethods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristicinformation R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress In-progress Future work Future Work
Sept. 3, 2015
Conclusions
38
Motivation
• Heuristics are often used when there is time pressure, high information acquisition costs, information overload, or ill-structured environments.
• A majority of previous decision support research does not apply to heuristic decision making
• There is potential for heuristic information acquisition to support decision making performance
Sept. 3, 2015 39
Contributions
Sept. 3, 2015 40
1. Define measures of incomplete information
2. Develop methods of heuristic information
restriction and guidance (R&G)
3: Examine the effect of distributions of
incomplete information
4: Compare heuristic information R&G
methods to normative methods
5: Examine the effect of distributions of
incomplete information on human decision
making
6: Implement the heuristic information
R&G methods in a decision support tool
A new understanding of heuristic decision
making with incomplete information
Completed In-Progress
How to implement this knowledge in
a decision support tool.
Future
Sept. 3, 2015
Acknowledgements• Dr. Karen Feigh, Dr. Zarrin Chua, and Rachel Haga for contributing to the research
• Members of the German Research Group and the Cognitive Engineering Center for their guidance and support
• The NSF Graduate Research Fellowship Program which funds my research
• The Office of Naval Research which funds the larger project which motivated this research
41