Upload
nico-nice
View
228
Download
5
Tags:
Embed Size (px)
Citation preview
The Decision Analysis and The Decision Analysis and Resolution (DAR) ProcessResolution (DAR) Process
Terry BahillSystems and Industrial EngineeringUniversity of [email protected]©, 2005-09, BahillThis file is located at
http://www.sie.arizona.edu/sysengr/slides/
Bahill2
CMMI The CMMI model is a collection of best practices
from diverse engineering companies. Improvements to our organization will come from
process improvements, not from people improvements or technology improvements.
CMMI provides guidance for improving an organization’s processes.
One of the CMMI process areas is Decision Analysis and Resolution, DAR.
Bahill3
DAR Programs and Functions select the decision
problems that require DAR and incorporate them in their program plans (e.g. SEMPs).
DAR is a BAE SYSTEMS common process. Common processes are tools that the user gets, customizes and uses.
DAR is invoked throughout the whole program lifecycle whenever a critical decision is to be made.
DAR is invoked by IPT leads on programs, financial analysts, program core teams, etc.
Invoke the DAR Process in Webster work instructions, in gate reviews, in phase reviews or with other triggers, which can be used anytime in the system life cycle.
Bahill4
WebsterBAE’s common processes are established by SP.12.15.02.
Bahill5
Typical decisions Decision problems that may require a formal
decision process Trade studies (eng_cat.shtml#GU0238) Bid/no-bid Make-reuse-buy (PW.10.01.01A017.html) Fagan inspection versus checklist inspection
(FM.05-1077.xls) Tool selection Vendor selection Cost estimating
Bahill6
Purpose“In all decisions you gain something and lose something. Know what they are and do it deliberately.”
Bahill7
A Simple Model for A Simple Model for Human Decision Making,Human Decision Making,
Called Image TheoryCalled Image Theory
Bahill8
References The following description of image theory is based
on Beach and Connolly (2005) and Bruce Gissing’s Roadmap to Business Excellence.
L. R. Beach and T. Connolly, The Psychology of Decision Making: People in Organizations, Sage Publications, Thousand Oaks, CA, 2005.
B. Gissing, The Roadmap to Business Excellence, http://sie.arizona.edu/sysengr/sie554/BruceGissing/RoadMap.ppt, 2005.
A. T. Bahill and B. Gissing, Re-evaluating systems engineering concepts using systems thinking, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, SMC-28(4): 516-527, 1998.
Bahill9
Image theory*
Decision Makers (DMs) code their knowledge into three images.
The value image contains principles of behavior. The trajectory image is the agenda of goals. The strategic image contains the plans for
implementing the goals.
Bahill10
The value imageconsists of the DM’s vision, mission, values, morals,
ethics, beliefs, evaluation criteria and standards for how things should be and how people ought to behave.
Collectively these are called principles. They limit
the goals that are worthy of pursuit and acceptable ways of pursuing these goals.
Potential goals and actions that contradict the principles will be unacceptable.
It is called the value image because it represents the DM’s vision about the state of events that conforms most closely to his or her principles.
Bahill11
The trajectory imageis the agenda of goals the DM wants to achieve. The goals are dictated by the problem statement,
principles, opportunities, desires, competitive issues and gaps encountered in the environment.
The goals are fed back to the value image. The DM’s goal agenda is called the trajectory
image, because it is his or her vision about how the future should unfold.
Bahill12
The strategic imagecontains the plans for implementing the goals. Each plan has two aspects:
tactics are the concrete behavioral aspects that deal with local environment conditions,
forecasts are the anticipation of the future that describe what might result if the tactics are successful.
The plans are also fed back to the value image. The collection of plans is called the strategic
image, because it represents the DM’s vision of what he or she is trying to do to achieve the goals on the trajectory image.
Bahill13
Framing*
means embedding observed events in a context that gives them meaning.
The DM uses contextual information to probe his or her memory to find image constituents that are relevant to the decision at hand.
This provides information about the goals and plans that were previously pursued in this context.
If a similar goal is being pursued this time, then the plan that was used before may be reused.
Bahill14
Two types of decisions Adoption decisions determine whether to add new
goals to the trajectory image or new plans to the strategic image.
Progress decisions determine whether a plan is making progress toward achieving a goal.
Bahill15
Adoption decisions A new goal or plan can be added if it is compatible
with the DM’s relevant principles, does not introduce unacceptable risk and does not interfere with existing goals or ongoing plans.
Adoption decisions are accomplished by screening potential goals and plans one by one in
light of relevant principles, existing goals and ongoing plans. If only one option passes screening, it is adopted.
If two or more options pass the screen, then a tradeoff study determines the best option from among the survivors.
Screening is the more common of these decision mechanism.
Bahill16
Progress decisionsuse the plan to forecast the future.
If that future includes achieving a goal, then the plan is retained.
If the forecast does not include achieving the goal, then the plan is rejected and a new plan is adopted in its place.
Bahill17
Two decision mechanisms The incompatibility test screens options based on
how well they fit the DM’s images. The profitability test focuses on the quality of the
outcomes associated with the options.
Bahill18
The incompatibility testscreens options (plans and goals) based on their
incompatibility with constituents* defined in the three images.
Each option’s incompatibility increases as a function of the weighted sum of the number of violations.**
Violations are defined as negations, contradictions, preventions, retardations or any other form of interference with the realization one of the images’ constituents.
If the weighted sum of the violations exceeds some rejection threshold, then the option is rejected, otherwise it is adopted.
Bahill19
Profitability test When more than one option survives the
incompatibility screen, the DM chooses the best using a profitability test.
The profitability test is not a single decision mechanism.
It is a repertory of strategies such as maximizing subjective expected utility, satisficing and performing tradeoff studies.
The selected strategy depends on characteristics of the choice, characteristics of the environment, characteristics of the DM.
Bahill20
Image theory for organizations*
Decisions in organizations are made by individual DMs, often forming a consensus.
So for organizational decisions, we can use the individual decision making model that we have just developed.
The only major addition is the need for a case for change.
Bahill21
The need for change*
People do not make good decisions.
A careful tradeoff study will help you overcome human ineptitude and thereby make better decisions.
Bahill22
Rational decisions*
One goal Perfect information The optimal course of action can be described This course maximizes expected value
This is a prescriptive model. We tell people that, in an ideal world, this is how they should make decisions.
Bahill23
Satisficing*
When making decisions there is always uncertainty, too little time and insufficient resources to explore the whole problem space.
Therefore, people cannot make rational decisions.
The term satisficing was coined by Noble Laureate Herb Simon in 1955.
Simon proposed that people do not attempt to find an optimal solution. Instead, they search for alternatives that are good enough, alternatives that satisfice.
Bahill24
Humans are not rational*1
Mark Twain said, “It ain’t what you don’t know that gets you into
trouble. It’s what you know for sure that just ain’t so.”
Humans are often very certain of knowledge that is false. What American city is directly north of Santiago
Chile? If you travel from Los Angeles to Reno Nevada, in
what direction would you travel? Most humans think that there are more words that
start with the letter r, than there are with r as the third letter.
Bahill25
Illusions*
We call these cognitive illusions. We believe them with as much certainty as we
believe optical illusions.
Bahill26
The Müller-Lyer Illusion*
Bahill27
Bahill28
ObjectiveProbability
SubjectiveProbability
EVRational Behavior V
Subjective Expected Value
Human Behavior
EExpected Utility
Value
Utility
Typical Estimate
0.00.0
1.0
1.0
Ideal Estimate
Ideal Estimate
1.00.00.0
1.0
Typical Estimate
Subjective Worth
Objective Value
Referencepoint
Gains
Losses
Objective Value
Subjective Worth Gains
LossesReference
point
Real Probability
Real Probability
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Bahill29
Humans judge probabilities poorly*
0.00.0
1.0
1.0
Ideal Estimate
Typical Estimate
Real Probability
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Bahill30
Monty Hall Paradox1*
Bahill31
Monty Hall Paradox2*
Bahill32
Monty Hall Paradox3*
Bahill33
Monty Hall Paradox4*
Bahill34
Monty Hall Paradox5*
Now here is your problem. Are you better off sticking to your original choice
or switching? A lot of people say it makes no difference. There are two boxes and one contains a ten-
dollar bill. Therefore, your chances of winning are 50/50. However, the laws of probability say that you
should switch.
Bahill35
Monty Hall Paradox6*
The box you originally chose has, and always will have, a one-third probability of containing the ten-dollar bill.
The other two, combined, have a two-thirds probability of containing the ten-dollar bill.
But at the moment when I open the empty box, then the other one alone will have a two-thirds probability of containing the ten-dollar bill.
Therefore, your best strategy is to always switch!
Bahill36
Utility We have just discussed the right column, subjective
probability. Now we will discuss the bottom row, utility
ObjectiveProbability
SubjectiveProbability
EVRational Behavior V
Subjective Expected Value
Human Behavior
EExpected Utility
Value
Utility
Typical Estimate
0.00.0
1.0
1.0
Ideal Estimate
Ideal Estimate
1.00.00.0
1.0
Typical Estimate
Subjective Worth
Objective Value
Referencepoint
Gains
Losses
Objective Value
Subjective Worth Gains
LossesReference
point
Real Probability
Real Probability
Sub
ject
ive
Pro
ba
bili
ty W
eig
htin
gS
ub
ject
ive
Pro
ba
bili
ty W
eig
htin
g
Bahill37
UtilityUtility is a measure of the happiness, satisfaction or reward a person gains (or loses) from receiving a good or service.
Utilities are numbers that express relative preferences using a particular set of assumptions and methods.
Utilities include both subjectively judged value and the assessor's attitude toward risk.
Bahill38
Risk Systems engineers use risk to evaluate and manage bad
things that could happen, hazards. Risk is measured with the frequency (or probability) of occurrence times the severity of the consequences.
However, in economics and in the psychology of decision making, risk is defined as the variance of the expected value, uncertainty.*
p1 x1 p2 x2 Risk, uncertainty
A 1.0 $10 $10 $0 none
B 0.5 $5 0.5 $15 $10 $5 medium
C 0.5 $1 0.5 $19 $10 $9 high
Bahill39
Ambiguity, uncertainty and hazards* Hazard: Would you prefer my forest picked
mushrooms or portabella mushrooms from the grocery store?
Uncertainty: Would you prefer one of my wines or a Kendall-Jackson merlot?
Ambiguity: Would you prefer my saffron and oyster sauce or marinara sauce?
Bahill40
Humans are not rational Even if they had the knowledge and resources,
people would not make rational decisions, because they do not evaluate utility rationally.
Most people would be more concerned with a large potential loss than with a large potential gain. Losses are felt more strongly than equal gains.
Which of these wagers would you prefer to take?*$2 with probability of 0.5 and $0 with probability 0.5$1 with probability of 0.99 and $1,000,000 with
probability 0.00000001$3 with probability of 0.999999 and -$1,999,997 with
probability 0.000001 They all have an expected value of $1
Bahill41
Gains and losses are not valued equally*
Gains
Losses
ObjectiveValue
Reference Point
SubjectiveWorth
Bahill42
Subjective expected utilitycombines two subjective concepts: utility and
probability. Utility is a measure of the happiness or
satisfaction a person gains from receiving a good or service.
Subjective probability is the person’s assessment of the frequency or likelihood of the event occurring.
The subjective expected utility is the product of the utility times the probability.
Bahill43
Subjective expected utility theorymodels human decision making as maximizing
subjective expected utility maximizing, because people choose the set of
alternatives with the highest total utility, subjective, because the choice depends on the
decision maker’s values and preferences, not on reality (e.g. advertising improves subjective perceptions of a product without improving the product), and
expected, because the expected value is used. This is a first-order model for human decision
making. Sometimes it is called Prospect Theory*.
Bahill44
ObjectiveProbability
SubjectiveProbability
EVRational Behavior V
Subjective Expected Value
Human Behavior
EExpected Utility
Value
Utility
Typical Estimate
0.00.0
1.0
1.0
Ideal Estimate
Ideal Estimate
1.00.00.0
1.0
Typical Estimate
Subjective Worth
Objective Value
Referencepoint
Gains
Losses
Objective Value
Subjective Worth Gains
LossesReference
point
Real Probability
Real Probability
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Bahill45
Why teach tradeoff studies? Because emotions, cognitive illusions, biases,
fallacies, fear of regret and use of heuristics make humans far from ideal decision makers.
Using tradeoff studies judiciously can help you make rational decisions.
We would like to help you move your decisions from the normal human decision-making lower-right quadrant to the ideal decision-making upper-left quadrant.
Bahill46
The Decision Analysis and Resolution The Decision Analysis and Resolution Proces (DAR)Proces (DAR)
Bahill47
Specific goals (SG)A specific goal applies to a process area and addresses the unique characteristics that describe what must be implemented to satisfy the process area. The specific goal for the DAR process area is
SG 1 Evaluate Alternatives.
Bahill48
Specific practices (SP) A specific practice is an activity that is
considered important in achieving the associated specific goal.
Practices are the major building blocks in establishing the process maturity of an organization.
Bahill49
SpecificPracticeNumber
DAR
Specific Practice Name
Example
1.1 Decide if formal evaluation process is warranted
When to do a trade study
1.2 Establish Evaluation Criteria
What is in a good trade study
1.3 Identify Alternative Solutions
1.4 Select Evaluation Methods
1.5 Evaluate Alternatives
1.6 Select Preferred Solutions
Bahill50
The Decision Analysis and Resolution (DAR) Process
SelectEvaluationMethods
EvaluateAlternatives
PreferredSolutions
SelectSolutions
EstablishEvaluation
Criteria
EvaluationCriteria
IdentifyAlternativeSolutions
ProposedAlternatives
SelectionProblem
Decide if Formal
Evaluation Process is Warranted
ProblemStatement S
Manage the DAR process
Recommendations
FormalEvaluations
These tasks are drawn serially, but they are not performed in a serial manner. Rather it is an iterative process with many unshown feedback loops.
Decision to Not Proceed
ExpertReview
Put in PAL
Present Results to Decision
Maker
Bahill51
When creating a process the most important facets are illustrating tasks that can be done in parallel suggesting feedback loops including a process to improve the process configuration management
Bahill52
A simple tradeoff study
Bahill53
Decisions Humans make four types of decisions:
Allocating resources among competing projects* Making plans, which includes scheduling Negotiating agreements Choosing amongst alternatives
Alternatives can be examined in series or parallel. When examined in series it is called sequential
search When examined in parallel it is called a tradeoff or
a trade study “Tradeoff studies address a range of problems
from selecting high-level system architecture to selecting a specific piece of commercial off the shelf hardware or software. Tradeoff studies are typical outputs of formal evaluation processes.”*
Bahill54
HistoryBen Franklin’s letter* to Joseph Priestly outlined one of the first descriptions of a tradeoff study.
Bahill55
Tradeoff Study ProcessTradeoff Study Process**
These tasks are drawn serially,but they are not performed in a serial manner. Rather, it is an iterative processwith many feedback loops, which are not shown.
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
∑
Bahill56
Decide if Formal Evaluation is NeededDecide if Formal Evaluation is Needed
Decide ifDecide if FormalFormalEvaluation isEvaluation is
Needed Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
Bahill57
Is formal evaluation needed? SP 1.1Companies should have polices for when to do
formal decision analysis. Criteria include When the decision is related to a moderate or high-
risk issue When the decision affects work products under
configuration management When the result of the decision could cause
significant schedule delays When the result of the decision could cause
significant cost overruns On material procurement of the 20 percent of the
parts that constitute 80 percent of the total material costs
Bahill58
Guidelines for formal evaluation, SP 1.1 When the decision is selecting one or a few
alternatives from a list When a decision is related to major changes in work
products that have been baselined When a decision affects the ability to achieve project
objectives When the cost of the formal evaluation is reasonable
when compared to the decision’s impact On design-implementation decisions when technical
performance failure may cause a catastrophic failure On decisions with the potential to significantly reduce
design risk, engineering changes, cycle time or production costs
Bahill59
Establish Evaluation CriteriaEstablish Evaluation Criteria
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Establish EvaluationEvaluation
CriteriaCriteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
Bahill60
Establish evaluation criteria* SP 1.2 Establish and maintain criteria for evaluating alternatives Each criterion must have a weight of importance Each criterion should link to a tradeoff requirement, i.e. a
requirement whose acceptable value can be more or less depending on quantitative values of other requirements.
Criteria must be arranged hierarchically. The top-level may be performance, cost, schedule and risk.
Program Management should prioritize these four criteria at the beginning of the project and make sure everyone knows the priorities.
All companies should have a repository of generic evaluation criteria.
Bahill61
What will you eat for lunch today? In class exercise. Write some evaluation criteria that will, help you
decide.*
Bahill62
Killer trades Evaluating alternatives is expensive. Therefore, early in tradeoff study, identify very
important requirements* that can eliminate many alternatives.
These requirements produce killer criteria.** Subsequent killer trades can often eliminate 90%
of the possible alternatives.
Bahill63
Identify Alternative SolutionsIdentify Alternative Solutions
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify Identify AlternativeAlternativeSolutionsSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
Bahill64
Identify alternative solutions, SP 1.3 Identify alternative solutions for the problem
statement Consider unusual alternatives in order to test the
system requirements* Do not list alternatives that do not satisfy all
mandatory requirements** Consider use of commercial off the shelf and in-
house entities***
Bahill65
What will you eat for lunch today? In class exercise. List some alternatives for today’s lunch.*
Bahill66
Select Evaluation MethodsSelect Evaluation Methods
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectSelectEvaluation Evaluation MethodsMethods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
Bahill67
Select evaluation methods, SP 1.4 Select the source of the evaluation data and the
method for evaluating the data Typical sources for evaluation data include
approximations, product literature, analysis, models, simulations, experiments and prototypes*
Methods for combining data and evaluating alternatives include Multi-Attribute Utility Technique (MAUT), Ideal Point, Search Beam, Fuzzy Databases, Decision Trees, Expected Utility, Pair-wise Comparisons, Analytic Hierarchy Process (AHP), Financial Analysis, Simulation, Monte Carlo, Linear Programming, Design of Experiments, Group Techniques, Quality Function Deployment (QFD), radar charts, forming a consensus and Tradeoff Studies
Bahill68
Collect evaluation data Using the appropriate source (approximations,
product literature, analysis, models, simulations, experiments or prototypes) collect data for evaluating each alternative.
Bahill69
Evaluate AlternativesEvaluate Alternatives
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateEvaluateAlternativesAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
Bahill70
Evaluate alternatives, SP 1.5Evaluate alternative solutions using the evaluation criteria, weights of importance, evaluation data, scoring functions and combining functions.
Evaluating alternative solutions involves analysis, discussion and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, or simulations may be needed to substantiate scoring and conclusions.
Bahill71
Select Preferred SolutionsSelect Preferred Solutions
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select Select PreferredPreferredSolutionsSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred Preferred SolutionsSolutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
Bahill72
Select preferred solutions, SP 1.6 Select preferred solutions from the alternatives
based on evaluation criteria. Selecting preferred alternatives involves weighing
and combining the results from the evaluation of alternatives. Many combining methods are available.
The true value of a formal decision process might not be listing the preferred alternatives. More important outputs are stimulating thought processes and documenting their outcomes.
A sensitivity analysis will help validate your recommendations.
Bahill73
Perform Expert ReviewPerform Expert Review
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
Perform Expert Review
Perform Expert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
∑
Bahill74
Perform expert review1
Formal evaluations should be reviewed* at regular gate reviews such as SRR, PDR and CDR or by special expert reviews
Technical reviews started about the same time as Systems Engineering, in 1960. The concept was formalized with MIL-STD-1521 in 1972.
Technical reviews are still around, because there is evidence that they help produce better systems at less cost.
The Perform Expert Review process is located at PS0303
Bahill75
Perform expert review2
Technical reviews evaluate the product of an IPT* They are conducted by a knowledgeable board of
specialists including supplier and customer representatives
The number of board members should be less than the number of IPT members
But board expertise should be greater than the IPT’s experience base
Bahill76
Who should come to the review? Program Manager Chief Systems Engineer Review Inspector Lead Systems Engineer Domain Experts IPT Lead Facilitator Stakeholders for this decision
Builder Customer Designer Tester PC Server
Depending on the decision, the Lead Hardware Engineer and the Lead Software Engineer
Bahill77
Present resultsPresent the results* of the formal evaluation to the original decision maker and other relevant stakeholders.
Bahill78
Put in the PAL Formal evaluations reviewed by experts should
be put in the organizational Process Asset Library (PAL) or the Project Process Asset Library (PPAL) (e.g. GDE 11 for M601)
Evaluation data for tradeoff studies come from approximations, analysis, models, simulations, experiments and prototypes. Each time better data is obtained the PAL should be updated.
Formal evaluations should be designed with reuse in mind.
Bahill79
Manage the DAR process The DAR Process Owner shall manage and
improve the DAR process. The DAR Process Owner will establish a change
control board and review the DAR Common Process on a regular basis. This is a high-level review of the DAR Common Process. This review must evaluate the activities, status and results of the DAR process. For instance, it might address use of and training for the many methods of performing DAR.
Bahill80
Closed Book Quiz, 5 minutes Closed Book Quiz, 5 minutes Fill in the empty boxesFill in the empty boxes
Problem StatementProblem
Statement
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
Formal Evaluations
Formal Evaluations
Preferred SolutionsPreferred Solutions∑
Bahill81
Tradeoff Study ExampleTradeoff Study Example
Bahill82
Example: What method should we use for evaluating alternatives?*
Is formal evaluation needed? SP 1.1 Check the Guidance for Formal Evaluations We find that many of its criteria are satisfied
including “On decisions with the potential to significantly reduce design risk … cycle time ...”
Establish evaluation criteria, SP 1.2 Ease of Use Familiarity
Killer criterion Engineers must think that use of the technique is
intuitive.
Bahill83
Example (continued)1 Identify alternative solutions, SP 1.3
Linear addition of weight times scores, Multiattribute Utility Theory (MAUT).* This method is often called a “trade study.” It is often implemented with an Excel spreadsheet.
Analytic Hierarchy Process (AHP)**
Bahill84
Example (continued)2 Select evaluation methods, SP 1.4
The evaluation data will come from expert opinion Common methods for combining data and
evaluating alternatives include: Multi-Attribute Utility Technique (MAUT),
Decision Trees, Analytic Hierarchy Process (AHP), Pair-wise Comparisons, Ideal Point, Search Beam, etc.
In the following slides we will use two methods: linear addition of weight times scores (MAUT) and the Analytic Hierarchy Process (AHP)*
Bahill85
Example (continued)3 Evaluate alternatives, SP 1.5
Let the weights and evaluation data be integers between 1 and 10, with 10 being the best. The computer can normalize the weights if necessary.
Bahill86
Multi-Attribute Utility Technique (MAUT)1
Criteria Weight of
Importance MAUT AHP
Ease of Use
8 4
Familiarity Sum of weight times score
Assess evaluation data* row by row
Bahill87
Multi-Attribute Utility Technique (MAUT)2
Criteria Weight* of Importance
MAUT AHP
Ease of Use
9 8 4
Familiarity 3 9 2 Sum of weight times score
99 42
The
winner
Bahill88
Analytic Hierarchy Process (AHP)
Verbal scale Numerical
value Equally important, likely or preferred
1
Moderately more important, likely or preferred
3
Strongly more important, likely or preferred
5
Very strongly more important, likely or preferred
7
Extremely more important, likely or preferred
9
Verbal scale Numerical
value Equally important, likely or preferred
1
Moderately more important, likely or preferred
3
Strongly more important, likely or preferred
5
Very strongly more important, likely or preferred
7
Extremely more important, likely or preferred
9
Bahill89
AHP, make comparisonsCreate a matrix with the criteria on the diagonal and
make pair-wise comparisons*
Ease of Use Ease of Use is moderately more important than Familiarity (3)
Reciprocal of 3 = 1/3 Familiarity
Bahill90
AHP, compute weights Create a matrix Square the matrix Add the rows Normalize*
1 1 23 3 3
1 3 1 3 2 6 8
1 1 2 2
0.7
. 5.6
5
0 27
1 1 23 3 3
1 3 1 3 2 6 8
1 1 2 2
0.7
. 5.6
5
0 27
Bahill91
In-class exercise Use these criteria to help select your lunch today.
Closeness, distance to the venue. Is it in the same building, the next building or do you have to get in a car and drive?
Tastiness, including gustatory delightfulness, healthiness, novelty and savoriness.
Price, total purchase price including tax and tip.
Bahill92
To help select lunch today1
closeness is ??? more important than tastiness, closeness is ??? more important than price, tastiness is ??? more important than price.
Closeness Tastiness Price
Closeness
Tastiness
Price
Bahill93
To help select lunch today2
closeness is strongly more important (5) than tastiness,
closeness is very strongly more important (7) than price,
tastiness is moderately more important (3) than price.
Closeness Tastiness Price
Closeness 1 5 7
Tastiness 1 3
Price 1
Bahill94
To help select lunch today3
1 5 7 1 5 7
3 12.3 29 44.3 0.731 1
1 3 1 3 0.8 3 7.4 11.2 0.195 5
0.4 1.4 3 4.8 0.081 1 1 1
1 17 3 7 3
Closeness Tastiness Price Weight of Importance
Closeness 1 5 7 0.73
Tastiness 1/5 1 3 0.19
Price 1/7 1/3 1 0.08
Bahill95
AHP, get scores Compare each alternative on the first criterion
1 12 2
1 2 1 2 2 4 6
1 1 1 2 3
0.67
0.33
1 12 2
1 2 1 2 2 4 6
1 1 1 2 3
0.67
0.33
Ease of Use MAUT In terms of Ease
of Use, MAUT is slightly preferred (2)
1/2 AHP
Bahill96
AHP, get scores2 Compare each alternative on the second criterion
1 15 5
1 5 1 5 2 10 0.83
0.17
12
1 1 0.4 2 2.4
1 15 5
1 5 1 5 2 10 0.83
0.17
12
1 1 0.4 2 2.4
Familiarity MAUT In terms of
Familiarity, MAUT is strongly preferred (5)
1/5 AHP
Bahill97
AHP, form comparison matrix**
Combine with linear addition*
Criteria Weight of
Importance MAUT AHP
Ease of Use
0.75 0.67 0.33
Familiarity 0.25 0.83 0.17 Sum of weight times score
0.71 0.29
The
winner
Bahill98
Example (continued)4
Select Preferred Solutions, SP 1.6 Linear addition of weight times scores (MAUT)
was the preferred alternative Now consider new criteria, such as Repeatability
of Result, Consistency*, Time to Compute Do a sensitivity analysis
Bahill99
Sensitivity analysis, simpleIn terms of Familiarity, MAUT was strongly preferred (5) over the AHP. Now change this 5
to a 3 and to a 7.
• Changing the scores for Familiarity does not change the recommended alternative.
• This is good.• It means the Tradeoff study is robust with
respect to these scores.
Final Score Familiarity MAUT AHP
3 0.69 0.31 5 0.71 0.29 7 0.72 0.28
Bahill100
Sensitivity analysis, analyticCompute the six semirelative-sensitivity functions, which are defined as
which reads, the semirelative-sensitivity function of the performance index F with respect to the parameter is the partial derivative of F with respect to times with everything evaluated at the normal operating point (NOP).
F
NOP
FS
FNOP
FS
Bahill101
Sensitivity analysis2
For the performance index use the alternative rating for MAUT minus the alternative rating for AHP*
F = F1 - F2 = Wt1×S11 + Wt2×S21 – Wt1×S12 –Wt2×S22
Criteria Weight of
Importance MAUT AHP
Ease of Use
Wt1 S11 S12
Familiarity Wt2 S21 S22 Sum of weight times score
F1 F2
Bahill102
Sensitivity analysis3
The semirelative-sensitivity functions*
1
2
11
21
12
22
11 12 1
21 22 2
1 11
2 21
1 12
2 22
0.26
0.16
0.50
0.21
-0.25
-0.04
FWt
FWt
FS
FS
FS
FS
S S S Wt
S S S Wt
S Wt S
S Wt S
S Wt S
S Wt S
S11 is the most importantparameter. So go back and reevaluate it.
Bahill103
Sensitivity analysis4
The most important parameter is the score for MAUT on the criterion Ease of Use
We should go back and re-evaluate the derivation of that score
Ease of Use MAUT In terms of Ease
of Use, MAUT is slightly preferred (2)
1/2 AHP
Bahill104
The Decision Analysis and Resolution (DAR) Process
SelectEvaluationMethods
EvaluateAlternatives
PreferredSolutions
SelectSolutions
EstablishEvaluation
Criteria
EvaluationCriteria
IdentifyAlternativeSolutions
ProposedAlternatives
SelectionProblem
Decide if Formal
Evaluation Process is Warranted
ProblemStatement S
Manage the DAR process
Recommendations
FormalEvaluations
These tasks are drawn serially, but they are not performed in a serial manner. Rather it is an iterative process with many unshown feedback loops.
Decision to Not Proceed
ExpertReview
Put in PAL
Present Results to Decision
Maker
Bahill105
Example (continued)5
Perform expert review of the tradeoff study. Present results to original decision maker. Put tradeoff study in PAL. Improve the DAR process.
Add some other techniques, such as AHP, to the DAR web course
Fix the utility curves document Add image theory to the DAR process Change linkages in the documentation system Create a course, Decision Making and Tradeoff Studies
Bahill106
Quintessential exampleA Tradeoff Study of Tradeoff Study Tools
is available at
http://www.sie.arizona.edu/sysengr/sie554/tradeoffStudyOfTradeoffStudyTools.doc
Bahill107
Generic goals (GG) Achievement of a generic goal in a process
area signifies improved control in planning and implementing the processes associated with that process area. Generic goals are called “generic” because the
same goal statement appears in (almost) all process areas.
Each process area has only one generic goal for each maturity level.
And the generic goal is different for each maturity level.
Bahill108
Maturity level 2 generic goal GG 2: The DAR process is institutionalized as
a managed process. A managed process is a performed process
that is planned and executed in accordance with policy; employs skilled people having adequate resources to produce controlled outputs; involves relevant stakeholders; is monitored, controlled, and reviewed; and is evaluated for adherence to its process description.
Bahill109
Maturity level 3 generic goal GG 3 The DAR process is institutionalized as a
defined process. A defined process is establish by tailoring the
selected process according to the organization’s tailoring guidelines to meet the needs of a project or organizational function. With a defined process, variability in how the process is performed across the organization is reduced and process assets, data, and learning can be effectively shared.
Bahill110
Generic practices (GP) Generic practices contribute to the achievement
of the generic goal when applied to a particular process area.
Generic practices are activities that ensure that the processes associated with the process area will be effective, repeatable, and lasting.
Bahill111
Generic practices1
GP 2.1: Establish an Organizational Policy,
Establish and maintain an organizational policy for planning and performing the DAR process.
The BAE solution SP.12.15.02 Organizational Business Practices
OM.12.15.02A001 Perform Decision Analysis and ResolutionRW.12.01.00A004 Perform Formal Evaluation
RF 1 Quantitative Methods for Tradeoff Analyses.doc…RF 12 Manage and Improve the DAR Process.doc
These documents are located at Users at Bluelnk\Bludfs001\Shared\Users\Bahill_AT\Draft DAR Process Docs
And O:\ENGR_LIB\SysPCRDocs\Reference Docs
Bahill112
Generic practices2
GP 3.1 Establish and maintain the description of a defined decision analysis and resolution process.
BAE company compliance documents SP.12.15.02 Organizational Business Practices
OM.12.15.02A001 Perform Decision Analysis and ResolutionRW.12.01.00A004 Perform Formal Evaluation
BAE program implementation evidenceTailoring reports, program plans and trade studies
with evidence of use of SP 1.2 to 1.6.
Bahill113
Generic practices3
GP 2.2: Plan the Process,
Establish and maintain the plan for performing the DAR process.
Bahill114
Generic practices4
GP 2.3: Provide Resources,
Provide adequate resources for performing the DAR process, developing the work products, and providing the services of the process.
GP 2.4: Assign Responsibility,
Assign responsibility and authority for performing the process, developing the work products, and providing the services of the DAR process.
GP 2.5: Train People,
Train the people performing or supporting the DAR process as needed.
Bahill115
Generic practices5 GP 2.6: Manage Configurations,
Place designated work products of the DAR process under appropriate levels of configuration management.
GP 2.7: Identify and Involve Relevant Stakeholders,
Identify and involve the relevant stakeholders of the DAR process as planned.
GP 2.8: Monitor and Control the Process,
Monitor and control the DAR process against the plan for performing the process and take appropriate corrective action.
Bahill116
Generic practices6
GP 3.2 Collect Improvement Information such as work products, measures, measurement results, and improvement information derived from planning and performing the decision analysis and resolution process to support the future use and improvement of the organization’s processes and process assets.
Bahill117
Generic practices7
GP 2.9: Objectively Evaluate Adherence,
Objectively evaluate adherence of the DAR process against its process description, standards, and procedures, and address noncompliance.
GP 2.10: Review Status with Higher Level Management,
Review the activities, status, and results of the DAR process with higher level management and resolve issues.
Bahill118
ExampleExamples of trade studies are given in
O:\ENGR_LIB\DAR\DAR Training\Web-based DAR Course\dar_index.html
Bahill119
Webster Tradeoff Study ReferencesUtility Curves (Trade-off Study) FM.05-994Evaluate Design Solutions RW.12.13.14A010Trade-off Study Matrix (template) FM.05-949
Bahill120
Webster DAR ReferencesOrganizational Business Practices SP.12.15.02Perform Decision Analysis and Resolution OM.12.15.02A001
Perform Formal Evaluation RW.12.01.00A004RF.QM Tradeoff AnalysesRF.Decide Formal EvaluationRF.Guide Formal EvaluationsRF.Other DAR MethodsRF.Establish Evaluation CriteriaRF.ID Alternative SolutionsRF.Select Evaluation MethodsRF.Evaluate AlternativesRF.Select Preferred SolutionsRF.Expert Review of Trade off StudiesRF.Retention Formal DecisionsRF.Manage Improve DAR
Bahill121
Bahill122
How to print To print this file, do this one time. View Color/grayscale Grayscale Settings Light grayscale Close grayscale view