Decision Making Guidebook

7/31/2019 Decision Making Guidebook

1/44

WSRC-IM-2002-00002

GUIDEBOOK TO

DECISION-MAKING

METHODS

Developed for the Department of Energy

Through a complex-wide collaborative effort under the auspices of the

Nuclear Materials Stewardship Initiative by:

Dennis Baker, WSRC/SRS

Donald Bridges, DOE-SR

Regina Hunter, SNL

Gregory Johnson, DOE-SR

Joseph Krupa, WSRC/SR

James Murphy, INEEL

Ken Sorenson, SNL

December 2001


2/44

Guidebook to DecisionMaking Methods WSRC-IM-2002-00002

ii

DISCLAIMER

This report was prepared as an account of work sponsored by an agency ofthe United States Government. Neither the United States Government nor anyagency thereof, nor any of their employees, makes any warranty, express or

implied, or assumes any legal liability or responsibility for the accuracy,completeness, or usefulness of any information, apparatus, product, or proc-ess disclosed, or represent that its use would not infringe privately ownedrights. Reference herein to any specific commercial product, process, orservice by trade name, trademark, manufacturer, or otherwise does not neces-sarily constitute or imply its endorsement, recommendation, or favoring bythe United States Government or any agency thereof. The views and opinionsof authors expressed herein do not necessarily state or reflect those of the

United States Government or any agency thereof.


3/44


iii

PREFACE

In the Affair of so much Importance to you, wherein you ask my Advice, I cannot for want ofsufficient Premises, advise you whatto determine, but if you please I will tell you how. When those

difficult Cases occur, they are difficult, chiefly because while we have them under Consideration,all the Reasons pro and con are not present to the Mind at the same time; but sometimes one Setpresent themselves, and at other times another, the first being out of Sight. Hence the various Pur-poses or Inclinations that alternately prevail, and the Uncertainty that perplexes us.

To get over this, my Way is, to divide half a Sheet of Paper by a Line into two Columns; writ-ing over the one Pro, and over the other Con. Then during three or four Days Consideration, I putdown under the different Heads short Hints of the different Motives, that at different Times occur tome,foror againstthe Measure. When I have thus got them all together in one View, I endeavor toestimate their respective Weights; and where I find two, one on each side, that seem equal, I strikethem both out. If I find a Reason pro equal to some two Reasons con, I strike out the three. If I

judge some two Reasons con, equal to some three Reasons pro, I strike out the five; and thus pro-ceeding I find at length where the Balance lies; and if after a Day or two of farther consideration,nothing new that is of Importance occurs on either side, I come to a Determination accordingly.

And, tho' the Weight of Reasons cannot be taken with the Precision of Algebraic Quantities, yet,when each is thus considered, separately and comparatively, and the whole lies before me, I think Ican judge better, and am less liable to make a rash Step; and in fact I have found great Advantagefrom this kind of Equation, in what may be called Moral or Prudential Algebra.1

B. Franklin, London, September 19, 17721

1 Appendix C-Further Reading 1,Forman and Selly

This guidebook introduces both a process and a selection of proven methods for disciplined decision-making so that the results are clearer, more transparent, and easier for reviewers to understand and accept.It was written to implement Recommendation 14 of the Integrated Nuclear Materials Management(INMM) Plan and set a standard for a consistent decision process. From this guidebook decision-maker(s) and their support staffs will learn:

the benefits of using a disciplined decision-making approach

prerequisites to the decision-making process

how to choose among several decision-making methods

how to apply the method chosen

This guidebook also presents examples of the decision-making methods in action and recommendssources of additional information on decision-making methods

This guidebook was compiled with input from a team experienced in the decision-making process fromthe Savannah River Site, Sandia National Laboratories, Idaho National Engineering and EnvironmentalLaboratory, and the U.S. Department of Energy.


4/44


iv

CONTENTS

PREFACE......................................................................................................................................................................................III

1.0 PURPOSE..........................................................................................................................................................................1

1.1 WHAT IS A DISCIPLINED DECISION-MAKING PROCESS?.................................................................................................11.2 WHY USE A DISCIPLINED DECISION-MAKING PROCESS?...............................................................................................11.3 WHEN SHOULD A FORMAL DECISION -MAKING METHOD BE USED?.............................................................................1

2.0 DECISION-MAKING PROCESS..............................................................................................................................2

2.1 STEP 1, DEFINE THE PROBLEM ........................................................................................................................................32.2 STEP 2, DETERMINE REQUIREMENTS..............................................................................................................................3

2.3 STEP 3, ESTABLISH GOALS...............................................................................................................................................42.4 STEP 4, IDENTIFY ALTERNATIVES................................................................................................................................... 42.5 STEP 5, DEFINE CRITERIA ................................................................................................................................................ 4

2.6 STEP 6, SELECT A DECISION-MAKING TOOL.................................................................................................................52.7 STEP 7, EVALUATE ALTERNATIVES AGAINST CRITERIA .............................................................................................5

2.8 STEP 8, VALIDATE SOLUTION(S) AGAINST PROBLEM STATEMENT ...........................................................................5

3.0 DECISION MAKING METHODS ............................................................................................................................6

3.1 PROS AND CONS ANALYSIS..............................................................................................................................................63.2 KEPNER-TREGOE (K-T)DECISION ANALYSIS..............................................................................................................6

3.3 ANALYTIC HIERARCHY PROCESS (AHP)........................................................................................................................73.4 MULTI-ATTRIBUTE UTILITY THEORY (MAUT)............................................................................................................83.5 COST-BENEFIT ANALYSIS................................................................................................................................................. 9

3.6 CUSTOM TAILORED TOOLS ..............................................................................................................................................9

4.0 SUMMARY..................................................................................................................................................................... 10

APPENDIX A DECISION-MAKING TOOLS AT WORK.........................................................................................11

APPENDIX B DECISION PROCESS AIDS .................................................................................................................... 27

APPENDIX C FURTHER READING................................................................................................................................40


5/44


1

1.0 Purpose

Decision-makers have to choose between alterna-tive actions every day. Often the alternatives andsupporting information presented is inadequate to

support or explain the recommended action. Thegoal of the Guidebook to Decision-Making Meth-ods is to help decision-makers and the decision

support staff choose and document the best al-ternative in a clear and transparent fashion.This guidebook will help all parties concernedknow what questions to ask and when to ask them.

1.1 What is a disciplined decision-making

process?

Good decisions can best be reached when everyone

involved uses a clearly defined and acknowledgeddecision-making process. A clear and transparentdecision process depends on asking and answeringenough questions to ensure that the final report willclearly answer the questions of reviewers andstakeholders. This guidebook provides:

An eight step decision-making process (Sec-tion 2)

Descriptions of specific decision methods(Section 3)

Examples of the specific decision methods in

action (Appendix A) Written aids, suggestions, and questions to

help implement the decision-making process(Appendix B), and

Supporting references for further reading (Ap-pendix C).

1.2 Why use a disciplined decision-making process?

For most familiar everyday problems, decisionsbased on intuition can produce acceptable results

because they involve few objectives and only oneor two decision-makers. In the DOE environment,problems are more complex. Most decisions in-volve multiple objectives, several decision-makers,and are subject to external review. A disciplined

and transparent decision-making process employ-ing credible evaluation methods will provide:

Structure to approach complex problems Rationale for decisions Consistency in the decision making process Objectivity Documented assumptions, criteria, and values

used to make decisions. and

Decisions that are repeatable, reviewable, re-visable, and easy to understand

Using such a disciplined approach can help avoidmisunderstandings that lead to questions about thevalidity of the analyses and ultimately slow prog-ress. Its use will set a baseline for continuous im-provement in decision making in the DOEnuclear

materials complex.

1.3 When should a formal decision-

making method be used?

The decision-making methods described in thisguidebook are readily applicable to a wide range ofdecisions, from ones as simple as picking a restau-rant for a special meal to those that are complicatedby interdepartmental government interfaces. Useof this decision-making process and supportingmethods is recommended any time decisions:

Require many reviews at different managementlevels

Involve more than one program

Require congressional line item approval

Affect new or redirected funding

Require approval for new facilities or upgrades toexisting facilities

Have alternatives with high technical risk

Have alternatives that appear equally viable

Require a decision to revise or discontinue work ona program

Have impact mainly in the future

Involve multiple or competing drivers, or Define data needed to support future decisions

In short this guide should be followed any time aclear, transparent, and understandable decision isdesired.


6/44


2

2.0 Decision-Making Process

First priority in making a decision is to establish

who are the decision-maker(s) and stakeholders inthe decision -the audience for the decision. Identi-fying the decision-maker(s) early in the processcuts down on disagreement about problem defini-tion, requirements, goals, and criteria.

Although the decision-maker(s) seldom will beinvolved in the day-to-day work of making evalua-tions, feedback from the decision-maker(s) is vitalat four steps in the process:

1. Problem definition [step 1]

2. Requirements identification [step 2]3. Goal establishment [step 3]4. Evaluation criteria development [step 5]

When appropriate, stakeholders should also beconsulted. By acquiring their input during theearly steps of the decision process, stakeholderscan provide useful feedback before a decision ismade.

Figure 1 shows the steps in the decision-makingprocess. The process flows from top to bottom,

but may return to a previous step from any point inthe process when new information is discovered.

It is the decision teams job to make sure that allsteps of the process are adequately performed.

Usually the decision support staff should in-clude the help of skilled and experienced ana-

lysts/facilitators to assist with all stages of thedecision process. Expert facilitation can help as-sure that all the steps are properly performed anddocumented. Their experience and expertise willhelp provide transparency to the decision making

process and help avoid misunderstandings that of-ten lead to questions about the validity of theanalyses which ultimately slow progress.

STEP 1

Define problem

STEP 2Determine the require-

ments that the solution to

the problem must meet

STEP 3Establish goals that solv-ing the problem should

accomplish

STEP 4Identify alternatives thatwill solve the problem

STEP 5Develop evaluation crite-

ria based on the goals

STEP 6Select a decision-making

tool

STEP 7Apply the tool to select a

preferred alternative

STEP 8Check the answer

to make sure itsolves the problem

Figure 1 General DecisionMaking Process


7/44


3

2.1 Step 1, Define the ProblemProblem definition is the crucial first step in mak-ing a good decision. This process must, as a mini-mum, identify root causes, limiting assumptions,system and organizational boundaries and inter-

faces, and any stakeholder issues. The goal is toexpress the issue in a clear, one-sentence prob-lem statement that describes both the initialconditions and the desired conditions. It is es-sential that the decision-maker(s) and support staffconcur on a written problem statement to ensurethat they all agree on what problem is going to besolved before proceeding to the next steps.

The key to developing an adequate problem state-ment is to ask enough questions about the problemto ensure that the final report will clearly answer

the questions of reviewers and stakeholders (seeFigure 2 below). When stakeholders are involved,it may be appropriate to have them review theproblem statement with its initial and desired stateto provide an external check before requirementsand goals are defined.

Some questions which may be helpful to the proc-ess are suggested Appendix B. For more informa-

tion, the reader can consult texts on problem defi-nition available from the business press.2

2.2 Step 2, Determine Requirements

2. Identify

Requirements &Goals

Problem statement:

functions, initialconditions,

desired state, etc.

List of absolute

requirements andgoals

Requirements are conditions that any acceptablesolution to the problem must meet. Requirementsspell out what the solution to the problem mustdo.For example, a requirement might be that a processmust (shall in the vernacular of writing require-ments) produce at least ten units per day. Any al-ternatives that produced only nine units per daywould be discarded. Requirements that dont dis-criminate between alternatives need not be used atthis time.

With the decision-makers concurrence, experts inoperations, maintenance, environment, safety,health and other technical disciplines typicallyprovide the requirements that a viable alternativemust meet. Aids for identifying requirements ap-pear in Appendix B. For more information, thereader can consult texts on requirements manage-ment available from the business press.3

__________________________________

2Appendix C-Further Reading 2, Folger and L eBlancand 3, Gause

3 Appendix C-Further Reading 4, Hammond, Keeney and

Raffia, and 5, Hooks and Farry

3

Reported

Symptom(s)

Customerand KeyStakeholder

Agreement

Root Cause

Problem Analysis:

Analyze conditions

Restate problem infunctional terms

Understand the

s stem

Identify possiblecauses

Determine rootcause

DocumentProblemOR

Reported

Problem

Identified

Problem

Clearly

Defined

Problem

Statement

no

yes

Figure 2. Problem Defini t ion:

Ask enough quest ions to be able to answer quest ions from others.


8/44


4

2.3 Step 3, Establish Goals

Goals are broad statements of intent and desirableprogrammatic values. Examples might be: reduceworker radiological exposure, lower costs, lowerpublic risk, etc. Goals go beyond the minimumessential must haves (i.e. requirements) to wants

and desires. Goals should be stated positively (i.e.what something should do, not what it shouldntdo). Because goals are useful in identifying supe-rior alternatives (i.e., define in more detail the de-sired state of the problem), they are developedprior to alternative identification.

Sometimes goals may conflict, but this is neitherunusual, nor cause for concern. During goal defi-nition, it is not necessary to eliminate conflictamong goals nor to define the relative importanceof the goals. The process of establishing goals may

suggest new or revised requirements or require-ments that should be converted to goals. In anycase, understanding the requirements and goals isimportant to defining alternatives. Aids for identi-fying goals appear in Appendix B.

2.4 Step 4, Identify Alternatives

4. Define

Alternatives

Problemstatement,

requirements,and goals

Potentialalternativesolutions

Alternatives offer different approaches for chang-ing the initial condition into the desired condition.The decision team evaluates the requirements andgoals and suggests alternatives that will meet therequirements and satisfy as many goals as possible.Generally, the alternatives vary in their ability tomeet the requirements and goals. Those alterna-tives that do not meet the requirements must bescreened out from further consideration. If an al-

ternative does not meet the requirements, three ac-tions are available:

1. The alternative is discarded2. The requirement is changed or eliminated3. The requirement is restated as a goal

The description of each alternative must clearlyshow how it solves the defined problem and how itdiffers from the other alternatives. A written de-scription and a diagram of the specific functionsperformed to solve the problem will prove useful.Aids for identifying alternatives appear in Appen-dix B.

2.5 Step 5, Define Criteria

5. DefineDiscriminating

Criteria

Problem statement,requirements,goals, andalternatives

Criteria withdefined measuresof effectiveness

Usually no one alternative will be the best for allgoals, requiring alternatives to be compared witheach other. The best alternative will be the one thatmost nearly achieves the goals. Decision criteriawhich will discriminate among alternatives mustbe based on the goals. It is necessary to define dis-criminating criteria as objective measures of thegoals to measure how well each alternativeachieves the project goals.

Each criterion should measure something impor-tant, and not depend on another criterion. Criteriamust discriminate among alternatives in a mean-ingful way (e.g., if the color of all alternatives isthe same or the user is indifferent to the color se-lection, then color should not be a criterion)4.

Criteria should be:

Able to discriminate among the alternatives Complete include all goals Operational meaningful to the decision

makers understanding of the implications ofthe alternatives

Non-redundant avoid double counting Few in number to keep the problem dimen-

sions manageable

Using a few real discriminators will result in amore understandable decision analysis product.However, every goal must generate at least one___________________________________

A summary of INMM goals and criteria appears in

Appendix B.


9/44


5

criterion. If a goal does not suggest a criterion, itshould be abandoned.

Several methods can be used to facilitate criteriaselection.

Brainstorming: Team brainstorming may be used

to develop goals and associated criteria. (Brain-storming is discussed in Appendix B.)

Round Robin: Team members are individuallyasked for their goals and the criteria associatedwith them. The initial elicitation of ideas shouldbe done non-judgmentally all ideas are recordedbefore criticism of any is allowed.

When members of the goal-setting group differwidely in rank or position, it can be useful to em-ploy the military method in which the lowest

ranking member is asked first to avoid being in-fluenced by the opinions of the higher-rankingmembers.

Reverse Direction Method: Team members con-sider available alternatives, identify differencesamong them, and develop criteria that reflectthese differences.

Previously Defined Criteria: End users,stakeholders, or the decision-maker(s) may pro-vide criteria.

Input from the decision-maker(s) is essential to

the development of useful criteria. Moreover,the decision-makers approval is crucial before

the criteria are used to evaluate the alterna-tives. Additional aids for defining criteria appearin Appendix B.

2.6 Step 6, Select a Decision-Making Tool

Section 3.0 introduces and describes these widelyemployed tools:

Pros and Cons Analysis Kepner-Tregoe Decision Analysis (K-T)

Analytic Hierarchy Process (AHP) Multi-Attribute Utility Theory Analysis (MAUT) Cost Benefit Analysis (CBA) Custom Tailored Tools

Some of these methods can be complicated anddifficult to apply. The method selection needs tobe based on the complexity of the problem andthe experience of the team. Generally, the simplerthe method, the better. More complex analysescan be added later if needed. Appendix A pro-vides step-by-step examples of these methods.

2.7 Step 7, Evaluate Alternatives againstCriteria

7. EvaluateAlternatives

Against Criteria

Collectedcriteria data foreachAlternative(model data,

research data)

Alternativeswith definedmeasures of

effectiveness

Alternatives can be evaluated with quantitative

methods, qualitative methods, or any combina-tion. Criteria can be weighted and used to rankthe alternatives. Both sensitivity and uncertaintyanalyses can be used to improve the quality of theselection process. Experienced analysts can pro-vide the necessary thorough understanding of themechanics of the chosen decision-making meth-odology. The step-by-step examples in AppendixA suggest some methods for performing theseevaluations. Additional aids for evaluating alter-natives appear in Appendix B.

2.8 Step 8, Validate Solution(s) against

Problem Statement

After the evaluation process has selected a pre-ferred alternative, the solution should be checkedto ensure that it truly solves the problem identi-fied. Compare the original problem statement tothe goals and requirements. A final solutionshould fulfill the desired state, meet requirements,and best achieve the goals within the values of thedecision makers. Once the preferred alternative

has been validated, the decision-making supportstaff can present it as a recommendation to thedecision-maker(s). A final report to the decision-maker(s) must be written documenting the deci-sion process, assumptions, methods, and conclu-sions recommending the final solution. AppendixB provides suggestions for the final report outline.


10/44


6

3.0 Decision Making Methods

Decision Analysis techniques are rational proc-esses/systematic procedures for applying criticalthinking to information, data, and experience inorder to make a balanced decision when the choicebetween alternatives is unclear. They provide or-ganized ways of applying critical thinking skillsdeveloped around accumulating answers to ques-tions about the problem. Steps include clarifyingpurpose, evaluating alternatives, assessing risksand benefits, and making a decision. These stepsusually involve scoring criteria and alternatives.This scoring (a systematic method for handling andcommunicating information) provides a commonlanguage and approach that removes decisionmaking from the realm of personal preference oridiosyncratic behavior (see Appendix B for scoringand weighting options).

The evaluation methods introduced here arehighly recommended. They are adaptable tomany situations, as determined by the complexityof the problem, needs of the customer, experienceof the decision team/analysts/facilitators, and thetime and resources available. No one decision-making method is appropriate for all decisions.The examples provided in Appendix A are in-tended to facilitate understanding and use of thesemethods.

3.1 Pros and Cons Analysis

Pros and Cons Analysis is a qualitative comparisonmethod in which good things (pros) and bad things(cons) are identified about each alternative. Lists ofthe pros and cons, based on the input of subjectmatter experts, are compared one to another foreach alternative. (See B. Franklins description onpage ii and the example in Appendix A.) The al-ternative with the strongest pros and weakest consis preferred. The decision documentation shouldinclude an exposition, which justifies why the pre-

ferred alternatives pros are more important and itscons are less consequential than those of the otheralternatives.

Pros and Cons Analysis is suitable for simple deci-sions with few alternatives (2 to 4) and few dis-criminating criteria (1 to 5) of approximately equalvalue. It requires no mathematical skill and can beimplemented rapidly.

3.2 Kepner-Tregoe (K-T) DecisionAnalysis

K-T is a quantitative comparison method in whicha team of experts numerically score criteria andalternatives based on individual judge-ments/assessments. The size of the team neededtends to be inversely proportional to the quality ofthe data available the more intangible and quali-tative the data, the greater the number of peoplethat should be involved.

In K-T parlance each evaluation criterion is firstscored based on its relative importance to the othercriteria (1 = least; 10 = most). These scores be-come the criteria weights (see K-T example in Ap-pendix A). Once the WANT objectives (goals)[have] been identified, each one [is] weighted ac-

cording to its relative importance. The most im-portant objective [is] identified and given a weightof 10. All other objectives [are] then weighted incomparison with the first, from 10 (equally impor-tant) down to a possible 1 (not very important).When the time comes to evaluate the alternatives,we do so by assessing them relative to each otheragainst all WANT objectives one at a time.5

The alternatives are scored individually againsteach of the goal criteria based on their relative per-formance. We give a score of 10 to the alterna-

tive that comes closest to meeting the objective,and score the other alternatives relative to it. It isnot an ideal that we seek through this comparativeevaluation. What we seek is an answer to thequestion: Of these (real and attainable) alterna-tives, which best fulfills the objective?6

A total score is determined for each alternative bymultiplying its score for each criterion by the crite-rion weights (relative weighting factor for eachcriterion) and then summing across all criteria.The preferred alternative will have the highest total

score.

_______________________________

5 Appendix C-Further Reading 6, Kepnerand Tregoe, P.92

6 Ibid, p.95


11/44


7

K-T Decision Analysis is suitable for moderatelycomplex decisions involving a few criteria. Themethod requires only basic arithmetic. Its maindisadvantage is that it may not be clear how muchbetter a score of 10 is than a score of 8, forexample. Moreover, total alternative scores may

be close together, making a clear choice difficult.7

3.3 Analytic Hierarchy Process (AHP)

AHP is a quantitative comparison method used toselect a preferred alternative by using pair-wisecomparisons of the alternatives based on their rela-tive performance against the criteria. The basis ofthis technique is that humans are more capable ofmaking relative judgements than absolute judge-ments. The Analytic Hierarchy Process is a sys-

tematic procedure for representing the elements ofany problem, hierarchically. It organizes the basicrationality by breaking down a problem into itssmaller and smaller constituent parts and thenguides decision makers through a series of pairwisecomparison judgements (which are documentedand can be reexamined) to express the relativestrength or intensity of impact of the elements inthe hierarchy. These judgements are then trans-lated to numbers (ratio scale estimates). The AHPincludes procedures and principles used to synthe-size the many judgements to derive priorities

among criteria and subsequently for alternativesolutions.8

Alternatives and criteria are scored using a pair-wise comparison method and mathematics (seeAHP example in Appendix A). The pair-wisecomparisons are made using a nine-point scale:

1 = Equal importance or preference3 = Moderate importance or preference of one

over another5 = Strong or essential importance or prefer-

ence7 = Very strong or demonstrated importance or

preference9 = Extreme importance or preference

_________________________________

Matrices are developed wherein each crite-rion/alternative is compared against the others. IfCriterion A is strongly more important comparedto Criterion B (i.e. a value of 5), then Criterion Bhas a value of 1/5 compared to Criterion A. Thus,for each comparative score given, the reciprocal isawarded to the opposite relationship. The priority

vector (i.e. the normalized weight) is calculatedfor each criterion using the geometric mean9 ofeach row in the matrix divided by the sum of thegeometric means of all the criteria (see example inAppendix A). This process is then repeated for thealternatives comparing them one to another to de-termine their relative value/importance for eachcriterion (i.e. determine the normalized alternativescore). The calculations are easily set up in aspreadsheet, and commercial software packagesare available.

HINT: The order of comparison can help simplifythis process. Try to identify and begin with themost important criterion and work through the cri-teria to the least important. When comparing al-ternatives try to identify and begin with the onewith the greatest benefits for each associated crite-rion.

To identify the preferred alternative multiply eachnormalized alternative score by the correspondingnormalized criterion weight, and sum the resultsfor all of an alternatives criteria. The preferred al-

ternative will have the highest total score.

AHP, like the other methods, can rank alternativesaccording to quantitative or qualitative (subjective)data. Qualitative/subjective criteria are based onthe evaluation teams feelings or perceptions abouthow an alternative ranks. The criteria weights andalternative comparisons are combined in the deci-sion synthesis to give the relative value (ra-tio/score) for each alternative for the prescribeddecision context. A sensitivity analysis can be per-formed to determine how the alternative selection

would change with different criteria weights. The

_________________________________

Appendix C-Further Reading 6, Kepner andTregoe

8 Appendix C-Further Reading 7, Saaty and Kearns,

p.19

The geometric mean is the nth root of the product of nscores. Thus, the geometric mean of the scores: 1, 2, 3, and10 is the fourth root of (1 x 2 x 3 x 10), which is the fourthroot of 60. (60)1/4 = 2.78. The geometric mean is less af-fected by extreme values than is the arithmetic mean. It isuseful as a measure of central tendency for some positivelyskewed distributions.


12/44


8

whole process can be repeated and revised, untileveryone is satisfied that all the important featuresneeded to solve the problem, or select the preferredalternative, have been covered.

AHP is a useful technique when there are multiplecriteria since most people cannot deal with more

than seven decision considerations at a time.10

AHP is suitable for decisions with both quantita-tive and qualitative criteria. It puts them in thesame decision context by relying on relative com-parisons instead of attempting to define absolutes.It facilitates discussion of the importance of criteriaand the ability of each alternative to meet the crite-ria. Its greatest strength is the analytical hierarchythat provides a structured model of the problem,mimicking the way people generally approachcomplex situations by allowing relative judgementsin lieu of absolute judgements. Another strength is

its systematic use of the geometric mean to definefunctional utilities based on simple comparisonsand to provide consistent, meaningful results. Thesize of AHP matrices make this method somewhatless flexible than either K-T or MAUT when newlydiscovered alternatives or criteria need to be con-sidered. Commercially available software, how-ever, can reduce this burden and facilitate thewhole process. Although software is not requiredfor implementation, it can be helpful especially if alarge number of alternatives (>8), or criteria (>5)must be considered.11

3.4 Multi-Attribute Utility Theory (MAUT)

MAUT is a quantitative comparison method usedto combine dissimilar measures of costs, risks, andbenefits, along with individual and stakeholderpreferences, into high-level, aggregated prefer-ences. The foundation of MAUT is the use of util-ity functions. Utility functions transform diversecriteria to one common, dimensionless scale (0 to1) known as the multi-attribute utility. Onceutility functions are created an alternatives raw

data (objective) or the analysts beliefs (subjective)can be converted to utility scores. As with theother methods, the criteria are weighted accordingto importance. To identify the preferred alternative,multiply each normalized alternatives utility score_________________________________identify the preferred alternative multiply eachnormalized alternatives utility score by its correbyby its corresponding criterion weight, and sum the

results for all of an alternatives criteria. The pre-ferred alternative will have the highest total score.

Utility functions (and MAUT) are typically used,when quantitative information is known about eachalternative, which can result in firmer estimates ofthe alternative performance. Utility graphs are

created based on the data for each criterion. Everydecision criterion has a utility function created forit. The utility functions transform an alternativesraw score (i.e. dimensioned feet, pounds, gallonsper minute, dollars, etc.) to a dimensionless utilityscore, between 0 and 1. The utility scores areweighted by multiplying the utility score by theweight of the decision criterion, which reflects thedecision-making support staffs and decision-makers values, and totaled for each alternative.The total scores indicate the ranking for the alter-natives.

The MAUT evaluation method is suitable for com-plex decisions with multiple criteria and many al-ternatives. Additional alternatives can be readilyadded to a MAUT analysis, provided they havedata available to determine the utility from theutility graphs. Once the utility functions have beendeveloped, any number of alternatives can bescored against them.

The Simple Multi Attribute Rating Technique(SMART) can be a useful variant of the MAUT

method. This method utilizes simple utility rela-tionships. Data normalization to define theMAUT/SMART utility functions can be performedusing any convenient scale. Five, seven, and tenpoint scales are the most commonly used. In aclassical MAUT the full range of the scoring scalewould be used even when there was no real differ-ence between alternatives scores. The SMARTmethodology allows for use of less of the scalerange if the data does not discriminate adequatelyso that, for example, alternatives which are notsignificantly different for a particular criterion can

be scored equally. This is particularly importantwhen confidence in the differences in data is low.In these cases, less of the range is used to ensurethat low confidence data differences do notpresent unwarranted large discriminations betweenthe alternatives. When actual numerical data are

10 Appendix C-Further Reading 8, Miller,p.81-97

11 Appendix C-Further Reading 9 and 10, Saaty


13/44


9

unavailable, subjective reasoning, opinions, and/orconsensus scoring can be substituted and docu-mented in the final report instead. Research hasdemonstrated that simplified MAUT decisionanalysis methods are robust and replicate decisionsmade from more complex MAUT analysis with ahigh degree of confidence.12

3.5 Cost-Benefit Analysis

Cost-Benefit Analysis (see example in AppendixA) is a systematic quantitative method of assess-ing the desirability of government projects or poli-cies when it is important to take a long view offuture effects and a broad view of possible side-effects.13 CBA is a good approach when the pri-mary basis for making decisions is the monetarycost vs. monetary benefit of the alternatives. Gen-

eral guidance for conducting cost-benefit and cost-effectiveness analyses is provided in the U.S. Of-fice of Management and Budget, OMB CircularNo. A-94, Guidelines and Discount Rates for

Benefit-Cost Analysis of Federal Programs.14 Thediscount rates for this methodology are updatedannually by the OMB.

The standard criterion for deciding whether a gov-ernment program can be justified on economicprinciples is net present value -- the discountedmonetized value of expected net benefits (i.e.,

benefits minus costs). Net present value is com-puted by assigning monetary values to benefits andcosts, discounting future benefits and costs usingan appropriate discount rate, and subtracting thesum total of discounted costs from the sum total ofdiscounted benefits. Discounting benefits and coststransforms gains and losses occurring in differenttime periods to a common unit of measurement.Programs with positive net present value increasesocial resources and are generally preferred. Pro-grams with negative net present value should_______________________________

12Appendix C-Further Reading 11, Edwards and Barron, 12,Goodwin and Wright, 4, Hammond, Keeney, and Raiffa

13 Appendix C-Further Reading 13, U.S. Office of Manage-ment and Budget, p. 15

14Ibid

generally be avoided. When benefits and costscan be quantified in dollar terms (as, for exampleavoided cost) over several years, these benefits canbe subtracted from the costs (or dollar outlays)and the present value of the benefit calculated.Both intangible and tangible benefits and costsshould be recognized. The relevant cost concept is

broader than the private-sector production andcompliance cost or government cash expenditures.Costs should reflect opportunity cost of any re-sources used, measured by the return to those re-sources in their most productive application else-where.15 The alternative returning the largest dis-counted benefit is preferred.

In Pros and Cons analysis cost is regarded intui-tively along with the other advantages and disad-vantages (high cost is a con; low cost is a pro).The other techniques provide numerical ranking of

alternatives based on either intangible (i.e. unableto be quantified in dollar terms) or tangible (quan-tifiable in dollar terms) benefits.16

3.6 Custom Tailored Tools

Customized tools may be needed to help under-stand complex behavior within a system. Verycomplex methods can be used to give straightfor-ward results. Because custom-tailored tools arenot off-the-shelf, they can require significant timeand resources for development. If a decision can-

not be made using the tools described previously,or the decision must be made many times employ-ing the same kinds of considerations, the decision-making support staff should consider employingspecialists with experience in computer modelingand decision analysis to develop a custom-tailoredtool.

_______________________________________

15Appendix C-Further Reading 13, U.S. Office of

Management and Budget16 Appendix C-Further Reading 14, Broadman,Greenberg, Vining, and Weimer; 15, Canada,Sullivan, and White; 16, Electric Power ResearchInstitute; 17, Snell; and 13, U.S. Office ofManagement and Budget.


14/44


10

4.0 Summary

The goal of this Guidebook to Decision-MakingMethods is to help decision-makers and their deci-sion support staffs choose and document the bestalternative in a clear and transparent fashion. The

decision-making methods described in this guide-book are readily applicable to a wide range of deci-sions, from ones as simple as picking a restaurantfor a special meal to those that are complicated byinterdepartmental government interfaces. Expertfacilitation can help assure that all the steps areproperly performed documented. Feedback fromthe decision-maker(s) is vital to the process.

The key to developing an adequate problem state-ment is to ask enough questions about the problemto ensure that the final report will clearly answer

the questions of reviewers and stakeholders. Re-quirements spell out what the solution to the prob-lem must do. Goals are useful in identifying supe-rior alternatives. The decision team evaluates therequirements and goals and suggests alternativesthat will meet the requirements and satisfy as manygoals as possible. The best alternative will be theone that most nearly achieves the goals. Criteriamust be developed to discriminate among alterna-tives in a meaningful way. The decision-makersapproval is crucial before the criteria are used toevaluate the alternatives.

Alternatives can be evaluated with quantitativemethods, qualitative methods, or any combination.

The evaluation methods introduced here arehighly recommended. They are adaptable tomany situations, as determined by the complexityof the problem, needs of the customer, experienceof the decision team / analysts / facilitators, and thetime and resources available. The decision-makingmethod selection needs to be based on the com-plexity of the problem and the experience of theteam. A final solution should fulfill the desiredstate, meet requirements, and best achieve thegoals within the values of the decision makers.

Once the preferred alternative has been validated,the decision-making support staff can present it as a recommendation to the decision-maker(s). A final report to the decision-maker(s)

must be written documenting the decision process,assumptions, methods, and conclusions recom-mending the final solution.


15/44


11

Appendix A Decision-Making Tools at Work

Step 1 Problem: Pick a replacement vehicle for the motor pool fleet

(The definition of the problem dictates the requirements. As the vehicle is for a motor pool, the requirementswill differ from those for a family car, for example.)

Step 2 Requirements:

1. Vehicle shall be made in U. S. A.2. Vehicle shall seat at least four adults, but no more than six adults3. Vehicle shall cost no more than $28,0004. Vehicle shall be new and the current model year

(Other requirements may be appropriate, but the above four will suffice for this example.)

Step 3 Goals:

Maximize passenger comfort Maximize passenger safety Maximize fuel-efficiency Maximize reliability Minimize investment cost

Step 4 Alternatives:

There are many alternatives but the requirements eliminate the consideration of a number of them:

Requirement 1 eliminates the products not manufactured in the USARequirement 2 eliminates vans, buses, and sports carsRequirement 3 eliminates high-end luxury carsRequirement 4 eliminates used vehicles

Despite the limitations imposed by the requirements, many alternatives remain. This example will evaluatefour, current, U. S. models against the goals:

For this simple problem the following quantitative data are available. This is how the four models stack up:

Arrow

Seats two adults in the front seat and three in the back seatRear seat leg and shoulder room 86 inchesNumber of safety stars 14Fuel consumption 21 mpgReliability 80Cost $26,000

Baton


16/44


12

Seats three adults in the front seat and three in the back seatRear seat leg and shoulder room 88 inchesNumber of safety stars 17Fuel consumption 19 mpgReliability 70Cost $21,000

Carefree

Seats two adults in the front seat and three in the back seatRear seat leg and shoulder room 80 inchesNumber of safety stars 15Fuel consumption 22 mpgReliability 65Cost $17,000

Dash

Seats three adults in the front seat and three in the back seatRear seat leg and shoulder room 89 inches

Number of safety stars 19Fuel consumption 21 mpgReliability 85Cost $24,000

Step 5 Criteria:

Maximize comfort will be based on the combined rear seat leg and shoulder room. (Note: frontseat passenger leg and shoulder room was found to be too nearly the same todiscriminate among the alternatives.)

Maximize safety will be based on the total number of stars awarded by the National Highway

Traffic Safety Administration for head-on and side impact.

Maximize fuel efficiency will be based on the EPA fuel consumption for city driving.

Maximize reliability will be based on the reliability rating given each vehicle by a consumer producttesting company.

Minimize Cost will be based on the purchase price.

Step 6 Decision-Making Tool Selection:

To demonstrate the application of the decision-making tools described in this Guidebook this problem will be

solved using each method. In a typical decision situation tool selection would depend on the mutual experi-ence of the decision team, any expert facilitator, and the complexity of the problem. Data, either quantitativeor qualitative, would be gathered and tabulated for evaluation against the defined criteria.


17/44


13

Step 7 Apply the Selected Method

PROS AND CONS ANALYSIS

Lists of the pros and cons, based on the input of subject matter experts, are compared one to another for eachalternative.

Table 1. Example of Pros And Cons Analysis

Arrow Baton

Pro Con Pro ConGood fuel efficiency Highest cost Next to most room Worst fuel efficiencyNext to best reliabili ty Fewest safety stars

Carefree Dash

Pro Con Pro ConLowest cost Next to fewest safety stars Most safety stars

Best fuel efficiency Least room Best reliabilityWorst reliability Good fuel efficiencyMost room

Best reliability

Step 8 Validate Solution:

Safety and reliability are the most important criteria. Dash is best in these areas. Dash scores pros in theother criteria, as well. Dash has five advantages and no disadvantages, so Dash is the preferred alternative.Dash meets all the requirements and solves the problem.


18/44


14


KEPNER-TREGOE DECISION ANALYSIS

In this example a team consensus approach based on the decision-making support staffs and decision-makers values was used to obtain both the criteria weight and the alternative score relative to each criterion.

The team was polled and the average score for each element, rounded to the nearest integer, obtained. TheTotal Score is the product of the Criteria Weight and the Alternative Score summed for the alternative.

Table 2. Example of Kepner-Tregoe Decision Analysis

Criteria/Want objectives

CriteriaWeight

Arrow Alter-nativeScore

Total Score

Comfort 5 86 in. rear seat leg and shoulder room, seats 5 6 30

Safety 10 14 stars 5 50Fuel efficiency 7 21 mpg 9 63

Reliability 9 80 9 81Cost 10 $26,000 5 50

Total 274Baton

Comfort 5 88 in. rear seat leg and shoulder room, seats 6 9 45Safety 10 17 stars 8 80

Fuel efficiency 7 19 mpg 8 56Reliability 9 70 7 63

Cost 10 $21,000 8 80Total 324

Carefree

Comfort 5 80 in. rear seat leg and shoulder room, seats 5 4 20Safety 10 15 stars 6 60

Fuel efficiency 7 22 mpg 10 70Reliability 9 65 5 45

Cost 10 $17,000 10 100Total 295

Dash

Comfort 5 89 in rear seat leg and shoulder room, seats 6 10 50

Safety 10 19 stars 10 100Fuel efficiency 7 21 mpg 9 63

Reliability 9 85 10 90Cost 10 $24,000 6 60

Total 363


The totals of the weighted scores show that the Dash most nearly meets the wants/goals (or put another way,has the most benefits). Dash meets all the requirements and solves the problem.


19/44


15


ANALYTICAL HIERARCHY PROCESS

In this example a team consensus approach based on the decision-making support staffs and decision-makers values was used to obtain the relative pair-wise comparisons for each criterion. The team was polled

and the average score for each comparison, rounded to the nearest integer, obtained. For example the teamconsensus was that Safety as compared to Comfort deserved a 7 - very strong or demonstrated importance orpreference.

HINT: The team first ranked the criteria in order of importance Safety, Cost, Reliability, Fuel efficiencyand Comfort and then compared them one to another (Table 3) to determine their relative importance (score)The basis for this ranking must be included in the final report. The ranking order presented here is not a re-quirement. Ranking order can vary depending on many factors.

Table 3. Example of Pair-Wise Comparison of CRITERIA

Safety- Comfort 7 Cost- Comfort 6 Reliability- Comfort 6 Fuel efficiency- Comfort 4

Safety- Fuel efficiency 4 Cost- Fuel efficiency 2 Reliability- Fuel efficiency 2Safety- Reliability 3 Cost- Reliability 1Safety- Cost 2

The remaining, or reciprocal, comparisons of the criteria are determined by inspection for incorporation in thecalculation of the normalized criteria weights in Table 4.

Table 4. Example of Calculating Priority Vector or Normalized Criteria Weights

Comfort Safety Fuel Efficiency Reliability Cost GeometricMean

NormalizeWeight

Comfort 1 1/7 1/4 1/6 1/6 0.251 0

Safety 7 1 4 3 2 2.787 0

Fuel efficiency 4 1/4 1 1/2 1/2 0.758 0Reliability 6 1/3 2 1 1 1.320 0

Cost 6 1/2 2 1 1 1.431 0

SUM= 6.546

The geometric mean is less affected by extreme values than is the arithmetic mean. It is the nth root of theproduct of n scores. Thus, the geometric mean of the scores: 1, 2, 3, and 10 is the fourth root of (1 x 2 x 3 10), which is the fourth root of 60. (60) = 2.78. It is useful as a measure of central tendency for some posi-tively skewed distributions. The normalized criterion weight is its geometric mean divided by the sum of thegeometric means of all the criteria. The geometric mean and normalized weights can be computed usingspreadsheet software as shown in Figure 1.

Next the team performed pair-wise comparisons of the alternatives with regard to each criterion. For examplethe team consensus was that Dash as compared to Baton with respect to comfort deserved a 1 - equal impor-tance or preference. The 1 inch difference in the comfort measurement between Dash and Baton was deemedto have no real impact while the difference in the comfort measurements between Dash and Arrow and between Dash and Carefree coupled with seating capacities were deemed to have a greater impact.


20/44


16

Figure 1. Example of Spreadsheet Set-up for AHP Matrices

AA BB CC DD EE FF GG HH

11 Comfort Safety Fuelefficiency

Reliability Cost Geometric Mean NormalizedWeight

22 Comfort 1 =1/7 =1 /4 =1/6 =1/6 =GEOMEAN(B2:F2) =+G2/$G$7

33 Safety 7 1 4 3 2 =GEOMEAN(B3:F3) =+G3/$G$7

44 Fuel efficiency 4 =1 /4 1 =1/2 =1/2 =GEOMEAN(B4:F4) =+G4/$G$7

55 Reliability 6 =1/3 2 1 1 =GEOMEAN(B5:F5) =+G5/$G$7

66 Cost 6 =1/2 2 1 1 =GEOMEAN(B6:F6) =+G6/$G$7

77 =SUM(G2:G6)

Table 5. Example of Pair-Wise Comparison of ALTERNATIVES With Respect to COMFORT

Dash - Baton 1 Baton - Arrow 2 Arrow - Carefree 3Dash - Arrow 4 Baton - Carefree 3Dash - Carefree 5

The remaining, or reciprocal, comparisons of the alternatives are determined by inspection for incorporationin the calculation of the normalized alternative scores in Table 6. Each alternative was compared in Tables 7- 14 to determine its normalized score for each criterion.

Table 6. Example of Calculating Priority Vector or Normalized Alternative Score With Respect toCOMFORT

Arrow Baton Carefree Dash GeometricMean

NormalizedScore

Arrow 1 1/2 3 1/4 0.78254229 0.160040343

Baton 2 1 3 1 1.56508458 0.320080687

Carefree 1/3 1/2 1 1/5 0.427287006 0.087385896Dash 4 1 5 1 2.114742527 0.432493074

Table 7. Example of Pair-Wise Comparison of ALTERNATIVES With Respect to SAFETY

Dash - Baton 2 Baton - Arrow 2 Carefree - Arrow 1Dash - Arrow 5 Baton - Carefree 3Dash - Carefree 4

Table 8. Example of Calculating Priority Vector or Normalized Alternative Score With Respect to

SAFETY

Arrow Baton Carefree Dash Geometric

Mean

Normalized

ScoreArrow 1 1/2 1 1/5 0.562341325 0.114052057

Baton 2 1 3 1/2 1.316074013 0.266921425

Carefree 1 1/3 1 1/4 0.537284965 0.108970215

Dash 5 2 4 1 2.514866859 0.510056303


21/44


17

Table 9. Example of Pair-Wise Comparison of ALTERNATIVES With Respect to FUEL EFFICIENC

Carefree - Baton 3 Dash - Arrow 1 Arrow - Baton 2Carefree - Arrow 1 Dash - Baton 2Carefree - Dash 1

Table 10. Example of Calculating Priority Vector or Normalized Alternative Score With Respect toFUEL EFFICIENCY


NormalizedScore

Arrow 1 2 1 1 1.189207115 0.277263153

Baton 1/2 1 1/2 1/2 0.594603558 0.138631576

Carefree 1 3 1 1 1.316074013 0.306842118

Dash 1 2 1 1 1.189207115 0.277263153

Table 11. Example of Pair-Wise Comparison of ALTERNATIVES With Respect to RELIABILITY

Dash - Baton 4 Arrow - Baton 3 Baton - Carefree 2

Dash - Arrow 2 Arrow - Carefree 4Dash - Carefree 6


RELIABILITY


NormalizedScore

Arrow 1 3 4 1/2 1.56508458 0.300049178

Baton 1/3 1 2 1/4 0.638943088 0.122494561

Carefree 1/4 1/2 1 1/6 0.379917843 0.072835704

Dash 2 4 6 1 2.632148026 0.504620557

Table 13. Example of Pair-Wise Comparison of ALTERNATIVES With Respect to COST

Carefree - Baton 3 Baton - Dash 3 Dash - Arrow 2Carefree - Dash 4 Baton - Arrow 4Carefree - Arrow 5


COST


NormalizedScore

Arrow 1 1/4 1/5 1/2 0.397635364 0.075972332

Baton 4 1 1/3 3 1.414213562 0.270200068Carefree 5 3 1 4 2.783157684 0.531750942

Dash 2 1/3 1/4 1 0.638943088 0.122076658


22/44


18

To identify the preferred alternative multiply each normalized alternative score by the corresponding normal-ized criterion weight, and sum the results for all of an alternatives criteria. The preferred alternative will havethe highest total score.

Table 15. Example of AHP Decision Analysis


NormalizedCriteriaWeight

Arrow NormalizedAlternative

Score

TotalScore

Comfort 0.038 86 in. rear seat leg and shoulder room, seats 5 0.160040343 0.0061

Safety 0.426 14 stars 0.114052057 0.0486Fuel efficiency 0.116 21 mpg 0.277263153 0.0322Reliability 0.202 80 0.300049178 0.0606

Cost 0.219 $26,000 0.075972332 0.0166Total 0.1641

Baton

Comfort 0.038 88 in. rear seat leg and shoulder room, seats 6 0.320080687 0.0122

Safety 0.426 17 stars 0.266921425 0.1137Fuel efficiency 0.116 19 mpg 0.138631576 0.0161

Reliability 0.202 70 0.122494561 0.0427

Cost 0.219 $21,000 0.270200068 0.0592Total 0.2259

Carefree

Comfort 0.038 80 in. rear seat leg and shoulder room, seats 5 0.087385896 0.0033Safety 0.426 15 stars 0.108970215 0.0464

Fuel efficiency 0.116 22 mpg 0.306842118 0.0356Reliability 0.202 65 0.072835704 0.0147

Cost 0.219 $17,000 0.531750942 0.1165Total 0.2165

Dash

Comfort 0.038 89 in rear seat leg and shoulder room, seats 6 0.432493074 0.0164Safety 0.426 19 stars 0.510056303 0.2173

Fuel efficiency 0.116 21 mpg 0.277263153 0.0322Reliability 0.202 85 0.504620557 0.1019

Cost 0.219 $24,000 0.122076658 0.0267Total 0.3945




23/44


19


MULTI-ATTRIBUTE UTILITY THEORY ANALYSIS

The team generated utility functions from evaluation studies of many comparable vehicles (see Figure 2)Note that the cost utility curve gives a value of 0 for all vehicles costing more than $28,000 which was one of

the requirements.

Figure 2. Examples of Utility Functions

Comfort

78

8082

84

86

88

90

92

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Utility

Rearleg

+shoulderroom(

in.)

Safety

13

14

15

16

17

18

19

20

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Utility

Tota

lStars

Reliability

0

20

40

60

80

100

120

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Utility

Reliab

ilityRating

Fuel Consumption

14

16

18

20

22

24

26

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Utility

MPG

(City)

Cost

28

26

24

22

20

18

16

14

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Utility

Cost

(X$1000)


24/44


20

In this example a team consensus approach based on the decision-making support staffs and decision-makers values was used to obtain the criteria weights which were then normalized to a 0 1 range. The teamwas polled and the average score for each element, rounded to the nearest integer, obtained and that value di-vided by the largest score. The Total Score is the product of the Criteria Weight and the Alternative Utilitysummed for the alternative.

Table 16. Example of MAUT Decision Analysis


CriteriaWeight

Arrow Alter-native

Utility*

WeightedUtility

ScoreComfort .5 86 in. rear seat leg and shoulder room, seats 5 0.375 0.1875Safety 1 14 stars 0.150 0.15

Fuel efficiency .7 21 mpg 0.300 0.21Reliability .9 80 0.700 0.63

Cost 1 $26,000 0.075 0.075Total 1.2525

Baton

Comfort .5 88 in. rear seat leg and shoulder room, seats 6 0.600 0.3

Safety 1 17 stars 0.550 0.55Fuel efficiency .7 19 mpg 0.175 0.1225

Reliability .9 70 0.600 0.54Cost 1 $21,000 0.275 0.275

Total 1.7875

Carefree

Comfort .5 80 in. rear seat leg and shoulder room, seats 5 0.050 0.025Safety 1 15 stars 0.275 0.275Fuel efficiency .7 22 mpg 0.400 0.28

Reliability .9 65 0.550 0.495Cost 1 $17,000 0.650 0.65

Total 1.725

Dash

Comfort .5 89 in rear seat leg and shoulder room, seats 6 0.700 0.35

Safety 1 19 stars 0.825 0.825Fuel efficiency .7 21 mpg 0.300 0.21Reliability .9 85 0.750 0.675

Cost 1 $24,000 0.150 0.15Total 2.21

*Extracted from the utility graphsUtility Score X Criterion Weight




25/44


21

MAUT can also accommodate utility functions that have an optimum utility. One example of this could be aprojectile targeting system. If, for example, the required range were 100 meters, systems that yielded resultsover or under that range would have lowered utility values compared to systems that yielded results nearer100 meters. An example of just such a utility curve is shown in Figure 3.

Figure 3. Example of Utility Function Having Optimum Utility

0

0.1

0.2

0.3

0.4

0.50.6

0.7

0.8

0.9

1

50

100

150

Data Value

Utility

Too low

Just right

Too high


26/44


22


SIMPLE MULTI ATTRIBUTE RATING TECHNIQUE (SMART)

As in the previous examples a team consensus approach based on the decision-making supportstaffs and decision-makers values was used to obtain the criteria weights which were then nor-

malized to a 0 1 range. The team was polled and the average score for each element, rounded tothe nearest integer, obtained. In this example (shown in Table 17) the team consensus value isdivided by the sum of the scores to obtain the Normalized Weight Factor for each criterion (goal).Table 18 repeats the raw data and criteria previously established.

Goal

N u m b e r G oa l Nam e

Goal Weight

Factor

Normalized

Weight

Factor

1 Max imize Passen er Comfor t 1 0.034

2 Maximize Passenger Safety 10 0.345

3 Maximize Fuel-efficiency 3 0.103

4 Maximize Rel iabi li t 5 0.172

5 Min imize Investment Cost 10 0.345

Table 17. Example Goal Weights

Table 18. Data used in normalized scoring.

Criteria DescriptionArrow Baton Carefree Dash

Combined rear

seat room

Based on the combined rear seat leg

and shoulder room. (Note: front seat

passsenger leg and shoulder room

was found not to discriminate.)

86 inches of leg and

shoulder room,Seats 5







Number of safety

stars

Based on the total number of stars

awarded by the National Highway

Traffic Safety Administration for head-

on and side impact.

14 stars 17 stars 15 stars 19 stars

City fuel economy

rate

Based on the EPA fuel consumption

for city driving.

21 mpg 19 mpg 22 mpg 21 mpg

Consumer

Reports reliability

rating

Based on the reliability rating given

by the consumer product testing

company.

80 70 65 85

Purchase cost Based on the purchase price. $26K $21K $17K $24K

Alternatives

Criteria


27/44


23

The team established rules of thumb for eachcriterion representing their utility functionsand then proceeded to score each alternativerounding to integer values as appropriate. A1-5 performance score scale is used where a

1 score always means worst performance(0) and a 5 score always means best per-formance (1). The team scored Dash andBaton the same for combined rear seat roombased on the fact that a 1 difference was in-significant. Since the SMART methodologyallows for use of less of the scale range if thedata does not discriminate adequately, theyalso scored Batons fuel economy as a 2 be-cause a lower rating wouldnt discriminateadequately. The normalized scores in Table19 are converted to utility values between 0

and 1 (1= 0, 2= .25, 3= .5, 4= .75, 5= 1.0) asshown in Figure 4. (Remember that the foun-

dation of MAUT is the use of utility func-tions to transform an alternatives raw scoreto a dimensionless utility score between 0and 1.) A weighted criterion score (notshown) is obtained by multiplying the utility

values by the Normalized Weight Factorsfrom Table 17. The weighted criterion scoresare summed into the overall final utilityscores shown at the tops of the columns inFigure 4.

Figure 4 displays the main tradeoffs acrossthe criteria using a Consumer Reports formatthat is easy for decision makers and others tounderstand. This presentation format pro-vides a different visual perspective of thesame/similar information as compared with

previous examples.

Criteria Scoring ExplanationArrow Baton Carefree Dash

Scoring

rule ofthumb

Combined rear

seat room5 for large space and 1 for small

space 4 5 2 53" per

point

Number of safety

stars5 for big number of stars, 1 for low

number of stars 1 4 2 5 1 star perpoint

City fuel economy

rate5 for high mpg, 1 for low mpg 4 2 5 4 1mpg per

point

Consumer

Reports reliability

rating

5 for high reliability rating, 1 for

low reliability rating 4 2 1 55 rating

points per

point

Purchase cost5 for low purchase cost, 1 for high

purchase cost 1 3 5 2 $2-$3Kper point

AlternativesTable 19. Normalized Scores


28/44


24


The shaded alternative in Figure 4, Dash, has the highest overall utility score and most nearly meets thewants/goals (or put another way, has the most benefits). Dash meets all the requirements and solvesthe problem.

Problem:

Scoring Method: 1 to 5 Scale

Arrow Baton Carefree Dash

CriteriaOverall Score:

23.2%Overall Score:

53.4%Overall Score:

54.3%Overall Score:

71.5%

Combined rear

seat room4.0 = 0.75 2.0 = 0.25 5.0 = 1.0

Number of safety

stars1.0 = 0 4.0 = 0.75 2.0 = 0.25 5.0 = 1.0

City fuel economyrate

4.0 = 0.75 2.0 = 0.25 5.0 = 1.0 4.0 = 0.75

Consumer

Reports reliability

rating4.0 = 0.75 2.0 = 0.25 1.0 = 0 5.0 = 1.0

Purchase cost

1.0 = 0 3.0 = 0.5 5.0 = 1.0 2.0 = 0.25

Pick a replacement vehicle for the motor pool.

Alternatives

1 2 3 4 5

BetterWorse

Figure 4. Final Evaluation Result, Dash has highest score.

5.0 = 1.0


29/44


25


COST-BENEFIT ANALYSIS

The object of a CBA is to calculate the Net Present Value of an alternative. OMB Circular No. A-94provides the following example of a Net Present Value calculation.17

QUANTIFIED BENEFIT-COST EXAMPLE

Assume a 10-year program which will commit the Government to the stream of real (or constant-dollar)expenditures appearing in column (2) of Table 20 and which will result in a series of real benefits appear-ing in column (3). The discount factor for a 7 percent discount rate is shown in column (4). The presentvalue cost for each of the 10 years is calculated by multiplying column (2) by column (4); the presentvalue benefit for each of the 10 years is calculated by multiplying column (3) by column (4). The presentvalues of costs and benefits are presented in columns (5) and (6) respectively.

Table 20. Example of CBA Calculation

Year since

Initiation,Renewal, orExpansion

(1)

Expected

yearly cost(2)

Expected

yearly benefit(3)

Discount

factors for7%(4)

Present

value ofcosts Col. 2

x Col. 4(5)

Present value

of benefitsCol. 3 x Col.

4(6)

1 $10.00 $ 0.00 0.9346 $ 9.35 $0.002 20.00 0.00 0.8734 17.47 0.003 30.00 5.00 0.8163 24.49 4.084 30.00 10.00 0.7629 22.89 7.635 20.00 30.00 0.7130 14.26 21.396 10.00 40.00 0.6663 6.66 26.657 5.00 40.00 0.6227 3.11 24.91

8 5.00 40.00 0.5820 2.91 23.289 5.00 40.00 0.5439 2.72 21.7610 5.00 25.00 0.5083 2.54 12.71Total $106.40 $142.41

NOTE: The discount factor is calculated as 1/(1 + i)t where i is the interest rate (.07) and t is the year(OMB Circular A-94 specifies a 7% discount rate).

The sum of column (5) is the total present value of costs and the sum of column (6) is the total presentvalue of benefits. Net present value is $36.01, the difference between the sum of discounted benefits andthe sum of discounted costs.


Only alternatives meeting all requirements should be considered. This calculation would be performedfor each alternative and the alternative with the largest Net Present Value would be preferred.

___________________________________

17 Appendix C-Further Reading 13, U.S. Office of

Management and Budget, p.18


30/44


26

The examples in Appendix A demonstrate that data resulting from any of these methodologies can begenerated and presented in a variety of formats ranging from hand reduction techniques to sophisticatedcomputer driven software packages. As discussed in section 1.8 of Appendix B sensitivity analyses canbe useful to the decision makers by allowing them to evaluate their values (Criteria/Goal Weights) tomake sure they select the alternative that best meets their values (honesty check). For example Figure 5shows how an increase in the purchase cost criterion Goal Weight Factor from the SMART example

could change the outcome. Increasing the cost Normalized Weight Factor from the current value of34.5% to 50% would result in Carefree having the highest overall utility score.

Three sample formats for presenting the Overall Score data from Figure 4, Final Evaluation Result, arepresented below a Bar Chart, a Pie Chart, and a Polar Chart. There are numerous additional ways inwhich the data from these examples can be presented. The decision team should pick the method thatbest displays their results and conclusions.

Bar Chart Pie Chart Polar Chart

Figure 5. Purchase Cost Sensitivity Analysis

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1Purchase cost Weight

Score

Arrow

Baton

Carefree

Dash

If weight = .5 for cost,

Carefree would be

preferred alternative

Current weight of

Cost criterion

0.232

0.5340.543

0.715

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

A rrow Baton Carefree Dash

UtilityScore

0.232

0.534

0.543

0.715

Arrow

Baton

Carefree

Dash

0.232

0.534

0.543

0.715

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8Arrow

Baton

Carefree

DashSeries1


31/44


27

Appendix B Decision ProcessAids

1.0 Introduction

This appendix provides helps, additional

information, and potential forms or productsto be developed during each of the processsteps defined in Section 2 of this Guidebook.

Usually the decision making support staff

should engage the help of, skilled and ex-

perienced analysts/facilitators to assistwith all stages of the decision process. Ex-pert facilitation can help assure that all thesteps are properly performed and docu-mented. Their experience and expertise willhelp provide transparency to the decision

making process and help avoid misunder-standings that often lead to questions aboutthe validity of the analyses which ultimatelyslow progress. Table B-8 (page 39) pro-vides some Decision Support PersonnelContacts at several DOE sites.

1.1 Brainstorming

Brainstorming is the chief tool used by thedecision support team to identify, analyze,and develop the problem and potential alter-

native solutions. Brainstorming is a tech-nique for using the subconscious to generateideas. Brainstorming tools like lateralthinking, affinity diagrams, and interrela-tionship digraphs can be used to aid the pro-cess. Creative, divergent thinking is essen-tial in this step. No ideas are critiqued untillater in the process and all ideas are re-corded. Once brainstorming has concluded,ideas can be accepted, refined, and com-bined with other ideas or cast aside as is ap-propriate.

1.2 Aid to Defining a Problem

The following suggestions/questions may beuseful in defining the scope and resourceneeds for a given problem.

1. Initial Problem Statement: Initial Undesired State

Desired State

2. List problem symptoms/triggers.

3. Scope of Evaluation how big is thesystem? Who is impacted by this prob-lem/issue?

4. How much time and funding is availableto perform the evaluation?

5. Who will be the project lead for the de-cision analysis?

6. Who are the personnel (what functionalareas are involved) needed for theevaluation?

7. Who will provide the decision processand facilitation support?

8. How will the decision team interact?

9. Who are the reviewers, decision-maker(s), and stake holders?

10. Is external review (e.g. DNFSB, etc.)anticipated.

11.Has the decision stalled out in the past?

12.What is the level of complexity? Simple, only a few parameters tell

the whole story

Complicated, many parameters anddata that are not known by one per-son

Complex interactions and quantita-tive data needed to support decision

Need an expert system developed tohelp make repetitive consistent deci-sions in area of problem

The Problem Definition Worksheet, FigureB-1 (page 34), presents some probing ques-tions which can be used to refine the prob-lem. Often, the initial problem descriptionis not adequate to ensure that a full solutionwill be found. These questions are not theonly probing questions, just a sampling ofsome of those used by DOE in the past.


32/44


28

Such questions are the essence of the trade-offs that may be found and discussed in theevaluation.

1.3 Aid to Identify Requirements andGoals

The essential task in this step is to segregatethe requirements from the goals. Require-ments are used to screen inadequate alterna-tives from further evaluation. The true testis to ask the question, If an otherwise goodalternative does not meet this requirement,should it be dismissed or considered? Ifthe answer is to dismiss the alternative, thisis truly a requirement. If the answer is toconsider the alternative, then this require-ment must be changed to a goal. Goals arenegotiable desirables. Tables B-1 and B-2(page 35) provide some examples of re-

quirements and goals.

Requirements may contain both strategicand functional elements that may or may notapply to a specific problem. Applicablestrategic objectives can generate either re-quirements or goals. Table B-3 (page 35)gives the strategic requirements to be con-sidered for the INMM Program.

1.4 Aid to Define Alternatives

Alternatives are distinct potential solutions,which convert the initial state to the desiredstate. Alternatives should differ from eachother in how they achieve the desired state.

Alternatives can be discovered in manyways. They can be developed throughbrainstorming, or from examining / chal-lenging constraints or requirements. Thisrequirement challenging can also lead to thediscovery that a requirement is really a goal.

Alternative definition typically occurs intwo phases: generation and refinement. Inthe generation phase, the decision supportteam focuses on the goals and requirementsin an effort to produce ideas on how to per-form the identified function. Each goal andrequirement is evaluated and alternativesfocused on meeting these goals and re-

quirements are suggested. The generationphase often results in infeasible concepts.

Next, in the refinement phase, the decisionsupport team combines and completes theconcepts so that they become alternativesworthy of consideration.

Alternatives must be defined in such a wayas to clearly explain how they will solve theparticular problem. Alternatives must bedefined at a level that enables comparativeanalysis. This may take a good written de-scription and a diagram of the specific func-tions performed to remedy the problem. Asimple functional diagram, as seen in FigureB-2 (page 36), can clarify an alternative.The diagram should be adequately detailedto describe the differences between this andthe other alternatives. Each definition

should enable the decision support team tounderstand how that alternative solves theproblem and how it differs from the otheralternatives.

1.5 Aid to Defining Discriminating

Criteria

Each goal is something desired from the al-ternative. Criteria are measures of effective-ness for each goal. The job of the criteria isto discriminate between the alternatives.

Some goals may require more than one cri-terion to fully capture the discrimination.Other goals may not have anything measur-able or estimable that discriminates. Thesegoals need to be set aside. Each criterionwill need a description and a unit of meas-ure/estimation definition. These definitionsmust be sufficient for sharing with the deci-sion-maker(s) and/or allow the team to per-form an initial qualitative evaluation. Goalsand criteria must be reviewed and approvedby the decision-maker(s) prior to alternative

evaluation.

Table B-4 (page 36) shows an example cor-relation between some goals and criteriaalong with criteria descriptions and an initialattempt as to what should be measurable orestimable. Maintaining the association of thegoals and criteria is important for two rea-


33/44


29

sons. First, it is important to remember whyand what a criterion is supposed to be meas-uring. Second, the association helps answerthe question, Are these all the criteria, andhow do you know all the essential criteriaare identified? Answering this questionrequires two steps: (1) confirm that all goalsare identified, and (2) confirm that discrimi-nating criteria have been identified for eachgoal.

Ultimately, criteria selection greatly influ-ences the selection of an alternative. A goodway to begin is to list all the advantages(pros) and disadvantages (cons) of each al-ternative. These pros and cons are then usedto suggest discriminating criteria for eachgoal. Each goal should have one or morecriteria. Each criterion should measuresomething important, and not be directly

dependent on another criterion.

Many DOE decisions are driven by similarforces, which results in similar criteria fordecisions. Generally, one or a few criteriaare selected for each goal as alternatives aredeveloped and potential discrimination dis-cussed. Fewer real discriminators will resultin a clearer, more understandable decisionanalysis product. Table B-7 (page 39) con-tains an initial set/starting point of goals andpotential discriminating criteria for INMM

Program decisions. Other programs andprojects need to define their own/other goalsand associated criteria as appropriate.

1.6 Aid toScreening Alternativesagainst Criteria

Alternative evaluation can be performed inmany ways, using many methods and tools.Be aware that the size of the team needed isproportional to the quality of the data avail-able the more intangible and qualitative

the data, the greater the number of peoplethat should be involved. It is important tohave a team with broad based expertise andto perform enough analysis to support a rec-ommendation or decision. Evaluations needto 1) avoid completion of analysis too soonsuch that the recommendation or decision isnot defensible but 2) avoid too much

evaluation and delaying the decision longerthan necessary. It is important to realize thatas the detail in an analysis increases, moredetailed data are required that generallycosts time and money. Maintaining a bal-ance between further evaluation and beingtechnically defensible deserves a lot of at-tention.

As alternatives are evaluated the criteria canbe either qualitative or quantitative in form.The first evaluation step is the screening outof alternatives because requirements cannotbe met. If a decision-maker(s) determinesthat one or more screened alternativesshould be retained and evaluated, the re-quirement causing it to be initially screenedout is moved to the goal list and discrimi-nating criteria defined for the new goal. Ifafter the requirement screening there are still

too many alternatives to fully evaluate, aqualitative analysis may be needed to reducethe number of alternatives needing data col-lection.

An example of screening requirements isshown in Table B-5 (page 37). This exam-ple is excerpted from an analysis of DOEowned spent nuclear fuel, which might be acandidate for Savannah River Site canyonprocessing. There are two requirements forevaluating the fuels as potential canyon can-

didates:

1. Compatibility with current canyon proc-esses. There must be a flow-sheet avail-able to process the fuel in an SRS can-yon.

2. Not currently qualified for the reposi-tory. DOE owned commercial fuel, forinstance, is identical with commercialspent nuclear fuel, and so there shouldbe no technical questions about its re-pository disposal.

Based on these two requirements oxidecommercial, oxide disrupted (TMI), metal(UZr), metal (U,UMo), N Rx and graphitefuel are screened out of the analysis becausethey do not meet one or both of the require-ments.


34/44


30

Evaluation of alternatives can occur in oneto three phases depending on complexity.

Phase I--Qualitative Analysis:

EvaluateAlternatives

Against Criteria

Collectedcriteria data foreachalternativemodel data,

research data)

Criteria withdefinedmeasures of

effectiveness

Phase I is generally performed first and mayinclude the Pro-Con method where the ad-vantages (pros) and disadvantages (cons) ofan alternative are described.

If knowledgeable experts make up theevaluation team, it can be useful to performa subjective analysis using qualitative scor-ing by each team member. Several of theevaluation methods can support this initialevaluation. This initial assessment evaluates

where perceptions of the criteria scores dif-fer across the decision support team. Theseareas of disagreement become the minimumset of criteria that need quantitative data orinformation collected. For those criteriawhere there is good agreement across thedecision support team, it may not be neces-sary to collect any additional data, depend-ing on the types of review expected and howuncontroversial the discrimination.

The qualitative analysis can also be used to

identify dominating alternatives. A knowl-edgeable team can set aside alternatives thatare clearly dominated by other alternativesprior to more sophisticated analysis. Thisprocess is called elimination by dominance.

A selection can be made based on thisanalysis if the winning alternative is veryclear and there is a single decision makerwith no external review expected. If this istrue, the evaluation step is complete and thesupporting information for the decision is

the results of the pro/con or qualitativeanalysis. If the pro/con analysis identifiedtradeoffs then a more sophisticated methodof analysis is required.

Phase II--Quantitative Analysis:

Get NeededQuantitative Data(build models if

necessary)

Additionalevaluationdata needsfor criteria

Modeled dataor research

information

Criteria judgements may require improve-ment through data collection and quantita-

tive analysis. The need for these data to bemore quantitative depends on the impor-tance of the data to the evaluation of the al-ternative. If the qualitative data is well un-derstood with little uncertainty, then gener-ating additional information may not be war-ranted. However, if there is significant un-certainty in one or more alternatives per-formance, or the cr

Documents

Decision Making Guidebook