Using the Analytic Hierarchy Process to Examine Judgment Consistency in a Complex Multiattribute Task

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 40, NO. 5, SEPTEMBER 2010 1105

Using the Analytic Hierarchy Process toExamine Judgment Consistency in a

Complex Multiattribute TaskJeonghwan Jin, Ling Rothrock, Senior Member, IEEE, Patricia L. McDermott, and Michael Barnes

Abstract—This paper investigates the impact of framing andtime pressure on human judgment performance in a complexmultiattribute judgment task. We focus on the decision processof human participants who must choose between pairwise al-ternatives in a resource-allocation task. We used the AnalyticHierarchy Process (AHP) to calculate the relative weights of thefour alternatives (i.e., C1, C2, C3, and C4) and the judgmentconsistency. Using the AHP, we examined two sets of hypothesesthat address the impact of task conditions on the weight priori-tization of choice alternatives and the internal consistency of thejudgment behavior under varying task conditions. The experimentsimulated the allocation of robotic assets across the battlefield tocollect data about an enemy. Participants had to make a judgmentabout which asset to allocate to a new area by taking into accountthree criteria related to the likelihood of success. We manipulatedthe information frame and the nature of the task. We foundthat, in general, participants gave significantly different weightsto the same alternatives under different frames and task condi-tions. Specifically, in terms of ln-transformed priority weights,participants gave significantly lower weights to C2 and C4 andhigher weight to C3 under gain frame than under loss frame,and also, under different task conditions (i.e., Tasks #1, #2, and#3), participants gave significantly higher weight to C4 in Task#1, lower weights to C1 and C4, higher weight to C3 in Task#2, and lower weight to C3 in Task #3. Furthermore, we foundthat the internal consistency of the decision behavior was worse,first, in the loss frame than the gain frame and, second, undertime pressure. Our methodology complements utility-theoreticframeworks by assessing judgment consistency without requiringthe use of task-performance outcomes. This work is a step towardestablishing a coherence criterion to investigate judgment undernaturalistic conditions. The results will be useful for the designof multiattribute interfaces and decision aiding tools for real-timejudgments in time-pressured task environments.

Index Terms—Analytic Hierarchy Process (AHP), framingeffect, judgment consistency, multiattribute task.

Manuscript received May 28, 2008; revised May 3, 2009. Date of publicationApril 26, 2010; date of current version August 18, 2010. This work wassupported in part by the U.S. Army Research Laboratory and in part by AlionScience and Technology. The views, opinions, and/or findings expressed in thispaper are those of the author(s) and should not be construed as an officialDepartment of the Army position, policy, or decision, unless so designatedby other documentation. This paper was recommended by Associate EditorE. J. Bass.

J. Jin and L. Rothrock are with the Harold and Inge Marcus Department ofIndustrial and Manufacturing Engineering, The Pennsylvania State University,University Park, PA 16802 USA (e-mail: [email protected]; [email protected]).

P. L. McDermott is with Alion Science and Technology, McLean, VA 22102USA (e-mail: [email protected]).

M. Barnes is with the U.S. Army Research Laboratory, Fort Huachuca,AZ 85613 USA (e-mail: [email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TSMCA.2010.2045119

I. INTRODUCTION

THIS paper evaluates the judgment effectiveness of humanparticipants in a dynamic resource-allocation task in a

command-and-control context. Specifically, we focus on theeffect of information framing and task complexity on judgmentperformance. Framing refers to the manner in which a choiceproblem is presented, and its effects on the decision processeshave been demonstrated in judgment and decision-makingliterature [1]. Framing effects have been found in loss/gainand sunk-cost problems [2], [3], in a multiattribute dynamicdecision-making environment [4], and across distinct task do-mains [5]. More recently, its effects have also been investigatedin medical decision-making processes [6], in financial decision-making processes [7]–[9], and in a risk-elicitation process [10].

Framing has been characterized in diverse manners in thedecision literature. Most decision researchers describe framingeffects in terms of deviation from an expected value that isbased on a decision maker’s utility function. For example, Kah-neman and Tversky [11] demonstrated that framing outcomesas losses versus gains can cause reversals in the selected choicethrough using a definition of risk as the deviation from theexpected utility. Whereas research efforts that rely upon utilitytheory are mathematically rigorous, the derived results may notbe representative of human capabilities [12]. In the light ofdecision-making research which suggests that humans derivesolutions that are satisfactory rather than optimal [13], [14] andthat humans utilize simple heuristics rather than elaborate com-putation methods [15], [16], utility-theoretic representations ofhuman judgment have become untenable.

In addition to objections about assumptions on human capa-bilities, there exists the difficulty of establishing a coherencecriterion for truth [17] in representative environments, i.e., itis difficult to assess the internal consistency of the observedbehavior with axioms prescribed by some normative modelwhen no normative model has been explicitly adopted by theresearch community.

Therefore, we propose a testable coherence criterion toaddress both information framing and task complexity. Wedesigned our experimental tasks based on Nygren’s work [4]in which participants performed three multiattribute dynamicdecision-making tasks (i.e., data-tracking, system-monitoring,and resource-management tasks) under different framing con-ditions of the payoff scheme (e.g., accumulating performancepoints under positive framing and losing performance pointsunder negative framing), and their task performances, such

1083-4427/$26.00 © 2010 IEEE

1106 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 40, NO. 5, SEPTEMBER 2010

Fig. 1. Representation of the resource-allocation problem.

as type of error, reaction time, and number and pattern ofresponses, were measured in terms of accuracy and latency.While Nygren [4] provides an analysis of task performancemeasures subject to framing effects, we investigate judgmentperformance subject to information framing and task com-plexity manipulations. We consider whether participants makeconsistent judgments within a specific frame, across loss andgain frames, and across tasks of different complexity. Wewill address consistency within a set of judgments as wellas the persistence of weight prioritization under different taskmanipulations.

II. PROBLEM STATEMENT

Military robots have revolutionized the battlefield as sensorsand weapons [18], [19], and the assessment of human allocationstrategies for limited assets is an area of active research. Fig. 1shows a graphical representation of the problem. A regionalcommander specifies Named Areas of Interest (NAIs) that needto be monitored for enemy activity for a particular time interval.Given that a new NAI has just been specified, the operator mustrearrange the existing robotic assets, shown as wedges, to createa new asset configuration that has sufficient coverage on allthe NAIs. In this problem, the coverage sufficiency of an assetis defined along three attributes. First, the timeliness attributeaddresses the probability that the asset will reach the new NAIwithin the defined time interval. Second, the longevity attributeconcerns the probability that the asset will be able to collectdata for the duration of the time interval. Third, the expectedcoverage attribute deals with the probability that the overall areais covered by all the assets.

The problem is hierarchically shown in Fig. 2. The primaryobjective of selecting a better asset configuration can be as-sessed using multiple attributes (i.e., timeliness, longevity, andexpected coverage), which can, in turn, be decomposed intospecific configurations (spatial arrangements of the asset, witheach configuration having its own attribute values).

III. USING THE AHP TO MEASURE JUDGMENT

In utility theory, a key problem is the elicitation of the utilityor worth of an attribute along some meaningful measurementscale. The problem is compounded in decisions involving mul-

Fig. 2. Hierarchical representation of judgment task.

tiple attributes. Therefore, a workable definition of consistencyalong the lines of the normative axioms of decision makingis not tenable. To circumvent the difficulties associated withutility theory, we used the Analytic Hierarchy Process (AHP).

The AHP is a general theory of measurement used toderive ratio scales from human judgments through multiplepaired comparisons in a hierarchically structured task [20]. Themethod of paired comparisons is not a new psychophysicaltechnique (for an introductory treatment, see [21]) and is afoundational element for modern psychophysics [22]. The AHPhas been used to assist decision making in applications rangingfrom bridge-design selection to product-pricing-strategy choice[23]. It has also been used in human-factor applications such asto rank order computer interfaces [24], to ascertain appropriateknowledge-elicitation methods [25], to select attributes fordesigning virtual environment systems [26], to isolate Gestaltgrouping principles to organize home pages [27], and to analyzethe decision process itself in multiattribute decisions [28].While a brief introduction to the AHP is presented in this paper,a more comprehensive tutorial on using the AHP is provided byMitta [24].

The first step in the AHP is to model the problem as ahierarchy, such as that shown in Fig. 2, containing the decisionobjective, the alternatives for reaching it, and the attributes forevaluating the alternatives. The next step is to establish priorityweights among the elements (i.e., alternatives and attributesin Fig. 2) of the hierarchy by making a series of judgmentsbased on pairwise comparisons of the elements. More specif-ically, participants make n(n − 1)/2 pairwise comparisons forn elements using a nine-point scale ranging from absolutedominance to equality. The results of the pairwise comparisonsare then arranged in a reciprocal matrix

A =

⎡⎢⎢⎣

1 w1/w2 · · · w1/wn

w2/w1 1 · · · w2/wn

...... · · ·

...wn/w1 wn/w2 · · · 1

⎤⎥⎥⎦ (1)

where wj (j = 1, 2, . . . , n) is the weight value of element j.Given a reciprocal matrix, the next step is to compute a

weight vector w, which represents the relative weights of theelements of the hierarchy. Saaty [20] showed that w must satisfy

Aw = λmaxw (2)

where λmax is the largest eigenvalue of A. Mathematically, wcan be obtained by normalizing the principle eigenvector of A.

JIN et al.: USING THE ANALYTIC HIERARCHY PROCESS 1107

TABLE IRANDOM INDEX (RI)

For a multilevel hierarchy, we can establish a hierarchicalvector of weights wh

i where i is an element of the hierarchy ina level h − 1. Each vector of weights can then be aggregatedinto a performance matrix for each level of the hierarchy Wh.Finally, a performance matrix containing priority weights of thelowest level of the hierarchy with respect to the overall objectiveU can be calculated as

U = W v,W v−1, . . . ,W 2 (3)

where v is the number of levels in the hierarchy. Details of thederivation can be found in [20] and [28].

In addition to calculating priority weights, the AHP is alsoused to assess judgment consistency. Since small changes in areciprocal matrix element imply a small change in λmax, thedeviation of the latter from n can be taken as a measure ofconsistency. Saaty [20] uses this deviation as a measure he callsthe consistency index, where

CI =λmax − n

n − 1. (4)

In other words, a set of choices is most consistent if CI = 0.The CI can then be compared with a random index (RI), whichis the CI value of a randomly generated comparison matrix,to determine if any inconsistency found is acceptable. This iscalled the consistency ratio (CR). CR is then calculated asfollows, and the RI values for a different number of n areshown in Table I:

CR =CI

RI. (5)

Once again, for a multilevel hierarchy, an aggregate CR canbe obtained by considering the CI and RI of each level. Forinstance, the CI of a three-level hierarchy M , like ours, can becalculated as

M = CI2 + W 2

⎡⎢⎢⎣

CI31

CI32

...CI3

m

⎤⎥⎥⎦ (6)

where CI2 is the CI value of the second level and CI3i is

the CI value for the ith element of the third level for melements. Similarly, the RI for a three-level hierarchy M̄ canbe calculated as

M̄ = RI2 + W 2

⎡⎢⎢⎣

RI31

RI32

...RI3

m

⎤⎥⎥⎦ (7)

where RI2 is the RI value for the number of elements inthe second level and RI3 is the RI value for the number of

elements in the third level. Finally, the consistency ratio of thehierarchy (CRH) can be calculated as

CRH =M

M̄. (8)

As with CR, consistency in judgment increases as CRHdecreases.

For readers who are not familiar with the AHP procedure, athorough example of demonstrating the use of (1)–(8) for ourproblem is provided in the Appendix.

We acknowledge that the AHP use has been controversial.Specifically, Dyer [29], [30] claimed that the AHP is flawedas a procedure because of the phenomenon of rank reversal inwhich an added alternative may cause the rank of two existingalternatives to be reversed. Harker and Vargas defended theAHP on the grounds that rank reversal is not a flaw becauseit is consistent with the nature of relative measurement [31] andthat the strength of the attribute should be preserved, not thechoice alternatives [32]. Nevertheless, Harker and Vargas [33]do stipulate that alternatives at level h of the hierarchy mustbe independent from attributes at level h − 1. This stipulationholds for our experiments. Dyer [30] and Harker and Vargas[33] provide additional viewpoints.

IV. RESEARCH HYPOTHESES

In this paper, we focus on the decision process of humanjudges who must choose between pairwise alternatives in aresource-allocation task. We regard the persistence of weightprioritization as the similarity of priority weights of choicealternatives over different experimental treatments. Moreover,we regard judgment consistency as the CRH value of a judge.

Using the AHP, we examined the following researchhypotheses.

1) Weight prioritization remains the same under gain or lossframing (H1).

2) Weight prioritization remains the same under differenttask types (H2).

3) Judgment consistency remains the same under gain orloss framing (H3).

4) Judgment consistency remains the same under differenttask types (H4).

The first two hypotheses deal with the impacts of framingand task conditions on the weight prioritization of alternatives.The second two hypotheses address the internal consistency ofthe decision behavior under varying task conditions.

V. EXPERIMENTAL DESIGN AND PROCEDURE

The experiment utilized a two-factorial within-subjects de-sign. One factor was framing and consisted of a gain frame anda loss frame. The other factor was task type and included a statictask (Task #1), a human-in-the-loop simulation task (Task #2),and a human-in-the-loop simulation task under time constraint(Task #3). Because of task differences and the amount of time


TABLE IIDEFINITIONS OF THREE ASSET ATTRIBUTES

UNDER GAIN AND LOSS FRAMES

pressure applied, our unit of analysis is the outcome of aparticipant under a specific framing condition. Each outcometakes two forms: 1) priority weights of four alternatives and 2)a CRH value.

Forty-eight Pennsylvania State University undergraduate stu-dents (19 women and 29 men) participated in the experi-ments. The students ranged in age from 19 to 24 years (μ =21.6 years). The experiment took approximately 2 h to com-plete, and participants were compensated with $20.

Each of the 48 participants performed the required tasksunder all six conditions of the two factors (i.e., 2 × 3 = 6).To avoid any learning or carryover effects, we counterbalancedthe experimental design such that half of the participants wereassigned to the gain frame first and then to the loss frame andthat the other half of the participants were assigned to the lossframe first and then to the gain frame. To further reduce learningeffects, the order of tasks presented to the participants undereach frame was randomized.

For each task presentation, the participant made a choicebetween two configurations of robotic asset allocation. Partic-ipants were instructed that robotic assets have already beenplaced across the battlefield to cover a specific NAI (see Fig. 1).However, a new NAI has been identified, and because of timeconstraints, there is only time enough to move one asset to thenew NAI. The system recommends the two best assets to bemoved to the new NAI, and the human decides which asset tomove. In deciding between configurations, participants neededto consider coverage sufficiency per attribute listed in Table II.The first three and the last three attributes in the table were usedto represent the coverage sufficiency of an asset configurationin the gain and loss frames, respectively. In essence, thereexist probabilistic relations among the attributes, such as U =1 − T, D = 1 − L, and EV = 1 − EC. For instance, if theTimeliness of an asset was 0.67 in the gain frame, then theUntimeliness of the same asset was 1 − 0.67 = 0.37 in the lossframe. These relations were used to frame the same assetconfigurations as gains or losses in the actual experiment.

To elicit the participants’ preference, we established twopossible values per attribute, namely, average and high in thegain frame (or average and low in the loss frame), and generatedonly four combinations per frame, as shown in Table III. While

TABLE IIIASSET ALLOCATION CONFIGURATIONS UNDER GAIN AND LOSS FRAMES

the number of possible combinations of the values of the threeattributes per frame is eight (i.e., 23 = 8), we used only fourof them because of the following: 1) the other possible fourcombinations, which are not included in Table III, are dominantover the generated four combinations, and 2) if they wereincluded, it would drastically increase the number of pairwisecomparisons for the generated configurations.

Based on the levels of the three attributes, we termed thefour combinations as C1–C4 in the gain frame (and as C5–C8

in the loss frame). In essence, considering the probabilisticrelations aforementioned, C1–C4 and C5–C8 are identical, butare framed as gains and losses, respectively.

Since the normal distribution has been widely used to rep-resent the lifetime of items or systems (e.g., Longevity in ourproblem), which exhibit failure due to wear or malfunction, andalso the variations in value (e.g., Timeliness and Expected Cov-erage in our problem) of electronic components or mechanicalsystem, we utilized the normal distribution as the probabilitydistribution of the three attributes and randomly generated theaverage values from a normal distribution N (μ = 0.50, σ =0.0567), the high values in the gain frame from N (μ =0.83, σ = 0.0567), and the low values in the loss frame fromN (μ = 0.17, σ = 0.0567).

A. Task #1

In Task #1, participants made a preference judgment ona nine-point scale between two asset configurations such asthe ones shown in Fig. 3. The display showed a gain-framingcondition and consisted of a map of the battlefield (left upper),an intelligence window that provided some detailed informationabout each scenario (left bottom), a decision aid that showed thetwo candidate configurations and the values of three attributesassociated with the two configurations (right upper), and aquestion to elicit the participants’ preference between the twoasset configurations (lower right). This display was static, andparticipants had no time limit to complete each question.

Note that the question posed in Fig. 3 looks like selecting oneof the two sensor assets (ARV-1 and MULE-1) recommendedby the system. In essence, the participant is choosing betweenasset configurations, because these are the configurations that


Fig. 3. Snapshot of Task #1’s graphical display.

define an asset’s values. For instance, if ARV-1 is chosen, thenthe overall Timeliness, Longevity, and Expected Coverage of aconfiguration would be high, average, and average, respectively,and the configuration corresponding to these levels is C1, aslisted in Table III. Similarly, a configuration corresponding tochoosing MULE-1 is C2.

Task #1 Procedure: First, a participant was randomly as-signed to either gain- or loss-framing condition and made atotal of 18 pairwise comparisons among the four asset con-figurations (i.e., C1–C4 or C5–C8, as shown in Table III)with regard to the three asset attributes listed in Table II. Inother words, if the participant was in the gain frame, he/shemade six pairwise comparisons among the four configurationswith regard to Timeliness, another six comparisons with regardto Longevity, and the other six comparisons with regard toExpected Coverage. A sample question follows.

Question: It has been determined that the following twoassets will be the best choice to allocate to NAI X. Withregard to Timeliness (or Untimeliness in the loss frame),which asset will you select?

Once the 18 comparisons are completed, each participantperformed three more pairwise comparisons among the threeattributes with respect to the overall objective. A sample ques-tion follows.

Question: Using a 9-point scale, compare the relativeimportance of the following two asset attributes with re-spect to the overall goal (i.e., select a better asset config-uration). a) Timeliness versus Longevity (or Untimelinessversus Disruptivity in the loss frame).

To complete Task #1, the participant was then assigned tothe other framing condition and made another 21 pairwisecomparisons (18 comparisons for other frames, 4 configurations

with regard to the three attributes, which had also been termeddifferently, and 3 comparisons for those three attributes).

B. Task #2

In Task #2, each participant answered pairwise comparisonquestions presented through a human-in-the-loop simulation, asshown in Fig. 4. The display showed a gain-framing conditionand consisted of information available in Task #1 to includea map, a decision aid, an intelligence window, and a pairwisecomparison question.

Compared with Task #1, Task #2 has four different character-istics: 1) The assets move from an initial position to a final oneduring each scenario; 2) the likelihood of an enemy presenceand its size are shown in the right upper corner of the mapand becomes more certain (i.e., approach unity) as a scenarioprogresses; 3) there was a time limit such that each simulationtrial had to be completed in 20 s; and 4) in deciding between thetwo configurations, each participant needed to consider all threeasset attributes simultaneously while the scenario progressed.

To measure judgment in this task, we did not stop thesimulation to query each participant. Instead, we extractedpreferences based on the values associated with each attributeon the configuration chosen, i.e., we asked which asset theywould actually move, thereby forcing them to consider allattributes simultaneously.

Task #2 Procedure: As in Task #1, a participant was firstrandomly assigned to either framing condition and then an-swered a series of pairwise choice questions for the samecandidate configurations used in Task #1. Once one frame iscompleted, the participant was assigned to the other frame andanswered another series of pairwise choice questions for thefour configurations, which had been framed differently.


Fig. 4. Snapshot of Task #2’s graphical display.

In this task, the total number of pairwise choices was 12 toaccount for all possible pairwise combinations of the four assetconfigurations and the two different frames (i.e., six pairwisecomparisons in the gain frame and the other six comparisons inthe loss frame). In addition, each participant compared the threeasset attributes with each other, in each frame, with respect tothe overall objective for six more comparisons.

C. Task #3

Task #3 was similar in all aspects to Task #2 with oneexception: The participant had only 10 s to view the decision aidand decide which asset to reallocate. This applied time pressureto the set of tasks. The Task #3 procedure was identical to theTask #2 procedure.

VI. MEASURES AND STATISTICAL ANALYSIS

Since the number of pairwise comparisons was differentacross the task types, we used outcome measures that resultedfrom the individual comparisons. Two dependent variableswere calculated from the participants’ data using the AHP. Oneis the priority weights wi (i = 1, 2, 3, 4) of the four configu-rations shown in Table III under either framing condition foreach task. The other is the CRH value under either framingcondition for each task.

Due to the unit sum constraint on the priority weights (i.e.,w1 + w2 + w3 + w4 = 1), meaning that the priority weights

are highly correlated, a traditional multivariate analysis ofvariance (ANOVA) (MANOVA) model could not be used toanalyze the priority weights. To resolve this high-correlationissue (and, eventually, to perform MANOVA and then ANOVAon the priority weights), we used the c-compositional analy-sis [34]–[37] and ln-ratio transformed the priority weights asfollows:

ratio1 = ln(

w1

w4

)

ratio2 = ln(

w2

w4

)

ratio3 = ln(

w3

w4

).

We did this because an ln-ratio transformation of the compo-sitional data makes the transformed variables independent, andstatistical tests can then be applied [34]. In the transformation,we designated w4, the priority weight of the fourth configu-ration, as the denominator in these ratios because the fourthconfiguration formed the smallest portion of the total weight(see Table IV), thus facilitating a better understanding of therelative contribution of the other configurations.

After the transformation, we checked the normality of thethree ratios and found that they were very close to normal withzero mean and independence. Then, we proceeded to performa two-way MANOVA test (shown below) on the three ratios


TABLE IVPRIORITY WEIGHTS OF FOUR CONFIGURATIONS UNDER GAIN AND LOSS FRAMES ACROSS TASKS

TABLE VUNIVARIATE TEST STATISTICS FOR ln-RATIO TRANSFORMED

WEIGHTS INCLUDED IN MANOVA

TABLE VIUNIVARIATE TEST STATISTICS FOR ln-TRANSFORMED

WEIGHTS INCLUDED IN MANOVA

in order to determine which ratio(s) had been significantlyaffected by the independent variables (i.e., framing conditionsand task types)⎛⎝ratio1

ratio2

ratio3

⎞⎠=Frame + Task Type + Frame × Task Type + error.

When the MANOVA results showed a significant effect offraming, we performed separate univariate ANOVA tests on thethree ratios and found that there were significant framing, tasktype, and interaction (frame × task type) effects for some ofthe ratios (see Table V). In addition, to further determine whichpriority weight has been changed significantly, we conductedunivariate ANOVA tests (shown below) on each of the ln-transformed priority weights (i.e., ln(w1), ln(w2), ln(w3), andln(w4)) and found that there were significant framing, tasktype, and interaction (frame × task type) effects for some ofthe ln-transformed priority weights (see Table VI)

ln(w1) = Task Type+error

ln(w2) = Frame+Frame×Task Type+error

ln(w3) = Frame+Task Type+error

ln(w4) = Frame+Task Type+Frame×Task Type+error.

Finally, through Tukey’s post hoc tests, we completed multi-ple comparison tests on the significant univariate tests.

To analyze the CRH values, we used a log transformation(base 10) on the CRH values because the variance acrosstasks was not constant. After transformation, we conducted anANOVA on the transformed CRH values, log10(CRH).

We note that because the focus of our work is on judgmentperformance rather than task performance, we did not examinetask-performance outcomes such as latency and accuracy.

VII. RESULTS

For all statistical tests, an alpha level of 0.05 was used.H1 and H2: Weight prioritization was analyzed by con-

ducting a MANOVA model on the ln-ratio-transformed priorityweights of the four alternatives for the task type (i.e., static,human-in-the-loop simulation, and human-in-the-loop simula-tion under time constraint) and framing (i.e., gain and loss)considered. Since the MANOVA results showed the significanteffects of framing (Wilks’ Λ = 0.376, F(3,45) = 24.86, andp < 0.001), task type (Wilks’ Λ = 0.396, F(6,184) = 18.06, andp < 0.001), and the interaction of task type and framing (Wilks’Λ = 0.689, F(6,184) = 6.28, and p < 0.001), we proceeded toconduct separate univariate ANOVA tests on the three ratios(Table V) and found that there was a significant differencebetween gain and loss frames for all ratios except ratio2 andthat the task type effect was also significant for all ratios.In addition, the interaction (frame × task type) effect wassignificant such that, for all three ratios, the difference betweengain and loss frames was greater in Task #1 than in those inTasks #2 and #3.

To further examine which priority weight has been changedsignificantly, we conducted univariate ANOVA tests on the ln-transformed priority weights (Table VI) and found that therewas a significant difference between gain and loss frames forall ln-transformed weights except ln(w1). We also found thatthe task type effect was also significant for all ln-transformedweights except ln(w2) and that, for the interaction (framing ×task type) effect (see Fig. 5), only ln(w2) and ln(w4) showedsignificant results such that, for ln(w2), which is the weightgiven to C2, the difference between the gain and loss frameswas increased as the complexity of task increased. In addition,for ln(w4), participants gave considerably different weights toC4 under different frames in Task #1, whereas in Tasks #2and #3, they gave similar weights to C4 under either framingcondition.

Through Tukey’s post hoc tests shown in Table VII, it wasalso found that the participants gave significantly differentweights under different frames and also under different taskconditions. Specifically, participants gave significantly lowerweights to C2 and C4 and higher weight to C3 under gain frame


Fig. 5. Interaction plots for (a) ln(w2) and (b) ln(w4).

TABLE VIITUKEY’S Post Hoc COMPARISONS OF ln-RATIO TRANSFORMED

WEIGHTS FOLLOWING SIGNIFICANT UNIVARIATE TESTS

than under loss frame and under different task conditions (i.e.,Tasks #1, #2, and #3), and participants gave significantly higherweight to C4 in Task #1, lower weights to C1 and C4 and higherweights to C3 in Task #2, and lower weight to C3 in Task #3.

H3 and H4: The CRH values were used as the dependentvariables to investigate judgment consistency under framingand task manipulations. The mean CRH and log10(CRH)values are shown in Table VIII.

We conducted an ANOVA on the log10(CRH) values for thetask type and framing considered and found the significant main

TABLE VIIICONSISTENCY RATIO OF THE HIERARCHY (CRH) VALUES

effects of task type, F(2,94) = 144.75,p < 0.001, and framing,F(1,94) = 28.26, p < 0.001. In particular, mean log10(CRH)values increased as task conditions became more time stressed.In other words, judgment consistency decreased as task con-ditions became more time stressed. Moreover, loss-framingconditions resulted in lower task consistency than gain-framingconditions. However, the interaction (task type × framing)effect was not significant.

VIII. DISCUSSION

The results conform to findings in traditional judgmentliterature, which rely on utility theoretic axioms. However,instead of deviation from an expected value, we offer an alter-native explanation in terms of judgment consistency betweenand within treatment conditions subject to framing and time-pressure effects.

A. Impact of Framing and Task Type on Weight Prioritization(H1 and H2)

Our results suggested that there exists a statistical effectof framing and task type on the weight prioritization of sen-sor configurations. The mean values of ln-ratio-transformedweights and ln-transformed weights show that participantsweighed the same alternatives differently under different framesand in different task conditions. Our findings are consistentwith the argument that dynamic decision environments impactjudgment behavior [38]. Specifically, our results are consistentwith findings in decision research that time pressure altersone’s choice behavior through inducing suboptimal judgmentheuristics [39]–[41].

B. Impact of Framing on Judgment Consistency (H3)

H3 and H4 address the internal consistency of the deci-sion behavior under framing and time-pressure conditions. Ourfindings suggest that CRH values represent the stability ofweight prioritization across treatments and can be used asthe potential first step for establishing a coherence criterionfor truth [17]. Although utility theory has served as such acriterion for internal consistency, we propose the use of CRHas an alternative criterion to assess judgment consistency ofparticipants. Specifically, we found judgment consistency tosignificantly worsen under loss compared with gain framing.Hence, it appears that the weight prioritization under lossframing is more volatile than that under gain framing. When


information is presented negatively in the loss frame, partic-ipants’ judgments about the relative importance of differentattributes were less likely to follow consistent internal rules. Wethink that this finding complements the stipulation of ProspectTheory [11] that the value function for losses has a differentslope than the corresponding function for gains. However, wealso acknowledge that further investigation is needed to confirmour notions.

C. Impact of Task Type on Judgment Consistency (H4)

We found consistency to deteriorate as the time available tochoose a configuration decreases. Like our findings on weightprioritization, our results on judgment consistency are consis-tent with findings that time pressure [40], [41] affects choicebehavior. Thus, our findings regarding the effect of time pres-sure are consistent with what would be expected if participantshad been using cognitively less demanding strategies. While wedo not have access to the heuristics used by the participants, wedo know that commonly acknowledged heuristics such as Take-the-Best [17], Elimination-by-Aspects [42], and Lexicographic[43] do not guarantee transitivity in preferences over multipleattributes.

In general, participants made significantly different choicesunder the gain versus loss frames under time pressure. Perhapsthe cause of the discrepancy is due to the formation of differentjudgment heuristics under time pressure. Interestingly, in spiteof the uneven number of pairwise comparisons between Task#1 and Tasks #2 and #3, the effect of time pressure (Task #3)was still significant. Nevertheless, we recommend that furtherinvestigation of the impact of uneven comparisons be made.

IX. CONCLUSION

In this paper, we have developed a methodology to assessjudgment performance in a real-time multiattribute judgmenttask. Specifically, we used the AHP to conduct pairwisecomparisons to evaluate human judgment consistency underframing and time-pressure effects. We found that framingsignificantly affected the prioritization of asset configurations.Moreover, we found judgment consistency to significantlyworsen under the loss frame. With regard to time pressure,we found that judgment consistency deteriorates as the time tochoose an asset configuration decreases.

This paper contributes to judgment literature by providing anevaluation of the impact of framing and task type on judgmentperformance. Through an understanding of the impact of taskconditions on configuration prioritization, we gained an abilityto assess the limitations of judgment performance. Addition-ally, through an understanding of the effects of task conditionson judgment consistency, we can analyze the reliability ofhuman judgment.

This paper differs from utility-theoretic frameworks likeProspect Theory in that we can assess judgment consistencywithout requiring the use of task-performance outcomes. More-over, we believe that this work is a step toward establishing acoherence criterion [17] through the use of the CRH to investi-gate judgment under naturalistic conditions. This is particularly

Fig. 6. Reciprocal matrices for choices made by a particular participant toselect a desirable asset configuration.

helpful in complex envisioned domains such as robotic assetallocation tasks in which tactics and procedures have not yetbeen developed and researchers cannot define the one “right”answer. In the absence of a right answer, the CRH providesa tool that system developers can use to determine if theanswers given using specific displays are internally consistent.An additional consideration, which we would like to investigatein the future, is the impact of framing and task type on subject-matter experts. In fact, we think that the tools offered by theAHP can help us to better measure expert consistency.

APPENDIX

EXAMPLE OF CALCULATING PRIORITY WEIGHTS AND

CONSISTENCY RATIO OF THE HIERARCHY

Suppose that a participant generated the reciprocal matrices(Fig. 6) for our problem, which is to select a better assetconfiguration with respect to the three asset attributes (i.e.,Timeliness, Longevity, and Expected Coverage).

To calculate the overall priority weights of the four con-figurations, we first need to calculate the weight vector w ofthe four matrices by normalizing the principle eigenvector ofthe corresponding matrix. Once calculated, the w for the fourmatrices (i.e., one for the three asset attributes and the other


three for the four configurations with respect to each attribute)would be

⎡⎣ 0.701

0.1930.106

⎤⎦

⎡⎢⎣

0.6040.2130.0640.119

⎤⎥⎦

⎡⎢⎣

0.6310.1820.0700.117

⎤⎥⎦

⎡⎢⎣

0.1270.2810.1200.463

⎤⎥⎦ .

In case of a multilevel hierarchy (like ours), these weightvectors are aggregated into a performance matrix Wh for eachlevel h of the hierarchy (see Fig. 2) for h = 2 to v (in our case,v = 3) as follows:

W 2 =

⎡⎣ 0.701

0.1930.106

⎤⎦ W 3 =

⎡⎢⎣

0.604 0.631 0.1270.213 0.182 0.2810.064 0.070 0.1200.119 0.117 0.463

⎤⎥⎦ .

In this case, W 2 reflects the relative weights of the threeattributes with respect to the objective, while W 3 representsa composite of the weights generated by comparing the fourconfigurations with respect to the three asset attributes.

Finally, using (3), the priority weights of the four configura-tions with respect to the overall objective U are calculated as

U = W 3W 2 =

⎡⎢⎣

0.5590.2140.0710.155

⎤⎥⎦ .

To calculate the consistency ratio of the hierarchy (CRH),we use (4)–(8). For example, for the reciprocal matrix of thethree asset attributes, CI and CR are calculated as follows:

CI =λmax − n

n − 1=

3.257 − 32

= 0.129

CR =CI

RI=

0.1290.58

= 0.222

where λmax is the largest eigenvalue of the reciprocal matrix(i.e., 3.257) and RI for n = 3 is 0.58 (see Table I).

The calculated CI and RI values for the other reciprocalmatrices are shown right below the corresponding matrices.

Finally, the CRH value is calculated as follows:

M =CI2 + W 2

⎡⎣CI3

1

CI32

CI33

⎤⎦

= 0.129 +

⎡⎣ 0.701

0.1930.106

⎤⎦

⎡⎣ 0.116

0.1480.473

⎤⎦

= 0.289

and if we take RI2 = 0.58 and RI3 = 0.90 from Table I

M̄ =RI2 + W 2

⎡⎣RI3

RI3

RI3

⎤⎦

= 0.580 +

⎡⎣ 0.701

0.1930.106

⎤⎦

⎡⎣ 0.900

0.9000.900

⎤⎦

= 1.480.

Hence

CRH =M

M̄=

0.2891.480

= 0.195.

REFERENCES

[1] P. J. Schoemaker, “Determinants of risk-taking: Behavioral and economicviews,” J. Risk Uncertainty, vol. 6, no. 1, pp. 49–73, Jan. 1993.

[2] A. Tversky and D. Kahneman, “The framing of decisions and the psychol-ogy of choice,” Science, vol. 211, no. 4481, pp. 453–458, Jan. 1981.

[3] D. Frisch, “Reasons for framing effects,” Org. Behav. Human DecisionProcess., vol. 54, no. 3, pp. 399–429, Apr. 1993.

[4] T. E. Nygren, “Framing of task performance strategies: Effects on perfor-mance in a multiattribute dynamic decision making environment,” Hum.Factors, vol. 39, no. 3, pp. 425–437, Sep. 1997.

[5] X. T. Wang, “Framing effects: Dynamics and task domains,” Org. Behav.Human Decision Process., vol. 68, no. 2, pp. 145–157, Nov. 1996.

[6] A. Moxey, D. O. Connell, P. McGettigan, and D. Henry, “Describing treat-ment effects to patients—How they are expressed makes a difference?”J. Gen. Intern. Med., vol. 18, no. 11, pp. 948–959, Nov. 2003.

[7] Z. Li, H. Li, and N. Li, “The experimentation of bounded rationalitybehavior in individual decision-making,” in Proc. 14th ICMSE, Harbin,China, Aug. 20–22, 2007, pp. 416–421.

[8] N. Miglietta and M. Remondino, “Modeling cognitive distortions ofbehavioural finance,” in Proc. Int. Conf. Comput. Intell., Model. Simul.(CSSim), Sep. 7–9, 2009, pp. 204–209.

[9] J. Li, G. Ren, and L. Liu, “An experimental study on investor’s statusquo bias,” in Proc. 16th ICMSE, Moscow, Russia, Sep. 14–16, 2009,pp. 1582–1593.

[10] M. Pahlman and A. Riabacke, “A study of framing effects in risk elicita-tion,” in Proc. CIMCA-IAWTIC, Nov. 28–30, 2005, pp. 689–694.

[11] D. Kahneman and A. Tversky, “Prospect theory: An analysis of decisionunder risk,” Econometrica, vol. 47, no. 2, pp. 263–291, Mar. 1979.

[12] D. Kahneman, “A perspective on judgment and choice: Mapping boundedrationality,” Amer. Psychol., vol. 58, no. 9, pp. 697–720, Sep. 2003.

[13] A. Newell and H. A. Simon, Human Problem Solving. Englewood Cliffs,NJ: Prentice-Hall, 1972.

[14] H. A. Simon, “Theories of decision-making in economics and behavioralscience,” Amer. Econ. Rev., vol. 49, no. 3, pp. 253–283, Jun. 1959.

[15] V. M. Chase, R. Hertwig, and G. Gigerenzer, “Visions of rationality,”Trends Cogn. Sci., vol. 2, no. 6, pp. 206–214, Jun. 1998.

[16] G. Gigerenzer and D. G. Goldstein, “Reasoning the fast and frugal way:Models of bounded rationality,” Psychol. Rev., vol. 103, no. 4, pp. 650–669, Oct. 1996.

[17] K. R. Hammond, “Generalization in operational contexts: What doesit mean? Can it be done?” IEEE Trans. Syst., Man, Cybern. A, Syst.,Humans, vol. SMCA-16, no. 3, pp. 428–433, May 1986.

[18] S. Harding. Army Demonstrates Future Combat SystemsArmy News Service. Sep. 28, 2005. [Online]. Available:http://www.rdecom.army.mil/rdemagazine/200510/itf_combat.html

[19] T. V. Brook, Drones Reshaping Iraq’s Battlefields. USA TODAY.Jul. 6, 2006. [Online]. Available: http://www.usatoday.com/tech/news/techinnovations/2006-07-06-uavs-iraq_x.htm

[20] T. L. Saaty, The Analytic Hierarchy Process. New York: McGraw-Hill,1980.

[21] J. P. Guilford, Psychometric Methods, 2nd ed. New York: McGraw-Hill,1954.

[22] G. A. Gescheider, Psychophysics: The Fundamentals, 3rd ed. Mahwah,NJ: Lawrence Erlbaum, 1997.

[23] T. L. Saaty and L. G. Vargas, Prediction, Projection and Forecasting:Applications of the Analytic Hierarchy Process in Economics, Finance,Politics, Games, and Sports. Boston, MA: Kluwer, 1990.

[24] D. Mitta, “An application of the analytic hierarchy process: A rank-ordering of computer interfaces,” Hum. Factors, vol. 35, no. 1, pp. 141–157, Mar. 1993.

[25] C.-J. Chao, G. Salvendy, and N. J. Lightner, “Development of a method-ology for optimizing elicited knowledge,” Behav. Inf. Technol., vol. 18,no. 6, pp. 413–430, Nov. 1999.

[26] K. M. Stanney, M. Mollaghasemi, L. Reeves, R. Breaux, andD. A. Graeber, “Usability engineering of virtual environments (VEs):Identifying multiple criteria that drive effective VE system design,” Int.J. Human-Comput. Studies, vol. 58, no. 4, pp. 447–481, Apr. 2003.

[27] S.-W. Hsiao and J.-R. Chou, “A gestalt-like perceptual measure for homepage design using a fuzzy entropy approach,” Int. J. Human-Comput.Studies, vol. 64, no. 2, pp. 137–156, Feb. 2006.


[28] E. E. Spires, “Using the analytic hierarchy process to analyze multi-attribute decisions,” Multivariate Behav. Res., vol. 26, no. 2, pp. 345–361,Apr. 1991.

[29] J. S. Dyer, “A clarification of ‘remarks on the analytic hierarchy process’,”Manage. Sci., vol. 36, no. 3, pp. 274–275, Mar. 1990a.

[30] J. S. Dyer, “Remarks on the analytic hierarchy process,” Manage. Sci.,vol. 36, no. 3, pp. 249–258, Mar. 1990b.

[31] T. L. Saaty, “An exposition on the AHP in reply to the paper ‘remarks onthe analytic hierarchy process’,” Manage. Sci., vol. 36, no. 3, pp. 259–268,Mar. 1990.

[32] P. T. Harker and L. G. Vargas, “Reply to ‘remarks on the analytic hierarchyprocess’,” Manage. Sci., vol. 36, no. 3, pp. 269–273, Mar. 1990.

[33] P. T. Harker and L. G. Vargas, “The theory of ratio scale estima-tion: Saaty’s analytic hierarchy process,” Manage. Sci., vol. 33, no. 11,pp. 1383–1403, Nov. 1987.

[34] J. Aitchison, The Statistical Analysis of Compositional Data. London,U.K.: Chapman & Hall, 1986.

[35] N. J. Aebischer, P. A. Roberson, and R. E. Kenward, “Compositionalanalysis of habitat use from radio-tracking data,” Ecology, vol. 74, no. 5,pp. 1313–1325, Jul. 1993.

[36] J. M. LaMontagne, R. M. R. Barclay, and L. J. Jackson, “Trumpeter swanbehavior at spring-migration stopover areas in Southern Alberta,” Can. J.Zoology, vol. 79, no. 11, pp. 2036–2042, Nov. 2001.

[37] J. L. Beck and J. M. Peek, “Diet composition, forage selection, andpotential for forage competition among elk, deer, and livestock onaspen–sagebrush summer range,” Rangeland Ecol. Manage., vol. 58,no. 2, pp. 135–147, 2005.

[38] R. Lipshitz, “Converging themes in the study of decision making in real-istic settings,” in Decision Making in Action: Models and Methods, G. A.Klein, J. Orasanu, R. Calderwood, and C. Zsambok, Eds. Norwood, NJ:Ablex, 1993, pp. 103–137.

[39] A. J. Maule and G. R. J. Hockey, “State, stress and time pressure,” inTime Pressure and Stress in Human Judgment and Decision Making,O. Svenson and A. J. Maule, Eds. New York: Plenum, 1993, pp. 83–101.

[40] J. W. Payne, J. R. Bettman, and E. J. Johnson, “Adaptive strategy selectionin decision making,” J. Exp. Psychol.: Learn., Memory, Cognition, vol. 14,no. 3, pp. 534–552, 1988.

[41] P. Wright, “The harassed decision maker: Time pressures, distractions,and the use of evidence,” J. Appl. Psychol., vol. 59, no. 5, pp. 555–561,1974.

[42] A. Tversky, “Elimination by aspects: A theory of choice,” Psychol. Rev.,vol. 79, no. 4, pp. 77–94, Jul. 1972.

[43] J. W. Payne, J. R. Bettman, and E. J. Johnson, “The adaptive decisionmaker: Effort and accuracy in choice,” in Insights in Decision Making:A Tribute to Hillel J. Einhorn, R. M. Hogarth, Ed. Chicago, IL: Univ.Chicago, 1990, pp. 129–153.

Jeonghwan Jin received the B.S. degree in industrialengineering from Jeonju University, Jeonju, Korea,in 1996 and the M.S. degrees in manufacturing sys-tem engineering from the University of Warwick,Coventry, U.K., in 1998. He is currently workingtoward the Ph.D. degree in the Harold and IngeMarcus Department of Industrial and Manufactur-ing Engineering, The Pennsylvania State University,University Park.

His research interests include human behaviormodeling, human–robot interaction, and decision-

making processes.

Ling Rothrock (SM’06) received the Ph.D. degreein industrial engineering from the Georgia Instituteof Technology, Atlanta, in 1995.

He was an Officer with the U.S. Army until 2000.He is currently an Associate Professor with theHarold and Inge Marcus Department of Industrialand Manufacturing Engineering, The PennsylvaniaState University, University Park. His research in-terests include human-in-the-loop simulations, be-havioral decision making, display visualization, andhuman–machine performance assessment. He has

published over 60 research articles in these areas.Dr. Rothrock is also a member of the Institute of Industrial Engineers.

Patricia L. McDermott received the M.S. degree inhuman factors engineering from Wright StateUniversity, Dayton, OH, in 2001.

She has over ten years of experience in the fieldof naturalistic decision making and human factorsand is currently with Alion Science and Technol-ogy, Boulder, CO. Her research interests includehuman–robot interaction and decision making underuncertainty.

Michael Barnes received the M.S. degree in experi-mental psychology from New Mexico State Univer-sity, Las Cruces.

He is currently with the U.S. Army ResearchLaboratory, Fort Huachuca, AZ, where he is thehuman–robot interaction Lead for the Army Tech-nology Objective in Safe Operations in UnmannedReconnaissance missions in Complex Environments.He was a Researcher with the Naval Air WarfareCenter Weapons Center, China Lake, CA, and theNaval Air Development Center, Warminster, PA, and

was also the Aegis Combat System human factor’s Unit Manager with theGovernment Services Division, General Electric Cooperation. He is currentlythe Chief of the Fort Huachuca field element where he supports cognitiveresearch in intelligence analysis, unmanned aerial vehicles, and visualizationtools for risk management. He has published over 70 research articles.

Documents

Using the Analytic Hierarchy Process to Examine Judgment Consistency in a Complex Multiattribute Task