Individual differences and the conundrums of user-centered design: Two experiments

Individual Differences and the Conundrums of User-Centered Design: Two Experiments

Bryce AllenSchool of Information Science and Learning Technologies, University of Missouri, 20 Rothwell,Columbia, MO, 65211. E-mail: [email protected]

Individual differences between users of information sys-tems can influence search performance. In user-cen-tered design it is important to match users with systemconfigurations that will optimize their performance. Twomatching strategies were explored in the first experi-ment: the capitalization match, and the compensatorymatch. Findings suggest that a compensatory match islikely to be encountered more frequently in designinginformation systems. Having determined an optimalmatch between users and system configurations, it isnecessary to find ways to ensure that users are guidedto the appropriate configuration. The second experimentexamined user selection of system configurations, andconcluded that users do not act to optimize systemconfiguration when they select features. This result sug-gests that information systems must have mechanismssuch as user models to direct users to optimal configu-rations. These experiments suggest some of the com-plexities and problems encountered in applying individ-ual differences research to user-centered design of in-formation systems.

Introduction

In the context of user-centered design of informationsystems, individual differences between users can be con-sidered to be differences in the resources that users bringwith them to information tasks. The effects of individualdifferences on search performance vary with the extent towhich these resources are required during the search pro-cess. Studies of individual differences in information con-texts, usually accomplished by logging search behaviors ofdifferent classes of users with operational information sys-tems, have shown a variety of relationships between indi-vidual characteristics and search behaviors. Different typesof cognitive resources, such as topic knowledge, searchskills, cognitive abilities, cognitive styles, and learningstyles, have been shown to be related to a variety of searchtactics and to tendencies to use certain information systemfeatures (Allen, 1991). More detailed research has been able

to isolate the specific search tasks in which the resources inquestion are employed (Allen, 1994). Investigations usingexperimental information systems in which different designfeatures are employed have been able to study the interac-tion between these design features and individual differ-ences in user characteristics. In other words, they havedemonstrated that it can be important to “match” users whohave certain characteristics (i.e., certain levels of resources)with certain design features.

There are, however, a number of ways to match userswith design features, and it is not always clear what ap-proach to customizing systems for users designers shouldadopt. The first experiment reported here investigated thisconundrum of user-centered design: how to match userswith system configurations. Once an optimal match betweenuser characteristics and design features has been identified,it becomes necessary to investigate ways in which users canbe directed to, or placed in, the optimal system configura-tion that matches their characteristics (i.e., the resourcesthey bring to the information task). The second experimentreported here investigated this second conundrum of user-centered design: how to ensure that users are directed tooptimal configurations. In tandem, these two experimentsprovide important additional evidence to incorporate indi-vidual differences into user-centered design.

Experiment 1

Introduction

Stanney and Salvendy (1995), based on previous modelsof Vicente and Williges (1988) and Messick (1976), sug-gested the “capitalization” and “compensatory” perspec-tives towards matching users with information system fea-tures. The capitalization approach suggests that only thoseindividuals who have higher levels of abilities will be suc-cessful in using specialized design features. For example,only those individuals who have higher levels of spatialabilities, such as spatial scanning and spatial orientation,should be able to make optimal use of interfaces that em-© 2000 John Wiley & Sons, Inc.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 51(6):508–520, 2000

ploy spatial representations of information such as datavisualization. The compensatory approach suggests thatsystem features can augment cognitive resources. Accord-ingly, users who have lower levels of abilities should bematched with system features that augment their individualresources to permit effective use of the information system.This approach suggests that individuals with lower levels ofspatial abilities might benefit more from spatial representa-tions of information because these representations will aug-ment their ability to visualize an information space. Thisdichotomy between compensation and capitalization doesnot mean that some system features might influence searchperformance in a way that is uniformly positive or negative.Rather, it means that in the absence of uniform effects, harddesign decisions must be made. It is in these circumstancesthat the dichotomy between compensation and capitaliza-tion becomes relevant to information system design. It isalso true that some design features are neutral in theirimpact on the search performance of some users. In suchcases, the decisions are easier. It is when a design featureprovides compensation for some users, while negativelyinfluencing the performance of others (for example), thatdesign decisions are most directly influenced by the type ofuser-system match.

This dichotomy between capitalization and compensa-tion is based on a tacit assumption that levels of cognitiveabilities such as perceptual speed and spatial scanning arerelatively stable in individuals. If it were possible to sub-stantially increase one’s cognitive abilities in the short runthrough learning, the compensation approach to designwould not be necessary. Designers could focus on capital-izing on the high levels of abilities, and count on learning toproduce high levels of abilities in all users. It is clear, infact, that some cognitive abilities can be learned. Logical(syllogistic) reasoning abilities, for example, can be im-proved in the short run. However, the abilities that were thefocus of this experiment seem more stable over time. Bothperceptual speed and spatial scanning have, for example,been associated with certain personality constructs from theCattell and Guilford inventories. To ensure that short-termlearning effects were not invalidating the theoretical basis ofthis research, learning effects were analyzed, and the resultsare reported below.

Another assumption about individual differences re-search such as this is that individuals use their abilities insimilar ways in different settings. For example, this researchtested levels of perceptual speed in users. It was assumedthat the perceptual speed displayed by participants in com-pleting the test was employed in subsequent searching of aninformation system. The evidence supporting this assump-tion is drawn from studies of cognitive abilities that show aconsistent effect on the performance of a variety of tasks,including information tasks. These results suggest that peo-ple do not turn their abilities off and on in a way that wouldcall into question research of this type.

The compensation and capitalization perspectivespresent a conundrum for information system designers who

wish to create usable systems. Should they focus on theresources that users are likely to have, and implement sys-tem features that capitalize on those cognitive resources?Or, should they focus on the areas of relative weakness incognitive resources found among their users, and implementdesign features that provide compensation for those weak-nesses? This experiment was designed to provide a morecomplete understanding of the extent to which differentdesign features might be used effectively by users withdifferent levels of cognitive resources.

Individual differences such as cognitive abilities are,however, not the only factor to be considered in resolvingthis first conundrum. The situation in which informationsystem use occurs, and more specifically, the task that isbeing accomplished by the user, must also be considered. Ina highly influential paper, Huber (1983) suggested thatcognitive style (and, presumably, related cognitive vari-ables) should not be used as a basis for information systemdesign. Rather, situational and task effects were consideredto be more predictive of system acceptance and use. Thecontroversy between “situationist,” “dispositionist,” and“interactionist” approaches in social psychology has spilledover into information science, with the result that researchinto cognitive variables alone may be regarded as incom-plete. Accordingly, an explicit task variable was added tothis investigation, and the effects of cognitive and taskvariables were analyzed and compared.

The hypotheses tested in this experiment were, in gen-eral, that there would be a significant interaction betweencognitive abilities and design features: in other words, thateither the compensation or the capitalization effect would befound when users with different levels of abilities useddifferent information systems. Higher level interactions in-volving task were also anticipated, although the complexityof the possible interactions between cognitive abilities,tasks, and design features did not permit specific hypothesesto be made about the nature of the interactions. Thesegeneral hypotheses gave rise to a large number of specifichypotheses, which were tested using the analytical methodsdescribed below.

Background

The cognitive abilities investigated in this research wereperceptual speed and spatial scanning. Perceptual speed isdefined as “speed in comparing figures or symbols, scan-ning to find figures or symbols, or carrying out other verysimple tasks involving visual perception” (Ekstrom, French,Harman, & Dermen, 1976, p. 123). Spatial scanning isdefined as “speed in exploring visually a wide or compli-cated spatial field” (Ekstrom et al., 1976, p. 155). Previousresearch (Allen, 1994) demonstrated that information re-trieval search effectiveness is influenced by perceptualspeed. Other spatial abilities seem to influence informationretrieval performance as well. Several studies (Allen, 1992;Borgman, 1989; Dumais & Schmitt, 1991; Greene, Devlin,Cannata, & Gomez, 1990; Vicente, Hayes, & Williges,

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—April 2000 509

1987) found that differences in abilities associated withspatial scanning were associated with differences in perfor-mance of information retrieval tasks.

Previous research has demonstrated that two-dimen-sional presentation of information has an influence on learn-ing. For example, knowledge maps facilitate learning(O’Donnell, 1993; Rewey, Dansereau, & Peel, 1991). Inthese maps, ideas are represented by words or phrases, andthe interrelationships among those ideas by arrows. Theseknowledge maps allow students to see an overall structurein educational content, or to represent their own understand-ing structurally. Similarly, the use of maps with text hasbeen shown to increase learning (Kulhavy, Stock & Kealy,1993; Kulhavy, Stock, Woodard, & Haygood, 1993;Rittschof, Stock, Kulhavy, & Verdi, 1994), as has the avail-ability of two-dimensional forms of data presentation(graphs and tables, for example) along with simple textualpresentations. Some of the learning effects associated withspatial representations may be attributed to dual coding(Paivio, 1990), while other effects may relate to depth ofprocessing or to the direction of attention to specific con-tents (Waddill & McDaniel, 1992).

The most interesting application of two-dimensional (orn-dimensional) data representation in information retrievalsystems occurs in systems that use data visualization toallow users to navigate through an information space to theinformation that interests them. Andrews (1995), Hearst(1995), and Wise et al. (1995) present interesting examplesof such data visualization, and important research such asthe VIBE project at Pittsburgh (Korfhage & Olsen, 1995)has advanced the technology of data visualization to thepoint that it can be considered for operational informationsystems. Similarly, current work at the Pacific NorthwestLabs on the SPIRE system (Hetzler, 1997; Miller, Hetzler,Nakamura, & Whitney, 1997) has demonstrated impressivecapabilities in data visualization.

The tasks used in this investigation were selected asrepresentative of a wide variety of tasks that users mightrequire information to accomplish. Because university stu-dents were the users selected for study, two tasks wereselected that would be familiar to these students, and thatwere sufficiently different in their focus to prompt differ-ences in performance of the information tasks.

Methods

The information systems

Four experimental systems were designed for use in thisexperiment. All of these systems used the same databaseconsisting of 668 bibliographic references. Of these, 140references were known from previous research to be relatedto the topic used in this investigation. An additional 528references from the general area of “family issues” wereadded to this core of potentially relevant references to createthe “Family Issues Database.” The mechanism for retriev-ing references from this database was a list of 466 terms.

As a means of providing a spatially oriented approach tothe term index, a word map was created. All of the signif-icant words from the terms were normalized by stemming,and the 100 most frequently occurring word roots wereentered into a word/reference matrix in which the cells werefrequencies of occurrence of the word stems in the terms ofthe reference. This matrix was reduced to a word/wordsimilarity matrix by calculating cosine similarity measuresbetween words. This similarity matrix served as input tomultidimensional scaling, which produced a two-dimen-sional word map.

In those systems that used the word map, the map waspresented in a scrollable window on the left side of thescreen, and the list of terms in a scrollable window on theright side of the screen. Clicking on any word of the wordmap caused a box to be drawn around the word, and causedthe term list to scroll to a term associated with that word. Inthose systems that did not use the word map, the term listwas presented in a large window by itself. Figure 1 illus-trates the word map interface.

Upon selecting a term from the term list, participantsviewed the references associated with that term. As analternative to the typical single-window presentation inwhich references are presented as a long paragraph withterms to identify data elements, a multiwindow data presen-tation screen was developed. On this screen, each elementof the bibliographic reference appeared in a separate boxthat was located in the same place for all references. Figure2 presents the multiwindow data screen.

Of the four information systems used in this research,one employed both the multiwindow display and the wordmap. Because both of these design features employed two-dimensional representations of information, this system wasthe most spatially oriented of the four. One system em-ployed the term list alone, and a single-window display,thus minimizing the spatial representation of information.One system employed the word map with the single-win-dow display, and one the multiwindow display with the termlist alone. In the discussion that follows, the word map andthe multiwindow display are referred to collectively as“design features.”

Spatial presentation of information such as that used inthis research always adds information that the user canemploy in searching. It is impossible to separate the “spa-tial” nature of such a design feature from the additionalinformation conveyed by the organization. Accordingly, theresults of this research should be taken as referring to boththe spatial organization and the extra information it conveysas inseparable effects.

Each search term that was entered by a participant in theresearch, each word from the word map that was clicked,each term from the term list that was selected, and eachreference that was viewed were recorded. In addition, par-ticipants’ judgments about the usefulness of each referenceviewed were recorded, as were the scrolling activities in theword map and the term list.

510 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—April 2000

Materials

To prepare for the information search, participants read ashort (two-page) document describing some of the mainresearch findings on the effects of viewing television vio-lence on aggressiveness in children. Spatial scanning abilitywas tested with two pencil-and-paper tests derived from theKit of Factor-Referenced Cognitive Tests (Ekstrom et al.,1976): the Maze Tracing Speed Test, and the Map PlanningTest. Perceptual speed was tested with two tests from thesame kit: the Number Comparison Test and the IdenticalPictures Test.

Participants

Eighty volunteer participants from the general studentpopulation of the University of Missouri participated in thisresearch. They were paid a sum of $5 for their participation,which lasted on average about 45 minutes.

Procedures

Participants first read the stimulus article on the topic tobe searched. They then completed the two tests of cognitiveabilities. Following the tests, they were given one of twosets of task instructions. The first participant received thefirst task instruction, the second participant the second in-

struction, and so on until 40 participants had received eachtask instruction. The first task condition was presented asfollows:

A few minutes ago you read an article on a topic. Now,assume that you are working a term paper assignment forone of your classes, which requires you to complete a10-page paper on this topic. To do this, you want to findadditional information about the topic. You will be search-ing an experimental information retrieval system to find afew good articles about the topic that you can include inyour term paper.

The second task condition was presented as follows:

A few minutes ago you read an article on a topic. Now,assume that you have been asked to write an article in thestudent newspaper on this topic. To do this, you want to findadditional information about the topic. You will be search-ing an experimental information retrieval system to find asmany articles as you can about the topic so that you canwrite a well-informed article.

Participants were then assigned to one of the four infor-mation systems, with the first participant assigned to the firstinformation system, the second participant assigned to thesecond information system, and so on until 20 participants

FIG. 1. The word map.


had been assigned to each information system. Becauseparticipants volunteered in no particular order, this ap-proach produced the same effect as random assignment toexperimental conditions. All participants were given indi-vidualized instruction in the use of the information systemto which they were assigned. This instruction was limited todetails of the manipulation of the windows and searchfeatures, and did not address questions about the topic beingsearched.

Participants searched the information system until theyfelt they had achieved the objective set for them in the taskinstruction. Every time they viewed a bibliographic refer-ence, the system asked whether the user would like to printthe reference or not. In the instructional sessions, it wasexplained to participants that they would normally onlyprint references that they felt to be useful to them. Aftercompletion of their search, participants completed a brief(one-page) questionnaire that asked them details about their

knowledge of and experience with information retrieval,their knowledge of the topic, their satisfaction with theirsearch, and any problems they experienced in using theinformation system. This questionnaire also asked partici-pants for a self-assessment of learning that occurred duringthe search.

Data Extraction and Analysis

As might be expected, scores obtained by participants onthe cognitive ability tests were correlated (see Table 1).

Given the significant correlations between these vari-ables, it was far from clear that the tests were providingreliable assessments of separate cognitive abilities. To en-sure that two different abilities were being tested, confir-matory factor analysis was used, in which two factors werederived, which accounted for 75% of the variance in thescores on the abilities tests. The two factors that emerged

FIG. 2. The multiwindow screen.

TABLE 1. Correlations between cognitive abilities scores.

Identical pictures Maze tracing Number comparison

Maze tracing r 5 0.33, p , 0.01Number comparison r 5 0.33, p , 0.01 r 5 0.39, p , 0.01Map planning r 5 0.38, p , 0.01 r 5 0.64, p , 0.01 r 5 0.45, p , 0.01


were somewhat different in configuration from those ex-pected. The identical pictures test alone loaded highly onone variable, while the other three tests loaded highly on thesecond variable. The factor loadings showed clearly that thenumber comparison test bore a greater resemblance to thespatial scanning tests than to the perceptual speed test. Toavoid ambiguity, scores on the number comparison testwere not used in the analyses reported here.

Following procedures established in previous research ofthis nature, references that were selected for printing bymore than half of the participants (who viewed the refer-ences) were considered relevant to the topic being searched.From this basis of operational relevance judgments, preci-sion and recall measures for each search were calculated. Inaddition, the detailed time-stamped logs produced by theexperimental information systems formed the basis for thecalculation of measures such as the number of relevantreferences viewed per minute. These measures were, for themost part, correlated with precision and recall, and so pro-vided process measures for these outcomes.

Two specific search tactics were identified in this re-search. The first, described as “vocabulary learning,” oc-curred when users selected terms from the term list that theyhad previously viewed in references. This search tactic hadbeen shown in previous research to be positively correlatedwith recall, and negatively correlated with precision. Evi-dence of the use of this search tactic was obtained by notingthe proportion of terms used in searches that had beenpreviously viewed in the bibliographic references. This pro-cess was done three different ways, to obtain slightly dif-ferent versions of the same measure. The first approach wasto simply note the proportion of terms used in searches thathad been viewed in references. The second approach was tocount only the terms used in searches that were viewed inreferences judged to be relevant. The third approach was tocount only the terms used in searches that were previouslyviewed more than once. It should be noted that the concep-tualization of this search tactic as “vocabulary learning” isto be distinguished from the amount of self-reported learn-ing derived from the questionnaire. There was, in fact, nosignificant correlation between use of this search strategyand the amount of self-reported learning.

The second search tactic noted in this investigation wasselectivity. Higher selectivity in searches occurred whenusers judged fewer references as relevant. This tactic waspositively correlated with precision, and negatively corre-lated with recall.

Confirmatory factor analysis of the data of this experi-ment showed that there were four groups of dependentvariables. Table 2 shows these variable groups, and theinterrelations of the variables.

It should also be noted that the two variables related tosearch performance derived from the questionnaire, self-reported learning, and self-reported level of difficulty expe-rienced with the search, were not correlated with any of theoutput variables outlined above. Accordingly, these self-report data were not included in the data analysis.

Analysis of these data was complex, given the correla-tions of some of the independent variables and the multiplecorrelations of the independent variables. The correlationsof the independent variables were dealt with by runningseparate analyses for each of the cognitive abilities. Thecorrelations of the dependent variables would have beenbest handled by multivariate analysis of variance, but thisalternative was not possible due to the mixed-factor design.As a result, analysis was accomplished by analysis of vari-ance, in which there were four independent variables: task(two levels, comprised of the term paper and the newspaperarticle tasks), word map used (two levels, Yes and No),display used (two levels, comprised of single window andmultiwindow), and cognitive ability (two levels, low andhigh). The cognitive ability variables were dichotomizedusing a median split to provide for more detailed interpre-tation of results, and were tested independently in separateANOVA models for each dependent variable. Because taskwas designed as a random factor, error terms were selectedto provide correct statistical tests for a mixed model design(Jackson & Brashers, 1994). This analysis, although con-servative, did increase the probability of family-wide type 1error. Accordingly, the findings reported below must betreated with some caution.

Findings

Participant profile

The eighty participants in this study represented a cross-section of the student population of a typical North Amer-

TABLE 2. Correlations among dependent variables.

Variables associated with recall

Dependent variable Correlation with recall

Number of references viewed 0.7947Number of references printed 0.5797Number of references not printed 0.7835Number of relevant references printed 0.8719

Variables associated with precision

Dependent variable Correlation with precision

Number of terms selected 20.6049Search time 20.6696Number of term list scrolls 20.3576

Variables associated with vocabulary learning

Dependent variableCorrelation with

vocabulary learning

Learning 2 0.8061Learning 3 0.8716

Variables associated with selectivity (% of references printed)

Dependent variable Correlation with selectivity

Relevant references not printed 20.6349


ican university. The average age was 23.1 years of age, andthere were more females (62.5%) than males. Students inthe humanities were underrepresented (7.5%), while thosein professional schools such as business and journalismwere overrepresented (28%). All levels of study werepresent, although only 8.8% of the participants were seniors.On the cognitive tests, the sample population scored wellwithin the established norms. Scores on the perceptualspeed tests were normally distributed. Scores on the spatialscanning tests were skewed towards the low end of thedistribution, with the highest skewness exhibited by thescores on the map planning test (skewness5 0.43). Thisdeviation from the normal distribution was not consideredto be a problem, because the scores were dichotomizedusing a median split for analysis.

Learning of cognitive abilities

As indicated above, short-term acquisition of higher lev-els of cognitive ability would call into question the theoret-ical basis for this research. As each test of cognitive abilitywas administered as two separate sections, it was possible totest for such a learning effect. The improvement in scoresbetween the first section and second section of each test wassignificant in two of the four tests (the picture comparisontest and the maze tracing speed test). The fact that improve-ment in scores was not consistent across all four of the testssuggests that the improvement noted in two of the testscould be attributed to the acquisition of test-taking skills,rather than to the improvement of the underlying cognitiveability. Accordingly, short-term improvements in cognitiveabilities were considered not be a factor in this investiga-tion.

Hypothesis tests

As described above, hypotheses were tested in this re-search using ANOVA. The results from these analyses arepresented below.

Recall

Individuals with higher levels of perceptual speedachieved higher recall (as indicated by the number of ref-erences viewed and the number of references printed). How-ever, of greater interest was the finding of significant inter-actions between cognitive abilities and system features.

Individuals with lower levels of perceptual speedachieved higher recall (as indicated by the number of ref-erences viewed) when they used the word map; individualswith higher levels of perceptual speed achieved higherrecall when they used the system without the word map.Similarly, there was a significant three-way interaction be-tween task, spatial scanning, and the use of the word map.Post hoc analysis revealed a simple two-way interactionbetween the cognitive ability and the design feature. In theterm paper task only, individuals with higher levels of

spatial scanning achieved higher recall (as indicated by thenumber of relevant references printed) when using the sys-tem without the word map, and much lower recall whenusing the system with the word map. Both of these resultssupport the compensatory model of user-system fit in thecase of the word map. Table 3 presents an example of aninteraction between system features and cognitive abilitiesin influencing a recall-related variable, which supports thecompensatory model.

On the other hand, individuals with lower levels ofperceptual speed achieved higher recall (as indicated by thenumber of references printed) when they used the single-window display; individuals with higher levels of percep-tual speed achieved higher recall when they used the systemwith the multiwindow display. This result supports thecapitalization model of user-system fit in the case of themultiwindow display.

Precision

Individuals with lower levels of spatial scanningachieved higher precision (as indicated by the number ofterms selected from the browse list) when using the multi-window display; individuals with higher levels of spatialscanning achieved higher precision when using the single-window display. This result supports the compensatorymodel of user-system fit in the case of the multiwindowdisplay.

Selectivity

There was one significant three-way interaction betweentask, spatial scanning, and display type. Post hoc analysisrevealed that there was a simple two-way interaction be-tween task and display, within those individuals who hadhigher scores on the spatial scanning test. In essence, themultiwindow display produced higher selectivity in the termpaper task, and lower selectivity in the newspaper articletask. Because this desirable effect was achieved only amongthose individuals with higher levels of spatial scanningability, this result supports the capitalization model of user-system fit in the case of the multiwindow display.

Vocabulary learning

Vocabulary learning was facilitated by the use of theword map. However, of greater interest were the interac-tions of the design features with user characteristics.

TABLE 3. Number of references viewed as influenced by word map useand perceptual speed.

Perceptual speed

Low High

Word map used No 38.84 63.43Yes 45.81 55.47

F(1, 1) 5 20825.48,p , 0.01.


Individuals with higher levels of cognitive abilities (per-ceptual speed or spatial scanning) used this search tacticmore when they employed the single-window display; userswith lower levels of cognitive abilities used this tactic morewhen they employed the multiwindow display. These find-ings support the compensatory model of user-system fit inthe case of the multiwindow display. Table 4 illustrates thisinteraction.

Individuals with higher levels of spatial scanning didabout the same amount of learning when they used the wordmap, but individuals with lower levels of spatial scanningdid much more learning when they used the word map. Thisresult also supports the compensatory model of user-systemfit, this time with regards to the word map.

Interactions between cognitive abilities and task

There were no significant interactions between cognitiveabilities and task observed in any of the analyses conductedin this experiment. This outcome suggests that cognitiveabilities and tasks are largely independent in their effects onsearch performance.

Interactions between design features

There were significant interactions between design fea-tures, influencing both precision and recall variables. Higherprecision (i.e., lower numbers of list scrolls) was associatedwith the combination of the word map and the multiwindowdisplay. Higher recall (measured by number of recordsprinted) was associated with the simple term list in combi-nation with the multiwindow display. The fact that theseinteractions occurred suggests that there is a collaborativeeffect among design features.

Compensation or Capitalization?

The task of designing information systems can be viewedas making choices available to users. Some of these choiceswill make sense from a compensation perspective: they willallow users with lower levels of abilities to conduct moreeffective searches. Others will make sense from a capitali-zation perspective: they will capitalize upon higher levels ofabilities in users. It is seldom possible to anticipate inadvance whether a design feature will contribute to usabilityfrom one perspective or the other. The results of this ex-

periment show that the word map compensated for lowerlevels of spatial abilities in users, facilitating higher levelsof recall than would otherwise have been achieved. Themultiwindow display capitalized on the higher levels ofspatial abilities in users, permitting higher levels of recall.But this display also compensated for lower levels of abil-ities, permitting higher levels of precision. It appears thatthe word map helped users with lower levels of abilities tovisualize the information space, and so to conduct moreeffective searches. At the same time, the multiwindow dis-play allowed users with all levels of abilities to adoptscanning strategies that produced effective results.

When examining a specific search tactic, learning ofsearch vocabulary, it was found that both design featurescompensated for lower levels of abilities. However, in thecase of another search strategy, selectivity, the multiwindowdisplay capitalized on higher levels of spatial abilities tofacilitate greater selectivity and thus more effective search-ing. Figure 3 shows these relationships between variables. Itillustrates that there were both two-way interactions be-tween system configurations and cognitive abilities, andthree-way interactions that also included the task variable.The relationships that are interpreted as supporting thecompensation perspective of user-system fit are shown asplain lines, those that support the capitalization perspectiveare shown as dotted lines.

The location of the two search tactics as interveningbetween system and users, on the one hand, and outcomeson the other illustrates that both capitalization and compen-sation can work indirectly, by influencing the choice ofsearch tactics.

Optimizing Design for Users and Tasks

From these findings it is possible to designate a systemconfiguration that would be optimal for users with specificlevels of abilities accomplishing specific tasks. For exam-ple, in the case of an individual with lower levels of abilitiesaccomplishing a task that requires high precision, the mul-tiwindow display should be used. For someone with higherlevels of abilities accomplishing the same task, a single-

FIG. 3. Capitalization and compensation effects.

TABLE 4. Learning as influenced by type of display and perceptualspeed.

Perceptual speed

Low High

Type of display Single window 0.13 0.23Multiwindow 0.16 0.17

F(1, 1) 5 647.42,p , 0.03.


window display should be employed. Someone with lowerlevels of abilities accomplishing a high-recall task shoulduse the word map in conjunction with the multiwindowdisplay to facilitate vocabulary learning. Users with higherlevels of abilities doing the same task would do bettersearches if they used a system without a word map, buteither display would facilitate higher recall.

Experiment 2

Introduction

Determining which system configuration is optimal forparticular users, or for specific tasks, or for task/user com-binations is just the first step in designing for usability. Thenext step must be to ensure that users are actually using theoptimal system configuration for them. There seem to betwo opposing approaches to ensuring a match between usersand systems. The first is based on user models. If a user canbe assigned to a user model on the basis of some diagnosisof the user’s tasks and abilities, it may be possible toconfigure the system for the user (Shapira, Shoval, & Ha-nani, 1997). The alternative is to present the user with ageneric system with user-selectable features, in the expec-tation that the user will select the system configuration thatis optimal for his or her abilities and tasks. The formeroption runs the risk of stereotyping users on the basis ofincomplete and approximate information about the usersand their tasks. User stereotypes of this type might wellcause a mismatch between users and system configurations.On the other hand, it is far from clear that users will knowenough about themselves or about their interactions withspecialized system features to select optimal configurations.

These opposing alternatives present another conundrumfor system designers. Should we present a flexible systemand trust to user preferences to create optimal search con-figurations, or should we try to match user features withsystem configurations through user models?

Purpose

This experiment was designed to explore how users,when given a flexible system with user-selectable options,select system configurations. The general objective was toascertain if users are likely to select system configurationsthat are optimal for the tasks they accomplish and for theirpersonal characteristics.

Background

There is considerable interest in user preferences as inputto information technology of various kinds. The approach ofusing stereotypes of users to generate system configurationshas been used successfully in a variety of settings. Forexample, Odubiyi et al. (1997) reported on an agent-basedinformation retrieval system that employs stereotypes toassign user preferences. Assignment of a user to a stereo-

type is based on an initial dialog in which questions aboutthe user’s information situation are posed by the system. Analternative approach uses system components that monitoruser behavior to deduce user preferences (Branting &Broos, 1997; Haynes, Sen, Arora, & Nadella, 1997). Al-though acquisition of user preferences by monitoring userbehavior seems to be an very desirable approach, leading totruly user-adaptive systems, many operational systems relyon users to specify their own preferences for system featuresand interaction modes. For example, the many optionsavailable to users of Microsoft products, usually selectedthrough a “preferences” menu, represent the most commonapproach to handling user preferences in operational sys-tems. Explicit user selection of system configurations wasthe approach tested in this research.

Methods

The information systems

The four experimental systems from the first experimentwere combined into a single system for this experiment. Thedatabase and terms remained the same. When the word mapwas activated, the interface presented an additional buttonlabeled “Hide Word Map.” When this button was clicked,the word map was replaced by the simple-term list. Thisscreen contained a button labeled “Show Word Map” that,when clicked, activated the word map. Similarly, the mul-tiwindow display included a button labeled “Single Win-dow” and the single-window display included a buttonlabeled “Multiwindow.” These allowed the user to toggleback and forth between the two display choices.

Materials

The materials used in this experiment were identical tothose used in the first experiment. The same stimulus articlewas used, and the same tests were used to assess cognitiveabilities of participants.

Participants

Eighty volunteer participants, different from the 80 whoparticipated in the first experiment, from the general studentpopulation of the University of Missouri participated in thisresearch. They were paid a sum of $5 for their participation,which lasted on average about 45 minutes. Extensive anal-ysis of demographics revealed that the group of studentswho participated in this experiment was identical in mostrespects with the group who completed the first experiment.

Procedures

The procedures used in the first experiment were repli-cated in the second experiment. The task condition waspresented using the same task instructions. Participantswere then randomly assigned to one of the four starting


configurations, the same as the four systems used in the firstexperiment. Instructions, search procedures, and question-naire were identical in this experiment. Training of usersfollowed the same pattern as the first experiment, focusingon how to use the system rather than on the topic beingsearched.

Data extraction and analysis

Two measures of user behavior in selecting system con-figurations were calculated. The first was a simple tally ofthe number of times users switched from one configurationto another. The second was a calculation of the proportionof time each user spent in each system configuration. Duringthis analysis, it became clear that exactly half of the partic-ipants had made some change in the system configurationwith which they were initially presented. From this fortu-itous circumstance, it was possible to compare the perfor-mance of the group who altered their system configurationwith those who did not make any changes. It was alsopossible to assess whether these two groups differed on anycognitive or task measures, and whether they differed fromparticipants in the first experiment, using ANOVA withScheffepost hoc analysis.

Findings

Use of configuration selection buttons

When users began in the configuration without the wordmap, 65% switched to the word map configuration for atleast part of their search. Of these, half switched back to theconfiguration without the word map for at least part of theremainder of their search. When users began in the config-uration with the word map, only 15% switched to theconfiguration without the word map. Two-thirds of thesereturned to the word map for at least part of the remainderof their search. On average, users spent 66% of their timeusing the word map, and 34% of their time using the simpleterm list.

When users began in the single-window configuration,20% switched to the multiwindow configuration for at leastpart of their search. Of these, 38% returned to the single-window configuration for at least part of the remainder oftheir search. When users began in the multiwindow config-uration, 12.5% switched to the single-window configurationfor at least part of their search. Of these, 40% returned to themultiwindow configuration for at least part of the remainderof their search. On average, users spend 52% of their timeusing the multiwindow display and 48% of their time usingthe single-window display.

Switching from one configuration to another was rela-tively limited. Half of the users stayed with the configura-tion to which they were randomly assigned. The next largestgroup of users (44%) switched configurations to one differ-ent configuration: these users experienced two of the pos-sible configurations. A small number of users (6%) changed

configurations often enough that they were able to use threeof the four possible configurations. No users employed allfour system configurations.

Selection of configurations appropriatefor abilities and tasks

There were no significant correlations between the cog-nitive abilities of users and the frequency with which theyswitched to different configurations, nor between cognitiveabilities of users and the proportion of their time spent usingspecific system configurations. Similarly, there were nosignificant differences between the average number ofswitches made to different configurations, nor in the averagetimes spent in each system configuration, based on the twotasks. Table 5 illustrates that there were no differencesbetween the cognitive abilities of users who made changesto the configuration of the information system and those ofusers who made no changes to the system to which theywere randomly assigned.

Table 6 demonstrates that there were few differences insearch outcomes between those users who changed theirsystem configuration and those who did not make changesto the system to which they were randomly assigned. Par-ticipants who made changes to their system configurationexhibited higher levels of vocabulary learning, and some-what greater selectivity in printing references. Comparingthe results of the first experiment with those of the secondproduced an unexpected finding. One would have expectedthat the greatest similarity in outcomes would occur be-tween participants in Experiment 1 (who had no choice as towhich system configuration to use) and those in Experiment2 who made no configuration changes. In fact, the oppositeoccurred. Individuals who were offered the choice of mak-ing system configuration changes, but opted not to changethe configuration, did not do as well in terms of recall andselectivity.

In making arguments from an absence of effects, it isalways important to note the power of the statistical tests todetect significant differences. In this analysis, the observedpower of the ANOVA ranged from 0.67 to 0.76 in thosetests in which significant effects were found. This powerlevel is high enough to suggest that the analysis wouldprobably have found any significant effects of configurationchanges on search outcomes.

TABLE 5. Cognitive abilities scores and configuration changes.

Ability test scores

Configurationchanges No changes

Mean St. Dev. Mean St. Dev.

Identical pictures test 60.25 12.2 55.4 12.6Maze tracing speed 18.4 7.7 16.2 5.6Number comparison test 42.7 10.2 39.6 8.3Map planning test 22.9 6.9 20.4 5.9


Discussion

The results that describe user activities in configuring thesystem present an interesting pattern. First, it appears clearthat users did not act in such a way as to take advantage ofcompensation effects. If this had been the case, the resultswould have shown that users with low levels of cognitiveabilities were selecting the word map or multiwindow con-figurations more frequently than those with higher levels ofabilities. Similarly, users with higher levels of abilitiesshould have been selecting the simple term list. The fact thatthere were no significant differences in configuration selec-tion that could be attributed to cognitive abilities shows thatusers were not matching their abilities with optimal systemconfigurations.

This conclusion is reinforced by the overall performancefigures. If participants who made changes had been acting tooptimize the match between their abilities or task and sys-tem configuration, one would have expected their perfor-mance to be superior to those who made no changes. Thosewho made changes in system configuration showed signif-icantly higher levels of learning and selectivity than thosewho did not make changes. It is possible that these differ-ences can be explained by higher levels of reflection andself-monitoring in searching. The users who made changesin their system configurations were, it appears, thinkingabout what they were doing in interacting with the infor-mation system. The same thoughtfulness about the interac-tion could have led to the higher level of adoption ofspecific learning strategies. However, it is clear that thesedifferences in adoption of specific search strategies had noimpact on any of the precision and recall measures.

In the second experiment, those users who were free toselect their own configuration but elected not to do so com-pleted lower recall searches than the users from the first ex-periment who were randomly assigned to configurations. It ispossible that the unwillingness to experiment with systemconfigurations reflected an inflexible approach to searching,which resulted in lower performance on the search task.

These results suggest that users were not selecting con-figurations to optimize performance. If this is true, there

must be an alternative explanation of the changes thatoccurred. The findings on configuration selection present analternative interpretation. Users were far more likely tochange the configuration to the word map than any otherfeature. Users had never seen such a device before, andaccordingly it had novelty value. Perhaps they wanted to tryit out and see how it worked. It is interesting to note that halfof those who did experiment with the word map switched tothe simple term list subsequently. But the overall findingssuggest that users tended to configure their system to try outa novel feature, rather than to optimize search conditions.The fact that relatively few users changed to the multiwin-dow display (which the first experiment showed can providesuperior results both for low-abilities users and for bothtasks) supports this conjecture. Most users were regularcomputer users, and could be expected to be familiar withmultiwindow displays, particularly because these are usedin many web sites.

The effect of novelty on system configuration decisionswas expected. It was also expected that at least some userswould be interested in exploring all possible system con-figurations before settling down to their preferred configu-ration. In fact, there was little evidence of exploratorybehavior. For the most part, users either stayed in theconfiguration to which they were assigned, or changed toone other configuration.

Conclusions

It is possible to associate specific combinations of designfeatures with superior performance on search tasks by userswith specific characteristics who are accomplishing specifictasks. In other words, both user characteristics and the taskthey are likely to be accomplishing should figure in usermodels. Using such user models, it is possible to customizeinformation systems to match the users and their activities.Based on the results presented here, which support theresults of previous investigations, it seems clear that the“compensatory match” is likely to be a productive approachto such customized information systems. In such systems,

TABLE 6. Configuration changes and search outcomes.

Search outcomes Experiment 1Experiment 2no changes

Experiment 2changes

Recall 0.34b 0.24b 0.28References viewed 51.1a 32.7a 39.5References printed 16.3a 11.3a 12.9References not printed 34.8a 21.4a 26.6Relevant references printed 9.5b 6.9b 7.3

Precision 0.32 0.28 0.27Scrolls in term list 263.9a 127.2a 175.3Terms selected 8.4 5.5 5.8Search time (minutes) 12.7 11.3 13.2

Learning 0.27 0.17b 0.3b

Selectivity (relevant references not printed) 3.6b 1.3b 2.3b

a p , 0.05.b p , 0.01.


users with lower levels of certain resources such as cogni-tive abilities will be matched with information systems withadvanced features such as data visualization that will allowthese users to perform at a higher level in their informationsearches. At the same time, there was some evidence of the“capitalization match” in these findings. It seems logicalthat those system features that make the greatest demandson the cognitive resources of the users will be of greatestutility to users with higher levels of cognitive abilities.

These conclusions lead system designers into a furtherconundrum. How can the system diagnose user character-istics and tasks to configure itself for optimal performance?Although some information systems have made attempts tobe self-configuring, it seems clear that user preferences,implemented and changed by users as they interact withinformation systems, would provide a simpler approach tosystem customization. This solution, although attractive inits simplicity, requires that users be able to recognize aconfiguration of design features that will produce optimalresults for them by augmenting their abilities where neces-sary, and capitalizing on their abilities where appropriate.Users must also have the ability to recognize design con-figurations that will hinder their performance. The results ofthe second experiment suggest that this approach will not bea successful one. In information systems of the type used inthis experiment, the reward offered to the user for selectingthe optimal configuration may not be clearly perceived bythe user. The benefits of increased learning or selectivity arenot tangible enough to motivate exploration of differentconfigurations to achieve those benefits.

A more directive manner of matching users with systemconfigurations seems to be called for. User models, includ-ing stereotypes, are the most straightforward ways of im-plementing such matching. It is, however, far from clearhow users can be assigned to models or stereotypes basedon differences in their cognitive abilities. It is possible thatsensible inferences could be drawn about their cognitiveabilities from user behaviors, and that systems could con-figure themselves in response to these inferences. The re-search reported here provides some indicators of the impli-cations of individual differences for the user-centered de-sign of information systems. But it also suggests some of theproblems that user-centered design will encounter in creat-ing information systems that respond to user characteristics.Research into individual differences in information sciencehas successfully demonstrated that information behaviorsand search performance are influenced by cognitive vari-ables. The challenge that remains is to demonstrate howinformation systems can make use of that knowledge.

References

Allen, B. (1991). Cognitive research in information science: Implicationsfor design. Annual Review of Information Science and Technology, 26,3–37.

Allen, B. (1992). Cognitive differences in end user searching of a CD-ROM index. In N. Belkin, P. Ingwersen, & A.M. Pejtersen (Eds.), SIGIR

’92: Proceedings of the fifteenth annual international ACM SIGIRconference on research and development in information retrieval (pp.298–309). New York: ACM.

Allen, B. (1994). Cognitive abilities and information system usability.Information Processing & Management, 30, 177–191.

Andrews, K. (1995). Visualising cyberspace: Information visualisationin the harmony internet browser. In N. Gershon, & S. Eick (Eds.),Information visualization proceedings (pp. 97–104). Washington:IEEE.

Borgman, C.L. (1989). All users of information retrieval systems are notcreated equal: An exploration into individual differences. InformationProcessing & Management, 25, 237–251.

Branting, L.K., & Broos, P.S. (1997). Automated acquisition of userpreferences. International Journal of Human-Computer Studies, 46(1),55–77.

Dumais, S.T., & Schmitt, D.G. (1991). Iterative searching in anonline database. In Proceedings of the Human Factors Society 35thannual meeting (pp. 398 – 402). Santa Monica, CA: Human FactorsSociety.

Ekstrom, R.B., French, J.W., Harman, H.H., & Dermen, D. (1976). Manualfor kit of factor-referenced cognitive tests. Princeton, NJ: EducationalTesting Service.

Greene, S.L., Devlin, S.J., Cannata, P.E., & Gomez, L.M. (1990). No IFs,ANDs or ORs: A study of database querying. International Journal ofMan-Machine Studies, 32, 303–326.

Haynes, T., Sen, S., Arora, N., & Nadella, R. (1997). Automated meetingscheduling system that utilizes user preferences. In Proceedings of theinternational conference on autonomous agents, 1997 (pp. 308–315).New York: ACM.

Hearst, M.A. (1995). TileBars: Visualization of term distribution informa-tion in full text information access. In Human factors in computingsystems: CHI ’95 conference proceedings (pp. 59–66). New York:ACM.

Hetzler, B. (1997). Beyond word relations. SIGIR Forum, 31(2), 28 –32.

Huber, G.P. (1983). Cognitive style as a basis for MIS and DSS designs:Much ado about nothing? Management Science, 29, 567–579.

Jackson, S., & Brashers, D.E. (1994). Random factors in ANOVA. Thou-sand Oaks, CA: Sage.

Korfhage, R.R., & Olsen, K.A. (1995). Image organization using VIBE, avisual information browsing environment. Proceedings of SPIE, 2606,380–388.

Kulhavy, R.W., Stock, W.A., & Kealy, W.A. (1993). How geographicmaps increase recall of instructional text. Educational Technology Re-search and Development, 41(4), 47–62.

Kulhavy, R.W., Stock, W.A., Woodard, K.A., & Haygood, R.C. (1993).Comparing elaboration and dual coding theories: The case of maps andtext. American Journal of Psychology, 106(4), 483–498.

Messick, S. (1976). Individuality in learning. San Francisco, CA: Jossey–Bass.

Miller, N., Hetzler, B., Nakamura, G., & Whitney, P. (1997). The need formetrics in visual information analysis. In The need for metrics in visualinformation analysis. New York: ACM.

Odubiyi, J.B., Kocur, D.J., Weinstein, S.M., Wakim, N., Srivastava, S.,Gokey, C., & Graham, J. (1997). SAIRE: A scalable agent-basedinformation retrieval engine. In Proceedings of the InternationalConference on Autonomous Agents, 1997 (pp. 292–299). New York:ACM.

O’Donnell, A. (1993). Searching for information in knowledge maps andtexts. Contemporary Educational Psychology, 18(2), 222–239.

Paivio, A. (1990). Mental representations: A dual coding approach. NewYork: Oxford.

Rewey, K.L., Dansereau, D.F., & Peel, J.L. (1991). Knowledge maps andinformation processing strategies. Contemporary Educational Psychol-ogy, 16(3), 203–214.

Rittschof, K.A., Stock, W.A., Kulhavy, R.W., & Verdi, M.P. (1994).Thematic maps improve memory for facts and inferences: A test of the


stimulus order hypothesis. Contemporary Educational Psychology,19(2), 129–142.

Shapira, B., Shoval, P., & Hanani, U. (1997). Stereotypes in informationfiltering systems. Information Processing & Management, 33(3), 273–287.

Stanney, K.M., & Salvendy, G. (1995). Information visualization: Assist-ing low spatial individuals with information access tasks through the useof visual mediators. Ergonomics, 36, 1184–1198.

Vicente, K.J., & Williges, R.C. (1988). Accommodating individual differ-ences in searching a hierarchical file system. International Journal ofMan-Machine Studies, 29, 647–688.

Vicente, K.J., Hayes, B.C., & Williges, R.C. (1987). Assaying and isolat-ing individual differences in searching a hierarchical file system. HumanFactors, 29, 349–359.

Waddill, P.J., & McDaniel, M.A. (1992). Pictorial enhancements of textmemory: Limitations imposed by picture type and comprehension skill.Memory and Cognition, 20(5), 472–482.

Wise, J.A., Thomas, J.J., Pennock, K., Lantrip, D., Pottier, M., Schur, A.,& Crow, V. (1995). Visualizing the non-visual: Spatial analysis andinteraction with information from text documents. In N. Gershon, & S.Eick (Eds.), Information visualization proceedings (pp. 51–58). Wash-ington: IEEE.


Documents

Individual differences and the conundrums of user-centered design: Two experiments