Cognitive-experiential modelling of web navigation and information architecture

Seminar Computational Cognitive Modelling of Web Navigation, Utrecht University, 19-20 March 2010 1

Cognitive-experiential

modelling of web navigation

and information architecture

Paul van Schaik

Mike Lockyer

Philip Barker

Raza Habib

Andy Price


Outline

• Problem and proposed solution

• Research framework

• A study of web navigation

• Computational Cognitive Modelling

for Improving the Information

Architecture of Large Web Sites


Where and what is Teesside?

• http://src-weblog.blogspot.com/

• http://www.stockton-rambling-club.org.uk/

• http://www.geograph.org.uk/search.php?i=11366536

• http://www.geograph.org.uk/







The problem• Poor usability and information architecture for

large web sites (intranet, Internet, other)

„Task success is up substantially with usability statistics from 2004. Bad information architecture[emphasis added] causes most of the remaining user failures‟ (Nielsen, 16/4/2009)

A proposed solutionCognitive-experiential modelling of web navigation and information architecture

http://www.useit.com/alertbox/ia-failures.html


Research framework

adapted from

Finneran and Zhang

(2003)

Person

Artefact Task

Web navigation process

Web navigation outcome


The influence of the experiential (1)

• Enhanced aesthetics increases performance under conditions of poor usability (Moshagen et al., 2009)

• Flow experience predicts performance over and above existing skills and knowledge (Engeser & Rheinberg, 2008)

• Modelling interaction experience to produce and represent HCI-knowledge and to guide system design - special issue of Interacting with Computers



• Experiential dynamic modelling of web navigation: ‘information seek cycle’ (David et al., 2007)



• ‘Information seek cycle’ (David et al., 2007)– Self-efficacy is enhanced by the successful execution

of information seeking goals in one cycle

– Reduces the perceived difficulty of information goals in the following cycle

– In addition, as a result of self-efficacy from previous cycles, more challenging goals are formulated in following cycles

– Effect on cognitive task performance not studied, but -given the nature of „virtuous circle‟ - enhanced performance would be expected

• Need for an integrated approach to studying cognitive and experiential factors in human-computer interaction


Artefact - web site complexity

• Increasing task difficulty as a function of– page complexity in terms of the number of navigation choices on

a web page (Gwidzka & Spence, 2006)

– structural complexity (Guo & Poole, 2009)

• The greater the number of links per page, the lower success rate (Blackmon et al., 2002)

• As artefact complexity increases, the balance between challenge and skill will be negatively affected and flow experience will decrease (Guo & Poole, 2009)

• Hypothesis 1: artefact complexity (page complexity) has a negative effect on the quality of task performance, flow experience and task outcome


Task - task complexity

• Task complexity (path length) has a negative effect on the quality of task performance (Gwizdka & Spence, 2006; van Oostendorp, Madrid & Puerta Melguizo; 2009)

• Possible mechanisms:– Increasing probability of selection error with path length

– increasing probability of error in relevance judgement with path length

• Hypothesis 2: task complexity (path length) has a negative effect on the quality of task performance, flow experience and task outcome


Person - intrinsic motivation (1)

• Intrinsic motivation as an individual-difference variable in web navigation– Disposition to engage in actions toward pursuits

“internal to the self, such as personal interest, enjoyment, and learning”: intrinsic motivation

– “external to the self, such as tangible rewards, interpersonal status, and the dictates of others”: extrinsic motivation (Hirschfeld et al., 2008, p. 155)

• Positive predictor of the process of task performance and task outcome in – flow experience in golf (Oh, 2001) and in athletics

(Stavrou, 2008)

– academic learning (Hirschfeld et al., 2008) and school learning (Vansteenkiste et al., 2008)



Theoretical accounts (Zapata-Phelan et al., 2009)– Activity, concentration, initiative, resilience and flexibility can

increase, as a result, enhancing task performance

– Intrinsic motivation has a stronger effect than external motivation on the persistence of effort, which has a strong effect on the performance of complex tasks or complex artefacts - but this effect may be less strong or disappear with less complex tasks and less complex artefacts

– In the domain of employment, internal (work) motivation is expected to have a positive effect on the quality of task performance

– Intrinsically motivated individuals (or individual with an „autotelic‟ personality) are those who engage in activities for the sake of the activities rather than in order to achieve some external goal

Therefore, these individuals should experience a higher level of flow experience, as confirmed by Asakawa (2004)



• Hypothesis 3: intrinsic motivation has a positive effect on the quality of task performance, flow experience and task outcome

• Hypothesis 4: the positive effect of intrinsic

motivation is moderated by artefact complexity: the

effect of intrinsic motivation is more positive with high

artefact complexity

• Hypothesis 5: the positive effect of intrinsic

motivation is moderated by task complexity: the

effect of intrinsic motivation is more positive with high

task complexity


Flow experience (1)• Web navigation process/task performance process:

experiential (including flow) and cognitive components

• ‘Holistic sensation that people feel when they act with total involvement’ (Csikszentmihalyi, 1990, p. 477)

• Nine dimensions of flow distinguished and measurement instruments developed (e.g. Jackson & Marsh, 1996); see also Pace (2004)

• Not a matter of ‘all or nothing’ - can experience a degree of flow on each dimension

• Flow is an independent positive predictor of task outcome in – computer-game playing (Murphy et al., 2008)

– mathematics performance (Heine, 1997; Engeser & Rheinberg, 2008),

– foreign-language performance (Engeser & Rheinberg, 2008)

– computer-based statistics performance (Vollmeyer & Imhof, 2007)


Dimension Description

Balance of challenge and skill “The person perceives a balance between the challenges of a

situation and one's skills, with both operating at a personally

high level.” (p. 18)

Mergence of action and

awareness

“The flow activity is so deep that it becomes spontaneous or

automatic.” (p. 18)

Goal clarity “Goals in the activity are clearly defined (...), giving the person

in flow a strong sense of what he or she is going to do.” (p. 19)

Feedback “Immediate and clear feedback is received, usually from the

activity itself, allowing the person to know he or she is

succeeding in the set goal.” (p. 19)

Concentration “Total concentration on the task at hand occurs when in flow”

(p. 19)

Control “A sense of exercising control is experienced, without the

person actively trying to exert control.” (p. 19)

Loss of self-consciousness “Concern for the self disappears during flow as the person

becomes one with the activity.” (p. 19)

Transformation of time “Time alters perceptibly, either slowing down or speeding up”

(p. 19)

Autotelic experience “Intrinsically rewarding experience. An activity is autotelic if it

is done for its own sake, with no expectation of some future

reward or benefit.” (p. 20)

Dimensions of flow experience (Jackson & March 1996)


Flow experience (2)

• Pathways for the positive effect of flow on

performance outcome (Engeser & Rheinberg, 2008)

– Flow is considered to be a „highly functional state‟; therefore,

should promote performance

– Flow is a driver of motivation for continued activity; leads people to

select higher challenges in order to experience flow again

• Hypothesis 6: flow experience has a positive effect on

task outcome, after controlling for the influence of

artefact complexity, task complexity and intrinsic

motivation


Flow experience (3)

• Given the motivating character of flow to continue task performance, the quality of task performance is a likely mediator

• Thus, flow experience has a positive effect on the quality of task performance and, thereby, a positive (indirect) effect on task outcome

• Hypothesis 7: flow experience has a positive

effect on the quality of task performance, after

controlling for artefact complexity, task

complexity and intrinsic motivation

• Hypothesis 8: the quality of task performance

has a positive effect on task outcome, after

controlling for artefact complexity, task

complexity, intrinsic motivation and flow


Artefact

complexity

Intrinsic

motivation

Task

complexity

Web navigation

performance

Flow

experience

Web navigation

outcome

H1H2

H4H5 H3

H7

H6H8

A cognitive-experiential model of web navigation


A modelling study of web navigation

• AimDemonstrate the need for an integrated approach (including cognitive and experiential factors in human-computer

interaction) to modelling web navigation

• Method– Test hypotheses, using a computer-controlled experiment, in

which artefact complexity (low or high) and task complexity (path length - low or high) were manipulated

– Test-users‟ intrinsic motivation measured as an individual-difference variable

– Series of information retrieval tasks - information-oriented realistic mock intranet site

– Measured task performance, flow experience and task outcome

– Participants: 114 undergraduate psychology students

– Information retrieval task

– Data analysis: partial-least squares (PLS) path modelling


Web site versions


Results - descriptives (1)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Simple (T) Complex (T)

Percentage correctly completed

Simple (A)

Complex (A)

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40


Percentage correct relative to total

Simple (A)

Complex (A)

0.00

10.00

20.00

30.00

40.00

50.00

60.00


Work load (SMEQ)

Simple (A)

Complex (A)

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50


Disorientation

Simple (A)

Complex (A)


Results - descriptives (2)

3.50

3.70

3.90

4.10

4.30

4.50

4.70

4.90

5.10

5.30

5.50


Flow - awareness-action merging

Simple (A)


Dependent variable Effect R2 Significance

Work load Artefact 0.25 0.30***Task 0.39***Intrinsic motivation

Artefact task

Artefact motivation

Task motivation

Artefact task motivation

Disorientation Artefact 0.15 0.21*Task 0.26*Intrinsic motivation

Artefact task

Artefact motivation

Task motivation


Action-awareness merging Artefact 0.20Task -0.34***Intrinsic motivation

Artefact task

Artefact motivation 0.25**Task motivation


Autotelic experience Artefact 0.33Task -0.21*Intrinsic motivation 0.56***Artefact task

Artefact motivation

Task motivation


Results - effects of artefact, task and person


Results: effects of task performance and

flow experience on task outcome (1)

• Percentage completed correctly

• Flow as a partial mediator of the effect of experimental manipulations on task performance


Results: effects of task performance and

flow experience on task outcome (2)

• Percentage completed correctly

• Task performance as a full mediator of the effect of flow on task outcome


Artefact

complexity

Intrinsic

motivation

Task

complexity

Web navigation

performance

Flow

experience

Web navigation

outcome

H1H2

H4H5 H3

H7

H6H8

Summary


Conclusion

• Aim Demonstrate the need for an integrated approach (including cognitive and experiential factors in human-computer interaction) to modelling web navigation

• Findings– Within the framework of the PAT model, cognitive and

experiential factors, together, do indeed influence task outcomes in web navigation

– In particular, artefact complexity, task complexity and intrinsic motivation have an effect on task performance, flow and task outcome (Hypotheses 1-3)

– Influence of flow moderated by artefact complexity (Hypothesis 4)

– Flow is a partial mediator of the effect of site- and task complexity on task performance (Hypotheses 1, 2, 7)

– Task performance is a complete mediator of the effect of flow on task outcome (Hypotheses 6-8)


Computational Cognitive Modelling for

Improving the Information Architecture

of Large Web Sites

Aim– Understand the information architecture of large web sites in

relation to a user‟s information-finding behaviour

– Use computational cognitive modelling techniques to aid in the analysis and design of information architectures for large web sites and to model a user‟s behaviour with these sites


The importance of information architecture

• The information architecture of web sites is the most important remaining factor causing usability problems for users (Nielsen, 2009)

• Information architecture– Conceptual (organisational) structure

• Headings and content across pages

• Links and content under headings on a page

– Navigation structure• Links

– Labelling system• Language used to describe headings and links


Information architecture (1)

Home page

Headings

Links

(Content)

Page 2

Headings

Links

(Content)

Page 3

Headings

Links

(Content)

Page 4

Headings

Links

(Content)

Page 5

Headings

Links

(Content)

Page 6

Headings

Links

(Content)


Information architecture (2)

• Aim Provide effective access to relevant online information resources (usually delivered through a web site)

• Role Provide the design of information for building a web site, but not address information presentation

• Non-computational techniques for informing the design of information architecture do not scale up

– Empirical psychological user-based

– Expert-based


Modelling information architecture

Modelling of information architecture (Katsanos et al. , 2008)• Used LSA to create new information architecture

• Design of small web sites

• No attempt made at • Modelling large web sites

• Automatically capturing of web sites

• Modelling information architecture explicitly

• Automatically creating web site with improved information architecture


Conceptual approach

• HCI-framework: PAT model

• Relational-database modelling of information architecture

• Information scentLSA

• Simulate users’ behaviourCoLiDeS

• Create improved information architecture as database model

• Automatically create web site with improved information architecture

• Offer service for analysing and improving web sites


Research question 1

• How can computational cognitive modelling techniques be used to analyse and aid the design of conceptual structures, labelling and navigation that underlie the structure of large web sites?

• Methods– Extract content (initially each of two) web sites using

software to be developed in this project

– Parse web pages and create a database model of information architecture

– Use LSA and CoLiDeS (and other techniques) to model, analyse and improve a site‟s information architecture for human information seeking in terms of conceptual structure, labelling scheme and navigation structure

– Build or automatically generate one or more alternative site versions with improved information architecture


Research question 2

• What is the effect of information architecture on people’s ability to find and use information within web sites and how is this effect moderated by task characteristics and individual differences between users?

• Methods– Conduct one or more web navigation experiments using different

versions of each web site (original and one or more improved)

– Use different types of task (e.g. varying in degrees of complexity)

– Measure participants‟ relevant individual-difference characteristics (e.g. spatial ability - Juvina & van Oostendorp, 2006; intrinsic motivation)

– Test the effects of information architecture (organisation, labelling and navigation) as moderated by task characteristics and individual differences on end-users‟ ability to complete information-seeking tasks and (flow) experience using AN(C)OVA


Progress (1)• Large web sites

– University intranet site

– University Internet site

– Without loss of generality, approach should be applicable to any web site

• Three-stage process

1. Capture pages, parse pages (links) and store in database1. Capturing

2. Read page URLs, parse (headings, links, text and images) and store (structure) in database

3. Analyse semantic similarity and term vector length and store in database

Use AutoCWW server at Colorado

• Threaded web application

• Hardware: Intel (R) Xeon (R) 2.53 GHz, 4GB RAM, Windows 7 Enterprise, 64 bit operating system, 4 CPUs

• Development platform: .NET


Progress (2)

• Modelling web pages - technical approach• Database schema for information architecture based on

cognitive model of web navigation (CoLiDeS)

• Parsing based on page template specific to web site

• Problem: badly coded („hand-written‟) web pages Takes time to check for inconsistencies

• Analysis of web pagesFrequency distribution of semantic similarity

• Heading-link

• Heading-paragraph

• Heading-image

• Link-link


Further work

• Complete capturing, parsing and modelling

• CoLiDeS simulation

• Shortest path analysis

• Improvement of captured and modelled web sites

• Experimental validationUse PAT model to model the effect of artefact (information architecture), task (web navigation task) and person (web user)

• Online service for web site analysis and improvement

Demonstrate the general applicability of the approach

• Automatic identification of web page/site template


Strengths and potential impact

• Explicitly modelling of information architecture

• Relational-database modelling to reflect cognitive model of web navigation

• Cognitive modelling to provide a credible basis for designing and automating the creation of information architecture

• Support for large web sites (intranet, Internet and other)

• Global/sitewide analysis of information architecture and web navigation


Summary

• Address the problem of poor usability and

information architecture for large web sites

by using a cognitive-experiential modelling

approach

• Research framework: PAT model

• A cognitive-experiential modelling study of

web navigation

• Project: computational cognitive modelling

for improving the information architecture

of large web sites


Final words

• ‘Someone who knows everything that can be

known has a lot of knowledge. But why

would he (/she) want to know everything?

Knowledge without a purpose is in fact non-

knowledge.’Toonder, M. (1980). The know-hat [De weetmuts].

In M. Toonder. There is something behind this

[Daar zit iets achter]. Amsterdam: De Bezige Bij.

• Norman, D. (2005). Human-centred design

considered harmful. Interactions, 12(4), 14-

19.

Documents

Cognitive-experiential modelling of web navigation and information architecture