Evaluating the Effectiveness of Interactive Map Interface Designs

4/24/2009

1

Evaluating the Effectiveness of Interactive Map Interface Designs:A Case Study with Eye Movement y y

Analysis

Arzu Çöltekin, Simone Garlandini, Benedikt Heil, Sara Irina Fabrikant

D t t f G hDepartment of GeographyUniversity of Zurich

Switzerland

This study is about..Tag cloud produced based on a journal article (http://www.geo.unizh.ch/~arzu/publications/ColtekinA_et_al_CaGIS2009.pdf) on this subject,

using http://www.wordle.net/

2009/04/22Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

4/24/2009

2

Overview

Question: How to evaluate interactive map interfaces designs?interfaces designs?

‐ A range of issues can be identified based on classical usability engineering methods‐Maps, however, changed a lot

‐More interactive‐Many more ‘interface’ elements in/on/around the cartographic display

‐ Could eye movement recording and analysis give us more information?

2009/04/22 1/26Çöltekin, Garlandini, Heil, Fabrikant ‐ EyeTrackUX 2009

Usability Engineering

• Testing, inspection, inquiry

• Metrics such as– Satisfaction

– Effectiveness (accuracy/success)

– Efficiency (time)

Especially what involves self reporting cannot beEspecially what involves self‐reporting cannot be trusted: Cognitive load, performance anxiety, social pressure


4/24/2009

3

Eye movements

• Investigated over 100 years

• When eyes move; ‐ Fixations ‐ eyes fixed to a point 50 to 500ms.

‐ Linked to (overt) attention

Saccades rapid movements between fixations

Figure: Yarbus, 1967

‐ Saccades– rapid movements between fixations‐ Sequences analyzed for understanding visual search process

Path (1) corresponds to free exploration. Path (2) was obtained when subjects were asked to judge the material status of the family, and path (3) when they were asked to guess the age of different individuals. Partially reproduced from Yarbus, A. L. (1967)


Eye movements for usability


4/24/2009

4

Case Study: A comparison of designsCase Study: A comparison of designs

• Informationally equivalent

• Computationally NOTComputationally NOT(Larkin and Simon, 1987)


Design – I: Natlas MapMakerwww.nationalatlas.gov/natlas/natlasstart.asp

43% of the screen is used for displaying the map. Queries are returned in a pop‐up window

6/262009/04/22Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

4/24/2009

5

Design – II: Carto.nethttp://www.carto.net/papers/svg/us_crime_2000/

72% of the screen is used for displaying the map. Small windows can be minimized and/or movedSmall windows can be minimized and/or moved.

2009/04/22 7/26Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

Case Study Overview

. Pilot studyE l i l lidit i t l t l

KISS: Keep it short and simple

Slide 9/30

. Ecological validity vs. experimental control

. N=30, between subject. 15 participants per stimuli

. 3 tasks, independent variables, within subjectAll subjects solve all tasks. All subjects solve all tasks

. Systematic rotation to avoid learning effects

. Cognitive walkthrough (AOIs)

2009/04/22Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

4/24/2009

6

TasksTwo close‐ended, one open‐ended

1) What is the number of assaults in Washington County (Maine) in the year 2000?

2) Which county in the state of Oregon has the highest murder rate in the year 2000?

3) Looking at the map of the U S A overall do you see a3) Looking at the map of the U.S.A, overall, do you see a relationship (if any) between poverty rates and burglaries in the year 2000?


Cognitive Walkthrough

• Locate the town– Zoom/pan

• Find the information– Find and understand the relevant menu buttons

• Carto.net: discover mini windows, open legend, discover mouse roll‐over behavior

• Natlas: select the right menu item, redraw map, legend (optional), identify, click on county

• Pilot study (3 users) agrees


4/24/2009

7

Stimuli: Designer Interviews

• Designers in decision making roles answered ti (th k !)our questions (thanks!)

– Small online questionnaire• Modified based on System Usability Scale (SUS)

– Phone interview

– Natlas usability‐tested

» Found some features too complex and removed

– Carto.net a student project (not tested)


ParticipantsN=30, 15 geographers 15 non‐geographers, 11 females, 19 males, average age:

28. No compensation offered.


4/24/2009

8

Apparatus

• Tobii X120 eye tracker (60Hz sampling l ti ) d T bii St di ft

Figure courtesy of tobii.com

resolution) and Tobii Studio software

• Morea for delivering SUS

• 24 inch flat screen– 1600x1200

2009/04/22 13/26Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

Procedure

• Welcome, sign consent form• Background questionnaire• Background questionnaire• Training for locations of relevant counties/states• Participant comfortable and in the field of view of the tracker

• Calibrate• Record as tasks are delivered verbally and solvedRecord as tasks are delivered verbally and solved

– 5 minute limit for successful task completion• SUS delivery• Thank participant


4/24/2009

9

ResultsOverall, confirming our hypothesis, participants are significantly more efficient (faster) using Carto.net’s interface, F=7.359, p=.011<.05, but significantly more effective (accurate) using Natlas, F=5.095 and p=.032<.05.


…do you see a relationship (if any) between poverty rates and burglaries in the year 2000?

12

4

6

8

10

Yes

Maybe

No

0

2

Natlas Cartonet


4/24/2009

10

• One design is faster, other more accurate – why?

• These results indicate both designs have advantages – where?have advantages where?

Eye movements can tell us a few things..

. more fixations may indicate a less efficient search strategy,

. longer fixations may indicate difficulty with the display plotting scan paths and fixations will allow documenting what. plotting scan paths and fixations will allow documenting what people look at, how often and how long (Goldberg and Kotval, 1999; Dix et al., 2004; Bojko, 2006)


Eye movement metrics

In this studyAlso

– Time to first fixation

– Fixation length

– Observation length

– Fixation count

Also:. Gaze plots. Density maps. Watching the records for qualitative observations

Not included:. Sequence analysis

– Fixation count

– Observation count

– Participant percentage


4/24/2009

11

2009/04/22 21Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

Density Maps

Density Maps

2009/04/22 22Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

4/24/2009

12

Eye movement analysis ‐ overview

• Fixation filter radius: 50 pixels, minimum fixation duration: 100 ms

• Overall fixation counts – Natlas (sum): 19554– Carto.net (sum): 17902

• Users may have a less efficient search strategy with Natlas in comparison with Carto.net.

• Overall mean fixation durationsNatlas (M 7715 1 s SD: 139 8 s)– Natlas (M= 7715.1 s, SD: 139.8 s)

– Carto.net (M= 6642.1 s, SD: 111.3 s) • Also may indicate that, overall, Natlas has a more complex interface.


AOI Analysis, Identified IssuesAOIs

Identify and Redraw Map buttons on Natlasmouse roll‐over behavior that reflects on the legend‐bar in Carto.net.

• Natlas (slower)

– Discovery (time to first fixation): • IdentifyM=116.2 s, SD=102.1 (67% of task time), Redraw mapM=50.6 s, SD=30.6 (27% of task time)

– First use (time to first mouse click): IdentifyM=186 s, Redraw Map M=54 s• Labeling of Redraw Map was more quickly understood by the users in comparison with the Identify

– 423 fixations before Identify and 302 fixations before Redraw Map • 75% of the participants spent time on the Menu area

– Redraw map button salient, yet 30% of the users have not used this button• Both redraw and identify buttons are probably too small: Identify 0.08%, Redraw Map 0.13%

– 100% of the participants who needed assistance (5 of 15) had trouble with the Identify button


4/24/2009

13

AOI Analysis, Identified Issues – cont’dCarto.net (less accurate)

– Mouse roll‐over• 80% attempted clicking with the mouse

• 40% used the right mouse click to explore what other options may be ‘hidden’

– The legend• 80% needed assistance needed help with legend

• Double column misleads? 48% looks at the first column longer (observation length M=24.5 s, SD=23.5, fixation length M=20.7 s SD=19.9) than the second column where the information is more relevant (observation length M=15.7 s, SD=19.6, fixation length M=13.3 SD=17.0).

– Only 26.6% have discovered that the mini‐windows can be moved on Carto.net.

Comparatively:

• Isolating the query‐returns reveals itself as a favorable design choice leading to more accurate results.

• Map size: the map is 43% of the screen for Natlas where it is 72% of the screen for Carto.net. – Lesser use of zoom buttons: fixation counts on Zoom‐in 0.7 and Zoom‐out 1.6 (Natlas fixated more

times). Fixation lengths (Zoom‐in difference M=0.3s, SD=0.4; Zoom‐out difference M=0.7, SD=1.0).


SUS: Sytem Usability ScaleParticipant attitudes

SUS - Questions

SUS-mean values over all participants ratings0=strongly disagree, 4=strongly agree

Question 1, F=4.95, p=0 034< 058 NOT cumbersome

7. Learnable quickly

6. NOT Inconsistent

5. Functions well integrated

4. NOT need tech support

3. Easy to use

2. NOT complex

1. Like to use freqeuently

Natlas-mean

Cartonet-mean

p=0.034<.05Question 4: F=7.22, p=.012<.05


0 1 2 3 4

10. NOT High learning curve

9. Felt confident

8. NOT cumbersome

4/24/2009

14

Observations & Interview• Participants prefer digital over paper maps when possible

– Interactivity and responsiveness listed• 33% of Natlas viewers and 13% Carto net viewers tried to leave the site within the33% of Natlas viewers and 13% Carto.net viewers tried to leave the site within the

first 5 minutes.

“Although it should not necessarily be the case always, I have a feeling that a poorly designed GUI (graphical user interface) is also an indication of a poorly implemented system”.

“They [legend information] did NOT always mean what IThey [legend information] did NOT always mean what I assumed. I think people don’t like reading too much, and it would be best if the legends would match what most people assume by default.”

• Only 7% of our observed map users user the Help button


Conclusions

• Visual attention is commonly considered a f t l tt tiproxy for mental attention

– We do learn more about displays by studying eye movement recordings

• Eye movement recordings are complex toEye movement recordings are complex to process, however valuable in identifying micro‐level visual behavior.


4/24/2009

15

Thank you for your attention!

Questions?

Contact information:

{arzu.coltekin, simone.garlandini, benedikt.heil, sara.fabrikant}@geo.uzh.ch

Dept. of Geography, University of Zurich, Winterthurerstr. 190, 8057 Zurich, Switzerland

2009/04/22 Thanks ☺Çöltekin, Garlandini, Heil, Fabrikant ‐

EyeTrackUX 2009

Documents

Evaluating the Effectiveness of Interactive Map Interface Designs