EyeR

8/13/2019 EyeR WWWpres

1/26

Eye Tracking Analysis of User

Behavior in WWW Search

Laura Granka

Thorsten Joachims

Geri Gay


2/26

why use eye-tracking forinformation retrieval?

Understand how searchers

evaluate online search results

Enhanced interface design More accurate interpretation of

implicit feedback (eg, clickthrough data)

More targeted metrics for evaluating retrieval

performanceFigure: popular regions are

highlighted through shadow-intensity
http://www.cs.cornell.edu/


3/26

key research questions

How long does it take searchers to select adocument?

How many abstracts do searchers look atbefore making a selection?

Do searchers look at abstracts ranked lowerthan the selected document?

Do searchers view abstracts linearly?

Which parts of the abstract are most likely to

be viewed?


4/26

what is eye-tracking?

Device to detect andrecord where and what

people look at

Multiple applications:reading, usability, visual

search, in both physical

and virtual contextsEye tracking device

Figure: Cornell HCI eye trackingconfigurationView of subjects pupil onmonitor; used for calibration


5/26

ocular indices for www tracking

Fixations: ~200-300ms; information is acquired

Saccades: extremely rapid movements between fixations

Pupil dilation: size of pupil indicates interest, arousal

Aggregate eye-tracking graphs depictviewing intensity in key regions

Scanpath output depicts pattern ofmovement throughout screen. Blackmarkers represent fixations.


6/26

experimental search tasks

Ten search tasksgiven to allparticipants

Search topicsincluded travel,

science, movies,local, television,college, and trivia

Searches evenly

split betweeninformational andnavigational tasks


7/26

experimental procedures

Users conducted liveGoogle searches

Users allowed to

search freely, with

any queries

Script removed all ad

content

Proxy stored allpages and log files

Figure: Specific zones werecreated around each result, enablingeye-movements to be analyzedspecific to the rankings


8/26

sample eye-tracking output


9/26


10/26



11/26



12/26



13/26

overall searching behaviorHow long does it take users to select a document?

0

1

2

3

45

6

7

8

9

10

cornell

jord

an

dudera

nch

timem

achi

ne

moun

tain

hous

ing

primary

antibio

tics

em

eril

greyho

und

meant

ime(s)

Overall mean: 5.7 seconds, St.D: 5.4

Time spent before a result is clicked

less difficultmore difficult task


14/26

overall viewing behavior

Total number of abstracts viewed per page

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10

Total number of abstracts viewed

frequency

Mean: 3.07 Median/Mode: 2.00

Most likely to view only twodocuments per results set

How many abstracts do we view, and in what order?

Notice dipafter page

break


15/26

overall viewing behaviorHow many abstracts do we view, and in what order?

Instance of arrival to each result

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Rank of re sult

m

eanfixationvalueofarrival

Results viewed linearly


16/26


Time spent viewing each abstract compared with the frequencythat each rank is selected. Error bars are 1 SE

How much time do we spend viewing each abstract?

0

20

40

60

80

100

120

140

160

180

1 2 3 4 5 6 7 8 9 10 11

Rank of result

#timesra

nkselected

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

mean

time(s)

# times result selected

time spent in abstract


17/26


Number of abstracts viewed above and below selected link

How thoroughly do we view the results set?


18/26

overall viewing behaviorWhat information in the abstract is most useful?

Title: 30%

Snippet: 43%

Category: 0.3%

URL: 21%Other: 5%(includes,cached, similar pages,description)

Percentage of time spent viewing each part of abstract

Other: 5.3%

Title: 30.5%

Category: 0.3%Snippet: 42.8%

URL: 21.1%


19/26

overall searching behavior

*Difficulty and satisfaction are ranked on a 1-10 scale; 10 meaning very difficult and very satisfied, respectively

0

1

2

3

4

5

6

7

8

9

Cornell

mansion

M ichael

Jordan

Dude

ranch

Time

M achine

Tallest

mountain

CM U

housing

NY

Primary

First

antibiotic

Emeril Greyhound

search task

ranking

Difficulty

Satisfaction

Search task difficulty and satisfaction with Google

less difficultmore difficult


20/26


Mean rank of selected doc: 2.66 Median/ Mode: 1.00

Task difficulty

influences rank of selected document and number of abstracts viewed

less difficultmore difficult

Cornell

mansion

M ichael

Jordan

Dude

ranch

Time

M achine

Tallest

mountain

CM U

housing

NY

Primary

First

antibiotic

Emeril Greyhound

search task


21/26


1. Michael Jordan statistician 20

2. Thousand acres dude ranch 11

2. One thousand acres dude ranch 11

3. 1000 acres dude ranch 9

4. Time machine movie 7

4. Carnegie mellon university graduate housing 7

5. Imdb 6

5. Emeril lagasse 6

5. First modern antibiotic 6

5. Greyhound bus 6

5. Carnegie mellon graduate housing 6

Top Query Terms Frequency


22/26


23/26

conclusions

Document selected in under 5 seconds Users click on the first promising link they see

Results viewed linearly

Top 2 results most likely to be viewed Users rather reformulate query than scroll

Task type and difficulty affect viewing

behavior Presentation of results affects selection


24/26

future researchImpact on advertising

With such fast selections being made, will searchers even view ads?

?

Ads most likely to be seen:

Difficult task

Ambiguous info needInformational query

Low searcher expertise


25/26

future research

Relevance judgments

Do we spent more time viewing relevant abstracts?

Do we click the first relevant abstract viewed?

Does pupil dilation increase for more relevantdocuments

If results were re-ranked, would

viewing behavior differ?


26/26

Documents

EyeR