Saccadic model of eye movements for free-viewing condition

Saccadic model

O. Le Meur

Visual attentionPresentation

Computationalmodels

Performance

Conclusion

Saccadic modelof eye movementPresentation

Existing saccadicmodels

Systematictendencies

Proposed model

Results

A focus on...

Limitations

Conclusion &Perspective

Saccadic model of eye movements forfree-viewing condition

Olivier Le Meur 1 Zhi Liu 1,2

[email protected]

1IRISA - University of Rennes 1, France

2School of Communication and Information Engineering, Shanghai University, China

April 28, 20151 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Preamble

The following slides rely heavily upon the following paper:ß O. Le Meur & Z. Liu, Saccadic model of eye movements for

free-viewing condition, Vision Research, 2015,doi:10.1016/j.visres.2014.12.026.

Acknowledgments:This work is supported in part by a Marie Curie International Incoming Fellowship within the 7th European Community FrameworkProgramme under Grant Nos. 299202 and 911202, and in part by the National Natural Science Foundation of China under GrantNos. 61171144 and 61471230.

2 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Outline

1 Visual attention

2 Saccadic model of eye movement

3 Conclusion & Perspective

3 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Visual Attention

1 Visual attentionI PresentationI Computational modelsI PerformanceI Conclusion

4 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Introduction to visual attention (1/3)

Natural visual scenes are clutteredand contain many different objects

that cannot all be processedsimultaneously.

Where is Waldo, the young boywearing the red-striped shirt...

Amount of information comingdown the optic nerve 108 − 109

bits per second

Far exceeds what the brain iscapable of processing...

5 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



WE DO NOT SEE EVERYTHING AROUND US!!!

Visual attentionPosner proposed the following definition (Posner, 1980). Visual atten-tion is used:

ß to select important areas of our visual field (alerting);ß to search a target in cluttered scenes (searching).

There are several kinds of visual attention:ß Overt visual attention: involving eye movements;ß Covert visual attention: without eye movements (Covert

fixations are not observable).

6 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Bottom-Up vs Top-Downß Bottom-Up: some things draw attention reflexively, in a

task-independent way (Involuntary, Very quick, Unconscious);ß Top-Down: some things draw volitional attention, in a

task-dependent way (Voluntary; Very slow; Conscious).

Computational models of Bottom-up overt visual attention

7 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Bottom-Up vs Top-Downß Bottom-Up: some things draw attention reflexively, in a

task-independent way (Involuntary, Very quick, Unconscious);ß Top-Down: some things draw volitional attention, in a

task-dependent way (Voluntary; Very slow; Conscious).

Computational models of Bottom-up overt visual attention

7 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Computational models of Bottom-up visualattention (1/3)

Most of the computation models of visual attention have beenmotivated by the seminal work of Koch and Ullmann (Koch andUllman, 1985).

ß a plausible computationalarchitecture to predict ourgaze;

ß a set of feature maps areprocessed in a massivelyparallel manner;

ß a single topographic saliencymap.

8 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Taxonomy of models:ß Information Theoretic

models;ß Cognitive models;ß Graphical models;ß Spectral analysis

models;ß Pattern classification

models;ß Bayesian models.

Extracted from (Borji and Itti, 2013).

9 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Cognitive models:

as faithful as possible tothe Human Visual System

(HVS)

ß inspired by cognitiveconcepts;

ß based on the HVSproperties.

Extracted from (Borji and Itti, 2013).

10 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Performance on still images (1/3)

The requirement of ground truth

ß Eye tracker:

ß A panel ofobservers;

ß An appropriateprotocol.

Adapted from (Judd et al., 2009).

11 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß Discrete fixation map f i for the ith

observer:

f i(x) =M∑

k=1δ(x− xk)

where M is the number of fixations and xk isthe kth fixation.

ß Continuous saliency map S :

S(x) =

(1N

N∑i=1

f i(x)

)∗Gσ(x)

where N is the number of observers and Gσ isa 2D Gaussian function.

More details in (Le Meur and Baccino, 2013).12 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Performance in terms of linear correlation, extracted from (Borjiet al., 2012).

ß more than 30 models.

13 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Conclusion (1/1)

The picture is much clearer than 10 years ago!BUT...

ß How far are we from human performance?∗

ß How to take advantage of top-down influences?∗

ß What is the best score?∗

ß What is the most representative dataset?∗

Important aspects of our visual system are clearly overlooked;Current models implicitly assume that eyes are equally likely tomove in any direction;Systematic biases are not taken into account;The temporal dimension is not considered (static saliency map)...

∗ Open challenges of visual attention, tutorial CVPR’13, Borji et al.

14 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Visual Attention

2 Saccadic model of eye movementI PresentationI Existing saccadic modelsI Systematic tendenciesI Proposed modelI ResultsI A focus on...I Limitations

15 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Presentation (1/1)

Eye movements are composed of fixations and saccades. A sequenceof fixations is called a visual scanpath.

The fundamental assumption is that scanpaths can be described by afirst-order Markov process, i.e. each eye fixation only depends on the

previous one.

ß The seminal work of (Ellis and Smith, 1985, Stark and Ellis,1981) described a probabilistic approach where the eyemovements are modelled as a first-order Markov process.

16 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Existing saccadic models (1/6)

ß Saliency models and eye movementmodels are combined (Itti and Koch,2000, Itti et al., 1998).

• winner-take-all algorithm,• an attended area is inhibited for

approximately 500-900 ms,• jumps from one salient location to

the next in approximately 30-70ms.the model is deterministic, given aset of of parameters, history ofrecent fixation locations...,the model does not reproduce thesystematic biases of ouroculomotor system.

17 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß (Brockmann and Geisel, 2000)’smodel is related to Levy flightswhich is a particular type ofrandom walk with a step lengththat follows a heavy-taileddistribution.

• The model is stochastic,• The model generates

scanpaths similar in theirnature to Levy flights, i.e.possess a power lawdependency in their magnitudedistribution..Saliency information is notused,Inhibition is not used.

Top: Model scanpath generated by afinite variance random walk

(Gaussian system); Bottom: Sameas top, except here the scanpath is a

Levy flights (Cauchian system).

18 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß (Boccignone and Ferraro, 2004) extended Brockmann’s work,and modeled eye gaze shifts by using Levy flights constrained bythe saliency.

• The model is based on a saliency map,• The jump has a higher probability to occur if the target site is

strongly connected in terms of saliency.Orientation bias is not used,Inhibition is not used.

19 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß (Wang et al., 2011)’s model is based on foveal images.• Visual working memory.• Distribution of saccade amplitudes is taken into account to jump

from one fixation to another.• Residual information are used to infer the next fixation location.

Orientation bias is not used,Time course of IoR is not considered.

20 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß (Tavakoli et al., 2013)• The model is based on a Markovian process of order one.• Visited locations are kept in order to inhibit immediate return of

attention to those locations.• Several parameters are learned from human eye tracking data,

such as p(d) which represents the probability of a jump withlength d. p() is a Gaussian mixture.Orientation bias is not used,Time course of IoR is not considered.

21 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß (Liu et al., 2013)• The model is based on a Markovian process of order one.• Saliency map.• Levy flight to reproduce the distribution of saccade amplitudes.• Semantic content (transition matrix learned by using Hidden

Markov Model).Orientation bias is not used,Time course of IoR is not considered.

Assuming a Markov process and independence between the threefactors:

22 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Systematic tendencies (1/7)

Saccade amplitudes & Saccade orientations on natural scenes

ß Short saccades around 1to 3 degrees of visualangle;

ß Short saccades are morefrequent than long ones.

23 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations




ß Anisotropic shape;ß More horizontal saccades

than vertical ones;ß Very few diagonal

saccades.

23 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations




ß Joint distribution ofsaccade amplitudes andorientations;

ß Strong horizontal biasand small saccade.

23 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Horizontal and vertical cross sections of the probability distributionfor horizontal saccades and vertical saccades

ß Saccades in the samedirection as theprecedent one, i.e. withan angle of 0, are morelikely than saccades inthe opposite direction;

ß Upward vertical saccadesare more likely thandownward saccades(consistent with (Greeneet al., 2014, Tatler andVincent, 2009)).

24 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations




From top-left to bottom-right: Le Meur, Bruce, Kootstra and Judd’sfixation datasets. 25 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Center bias on natural scenes

ß Visual fixations areneither distributedevenly nor randomly.

26 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Saccade amplitudes & Saccade orientations on webpages

ß Joint distribution ofsaccade amplitudes andorientations;

ß Strong horizontal bias inthe rightward direction;

ß F-shaped pattern.

From (Shen and Zhao, 2014)’s eye fixation dataset.27 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (1/10)

So, what are the key ingredients for successfully designing a saccadicmodel?

ß The model has to be stochastic: the subsequent fixation cannot becompletely specified (given a set of data).

ß The model has to generate plausible scanpaths that are similar tothose generated by humans in similar conditions: distribution ofsaccade amplitudes and orientations, center bias...

ß Inhibition of return has to be considered: time-course, spatial decay...

ß Fixations should be mainly located on salient areas.

28 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (2/10)

Let I : Ω ⊂ R2 7→ R3 an input image and xt a fixation point at timet.

We consider the 2D discrete conditional probabilityp (x|xt−1, · · · , xt−T) which is composed of three terms:

p (x|xt−1, . . . , xt−T) ∝ pBU (x)pB(d, φ)pM (x, t) (1)

ß pBU : Ω 7→ [0, 1] is the grayscale saliency map;ß pB(d, φ) represents the joint probability distribution of saccade

amplitudes and orientations. d is the saccade amplitude between twofixation points xt and xt−1 (expressed in degree of visual angle), andφ is the angle (expressed in degree between these two points);

ß pM (x, t) represents the memory state of the location x at time t.This time-dependent term simulates the inhibition of return.

29 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (3/10)Bottom-up saliency map

ß pBU is the bottom-up saliency map.• Computed by GBVS model (Harel et al., 2006). According to

(Borji et al., 2012)’s benchmark, this model is among the bestones and presents a good trade-off between quality andcomplexity.

• pBU (x) is constant over time. (Tatler et al., 2005) indeeddemonstrated that bottom-up influences do not vanish over time.

30 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (4/10)Joint probability distribution of saccade amplitudes and orientations

ß pB(d, φ) represents the joint probability distribution of saccadeamplitudes and orientations.

• We define di and φi the distance and the angle between eachpair of successive fixations respectively (several eye fixationdatasets were used).

pB(d, φ) = 1n

n∑i=1

Kh(d − di , φ− φi) (2)

where, n is the total number of samples (n = 105727) and Kh isa two-dimensional Gaussian kernel. h = (hd , hφ) is the kernelbandwidth.Separate bandwidths were used for angle and distancecomponents. The two bandwidth parameters are chosenoptimally based on the linear diffusion method proposedby (Botev et al., 2010).

31 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (5/10)Joint probability distribution of saccade amplitudes and orientations

ß pB(d, φ) represents the joint probability distribution of saccadeamplitudes and orientations.

32 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (6/10)Memory effect and inhibition of return (IoR)

ß pM (x, t) represents the memory effect and inhibition of return(IoR) of the location x at time t.

• We assume that the IoR effect disappears after T = 8 fixations.Considering that a fixation duration lasts 300 ms on average, anattended location could be refixated after 2.4 seconds (Mannanet al., 1997, Samuel and Kat, 2003);

• We also assume that the spatial IoR effect declines as aGaussian function Φσi (d) with the Euclidean distance d fromthe attended location (Bennett and Pratt, 2001);

• The temporal decline of the IoR effect is simulated by a simplelinear model.

33 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß pM (x, t) represents the memory effect and inhibition of return(IoR) of the location x at time t:

pM (x, t) =

1, t = 0⌊

pM (x, t − 1)− I (y|xt) +ξ∑

k=1

R(y|xt−k)

⌋otherwise

(3)where y ∈ Ω represent all the possible locations in the input image. b.c isused to clip the value to the range [0, 1]. ξ is the number of visual fixationsto consider: ξ is equal to min(T , t − 1) with T the number of fixationsneeded for an area to recover its initial saliency.

I and R represent the inhibition and the recovery functions, respectively:

I (y|z) = Φσi (‖y − z‖) (4)

R(y|z) = Φσi (‖y − z‖)T (5)

where Φσi (d) is the 2D Gaussian representing the spatial effect of the IoR.34 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Illustration of the temporal evolution of pM (x, t) for an image definedover the space Ω = [1...256]× [1...256] and for an unique fixationpoint having the spatial coordinate (128, 128). This fixation isrepresented by the red cross. We consider, in this example, that thenumber of fixation T to recover the original saliency is equal to 4.

35 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (9/10)Selecting the next fixation point

ß Optimal next fixation point (Bayesian ideal searcher proposedby (Najemnik and Geisler, 2009)):

x∗t = arg maxx∈Ω

p (x|xt−1, · · · , xt−T) (6)

Problem: this approach does not reflect the stochastic behaviorof our visual system and may fail to provide plausiblescanpaths (Najemnik and Geisler, 2008).

ß Rather than selecting the best candidate, we generate Nc = 5random locations according to the 2D discrete conditionalprobability p (x|xt−1, · · · , xt−T).The location with the highest saliency gain is chosen as the nextfixation point x∗t .

36 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Proposed model (10/10)Selecting the next fixation point

Probability of saccade targeting in retinocentric space by considering thatthe previous fixation xt−1 is the centre of the plot. The red crosses are theNc candidates that have been randomly drawn according to the shownprobability. From left to right: Nc = 5, 10, 15.

When Nc = 1, the randomness is maximal, whereas, when Nc increases,the stochastic behavior of the proposed method is getting less important.

37 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Results (1/8)

The relevance of the proposed approach is assessed with regard tothe plausibility, the spatial precision of the simulated scanpath

and ability to predict saliency areas.

ß Do the generated scanpaths present the same oculomotor biasesas human scanpaths?

ß What is the similarity degree between predicted and humanscanpaths?

ß Could the predicted scanpaths be used to form relevant saliencymaps?

38 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Results (2/8)Are the simulated scanpaths plausible?

ß Protocol:• We assume that the simulated scanpaths are obtained in a

context of purely free viewing ⇒ top-down effects are not takeninto account.

• For each image in Bruce’s and Judd’s datasets, we generate 20scanpaths, each composed of 10 fixations ⇒ 224600 generatedvisual fixations.

• We assume that the visual fixation duration is constant. So,considering an average fixation duration of 300ms, 10 fixationsrepresent a viewing duration of 3s.

• Bottom-up saliency maps are computed by GBVS model (Harelet al., 2006).

39 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Top row: Bruce’s dataset. Bottom row: Judd’s dataset.

40 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



Impact of the oculomotor constraints (spatial and orientation),WTA+IoR

41 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Results (5/8)What is the similarity degree between predicted and human scanpaths?

There are a few methods for comparing scanpaths:string-edit (Privitera and Stark, 2000), Dynamic Time Warpalgorithm (DTW) (Gupta et al., 1996, Jarodzka et al., 2010). Moredetails in (Le Meur and Baccino, 2013).

ß We use DTW’s method.

ß For a given image, 20 scanpaths each composed of 10 fixationsare generated. The final distance between the predictedscanpath and human scanpaths is equal to the average of the 20DTW scores.

The closer to 0 the value DTW , the more similar the scanpaths.

42 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Results (6/8)What is the similarity degree between predicted and human scanpaths?

ß Five models are evaluated.ß The error bars correspond to the SEM (Standard Error of the Mean).ß DTW = 0 when there is a perfect similarity between scanpaths.ß There is a significant difference between the performances of the proposed

model and (Boccignone and Ferraro, 2004)’s model (paired t-test,p << 0.01).

ß As expected, the lowest performances are obtained by the random model.43 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Results (7/8)Scanpath-based saliency map

ß We compute, for each image, 20 scanpaths, each composed of10 fixations.

ß For each image, we created a saliency map by convolving aGaussian function over the fixation locations.

(a) original image; (b) human saliency map; (c) GBVS saliency map; (d)GBVS-SM saliency maps computed from the simulated scanpaths.

44 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Results (8/8)Scanpath-based saliency map

45 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Saliency map, randomness and webpages (1/3)

ß Influence of the saliency map:

Top2-SM: we aggregated the saliency maps of GBVS and RARE2012 modelsthrough a simple average. (Le Meur and Liu, 2014) demonstrated that a simpleaverage of the top 2 saliency maps, computed by GBVS and RARE2012 models,significantly outperforms the best saliency models.

46 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß Randomness:

The maximal randomness is obtained when Nc = 1.

47 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations



ß Is the model valid for webpages? By using the eye fixationdataset of (Shen and Zhao, 2014), we show that oculomotorbiases on webpages differ from those observed on natural scenes:

(a) Distribution of saccade amplitudes; (b) distribution of saccadeorientations; (c) joint distribution of saccade orientations and amplitudes.

ß There is a strong tendency for making horizontal small saccades in therightward direction. This tendency is known as the F-bias (Buscher et al.,2009).

48 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Limitations of the proposed model

Still far from the reality...

ß We do not predict the fixation durations. Some models could beused for this purpose (Nuthmann et al., 2010, Trukenbrod andEngbert, 2014).

ß Second-order effect. We assume that the memory effect occursonly in the fixation location. However, are saccades independentevents? No, see (Tatler and Vincent, 2008).

ß High-level aspects such as the scene context are not included inour model.

ß Should we recompute the saliency map after every fixations?Probably yes...

ß Randomness (Nc) should be adapted to the input image. Bydefault, Nc = 5.

ß Is the time course of IoR relevant? Is the recovery linear?

49 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Conclusion

3 Conclusion & Perspective

50 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Conclusion & Perspective

Improvementsß Dealing with our model’s limitations.ß A new metric to evaluate the similarity between scanpaths.ß How to integrate top-down effects?ß How to attribute greater meaning to fixations (Bruce, 2014,

Follet et al., 2011, Unema et al., 2005)Applications

ß How can we use predicted scanpath in computer visionapplications?

ß Is there an application to computational medicine?

51 / 52

Saccadic model

O. Le Meur


Computationalmodels

Performance

Conclusion




Proposed model

Results

A focus on...

Limitations


Thanks for your attention

52 / 52

Saccadic model

O. Le Meur

References

ReferencesP. J. Bennett and J. Pratt. The spatial distribution of inhibition of return:. Psychological Science, 12:76–80, 2001.

G. Boccignone and M. Ferraro. Modelling gaze shift as a constrained random walk. Physica A: Statistical Mechanics and itsApplications, 331:207 – 218, 2004. ISSN 0378-4371. doi: http://dx.doi.org/10.1016/j.physa.2003.09.011.

A. Borji and L. Itti. State-of-the-art in visual attention modeling. IEEE Trans. on Pattern Analysis and Machine Intelligence, 35:185–207, 2013.

A. Borji, D. N. Sihite, and L. Itti. Quantitative analysis of human-model agreement in visual saliency modeling: A comparativestudy. IEEE Transactions on Image Processing, 22(1):55–69, 2012.

Z.I. Botev, J.F. Grotowski, and D. P. Kroese. Kernel density estimation via diffusion. The annals of Statistics, 38(8):2916–2957,2010.

D. Brockmann and T. Geisel. The ecology of gaze shifts. Neurocomputing, 32(1):643–650, 2000.

Neil D. B. Bruce. Towards fine-grained fixation analysis: Distilling out context dependence. In Proceedings of the Symposium onEye Tracking Research and Applications, ETRA ’14, pages 99–102, New York, NY, USA, 2014. ACM. ISBN978-1-4503-2751-0. doi: 10.1145/2578153.2578167. URL http://doi.acm.org/10.1145/2578153.2578167.

Georg Buscher, Ed Cutrell, and Meredith Ringel Morris. What do you see when you’re surfing? using eye tracking to predict salientregions of web pages. In Proceedings of CHI 2009. Association for Computing Machinery, Inc., April 2009. URLhttp://research.microsoft.com/apps/pubs/default.aspx?id=76826.

S. R. Ellis and J. D. Smith. Patterns of statistical dependency in visual scanning, chapter Eye Movements and Human InformationProcessing, pages 221–238. Elsevier Science Publishers BV, (eds) Amsterdam, North Holland Press, 1985.

B. Follet, O. Le Meur, and T. Baccino. New insights into ambient and focal visual fixations using an automatic classificationalgorithm. i-Perception, 2(6):592–610, 2011.

Harold H. Greene, James M. Brown, and Barry Dauphin. When do you look where you look? a visual field asymmetry. VisionResearch, 102(0):33 – 40, 2014. ISSN 0042-6989. doi: http://dx.doi.org/10.1016/j.visres.2014.07.012. URLhttp://www.sciencedirect.com/science/article/pii/S0042698914001710.

L. Gupta, D. L. Molfese, R. Tammana, and P. G. Simos. Nonlinear alignment and averaging for estimating the evoked potential.IEEE Transactions on Biomedical Engineering, 43(4):348–356, 1996.

J. Harel, C. Koch, and P. Perona. Graph-based visual saliency. In Proceedings of Neural Information Processing Systems (NIPS),2006.

52 / 52

http://doi.acm.org/10.1145/2578153.2578167

http://research.microsoft.com/apps/pubs/default.aspx?id=76826

http://www.sciencedirect.com/science/article/pii/S0042698914001710

Saccadic model

O. Le Meur

References

L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40(10-12):1489–1506, May 2000.

L. Itti, C. Koch, and E. Niebur. A model for saliency-based visual attention for rapid scene analysis. IEEE Trans. on PAMI, 20:1254–1259, 1998.

H. Jarodzka, K. Holmqvist, and K. Nystrom. A vector-based, multidimensional scanpath similarity measure. In ETRA, pages211–218, 2010.

T. Judd, K. Ehinger, F. Durand, and A. Torralba. Learning to predict where people look. In ICCV, 2009.

C. Koch and S. Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. Human Neurobiology, 4:219–227, 1985.

O. Le Meur and T. Baccino. Methods for comparing scanpaths and saliency maps: strengths and weaknesses. Behavior ResearchMethod, 45(1):251–266, 2013.

O. Le Meur and Z. Liu. Saliency aggregation: Does unity make strength? In ACCV, 2014.

H. Liu, D. Xu, Q. Huang, W. Li, M. Xu, and S. Lin. Semantically-based human scanpath estimation with hmms. In ICCV, 2013.

S.K. Mannan, K.H. Ruddock, and D.S. Wooding. Fixation patterns made during brief examination of two-dimensional images.Perception, 26(8):1059–1072, 1997.

J. Najemnik and W.S. Geisler. Eye movement statistics in humans are consistent with an optimal strategy. Journal of Vision, 8(3):1–14, 2008.

J. Najemnik and W.S. Geisler. Simple summation rule for optimal fixation selection in visual search. Vision Research, 42:1286–1294, 2009.

Antje Nuthmann, Tim J. Smith, Ralf Engbert, and John M. Henderson. CRISP: A Computational Model of Fixation Durations inScene Viewing. Psychological Review, 117(2):382–405, April 2010. URLhttp://www.eric.ed.gov/ERICWebPortal/detail?accno=EJ884784.

M. I. Posner. Orienting of attention. Quarterly Journal of Experimental Psychology, 32:3–25, 1980.

C. M. Privitera and L. W. Stark. Algorithms for defining visual regions-of-interest: Comparison with eye fixations. IEEETransactions on Pattern Analysis and Machine Intelligence (PAMI), 22:970–982, 2000.

A.G. Samuel and D. Kat. Inhibition of return: a graphical meta-analysis of its time course and an empirical test of its temporaland spatial properties. Psychonomic Bulletin & Review, 10(4):897–906, 2003.

Chengyao Shen and Qi Zhao. Webpage saliency. In ECCV. IEEE, 2014.52 / 52

http://www.eric.ed.gov/ERICWebPortal/detail?accno=EJ884784

Saccadic model

O. Le Meur

References

L. W. Stark and S. R. Ellis. Scanpaths revisited: cognitive models direct active looking, pages 193–226. Lawrence ErlbaumAssociates, Hillsdale, NJ, 1981.

B.W. Tatler and B. T. Vincent. The prominence of behavioural biases in eye guidance. Visual Cognition, Special Issue: EyeGuidance in Natural Scenes, 17(6-7):1029–1059, 2009.

B.W. Tatler and B.T. Vincent. Systematic tendencies in scene viewing. Journal of Eye Movement Research, 2:1–18, 2008.

B.W. Tatler, R. J. Baddeley, and I.D. Gilchrist. Visual correlates of fixation selection: effects of scale and time. Vision Research,45:643–659, 2005.

H.R. Tavakoli, E. Rahtu, and J. Heikkika. Stochastic bottom-up fixation prediction and saccade generation. Image and VisionComputing, 31:686–693, 2013.

HansA. Trukenbrod and Ralf Engbert. Icat: a computational model for the adaptive control of fixation durations. PsychonomicBulletin & Review, 21(4):907–934, 2014. ISSN 1069-9384. doi: 10.3758/s13423-013-0575-0. URLhttp://dx.doi.org/10.3758/s13423-013-0575-0.

P.J.A. Unema, S. Pannasch, M. Joos, and B.M. Velichkovsky. Time course of information processing during scene perception: therelationship between saccade amplitude and fixation duration. Visual Cognition, 12(3):473–494, 2005.

W. Wang, C. Chen, Y. Wang, T. Jiang, F. Fang, and Y. Yao. Simulating human saccadic scanpaths on natural images. In CVPR,2011.

52 / 52

http://dx.doi.org/10.3758/s13423-013-0575-0