Modeling Combined Proximity-Similarity Effects in Visual Search Tamar Avraham* Yaffa Yeshurun**...

Modeling Combined Modeling Combined Proximity-Similarity Proximity-Similarity

Effects in Visual SearchEffects in Visual Search

Tamar Avraham*Tamar Avraham*Yaffa Yeshurun**Yaffa Yeshurun**

Michael Lindenbaum*Michael Lindenbaum*

*Computer science dept., Technion, Israel*Computer science dept., Technion, Israel**Psychology dept., Haifa University, Israel**Psychology dept., Haifa University, Israel

OutlineOutline► The FLNN and COVER models for computer visionThe FLNN and COVER models for computer vision

Avraham & Lindenbaum, IEEEAvraham & Lindenbaum, IEEE--PAMI 2006 PAMI 2006

► Study 1:Study 1:

Adapting the models for human performanceAdapting the models for human performanceAvraham, Yeshurun & Lindenbaum , Journal of Vision 2008Avraham, Yeshurun & Lindenbaum , Journal of Vision 2008

► Study 2:Study 2:

Extending the models to account for spatial Extending the models to account for spatial organization effects.organization effects.

The FLNN modelThe FLNN model

feature extraction

ix

Farthest Labeled Nearest Neighbor

feature space

Avraham & Lindenbaum, IEEE-PAMI 2006

0° 30° 60° orientation

T = 0° D = 30°, 60°

T D D

The FLNN model – contThe FLNN model – cont..

1

2

3

4

Avraham & Lindenbaum, IEEE-PAMI 2006

Alternative parallel explanation: dynamic priority map

► Homogeneous distractors Homogeneous distractors

► Clustered distractors Clustered distractors –– maximum one from each maximum one from each clustercluster

Qualitative model behaviorQualitative model behavior

D

T

D

T

D

D

1

2

3

1

2

– pop-out behavior

Avraham & Lindenbaum, IEEE-PAMI 2006

Visual Search DifficultyVisual Search Difficulty►Search difficulty depends on two factors:Search difficulty depends on two factors:

T-D similaritiesT-D similarities D-D similaritiesD-D similarities

►Quantitative measures of search Quantitative measures of search difficultydifficulty the the saliencysaliency measure (measure (Rosenholtz 99Rosenholtz 99)) our our COVERCOVER measure measure

D

T

easy

T

difficult

Duncan & Humphreys 89 similarity theory

Avraham & Lindenbaum, IEEE-PAMI 2006

The COVER measureThe COVER measure

Avraham & Lindenbaum, IEEE-PAMI 2006

► A minimum-d-cover (Kolmogorov 61): the minimum number of spheres with diameter d covering all points

d

The COVER measureThe COVER measure

D

T

dT

D

T

D

DdT

COVER = 1 COVER = 3

T-D similarity effects D-D grouping

Avraham & Lindenbaum, IEEE-PAMI 2006

► If d = dT (the minimum T-D distance),

minimum-d-cover (COVER) the search difficulty

D

T

D

DdT

COVER = 2

COVER = an inherent limitation COVER = an inherent limitation

for all models/algorithmsfor all models/algorithms

FLNN Performance ≥ COVERFLNN Performance ≥ COVER

COVER and FLNNCOVER and FLNN

Avraham & Lindenbaum, IEEE-PAMI 2006

Study 1:Study 1:

Testing the ability of COVER and FLNN Testing the ability of COVER and FLNN to predict human performanceto predict human performance

Avraham, Yeshurun & Lindenbaum, Journal of Vision 2008

►COVER with internal noise: COVER with internal noise:

Low noiseLow noise

Higher noiseHigher noise

►Other visual search modelsOther visual search models Temporal-serial (Temporal-serial (Bergen&Juletz 83’Bergen&Juletz 83’) ) Signal-Detection-Theory (Signal-Detection-Theory (Palmer et. al. 93’, Eckstein 00’Palmer et. al. 93’, Eckstein 00’)) Target-Saliency model Target-Saliency model ((Rosenholtz Rosenholtz 9999’’)) Best-Normal Best-Normal ((Rosenholtz Rosenholtz 0101’’)) RCref RCref ((Rosenholtz Rosenholtz 0101’’))

dT

T

dT

T

Avraham, Yeshurun & Lindenbaum, Journal of Vision 2008

Study 1Study 1

Manipulated T-D and D-D similarityAccuracy. 2IFC.

Avraham, Yeshurun & Lindenbaum, Journal of Vision 2008

Participant Saliency (Rosenholtz

99)

COVER

Exp 1 A.P. 0.812 0.999*

Y.B. 0.538 0.998*

D.A. 0.572 0.997*

V.S. 0.570 0.999*

A.P.Z. 0.510 1*

Exp 2 A.D. - 0.926*

A.A. - 0.903*

M.D. - 0.962*

L.F. - 0.873

Exp 3 D.A. 0.900 0.862

S.M. 0.945 0.883

E.D. 0.996* 0.993*

G.S. 0.880 0.997*

Exp 4 R.A. - 0.967*

O.R. - 0.956*

R.I. - 0.979*

A.O. - 0.951*

COVER: Prediction COVER: Prediction ComparisonComparison

Avraham, Yeshurun & Lindenbaum, Journal of Vision 2008

Good correlation between

accuracy and COVER

correlation between accuracy and measure

FLNN - Prediction FLNN - Prediction ComparisonComparison

Avraham, Yeshurun, Lindenbaum VSS 2011

FLNN best in 2 test.Lowest 2/df values

Avraham, Yeshurun & Lindenbaum, Journal of Vision 2008

Study 1 SummaryStudy 1 Summary

► FLNN and COVER predict T-D and D-D similarity FLNN and COVER predict T-D and D-D similarity effects better than other prominent effects better than other prominent computational models.computational models.

► The models quantify grouping-by-similarity The models quantify grouping-by-similarity involved in visual search, by suggesting that the involved in visual search, by suggesting that the degree of within-group heterogeneity depends degree of within-group heterogeneity depends on the T-D similarity.on the T-D similarity.

Avraham, Yeshurun, Lindenbaum VSS 2011

Study 2:Study 2:

Spatial Organization Effects Spatial Organization Effects

test and model how the effects of grouping by proximity and grouping by similarity

are combined in the context of visual search

Manipulating Spatial Manipulating Spatial OrganizationOrganization

Experiment 1Experiment 1T=0° D = 23°, 47°, 70°T=0° D = 23°, 47°, 70°

The same 30 elements in all conditionsThe same 30 elements in all conditions

Avraham, Yeshurun, Lindenbaum VSS 2011

no clusters 6 clusters 3 clusters

condition 1 condition 2 condition 3

Does spatial organization matter?

Experiment 1: ResultsExperiment 1: Results

► Previous models do not account for this significancePrevious models do not account for this significance

Avraham, Yeshurun, Lindenbaum VSS 2011

Spatial Organization Effects - Spatial Organization Effects - ModelingModeling

► One possibility: a multi scale approachOne possibility: a multi scale approach((e.g., Itti et al. 1998, Rosenholtz et al. 2007e.g., Itti et al. 1998, Rosenholtz et al. 2007))

How to combine the measure over scales? How to combine the measure over scales?

max? weight and sum?max? weight and sum?Avraham, Yeshurun, Lindenbaum VSS 2011

COVER=3

COVER=2

COVER=3

no clusters no clusters 3 3 clustersclusters

Spatial Organization Effects – Spatial Organization Effects – ModelingModeling

► Our models need only some measure of Our models need only some measure of distance between each two elementsdistance between each two elements

► Combine Combine feature difference feature difference and and spatial spatial distance distance

Avraham, Yeshurun, Lindenbaum VSS 2011

,i jD

feature, spatial(1 )i jD d d

Indicates the relative effect of the forces

feature,i jD d

Spatial Organization Effects – Spatial Organization Effects – ModelingModeling

► AdvantagesAdvantages: : same treatment for similarity and proximity same treatment for similarity and proximity understand and quantify the relative effect of eachunderstand and quantify the relative effect of each

► Questions to answer in this study:Questions to answer in this study: Will this enable our models to account for the Will this enable our models to account for the

spatial organization effect?spatial organization effect? What is the value of ?What is the value of ? Is stable or stimuli dependent?Is stable or stimuli dependent?

Avraham, Yeshurun, Lindenbaum VSS 2011

, feature spatial(1 )i jD d d

FLNN predictionsFLNN predictions

► Conclusions:Conclusions: Combination of feature distance and spatial distance is Combination of feature distance and spatial distance is

essential for predictionessential for prediction Limited possibilities: implies that the model is informativeLimited possibilities: implies that the model is informative Relates to previous findings regarding the combined effects Relates to previous findings regarding the combined effects

of proximity and similarity on perceptual groupingof proximity and similarity on perceptual grouping

((e.g., Kobovy & van den Berg 2008e.g., Kobovy & van den Berg 2008))Avraham, Yeshurun, Lindenbaum VSS 2011

0.35 prediction prediction withwith

predictive ability vs. predictive ability vs.

Maximum value of 2 to pass the 2 test

Preliminary: Is stable Preliminary: Is stable or stimuli dependent?or stimuli dependent?

► Experiment 2: Experiment 2: Manipulate the number of distractor typesManipulate the number of distractor types

► Experiment 3:Experiment 3:Manipulate the distractors varianceManipulate the distractors variance

Avraham, Yeshurun, Lindenbaum VSS 2011

Exp 2 (preliminary): Exp 2 (preliminary): manipulating the number of manipulating the number of

distractor typesdistractor types

Avraham, Yeshurun, Lindenbaum VSS 2011

2 distractor types(15°, 30°)

4 distractor types(15°, 30°, 45°, 60°)

4 clustersno clusters

Avraham, Yeshurun, Lindenbaum VSS 2011

D = 15°, 30°

clustered

not clustered

Exp 3 (preliminary): Exp 3 (preliminary): manipulating distractors manipulating distractors

variancevarianceD = 15°, 45°D = 15°, 60°

Avraham, Yeshurun, Lindenbaum VSS 2011

Experiment 3 ResultsExperiment 3 ResultsExperiment 2 Experiment 2 ResultsResults

Experiment 2 and 3: Experiment 2 and 3: (preliminary) Results(preliminary) Results

Avraham, Yeshurun, Lindenbaum VSS 2011

0.35 Experiment 2 and 3: Experiment 2 and 3:

FLNN Predictions withFLNN Predictions withExperiment 3 PredictionsExperiment 3 PredictionsExperiment 2 Experiment 2

PredictionsPredictions

SummarySummary► A study of the effects of spatial organization on A study of the effects of spatial organization on

visual searchvisual search

► The FLNN model can predict effects of grouping The FLNN model can predict effects of grouping by similarity and grouping be proximityby similarity and grouping be proximity

► As it uses an explicit combination of feature As it uses an explicit combination of feature difference and spatial distance, it can help us difference and spatial distance, it can help us understand the relative effect of similarity and understand the relative effect of similarity and proximity on visual searchproximity on visual search

Avraham, Yeshurun, Lindenbaum VSS 2011

Thank you Thank you

Tamar AvrahamTamar AvrahamMichael LindenbaumMichael Lindenbaum

Yaffa YeshurunYaffa Yeshurun