Context for low-level saliency detection

Devi Parikh, Larry Zitnick and Tsuhan Chen

For what can context be used?

• So far higher level tasks

• What about lower level tasks?

• Picking out salient (representative) patches in an image?

Set upBag-of-words paradigm

Sample image Classify

SVM

Build histogram

?

Sample image

Saliency• Interest point detectors

– [Lowe 2004, Harris 1988, Kadir-Brady 2001, etc.]

• Uniform

• Discriminative– [Nowak et al., ECCV 06, Vidal-Naquet et al., ICCV 2003]

• Contextual– Co-occurrence based– Relative location based

Contextual saliency

n

jiji xx

n 1

o |φ1S

abjb

n

j

m

a

m

biai wwywyw

n|p|p|p1

1 1 1

o

S

Association of patch i to

word a

Association of patch j to

word b

Likelihood of word b given

word a

MLE from images

Normal distribution

Normal distribution

Occurrence based

Similarly, relative location based

Datasets coast forest highway inside-city mountain open-country street tall-building

cars bicycles motorbikes people

[Oliva Torralba IJCV 2001]

Pascal-01

Features• Scene recognition– Color information– Some gradient information inherent

• Object recognition– SIFT

Results

Results

Saliency maps

Saliency maps

Sampling strategies

• Sorting

• Random sampling

• Sequential sampling

Sequential sampling

Sequential sampling

Sequential sampling

Results

Contributions

• Context can be leveraged for low-level tasks

• Outperform several existing saliency measures

• Sparse representation was found to be more accurate

Discussion• Discrminative vs. contextual saliency

• Saliency is a subjective term: task and domain dependent– Representative (usual)– Interesting (unusual)– Generic defintion: Informative

• Contextual saliency is unsupervised but is dataset dependent