Olympic weight lifter Olympic victory Olympic
achievement
Olympic weight lifter Olympic victory Olympic
achievement
Description based on “salient” pixels only
Itti & Koch, 2001Bruce Tsotsos, 2009Judd et al., 2009Previous talk of the session
Liu et al, 2007
Grab-Cut, Rother et al., 2004
Our goal: Convey the image content
Stas GofermanLihi Zelnik-ManorAyellet Tal
1. Principles for context-aware saliency
2. A saliency detection algorithm
3. Applications:• Image retargeting
1. Principles for context-aware saliency
2. A saliency detection algorithm
3. Applications:• Image retargeting• Collages
Following perceptual properties
Local low-level factors• Contrast• Color• …
Walther & Koch, 2006
Global considerations• Maintain unique features
Hou & Zhang, 2007
Local & global• Should be multi-scale
Liu et al, 2007
Visual organization (Gestalt)• Few centers of gravity• Position is important!!
Our foci
High-level• Faces• Objects• People• …
Judd et al, 2009
Low-level
With face detection
Our result
Our result
LocalWalther & Koch, 2006
GlobalHou & Zhang, 2007
Local + globalLiu et al, 2007
The steps of our algorithm
Principles 1-2:Unique appearance salient
salient
Not salient
Principles 1-2:Unique appearance salient
Principles 1-2:Unique appearance salient
( , )color i jd p p Euclidean distance between colors of patches at pi & pj
Principles 1-2:Unique appearance salient
high ( , )color i jd p p j
salient
Principle 3:Position is important!
Similar patches both near and far
Not salient
Principle 3:Position is important!
Similar patches near
Salient
Principle 3:Position is important!
( , )position i jd p p Normalized Euclidean distance between positions of pi & pj
Distance between a pair of patches:
( , )( , )
1 ( , )color i j
i jposition i j
d p pd p p
c d p p
Distance between a pair of patches:
( , )( , )
1 ( , )color i j
i jposition i j
d p pd p p
c d p p
salient
High j
Distance between a pair of patches:
( , )( , )
1 ( , )color i j
i jposition i j
d p pd p p
c d p p
salient
High for K most similar
k
rq K most similar patches at scale r
1
11 exp ( , )
i j
Kr r ri
k
S d p qK
1
11 exp ( , )
i j
Kr r ri
k
S d p qK
Salient at:• Multiple scales foreground• Few scales background
1
1 Mrr
i ir r
S SM
Scale 1 Scale 4
Principle 3:• Few centers of gravity
Context
Foci =
Include distance map
0.8iS
1 ( )focid i
X
iS
Realizing Principles 1,2,3 at multiple scales
ˆ 1 ( )i i fociS S d i
Principle 4:• Faces• Objects• …
Excluded from this talk
Single-scale saliency
Multiple scales
Final saliency
1
11 exp ( , )
i j
Kr r ri
k
S d p qK
ˆ 1 ( )i i fociS S d i
1
1 Mrr
i ir r
S SM
X
…
Walther & Koch, 2006 Hou & Zhang, 2007
Our result
Walther & Koch, 2006 Hou & Zhang, 2007
Our result
Walther & Koch, 2006 Hou & Zhang, 2007
Our result
Walther & Koch, 2006 Hou & Zhang, 2007
Our result
Walther & Koch, 2006 Hou & Zhang, 2007
Our result
Walther & Koch, 2006 Hou & Zhang, 2007
Our result
Database of Hou & Zhang
Liu et al, 2007
Our result
Image retargetingCollage
Seam Carving Our result
Seam Carving Our result
Seam Carving Our result
New definition: Context-aware saliency
Algorithm:Based on 4 perceptual principles
Applications
salient
Not salient
…