WhittleSearch: Image Search with Relative Attribute FeedbackCVPR 2012

Adriana Kovashka Devi Parikh Kristen Grauman

University of Texas at Austin Toyota Technological Institute Chicago (TTIC)

Approach

•Dataset

•At each iteration, the top K < N ranked images

Binary Relevance Feedback

Binary Relevance Feedback

•Current reference set

•Score function

Relative Attribute Feedback

Learning Relative Attributes

•Mechanical Turk (MTurk)

Updating the scoring function from feedback•Feedback form

▫“ What I want is more/less/similarly m than image Itr ”

Three cases

Score function of Relative Attribute Feedback

•Take the intersection of all F feedback

Hybrid Feedback Approach

x denotes the Cartesian product

Experimental Results

Datasets• Shoes

▫14,658 shoe images from like.com▫attributes—‘pointy-at-the-front’, ‘open’, ‘bright-

in-color’, ‘covered-with-ornaments’, ‘shiny’, ‘highat-the-heel’, ‘long-on-the-leg’, ‘formal’, ‘sporty’, and ‘feminine’

• PubFig▫the Public Figures dataset of human faces▫772 images from 8 people and 11 attributes

• OSR▫the Outdoor Scene Recognition dataset of

natural scenes▫2,688 images from 8 categories and 6 attributes

Datasets

•Training set ▫750 triplets of images (i , j , k) from each

dataset

•Score correlation▫Normalized Discounted Cumulative Gain at

top K (NDCG@K)▫K = 50

Iteration experiments on the three datasets

Amount of feedback

Ranking accuracy with first feedback

•Attribute meaning▫“amount of perspective” on a scene is less

intuitivethan “shininess” on shoes