Embed Size (px)
DESCRIPTION
Relevance Feedback for the Earth Mover‘s Distance. Marc Wichterich , Christian Beecks, Martin Sundermeyer, Thomas Seidl Data Management and Data Exploration Group RWTH Aachen University, Germany. Introduction. Distance-based Adaptable Similarity Search - PowerPoint PPT Presentation
Citation preview
Relevance Feedback for the Earth Mover‘s Distance / 21I9CHAIR OF COMPUTER SCIENCE 9DATA MANAGEMENT AND EXPLORATION
Relevance Feedback for theEarth Mover‘s Distance
Marc Wichterich, Christian Beecks, Martin Sundermeyer, Thomas Seidl
Data Management and Data Exploration GroupRWTH Aachen University, Germany
Relevance Feedback for the Earth Mover‘s Distance / 21
Introduction Distance-based Adaptable Similarity Search Similarity of objects defined by distance function Small distance → similar, large distance → dissimilar Query by example: user-given object, find similar ones Query and distance only approximate descriptions of
user’s desired result If delivered result does not meet expectations:
Bad query? Bad distance? Bad database? How to do it better? Relevance Feedback attempts to adapt query/similarity model
based on simple user input (result relevancy)
2
Relevance Feedback for the Earth Mover‘s Distance / 213
Relevance Feedback
userDB
query, feedback
results
feedback system
similarity model
Photo: Flickr / Caro Wallis
Earth Mover’s Distance
RF for EMD
RFEMD
Relevance Feedback for the Earth Mover‘s Distance / 21
Overview Introduction Adaptive Similarity Model
Feature Signatures The Earth Mover’s Distance
Relevance Feedback for the Earth Mover’s Distance Experimental Evaluation Conclusion
4
Relevance Feedback for the Earth Mover‘s Distance / 21
Similarity Model – Feature Signatures
5
x
y
color
Relevance Feedback for the Earth Mover‘s Distance / 21
Similarity Model – Earth Mover’s Distance Introduced in Computer Vision by Rubner et al. Used in many differing application domains Idea: transform features of Q into features of P EMD: minimum of transformation cost
6
Q Px
y
x
y
Relevance Feedback for the Earth Mover‘s Distance / 21
Feature Transformation
7
Relevance Feedback for the Earth Mover‘s Distance / 21
EMD – Formal Definition Modeled as linear optimization (transportation problem)
8
Relevance Feedback for the Earth Mover‘s Distance / 21
Overview Introduction Adaptive Similarity Model Relevance Feedback for the Earth Mover’s Distance
The Feedback Loop Query Adaptation Heuristic EMD Adaptation Optimization-based EMD Adaptation
Experimental Evaluation Conclusion
9
Relevance Feedback for the Earth Mover‘s Distance / 21
The Feedback Loop
10
user
DB
query, feedback
results
feedback system
similarity model
yes
exit
start
get query
adapt distance
no get feedback
adapt query
retrieve results
display results
satisfied?
?
Relevance Feedback for the Earth Mover‘s Distance / 21
Query Adaptation Input: signatures from relevant objects Output: new query signature Idea: cluster signature elements Refinements by Rubner:
Only keep clusters with elements from majority of signatures
Reweight resulting signatureaccordingly
Combine with fixed gd L2 and call it „Query-by-Refinement“
„Query-by-Refinement“ is baseline for our evaluation We adapt EMD via ground distance
11
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Relevance Feedback for the Earth Mover‘s Distance / 21
Heuristic EMD Adaptation 1 Approach: pick gd based on feedback gd should reflect user preferences:
Don’t care if blue cluster at upper half of image is moved left/right
Do care if it is moved vertically Use variance information in relevant feedback
Low variance → assume user cares High variance → assume user does not care
Measure variance in feedback locally around query signature elements ci
(Q). Define gd: c(Q) x FS → R ( )
12
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Relevance Feedback for the Earth Mover‘s Distance / 21
Heuristic EMD Adaptation 2 Not 1 but m distance functions: gdi(ci
(Q),y) = ((ci(Q)- y) Vi (ci
(Q)- y)T)½
Weighted Euclidean Distances (weights on diagonal of Vi) Vi : inverted variance for ci
(Q) per feature space dimension
13
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Relevance Feedback for the Earth Mover‘s Distance / 21
Optimization-Based EMD Adaptation 1 Aim: Pick best possible gd. Failback: Find a good one. Q: When is gd good? A: If ranking it produces is good. New Q: When is a ranking of DB good?
Given ground truth, a number of measures exist We used “average precision at relevant positions”
We have ground truth for part of the DB: feedback Idea: test candidates for gd on feedback
14
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Ranking Avg. Precision1 1 1 1 0 0 0 0 1.000
1 1 0 1 0 1 0 0 0.854
0 0 0 0 1 1 1 1 0.365
Relevance Feedback for the Earth Mover‘s Distance / 21
Optimization-Based EMD Adaptation 2 Optimization:
Optimization variable: gd Objective function: avgPrec(EMDgd , q, Feedback) Constraints: m weighted Euclidean distances
Analytic optimization with closed form for weights infeasible (ranking/sorting, EMDs in objective function)
Probabilistic optimization via Simulated Annealing Start with some initial solution Move in solution space Compute objective function Adopt solution with certain probability Iterate & turn more greedy
15
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Relevance Feedback for the Earth Mover‘s Distance / 21
Optimization-Based EMD Adaptation 3 Optimization for EMD based on Feedback:
Solution: weights for m weighted Euclidean distances Initial solution: given by heuristic Moving: redistribute weights per Euclidean distance Objective function: avgPrec(EMDgd , q, Feedback)
Results for EMDgd on DB?
16
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Relevance Feedback for the Earth Mover‘s Distance / 21
Overview Introduction Adaptive Similarity Model Relevance Feedback for the Earth Mover’s Distance Experimental Evaluation Conclusion
17
Relevance Feedback for the Earth Mover‘s Distance / 21
Experimental Evaluation: Databases
18
72,000 images in ALOI DB ~60,000 images in COREL DB
Relevance Feedback for the Earth Mover‘s Distance / 21
Experimental Evaluation: ALOI
19
Heu
ristic
Ada
ptat
ion
Opt
imiz
atio
n-ba
sed
Que
ry-b
y-R
efin
emen
t
Relevance Feedback for the Earth Mover‘s Distance / 21
Experimental Evaluation: COREL
20
Heu
ristic
Ada
ptat
ion
Opt
imiz
atio
n-ba
sed
Que
ry-b
y-R
efin
emen
t
Relevance Feedback for the Earth Mover‘s Distance / 21
Experimental Evaluation
21
After 5 iterations of looking for doors in COREL:
(a) Query-by-Refinement (b) Heuristic (c) Optimization-Based
pos 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Relevance Feedback for the Earth Mover‘s Distance / 21
Conclusion Exploited adaptability of the EMD in RF framework Goal: Improve similarity search results Techniques:
Baseline: fixed ground distance Statistics-based heuristic adaptation Optimization-based adaptation
Evaluation: Experiments on two image datasets More relevant objects in fewer iterations
Techniques extensible to other adaptable distance functions
22
