View
46
Download
4
Category
Preview:
Citation preview
Exploiting Visual-based Intent Classification for Diverse Social Image Retrieval
Bo Wang1, Martha Larson1,2 Delft University of Technology, the Netherlands1
Radboud University, the Netherlands1,2
Query Ambiguity Topic Coverage Sub-topic Retrieval,IA-Select…
Redundancy Novelty
Maximal Marginal Relevance,
Varies visual-feature based unsupervised learning algorithms
P@20 CR@20 F1@20
Pearson’s coefficient 0.049 0.044 0.061
p-value 0.653 0.690 0.575
Pearson’s coefficient between query clarity score and Flickr Baseline
Broad Latent Aspects: 1. Broad latent aspects apply to a broad set of queries. 2. User queries frequently leave these aspects unspecified.
The sum of the choices made by photographers on exactly how to portray the subject matter that they have decided to photograph.
— Riegler et al.
A sailing boat within vast,
unending space.A sailing boat as an
object.
The characteristics of a group of
people on sailing boat
Other information source related to
sailing boat
Tag Based Search Engine based on YFCC100M
81 NUS-Wide concepts
Top 200 Documents
15618 Images
Examine in turn
Preliminary Intent Class
Exist?Yes
No
Introduce new intent class.
VGG Net Chop off classification layer
Softmax classifier with cross-entropy loss
on 15618 images
Intent class: Candid Probability: 73.60/%
Intent class: Social Event Public Probability: 89.36/%
71% Accuracy
Runs TF_IDF Reranking Feature Clustering
Visual (run1) FALSE CNN_Features K-means
Text_rerank + Text (run2) TRUE
Weighted Word Embedding Aggregation
K-means
Text_rerank + Visual (run3) TRUE CNN_Features K-means
Text_Rerank + Intent (run4) TRUE CNN_Feature Intent
Data Set Evaluation Visual (1) Text-rerank + text (2)
Text-rerank + visual (3)
Text-rerank + intent (4)
dev P@20 61.52% 67.72% 67.72% 67.69%
dev CR@20 49.29% 52.36% 53.61% 55.61%
dev F1@20 54.73% 59.05% 59.83% 61.07%
test P@20 66.01% 70.36% 70.71% 72.62%
test CR@20 56.98% 61.42% 58.09% 61.25%
test F1@20 58.30% 63.43% 61.21% 64.62%
Pros and Cons• Intent-based diversification has the advantage of better understandability. • Do not necessarily need to fine-tune the hyper parameters. • Faster than unsupervised approaches.
• Single annotator bring subjectivity of intent classes.
Conclusions
• We point out ambiguity and redundancy removal might not work. • Broad latent aspects might help. • Proposed intent-based approach. • Intent-based search result diversification is able to bring high performance
with several extra benefits.
• http://www.wangbo.info/pdf/intent.pdf • http://www.wangbo.info/ACMMM-MUSA-2017/
Recommended