Slide 1

1 Liyan Zhang et al. Liyan Zhang, Dmitri V. Kalashnikov, Sharad Mehrotra Department of Computer Science University of California, Irvine A Unified Framework for Context Assisted Face Clustering Slide 2 2 Liyan Zhang et al. Introduction Explosion of Media Data Human is Center Face Clustering Face Tagging User Feedback Slide 3 3 Liyan Zhang et al. Outline Introduction to Face Clustering Traditional Approaches for Face Clustering The Proposed Context Assisted Framework Experimental Results Conclusions and Future Work Slide 4 4 Liyan Zhang et al. Face Appearance based Approach Facial Features Face Similarity Graph Clustering Algorithm Detected Faces Clustering Results Slide 5 5 Liyan Zhang et al. Appearance based Face Clustering Results Good Clustering Results High Precision,High Recall Tight Clustering Threshold High Precision, Low Recall loose Clustering Threshold Low Precision, High Recall Too Much Merging Work! Slide 6 6 Liyan Zhang et al. Drawbacks of Facial Similarities Same People Look Different Different PoseDifferent Expression Different IlluminationDifferent Occlusion Different People Look The Same BoyGirlBoyGirl Slide 7 7 Liyan Zhang et al. Context Information Helps Common Scene: Geo Location Captured Time Image Background Social Context: People Co-occur Human Attributes: Age Ethnicity Gender Hair Clothing: Cloth color Slide 8 8 Liyan Zhang et al. Related Work [1] Y. J. Lee and K. Grauman. Face discovery with social context. In BMVC, 2011. [3] N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011. [2] A. Gallagher and T. Chen. Clothing cosegmentation for recognizing people. In IEEE CVPR, 2008. People Co-occurrence [1] Clothing [2] Human Attributes [3] Heterogeneous Context FeatureSingle Context Type Face LevelCluster Level Context Prior work Context Heterogeneous Single Slide 9 9 Liyan Zhang et al. The Framework Photo Collection Detected Faces Initial Clusters : High Precision, Low Recall Iterative Merging cont Common Scene People Co-occurrence Human Attributes Clothing Final Clusters: High Precision, High Recall Slide 10 10 Liyan Zhang et al. Context Features Extraction Cluster level Common ScenePeople Co-occurrenceHuman AttributesClothing Context Similarities Context Constraints Integrate Same? Diff ? Slide 11 11 Liyan Zhang et al. Common Scene Image captured time, camera model, image visual features Common Scene Same people I1I1 I2I2 I3I3 I1I1 I2I2 C1C1 C2C2 Slide 12 12 Liyan Zhang et al. People Co-occurrence same diff Slide 13 13 Liyan Zhang et al. People Co-occurrence Cluster Co-Occurrence Graph 1 1 1 1 1 1 I2I2 I1I1 1 1 1 1 1 f 1 f 2 f 3 f 4 f 5 f 6 f 7 f 8 Slide 14 14 Liyan Zhang et al. Human Attributes samediff 73-D N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011. Slide 15 15 Liyan Zhang et al. Human Attributes Attri bute C5 f 1 f 2 f 3 Attri bute Similar? C5 cosine Only One Child Many Children! AGE attribute Bootstrapping: Learn Weights From Dataset! Different Attributes Different Weights Slide 16 16 Liyan Zhang et al. Human Attributes FaceAttributesLabel C1C1 C1C1 C1C1 ~ C 1 Attri bute C5 f 1 f 2 f 3 Attri bute f 4 f 5 f 6 f 7 f 8 diff Attri bute Train Classifier C1 ? Slide 17 17 Liyan Zhang et al. Clothing Similarity from clothes Cloth color hist similarity Time diff Time slot threshold Time Sensitive! DiffSame Time diff > S Slide 18 18 Liyan Zhang et al. Context Features Cluster-Level Context Similarities Common ScenePeople Co-occurrence Human Attributes Clothing Slide 19 19 Liyan Zhang et al. Context Features Cluster-Level Context Constraints Diff Co-occurred people Diff Distinct Attributes Time & Location Time: t s Indoor Time: (t+1)s Outdoor Diff Slide 20 20 Liyan Zhang et al. Single Context Feature Fails Common Scene People Co-occurrence Diff Same Different Attributes Different Clothing Co-occurred People Diff Same Slide 21 21 Liyan Zhang et al. Integration is Required Aggregation? Set a rule? Context Similarities Context Constraints Integrate ? Context Features Merge Not Y=a +b +c +d +e + The importance of features differ with different dataset! Learn Rules from Each Dataset! Slide 22 22 Liyan Zhang et al. How to Learn Rules? Photo Collection Split Initial clusters Same Pairs Manually Label Learning Rules Apply Rules Training Dataset Initial Clusters : High Precision, Low Recall Facial Features cont Common Scene People Co-occurrence Human Attributes Clothing Context Similarity & Constraint Learn from data Itself! Apply rules Learning Rules Training Dataset Automatic Label Context Constraints Diff Pairs Bootstrapping Slide 23 23 Liyan Zhang et al. pairsLabel Same Diff Cost-sensitive DTC Splitting Training Diff Example of Automatic Labeling same diff same diff same Slide 24 24 Liyan Zhang et al. pairsLabel Same Diff Cost-sensitive DTC pairspredict Same Diff Same Diff ( ): 5 same ( ): 1 same Splitting Training Predicting Diff 1 st SplittingTraining--Predicting Slide 25 25 Liyan Zhang et al. pairsLabel Same Diff Cost-sensitive DTC pairspredict Same Diff ( ): 4 same ( ): 0 same Splitting Training Predicting Diff 2 nd SplittingTraining--Predicting Slide 26 26 Liyan Zhang et al. Combine results C1-C3: 5 same C2-C3: 1 same C1-C3: 4 same C2-C3: 0 same C1-C3: 9 same C2-C3: 1 same Merge C1-C3 pairspredict Same Diff Same Diff pairspredict Same Diff 1 st Time 2 nd Time Slide 27 27 Liyan Zhang et al. Unified Framework Pure clusters splittingtrainingprediction splittingtrainingprediction splittingtrainingprediction Final Decision Extracted Faces Merge Pairs? YES No Results Iterative Merging Photo Album Facial Context Features Slide 28 28 Liyan Zhang et al. Experiment Datasets Gallagher Wedding Surveillance Slide 29 29 Liyan Zhang et al. Evaluation Metrics B-cubed Precision and Recall Slide 30 30 Liyan Zhang et al. Performance Comparison Facial Features Photo Album Context Features Pure Clusters Splitting Training Predicting Process Merge Decision Update Our Approach: Precision Recall Picasa: Cluster Threshold 50 95 Different Clusters Affinity Propagation: Context Similarities Aggregation Facial Similarities Different Parameter: p Different Clusters Slide 31 31 Liyan Zhang et al. Results Slide 32 32 Liyan Zhang et al. Results High Precision Higher Recall Slide 33 33 Liyan Zhang et al. Results High Precision, 662 clusters 31 Real Person, 631 Merging High Precision, 203 clusters 31 Real Person, 172 Merging 4 Times Slide 34 34 Liyan Zhang et al. Results Less Clusters Less Manual Merging Slide 35 35 Liyan Zhang et al. Results Slide 36 36 Liyan Zhang et al. Conclusion and Future Work Heterogeneous Context Features Context Constraint Co-occur People Distinct Attributes Time & Space Context Similarity Common Scene People Co-occur Human Attributes Clothing Single Context Feature Context Similarity Prior work Our Approach Efficiency?User Feedback? Break points for precision dropping? Future work Bootstrapping Integration Iterative Merging High precision High recall Slide 37 37 Liyan Zhang et al. Thank you! Questions?