View
216
Download
0
Category
Tags:
Preview:
Citation preview
...
...
(c)Distance matrix between blocks
(d) Integer-value constrained flow matrix between blocks
...
(a) Query image (b) Near duplicate image
...
Problem Statement
A pair of images or videos in which one is close to the exact duplicate of the other, but different in conditions related to capture, edits, and rendering.
......
Problem: Spatial shift and scale variations
Tasks and Applications
• Near Duplicate Retrieval (NDR): – Copyright infringement detection – query-by-example application
• Near Duplicate Detection (NDD)– Link news stories and group them into threads – Filter out the redundant near duplicate images or videos in the
top results from text keywords based web search
Prior Work
• Attributed Relational Graph (ARG) matching: ACM Multimedia 2004
• Point set matching: ACM Multimedia 2004
• One-to-one symmetric matching algorithm: T-MM 2007 and ACM Multimedia 2007
• Large-scale near duplicate detection: CIVR 2007
Prior Work: Spatial pyramid match kernel
First quantize descriptors (SIFT) into words, then do one pyramid match per word in image coordinate space.
Lazebnik, Schmid & Ponce, CVPR 2006
•Fusion of information from different levels.
•Alignment of different subclips (Level-1 as an example)
EMD DistanceMatrix between
Sub-clips
Integer-valueAlignment
Smoke Fire
Smoke
Level-0 Level-0
Level-1
Level-1
Level-1
Level-1
•Temporally Constrained Hierarchical Agglomerative Clustering
Fire
Temporal Pyramid Matching for Event Recognition in News Video
Level-2
Level-2
Level-2
Level-2
D. Xu & S.-F. Chang, CVPR 2007 and T-PAMI 2008
Earth Mover’s Distance (EMD)
dij
Supplier P is with a given amount of goods
Receiver Q is with a given limited capacity
Weights: Solved by linear programming
1/m1/2m1/2
m
...
...
(c)Distance matrix between blocks
(d) Integer-value constrained flow matrix between blocks
...
(a) Query image (b) Near duplicate image
...
Spatially Aligned Pyramid Matching
Non-overlapped and overlapped partition at multiple-levels:
Divide images into non-overlapped blocksDivide images into overlapped blocks with size equaling of the original image (in width and height) sampled at a fixed interval, say 1/8 of the image width and height.
4l 1
2l
First Stage Matching
...
...
(c)Distance matrix between blocks
(d) Integer-value constrained flow matrix between blocks
...
(a) Query image (b) Near duplicate image
...Objective: Compute the pairwise distances between any two blocks and .
Solution: We represent each block as a bag of orderless SIFT descriptors and use EDM distance to measure the similarity between two sets of descriptors of unequal cardinality.
rx cy
Jianguo Zhang et al. Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV, 2007
Second Stage Matching (1)
Objective: Align the blocks from one query image x to corresponding blocks in its near duplicate image y.
SAPM (our work): One block may be matched to another block at a different position and/or scale level to robustly handle piecewise spatial translations and scale variations.SPM: fixed block-to-block matching
Second Stage Matching (2)
Eq (3) can be reestablished from Eq (4): (Assume R<C)
1) adding C − R virtual blocks in image x, 2) setting , for all r satisfyingR < r ≤ C.
0rcD
Solution: Integer-flow EMD
Comparison of SAPM, SPM and TPM
Three blocks in the query images (i.e.,(a)) and their matched counterparts in nearduplicate images (i.e.,(b), (c), (d)) are highlighted and associated by the same color outlines.
Third Stage Matching
• Extension to Near Duplicate Video Identification:
One video clip V1 comprises {x(1), x(2), …, x(M)}, where x(i) is the i-th
frame and M is the total number of frames of V1; Another video clip V2
comprises {y(1), y(2),…, y(N)}, where N and y(j) are similarly defined.
Solution: temporal matching with EMD again.
Discussion
• If the query image was divided into non-overlapped blocks (e.g., L2-N) and the corresponding database images were divided into overlapped blocks (e.g. L2-O) at the same level, spatial shifts and some degree of scale change are addressed (e.g., )
• a broad range of scale variations is considered by matching the query image and the database images at different levels (e.g., )
• Ideally, SAPM can deal with any variations from spatial shift and scale variation by using more denser scales and spatial spacings.
2 2( )L N L OS x y
1 2( )L N L OS x y
Near Duplicate Retrieval and Detection
NDR: We directly fuse the distances from different levels :
NDD: Generalized Neighborhood Component Analysis (GNCA)
We use p = {0, 1, 2, 3, 4} to indicate partitions designated as level-0 non-Overlapped (L0-N), level-1 non-overlapped (L1-N), level-1 overlapped (L1-O), level-2 non-overlapped (L2-N), and level-2 overlapped (L2-O).
Experiments: Three Datasets
• Columbia Near Duplicate Image Database: TRECVID 2003 corpus
• New Image Dataset: TRECVID 2005 and 2006 corpus– 150 near duplicate pairs (300 images) and 300 non-duplicate
images
• New Video Dataset: TRECVID 2005 and 2006 corpus– 50 near duplicate pairs (100 videos) and 200 non-duplicate
videos
The images are collected from real broadcast news (rather than
edits of the same image by the authors).
Comparison of SAPM with SPM and TPM forImage NDR
Columbia Database
New Image Dataset
SAPM and GNCA for Image NDD
Performance Measure: Equal Error Rate (EER)
SAPM+NCA and SAPM+GNCA: 20 positive and 80 negative samples to train the projection matrices in NCA and GNCA, another 40 positive and 160 negative samples were used for SVM training. SPM, TPM and SAPM: all training samples (60 positive and 240 negative) were used for SVM training.
Test samples: 90 (positive) and 4840 (negative).
Comparison of SAPM+TM, SPM+TM and TPM+TM for Video NDR
1: Single-level L0-N->L0-N; 2: Single-level L1-N ->L1-N (or L1-O); 3:Multi-level.
Two weighting schemes in temporal matching: normalized weight (NW) and unit weight (UW)
Conclusion
• A multi-level spatial matching framework for image and video near duplicate identification.
• GNCA outperforms NCA for near duplicate detection.
Recommended