View
219
Download
0
Category
Tags:
Preview:
Citation preview
Global and Efficient Self-Similarity for Object Classification and Detection
CVPR 2010
Thomas Deselaers and Vittorio Ferrari
Conventional Image DescriptorsMeasure direct image properties
gradients
colors
2
Self-Similarity vs Conventional Descriptors
[Shechtman, Irani CVPR 07]
Assumption of conventional image descriptors• There is a direct visual property shared by images of objects of the same class
(e.g. colors, gradients, …).• This property can be used to compare images.
Self-similarity:• Indirect property: geometric layout of repeating patches within an image• More general property
3
Local Self-Similarity Descriptors
4[Shechtman, Irani CVPR 07]
Using Local Self-Similarity Descriptors
Applications: object recognition, image retrieval, action recognition• Ensemble matching [Shechtman CVPR 07]• Nearest neighbor matching [Boiman CVPR 08]• Bag of local self-similarities
[Gehler ICCV09, Vedaldi ICCV09, Hörster ACMM08, Lampert CVPR09, Chatfield ICCV09 WS]
1. Compute LSS descriptors for an image2. Assign the LSS descriptors to a codebook3. Represent the image as a histogram of LSS descriptors
5
Self-Similarity goes Global
Capture long-range self-similarities and their spatial arrangement
6
Self-Similarity goes Global
Capture long-range self-similarities and their spatial arrangement
7
compute self-similarity between all pairs of
pixels
compute self-similarity between all pairs of
pixels
Global Self-Similarity Tensor
4D self-similarity tensor
Note: local self-similarities included8
Problems with the GSS Tensor
• Computation time:• Memory requirement:
Aim: Reduce both9
1111
∼ 80GB ∼ 20h
Outline• Efficient global self-similarity tensor
• Global self-similarity descriptors– Bag of correlation surfaces– Self-similarity hypercubes
• Detection with self-similarity hypercubes– Efficient sliding window– Efficient subwindow search
• Experiments– Global self-similarity better than local self-similarity– Complementary to conventional descriptors– Object detection possible
10
Efficient Global Self-Similarity TensorFind an efficient approximation to
Quantize patches according to codebook
If two patches are assigned to the same prototype, they are similar
Reduces runtime to speedup: 11750
Efficient Global Self-Similarity
Two patches are only similar if they are assigned to the same prototype
Reduces memory to reduction: 12
Patch Prototype CodebooksRemember: Self-similarity encodes image content indirectly
Image-specific codebooks can be smaller than conventional ones
see paper for more generic codebooks and extensive evaluation 13
• Self-similarity hypercubes: now
• Bag of correlation surfaces: only in the paper
Global Self-Similarity DescriptorsSo far:• Compact GSS computed efficiently
Now: • Descriptors that can be used in machine learning classifiers• Fixed dimensionality• Compact representation
14
Self-Similarity Hybercubes
SSH of size
15
SSHs for Detection• Computing SSH naïvely requires operations
• Sliding windows has to evaluate many windows
16
operations
Efficient Computation of SSHs
Compute integral self-similarity tensor:
operations to compute SSHfor an image window
17
∼5000x speedup
160000
can be obtained using 16 lookups in
Efficient Subwindow Search for SSH
18
• Derive an upper bound on the score of a set of windows
• Section 5.2 in our paper
• Similar to [Lampert PAMI09]
Experiments: Object classificationPASCAL 07 objects– 9608 cropped images of objects from PASCAL 07 – 20 classes
Task: Classify each test image into one of 20 classesModel: Linear SVMTrain: train+val Test: test
19
Classification on the PASCAL 07 objects set
+ GSS outperform LSS+ Self-Similarity is truly complementary to conventional descriptors
20
clas
sific
ation
acc
urac
y [%
]
Experiments: Object detectionETHZ Shape Classes– 255 images– 5 classes (apple logos, bottles, giraffes, mugs, swans)
Task: Detect objects in imagesDetector: Linear SVM, sliding windows
21
e.g. [Ferrari CVPR07, Maji CVPR09]
Detection Results
+ SSH outperforms BOLSS+ it is possible to use GSS for detection with good results
22
BoLSS SSH
apple logos 10.0 80.0
bottles 10.7 96.4
giraffes 23.4 85.1
mugs 6.5 67.7
swans 17.6 70.6
Average 13.6 80.0
DR at FPPI 0.4
apple logos
bottlesgiraffes
mugs
swans
}
} SSH
BoLSS
FPPI 0.4
Comparison results (avg):[Ferrari CVPR07]: 71.9[Maji CVPR09]: 93.2
… many more
DR
at 0
.5 P
ASCA
L ov
erla
p
Runtimes for Computing Descriptors• 200x200 image• GSS tensor – directly: 5512s ( 1.5 hours)∼– using our method: 81s ( 1.5 minutes)∼
• Computing descriptors: few seconds• Our method: 70x speedup
• For Reference:– GIST: 0.4s– BOLSS: 0.7s
23
Runtimes for Detection
Given the prototype assignment map (80s) (once only)
SSH sliding window: 30s/img (once per class)
For Comparison– Computing direct GSS tensor for 25000 windows: 4 years/img
Speedup: ∼1 million ⇒ Using our methods, GSS can be used for object detection
For Reference:– Felzenszwalb PAMI 09: 5s.
24
June 2014
Global and Efficient Self-Similarity for Object Classification and Detection
CVPR 2010
Thomas Deselaers and Vittorio Ferrari
Feasible
Conclusion• self-similarity should be considered globally– Global self-similarity performs better than local self-similarity
• truly complementary to conventional descriptors
• global self-similarity is feasible – efficient computation of self-similarity– two descriptors based on self-similarity
• global self-similarity for detection
• code will be available soon
26
Thank you for your attention
Thank you for your attention
Recommended