C. Ventura X. Giró-i-Nieto V. Vilaplana F. Marqués K. McGuinness N. O’Connor
Improving Spatial Codification in Semantic Segmentation
Outline
● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions
Introduction● Object segmentation vs Class segmentation
Image Object segmentation
Class segmentation
3
Outline
● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions
Related Work● The Visual Extent of
an Object [1]
5[1] Uijlings et al, The VIsual Extent of an Object. IJCV’12
1st Contribution● Using a Figure-Border-Ground spatial pooling with object
candidates
6
Figure-Ground spatial pooling
Figure-Border-Ground spatial pooling
Related Work● Beyond bags of features:
Spatial pyramid matching for recognizing natural scene categories [1]
7[1] Lazebnik et al, Beyond bags of features: Spatial pyramid matching for recognizing natural scenes. CVPR’06
Related Work● Variations of SPM
○ Non-arbitrary division■ Object-centric pooling [1]■ Object confidence map
partition [2]○ SPM over bounding boxes [3]
[4]
[1] Russakovky et al, Object-centric spatial pooling for image classification. ECCV’12[2] Chen et al, Hierarchical Matching with Side Information for Image Classification. CVPR’12[3] Arbeláez et at, Semantic segmentation using regions and parts. CVPR’12[4] Gu et al, Multi-component models for object detection. ECCV’12 8
2nd Contribution● Applying a contour-based spatial pyramid (SP)
○ Crown-based SP○ Cartesian-based SP
9
Crown-basedspatial pyramid
Cartesian-basedspatial pyramid
Outline
● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions
Architecture● Architecture proposed and released in [1]
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
Train Test
DataBaseObject
CandidatesFeature
Extraction
Test
Model
Prediction
Evaluation
AAC
Ground Truth
Train
CPMC SIFT-based features (O2P)
11
Outline
● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions
Experiments● Experiments with ideal object
candidates○ Train set: train11○ Test set: val11
F[1] F-B F-G[1] F-B-G
eSIFT 63.9 66.2 66.4 68.6
eMSIFT 64.8 68.9 67.6 70.8
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 13
Experiments● Experiments with ideal object
candidates○ Train set: train11○ Test set: val11
F F-B F-B-G
non SP 64.8 [1] 68.9 70.8
crown-based SP 68.7 71.1 71.7
Cartesian-based SP 67.7 71.6 72.7
Figure SP(Figure) Border Ground AAC
eSIFT+eMSIFT+eLBP eSIFT 72.98 [1]
eSIFT+eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 73.84
eSIFT+eMSIFT+eLBP eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 75.86
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’1214
Experiments● Experiments with CPMC object candidates
○ Train set: train11○ Test set: val11
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
Figure SP(Figure) Border Ground AAC
eSIFT eSIFT 28.6 [1]
eSIFT eSIFT eSIFT 34.8
eSIFT+eMSIFT+eLBP eSIFT 37.2 [1]
eSIFT eSIFT eSIFT eSIFT 37.4
eSIFT+eMSIFT+eLBP eSIFT eSIFT eSIFT 39.6
15
Experiments● Experiments with CPMC object
candidates in comp5 challenge○ Train set: trainval11 /
trainval12○ Test set: test11 / test12
F-G [1] F-B-G SP(F)-B-G
PASCAL VOC11
38.8 43.8 40.3
PASCAL VOC12
39.9 42.2 40.8
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’1216
Experiments● Experiments with MCG object
candidates [1]○ Train set: train11○ Test set: val11
F-G[2] F-B-G SP(F)-B-G
CPMC 37.2 38.9 39.6
MCG 30.9 34.1 36.1
[1] Arbeláez et al, Multiscale Combinatorial Grouping. CVPR’14[2] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 17
Experiments● Qualitative results F-B-G spatial pooling with CPMC
18
F-G F-B-G F-G F-B-G
aeroplane
bicycle bicycle
birdcat
motorbike boat
bottle
bus bus
motorbike
car
chaircat
chair chair
horse
cow
bird
Experiments● Qualitative results F-B-G spatial pooling with CPMC
19
chair
dining table
cow dog
person
horse
person motorbike
motorbikemotorbike
person
plotted plant bottle
sheep
sofa dog
bus
train train
tvmonitor
F-G F-B-G F-G F-B-G
Outline
● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions
Conclusions● 2 proposals beyond the classic Figure-Ground pooling
○ Figure-Border-Ground spatial pooling■ Extended to realistic scenario with CPMC object
candidates○ A novel contour-based spatial pyramid has been introduced
■ Cartesian-based spatial pyramid■ Crown-based spatial pyramid
● Validation of both proposals also for MCG object candidates
21
Related Work● The Visual
Extent of an Object (Uijlings et al, IJCV’12)
23