Joint Optimisation for Object Class Segmentation and Dense
Stereo Reconstruction ubor Ladick, Paul Sturgess, Christopher
Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin,
Philip H.S. Torr Oxford Brookes University
http://cms.brookes.ac.uk/research/visiongroup/ Slide 2 Black Box
Solver Left Camera Image Right Camera Image Object Class
Segmentation Dense Stereo Reconstruction Joint Object Class
Segmentation and Dense Stereo Reconstruction Slide 3 Black Box
Solver Left Camera Image Right Camera Image Object Class
Segmentation Dense Stereo Reconstruction Joint Object Class
Segmentation and Dense Stereo Reconstruction Objective : Joint
Estimation Slide 4 Dense Stereo Reconstruction Left Camera
ImageRight Camera ImageDense Stereo Result For each pixel assigns a
disparity label : y Disparities from the discrete set {0, 1,.. D}
Slide 5 Dense Stereo Reconstruction Unary Potential Unary Cost
dependent on the similarity of patches, e.g.cross correlation
Disparity = 0 Slide 6 Dense Stereo Reconstruction Unary Potential
Unary Cost dependent on the similarity of patches, e.g.cross
correlation Disparity = 5 Slide 7 Dense Stereo Reconstruction Unary
Cost dependent on the similarity of patches, e.g.cross correlation
Unary Potential Disparity = 10 Slide 8 Dense Stereo Reconstruction
Unary Cost dependent on the similarity of patches, e.g.cross
correlation Unary Potential Disparity = 15 Slide 9 Dense Stereo
Reconstruction Pairwise Potential Encourages label consistency in
adjacent pixels Cost based on the distance of labels Linear
TruncatedQuadratic Truncated Slide 10 Dense Stereo Reconstruction
Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler
et al. CVPR09) Original ImageInitial Solution Slide 11 Dense Stereo
Reconstruction Graph-Cut based Range-move inference (Kumar et al.
NIPS09, Veksler et al. CVPR09) Original ImageInitial SolutionAfter
1 st expansion Final solution Slide 12 Dense Stereo Reconstruction
Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler
et al. CVPR09) Original ImageInitial Solution After 2 nd expansion
After 1 st expansion Slide 13 Dense Stereo Reconstruction Graph-Cut
based Range-move inference (Kumar et al. NIPS09, Veksler et al.
CVPR09) Original ImageInitial Solution After 2 nd expansion After 1
st expansion After 3 rd expansion Slide 14 Dense Stereo
Reconstruction Graph-Cut based Range-move inference (Kumar et al.
NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After
2 nd expansion After 1 st expansion After 3 rd expansionFinal
solution Slide 15 Dense Stereo Reconstruction Does not work for
Road Scenes ! Original Image Dense Stereo Reconstruction Slide 16
Does not work for Road Scenes ! Patches can be matched to any other
patch for flat surfices Different brightness in cameras Slide 17
Dense Stereo Reconstruction Does not work for Road Scenes ! Patches
can be matched to any other patch for flat surfices Different
brightness in cameras Could object recognition for road scenes
help? Recognition of road scenes is relatively easy (Sturgess et
al., BMVC09) Slide 18 Object Class Segmentation Aims to assign a
class label for each pixel of an image Classifier trained on the
training set Evaluated on never seen test images Slide 19 Object
Class Segmentation Unary Potential Likelihood of a pixel taking a
label (Shotton et al. ECCV06, He et al, CVPR04, Ladick et al. ICCV
09) Slide 20 Object Class Segmentation Pairwise Potential Contrast
sensitive Potts model Encourages label consistency in adjacent
pixels Slide 21 Object Class Segmentation Higher Order Potential
Encouraging consistency in superpixels (Kohli et al. CVPR08)
Merging information at different scales (Ladick et al. ICCV09)
Slide 22 Object Class Segmentation Graph-Cut based -Expansion
inference (Boykov et al. ICCV99) Original ImageInitial solution
grass Slide 23 Object Class Segmentation Graph-Cut based -Expansion
inference (Boykov et al. ICCV99) Original ImageInitial
solutionBuilding expansion grass building Slide 24 Object Class
Segmentation Graph-Cut based -Expansion inference (Boykov et al.
ICCV99) Original ImageInitial solutionBuilding expansion Sky
expansion grass building sky Slide 25 Object Class Segmentation
Graph-Cut based -Expansion inference (Boykov et al. ICCV99)
Original ImageInitial solutionBuilding expansion Sky expansion Tree
expansion grass building sky tree Slide 26 Object Class
Segmentation Graph-Cut based -Expansion inference (Boykov et al.
ICCV99) Original ImageInitial solutionBuilding expansion Sky
expansion Tree expansionFinal Solution grass building sky tree
aeroplane Slide 27 Object Class Segmentation vs. Dense Stereo
Reconstruction sky Object class and 3D location are mutually
informative Sky always in infinity (disparity = 0) Slide 28 Object
Class Segmentation vs. Dense Stereo Reconstruction buildingcarsky
Object class and 3D location are mutually informative Sky always in
infinity (disparity = 0) Cars, buildings & pedestrians have
their typical height Slide 29 Object Class Segmentation vs. Dense
Stereo Reconstruction roadbuildingcarsky Object class and 3D
location are mutually informative Sky always in infinity (disparity
= 0) Cars, buses & pedestrians have their typical height Road
and pavement on the ground plane Slide 30 Object Class Segmentation
vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and
3D location are mutually informative Sky always in infinity
(disparity = 0) Cars, buses & pedestrians have their typical
height Road and pavement on the ground plane Buildings and pavement
on the sides Slide 31 Object Class Segmentation vs. Dense Stereo
Reconstruction roadbuildingcarsky Object class and 3D location are
mutually informative Sky always in infinity (disparity = 0) Cars,
buses & pedestrians have their typical height Road and pavement
on the ground plane Buildings and pavement on the sides Both
problems formulated as CRF Joint approach possible? Slide 32 Joint
Formulation Each pixels takes label z i = [ x i y i ] L 1 L 2
Dependency of x i and y i encoded as a unary and pairwise
potential, e.g. strong correlation between x = road, y = near
ground plane strong correlation between x = sky, y = 0 Correlation
of edge in object class and disparity domain Slide 33 Joint
formulation Weighted sum of object class, depth and joint potential
Joint unary potential based on histograms of height Unary Potential
Object layer Disparity layer Joint unary links Slide 34 Joint
Formulation Object class and depth edges correlated Transitions in
depth occur often at the object boundaries Pairwise Potential
Object layer Disparity layer Joint pairwise links Slide 35 Joint
Formulation Slide 36 Standard -expansion Each node in each
expansion move keeps its old label or takes a new label [x L1, y L2
], Possible in case of metric pairwise potentials Inference Slide
37 Standard -expansion Each node in each expansion move keeps its
old label or takes a new label [x L1, y L2 ], Possible in case of
metric pairwise potentials Inference Too many moves! ( |L1| |L2| )
Impractical ! Slide 38 Projected move for product label space One /
Some of the label components remain(s) constant after the move Set
of projected moves -expansion in the object class projection
Range-expansion in the depth projection Inference Slide 39 Leuven
Road Scene dataset Contained 3 sequences 643 pairs of images We
labelled 50 training + 20 test images Object class (7 labels)
Disparity (100 labels) Available on our website
http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip
Dataset Left cameraRight cameraObject GTDisparity GT Slide 40
Qualitative results Large improvement for dense stereo estimation
Minor improvement in object class segmentation Original ImageObject
GTObject ResultDisparity GTDisparity AloneDisparity Jointly Slide
41 Quantitative disparity results Dependency of the ratio of
correctly labelled pixels within the maximum allowed error delta
Slide 42 Application to monocular sequences Making method (close
to) real time Application to multi-view problems Optical flow /
motion estimation On-going and Future Work Slide 43 First dataset
with both object class and disparity labels Joint estimation
improves significantly disparity results Projected moves make
inference much faster Questions ? Summary
