Embed Size (px)
Citation preview
Dense image matching and surface reconstruction when photogrammetry meets computational geometry
Florent Lafarge
Sophia Antipolis - France
Problem statement
June 2014
Given (at least) two (calibrated) images observing
the same scene, what type of 3D information can
we extract ?
1
Problem statement
June 2014
Given (at least) two (calibrated) images observing
the same scene, what type of 3D information can
we extract ?
1
=2 : Stereo Matching
>2 : Multi-View Stereo (MVS)
Problem statement
June 2014
Given (at least) two (calibrated) images observing
the same scene, what type of 3D information can
we extract ?
1
Intrinsics
(focal length, radial distortion..)
Extrinsics
(optical center, camera orentation)
Overview
June 2014
Stereo matching
Multi-View Stereo
Beyond free-form surface reconstruction
2
June 2014
1. Stereo matching
P
optical center
left camera
F
Principle
optical center
right camera
In the 3D world,
June 2014 3
A
P
optical center
left camera
F
Principle
optical center
right camera
In the 3D world,
June 2014 3
A
P
Principle
optical center
left camera
optical center
right camera
In the 3D world,
June 2014 3
A
P
optical center
left camera
optical center
right camera
In the 3D world,
Principle
June 2014 3
A
P
optical center
left camera
optical center
right camera
In the 3D world,
Principle
epipolar plane
June 2014 3
A
P
optical center
left camera
optical center
right camera
In the 3D world,
Principle
epipolar plane
epipolar lines
June 2014 3
A
?
optical center
left camera
optical center
right camera
In the 3D world,
Principle
P
June 2014 3
A
A
?
optical center
left camera
optical center
right camera
In the 3D world,
Principle
P
June 2014 3
A
?
optical center
left camera
optical center
right camera
In the 3D world,
Principle
matching
cost
≠
June 2014 3
A
?
optical center
left camera
optical center
right camera
In the 3D world,
Principle
≠
matching
cost
June 2014 3
A
?
optical center
left camera
optical center
right camera
In the 3D world,
Principle
≠
matching
cost
June 2014 3
A
optical center
left camera
optical center
right camera
In the 3D world,
Principle
matching
cost
June 2014 3
A
P
optical center
left camera
optical center
right camera
In the 3D world,
Principle
matching
cost
June 2014 3
Principle Z
June 2014 4
Principle Z
June 2014 4
Principle Z
June 2014 4
Depth map
Principle
left image
(reference) right image
Base B Z
depth z
focal
length f
June 2014 4
Principle Z
left image
(reference) right image
June 2014 4
Z
disparity d
Principle
left image
(reference) right image
June 2014 4
Z
disparity d
Base B
depth z
focal
length f
d
fBz
Principle
left image
(reference) right image
June 2014 4
June 2014
What matching cost ?
C( , ) 0 and C( , ) > 0
June 2014
What matching cost ?
C( , ) 0 and C( , ) > 0
Sum of Absolute Distances (SAD)
Sum of Square Differences (SSD)
Normalized Cross Correlation (NCC)
Mutual Information (MI)
… potentially object-based, eg line segments
[Hirschmuller and Scharstein, Evaluation of cost functions for stereo
matching, CVPR 2007]
June 2014
What matching constraints ?
Uniqueness
Ordering
Spatial coherence (or smoothness)
Sharp feature preservation
…
7
June 2014
What mechanism ?
Local
greedy local matching, eg pixel-per-pixel matching
Semi-global
spatial coherence along multiple 1D paths, eg SGM
Global
spatial coherence in 2D domain, eg MRF-based
8
June 2014
http://vision.middlebury.edu/stereo/
Middlebury benchmark
150 algorithms evaluated
9
June 2014
2. Multi-View Stereo
Principle
Stereo Matching • 2 input images
• 2.5D output representation
(depth maps)
June 2014 10
Principle
Stereo Matching • 2 input images
• 2.5D output representation
(depth maps)
Multi-View Stereo (MVS) • N input images
• 3D output representation
(eg meshes)
June 2014 10
Principle
Stereo Matching • 2 input images
• 2.5D output representation
(depth maps)
Multi-View Stereo (MVS) • N input images
• 3D output representation
(eg meshes)
How to reconstruct the 3D shape of
an object from N calibrated images?
June 2014 10
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
June 2014 11
Depth map fusion
A natural extension of Stereo Matching
time-consuming
occlusions & conflicts
sharp features
June 2014 11
Point cloud generation
Why not using 3D points instead of depth maps ?
June 2014 12
Point cloud generation
Why not using 3D points instead of depth maps ?
For each pair of images,
Keypoint extraction
(SIFT, Harris corners..)
June 2014 12
Point cloud generation
Why not using 3D points instead of depth maps ?
For each pair of images,
Keypoint extraction
(SIFT, Harris corners..) Keypoint descriptor
matching
June 2014 12
Point cloud generation
Why not using 3D points instead of depth maps ?
For each pair of images,
Keypoint extraction
(SIFT, Harris corners..) Keypoint descriptor
matching
June 2014 12
Point cloud generation
Why not using 3D points instead of depth maps ?
For each pair of images,
natural unit element in 3D
fast
capture well sharp features
Keypoint extraction
(SIFT, Harris corners..) Keypoint descriptor
matching
June 2014 12
Point cloud generation
At the scale of a facade
June 2014 13
Input images Keypoints detected in one image
3D point cloud
Point cloud generation
At the scale of a facade
June 2014 13
Input images Keypoints detected in one image
3D point cloud
Noise &
outliers !
Point cloud generation
At the scale of a city
June 2014 14
Surface reconstruction
From (defect-laden) point cloud to surface/volume representations
June 2014 15
Data-structure : mesh with triangular facets
with borders
watertight
Surface reconstruction
An old topic in computer graphics
June 2014 16
Surface reconstruction
An old topic in computer graphics
but without considering photo-consistency
[Berger et al, State of the art in surface reconstruction
from point clouds, Eurographics 2014]
June 2014 16
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
June 2014 17
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
1. Point cloud generation
2. Visibility consistent mesh extraction
3. Variationnal refinement with photo-consistency
June 2014 17
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
2. Visibility consistent mesh extraction
Partition the space with a 3D Delaunay triangulation
June 2014 18
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
2. Visibility consistent mesh extraction
Partition the space with a 3D Delaunay triangulation
June 2014 18
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
2. Visibility consistent mesh extraction
Partition the space with a 3D Delaunay triangulation
Label each Delaunay cell as inside or outside the observed
objects using visibility
+1 +1
+1 +1 -1
Before the vertex: +1 (outside)
After the vertex: -1 (inside)
June 2014 18
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
2. Visibility consistent mesh extraction
Partition the space with a 3D Delaunay triangulation
Label each Delaunay cell as inside or outside the observed
objects using visibility
out in out
out out
out
in
June 2014 18
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
2. Visibility consistent mesh extraction
Partition the space with a 3D Delaunay triangulation
Label each Delaunay cell as inside or outside the observed
objects using visibility
out in out
out out
out
in
June 2014 18
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
2. Visibility consistent mesh extraction
Partition the space with a 3D Delaunay triangulation
Label each Delaunay cell as inside or outside the observed
objects using visibility
June 2014 18
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
3. Variationnal refinement with photo-consistency
June 2014 19
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
3. Variationnal refinement with photo-consistency
June 2014 19
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
3. Variationnal refinement with photo-consistency
Minimize the reprojection error of image 1 into image 2
according to the surface S
June 2014 19
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
June 2014 20
Focus on [Vu et al, High accuracy and visibility-consistent
dense multiview Stereo, PAMI 2012]
Surface reconstruction
now a company
June 2014 21
June 2014
3. Beyond free-form surface
reconstruction
1
What can we do next ?
June 2014 22
What can we do next ?
June 2014
More compact and meaningful mesh
22
What can we do next ?
June 2014
More compact and meaningful mesh
High model complexity
11M facets
22
What can we do next ?
June 2014
More compact and meaningful mesh
Free of geometric structure
22
planar surface
What can we do next ?
June 2014
More compact and meaningful mesh
Free of geometric structure
Free of semantics
roof
facade
22
1st idea: Hybrid surfaces
June 2014
High model complexity
No structure
Free-form reconstruction
(Poisson algorithm)
23
June 2014
If you can detect the right
set of geometric primitives
23
1st idea: Hybrid surfaces
June 2014
and the right primitive
adjacencies
23
1st idea: Hybrid surfaces
June 2014
low model complexity
structure-aware
23
1st idea: Hybrid surfaces
June 2014
low model complexity
structure-aware
But No guarantee of finding the right primitives and right adjacency graph
23
1st idea: Hybrid surfaces
June 2014
low model complexity
structure-aware
But No guarantee of finding the right primitives and right adjacency graph
No guarantee that the object can be entirely explained by primitives
23
1st idea: Hybrid surfaces
June 2014
We need both structure and free-form!
Structure: primitive assembling for describing regular components
Free-form: smooth mesh for preserving the details
23
1st idea: Hybrid surfaces
Hybrid surfaces (by point set structuring)
June 2014 24
June 2014
Hybrid surfaces (by point set structuring)
24
June 2014
Hybrid surfaces (by point set structuring)
24
June 2014
Texte courant dunt am iriure commolut eumsandio odolor sectem
Hybrid surfaces (by point set structuring)
24
25
Hybrid surfaces (by point set structuring)
highly freeform (no structure)
high accuracy
high running time
Hybrid surfaces (by point set structuring)
25
structure ↑ (freeform ↓ )
accuracy ↓
running time ↓
Hybrid surfaces (by point set structuring)
25
fully structured
low accuracy
low running time
Hybrid surfaces (by point set structuring)
25
26
Hybrid surfaces (by point set structuring)
Reduction of the structure complexity while preserving details
June 2014
Hybrid surfaces (by shape sampling)
27
MVS images and a rough initial surface
June 2014
Hybrid surfaces (by shape sampling)
27
MVS images and a rough initial surface
Goal: refine the surface while sampling geometric primitives (planes,
cylinders…) by Jump-Diffusion
June 2014
Hybrid surfaces (by shape sampling)
27
MVS images and a rough initial surface
Goal: refine the surface while sampling geometric primitives (planes,
cylinders…) by Jump-Diffusion
June 2014
Hybrid surfaces (by shape sampling)
28
June 2014
Hybrid surfaces (by shape sampling)
29
Hybrid surfaces (by planimetric map)
June 2014 30
3D Primitive detection
Hybrid surfaces (by planimetric map)
June 2014 30
3D Primitive detection 2.5D planimetric map
Hybrid surfaces (by planimetric map)
June 2014 30
3D Primitive detection 2.5D planimetric map
hybrid surface
June 2014
3D-model
Aerial image (from Google Maps)
Hybrid surfaces (by planimetric map)
31
June 2014
Hybrid surfaces (by planimetric map)
32
2nd idea: 3D-Block assembling
June 2014
A building = an assemblage of
elementary urban 3D-models
33
2nd idea: 3D-Block assembling
June 2014
A building = an assemblage of
elementary urban 3D-models
33
Elevation map (DEM) Polygonalization of building
footprints by point process
2nd idea: 3D-Block assembling
June 2014
A building = an assemblage of
elementary urban 3D-models
33
Elevation map (DEM) Polygonalization of building
footprints by point process
2nd idea: 3D-Block assembling
June 2014
A building = an assemblage of
elementary urban 3D-models
33
Elevation map (DEM) Polygonalization of building
footprints by point process
Building reconstruction by
3D-block sampling
2nd idea: 3D-Block assembling
June 2014
A building = an assemblage of
elementary urban 3D-models
33
Elevation map (DEM) Polygonalization of building
footprints by point process
Building reconstruction by
3D-block sampling
June 2014
Pixel resolution: 0.70 m Pixel resolution: 0.10 m
34
3D-Block assembling
Urban 3D models with different Levels Of Details
June 2014 35
3rd idea: LOD generation
Urban 3D models with different Levels Of Details
3rd idea: LOD generation
June 2014 35
1. Semantical segmentation of dense mesh into 4 classes of interest
LOD generation
June 2014
Classification Abstraction Reconstruction
roof
facade
tree
ground
36
1. Semantical segmentation of dense mesh into 4 classes of interest
Geometric descriptors
LOD generation
June 2014 36
1. Semantical segmentation of dense mesh into 4 classes of interest
Geometric descriptors
Markov Random Fields with feature-preserving potential
LOD generation
June 2014 36
1. Semantical segmentation of dense mesh into 4 classes of interest
Geometric descriptors
Markov Random Fields with feature-preserving potential
LOD generation
June 2014 36
LOD generation
June 2014
2. Abstraction of urban objects
Planar hypotheses
globally consistent
Tree localization
37
LOD generation
June 2014
2. Abstraction of urban objects
Detection of planar primitives with global regularities
(parallelism, orthogonality, Z-symmetry, coplanarity)
37
LOD generation
June 2014
2. Abstraction of urban objects
Detection of planar primitives with global regularities
Iconization of trees and superstructures
37
3. 3D modeling at various LOD
LOD generation
June 2014
Classification Abstraction Reconstruction
38
LOD0
LOD1
LOD2
LOD1
3. 3D modeling at various LOD
3D arrangement of planar primitives filtered at a given LOD
LOD generation
June 2014
Classification Abstraction Reconstruction
38
3. 3D modeling at various LOD
3D arrangement of planar primitives filtered at a given LOD
Min-cut formulation with a discrete cost estimation
LOD generation
June 2014
Classification Abstraction Reconstruction
38
June 2014
Citer mes papiers avec une photo
LOD generation
39
June 2014
LOD generation
40
1 km² of Paris
input mesh: 11M facets
output: 175K facets (LOD2)
References
June 2014 41
Hybrid surfaces by point set structuring Surface reconstruction through point set structuring, Eurographics’13
Hybrid surfaces by shape sampling A hybrid multi-view stereo algorithm for modeling urban scenes, IEEE PAMI 2013
Hybrid surfaces by planimetric map Creating large-scale city models from 3D-point clouds: A robust approach with hybrid
representation, IJCV 2012
3D-block assembling Structural approach for building reconstruction from a single DSM, IEEE PAMI 2010
LOD generation Recently submitted..
June 2014
Thank you!
![Using Transactions in Delaunay Mesh Generation2. Delaunay Mesh Generation A Delaunay mesh is a mesh over a set of points which satisfies the Delaunay property [4]. This property,](https://img.pdfslide.net/doc/110x75/5e78132d55760c30656ba589/using-transactions-in-delaunay-mesh-generation-2-delaunay-mesh-generation-a-delaunay.jpg)
