Multi-View Reconstruction Preserving Weakly-Supported Surfaces

(CVPR 2011)M. Jancosek and T. Pajdla

Czech Technical University in Prague

Presenter : Jia-Hao Syu

04/21/23 1

Motivation

04/21/23 2

Outline

• Related Work[15]– System diagram

• Weakly-Supported Surfaces• Idea• Modified weights• Results• Conclusion

04/21/23 3

Related Work

• [15] P. Labatut, J. Pons and R.Keriven, “Robust and efficient surface reconstruction from range data”, In Computer Graphics Forum, 2009

• Target : Reconstruct a surface from a set of merged scans (noisy and outliers)04/21/23 4

System Diagram

3D cloud points for each cameras

Combine to one 3D cloud points

Delaunay tetrahedralization

of a cloud point

3D Delaunay Triangulation

Surface Reconstruction by graph-cut method

04/21/23 5

System Diagram

3D cloud points for each cameras

Combine to one 3D cloud points

Delaunay tetrahedralization

of a cloud point

3D Delaunay Triangulation

Surface Reconstruction by graph-cut method

04/21/23 6

3D Scanning Technique

• Contact• Non-Contact– Time-of-flight camera : a range imaging camera

system that resolves distance based on the known speed of light

D : distance c : speed of lightt : time for round-trip between A and B

04/21/23 7

System Diagram

3D cloud points for each cameras

Combine to one 3D cloud points

Delaunay tetrahedralization

of a cloud point

3D Delaunay Triangulation

Surface Reconstruction by graph-cut method

04/21/23 8

3D Cloud Points

• Acquire a depth map of each camera by 3D scanning technique

• Compute depth maps of a 3D cloud points to every camera by plane-sweeping method

04/21/23 9

• Pick one pixel P with depth d

Plane-sweeping Method

P

d

Reference Image

04/21/23 10

• Find the nearest n target cameras(ex. n = 4)

Plane-sweeping Method

P

d

Reference Image

04/21/23 11

• Compute photo consistency by normalized cross-correlation(NCC)

Plane-sweeping Method

P

d

Reference Image

04/21/23 12

Target Image

Plane-sweeping Method

• Normalized Cross-Correlation

04/21/23 13

The formulation is from wikin : the pixel numberf(x,y) : reference imaget(x,y) : target image

The value of NCC is between -1 and 1

One 3D Cloud Points

• Compute photo-consistence between reference and target image

• One 3D cloud points can be built

• Get all depth-maps of each camera by using a related camera matrix

04/21/23 14

System Diagram

3D cloud points for each cameras

Combine to one 3D cloud points

Delaunay tetrahedralization

of a cloud point

3D Delaunay Triangulation

Surface Reconstruction by graph-cut method

04/21/23 15

2D Delaunay Triangulation

• Three points can draw a triangle

• Add one more point

04/21/23 16

or

2D Delaunay Triangulation

• Draw a circumcircle of triangle

04/21/23 17

or

• Give a set of P point in 2D

2D Delaunay Triangulation

04/21/23 18

2D Delaunay Triangulation

• No point in P is inside the circumcircle of any triangle

3D Delaunay triangulation : no point in P is inside the circumsphere of any tetrahedralization

04/21/23 19

3D Delaunay triangulation

• Example for 3D Delaunay triangulation

04/21/23 20

System Diagram

3D cloud points for each cameras

Combine to one 3D cloud points

Delaunay tetrahedralization

of a cloud point

3D Delaunay Triangulation

Surface Reconstruction by graph-cut method

04/21/23 21

Surface Reconstruction

• We build the 3D Delaunay triangulation

• How do you reconstruct surface of the object?

• Concept : 3D cloud points are dense near the object surface (cost is small)

• S-t graph cut algorithm

04/21/23 22

04/21/23

• Source and sink become separated the node of set by a cut

• the cost of a cut :

• Minimum cut : a cut whose cost is the least over all cuts

S-t Graph Cut

23

Define Parameters

• Node : Delaunay tetrahedralization• Edge : triangulation between adjacent

tetrahedralizations • s(source) : outside of the surface• t(sink) : inside of the surface

04/21/23 24

P

S-t Graph Cut Algorithm

• Perform a Delaunay Triangulation of the 3d point cloud

04/21/23 25

S-t Graph Cut Algorithm

• Add a node P from left tetrahedralization

04/21/23 26

P

S-t Graph Cut Algorithm

• Add a node Q from right tetrahedralization

04/21/23 27

P Q

S-t Graph Cut Algorithm

• Add two s and t nodes

04/21/23 28

P Q

s

t

S-t Graph Cut Algorithm

04/21/23 29

P Q

s

t

10

100

0

3

S-t Graph Cut Algorithm

04/21/23 30

Outside the surface

inside the surface

This is the surface we want

Assigned Weight

• 3D cloud points to camera center(line of sight)

04/21/23 31

04/21/23 32

Formulation of Cost Function

: Visibility Information from points, cameras

: Quality of reconstructed surface in terms of size of triangles

04/21/23 33

Weakly Supported Surfaces

Not photo consistent surface : Low-textured walls, windows, cars and ground planes

Idea

• Other information to reconstruct weakly supported surface

• Visual Hull

04/21/23 35

Idea

• Define free-support-space

• Highly-supported-free space : union of dense 3D points

• Weakly-supported surface with weakly sampled by 3D points are close to the highly-supported-free space

04/21/23 36

Free-space-support

04/21/23 37

pi

pj

T

r

Original 3D cloud points

X : 3D cloud points(before photo-consistence)

Target

• Large Jump in Free Space Support as we go from outside to inside.

• Next, I give a example of weight assumption

Old T-weights

Modified Weights

Setting up t-weight

04/21/23 41

System and Spend Time

DataSet/Method Baseline[CFG 09](mins) Ours(mins)

Castle 30 32

Dragon 90 94

• System OS : 64-bit Win7 CPU : Inter Core i7 RAM : 12GB

• Dataset Castle data : 30 images with 3072*2048 resolution Dragon data : 114 images with 1936*1296 resolution

Results

INPUT IMAGE POINT CLOUD

CFG-09 OUR METHOD

Results

INPUT IMAGE POINT CLOUD

CGF-09 OUR METHOD

Demo Video

• Images :http://www.youtube.com/watch?

v=uwluzq5LUn0&feature=player_embedded• Demo videohttp://www.youtube.com/watch?v=UgkB7ITpNaE&feature=player_embedded

04/21/23 45

Conclusion

• Resolve weakly-supported surface by using the information of free-support-space

04/21/23 46

Reference

• M. Jancosek and T. Pajdla, “Multi-View Reconstruction Preserving Weakly-Supported Surface”, IEEE Conference on Computer Vision and Pattern Recognition, 2011

• P. Labatut, J. Pons and R.Keriven, “Robust and efficient surface reconstruction from range data”, In Computer Graphics Forum, 2009

• P. Labatut, J. Pons, and R. Keriven., “Efficient Multi-view Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts”, International Conference on Computer Vision, 2007

04/21/23 47

Reference

• M. Jancosek and T. Pajdla , ”Hallucination-free multi-view stereo.”, In RMLE , 2010

• M. Jancosek and T. Pajdla, “Removing hallucinations from 3D reconstructions”, Technical Report CMP CTU, 2011

04/21/23 48