Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography
KAIST Universidad de Zaragoza, I3A †
† † †§Giljoo Nam Joo Ho Lee Diego Gutierrez Min H. Kim
§
2/41
Input & Output
Unstructured flash photographs 3D geometry
3/41
Input & Output
Spatially-varying BRDFUnstructured flash photographs
4/41
Input & Output
Reproduction in a virtual environmentUnstructured flash photographs
5/41
Previous Work
[Holroyd et al., 2010] [Schwartz et al., 2013]
[Lensch et al., 2003] [Tunwattanapong et al., 2013]
6/41[Riviere et al., 2015]
[Ren et al., 2011]
Previous Work
[Aittala et al., 2015]
[Hui et al., 2017]
7/41
Challenges
• Limited sampling angles for BRDF acquisition
objectcamera &
light
How can we reconstruct a full BRDF from retro-reflected observations?
8/41
Challenges
• Simultaneous acquisition of SVBRDF and 3D geometry
objectcamera &
light
How can we ensure 3D-2D correspondence?
same point
9/41
BRDF Acquisition from Flash Photography
objectcamera &
light
n
o
df
h
hq
dq
i
( , ) ( , , )h d df f q q f=i o
• Parameter space for isotropic BRDFs
a 3D BRDF data
dq
hq
df
[Rusinkiewicz 98] [Matusik 02]
10/41
BRDF Acquisition from Flash Photography
a full 3D BRDF
dq
hq
df
subsample(flash photography)
hq
df
5dq » °object
camera &light
• Limited BRDF sampling angle in flash photography
11/41
• Cook-Torrance BRDF with 1D data-driven NDF
BRDF Model Validation
• Cook-Torrance BRDF with 1D data-driven NDF
( ) ( )( )( ) ( , , ) ( , ),4
dsD G Ff r r
p= +
× ×h n i o h ii on i n o
reconstructed BRDF
dq
hq
df
subsample(flash photography)
hq
df
5dq = °
a full 3D BRDF
dq
hq
dfC-T BRDFw/ 1D NDF
fit BRDF
[MERL BRDF dataset]
12/41
BRDF Model Validation
• Cook-Torrance BRDF with 1D data-driven NDF
( ) ( )( )( ) ( , , ) ( , ),4
dsD G Ff r r
p= +
× ×h n i o h ii on i n o
reference reconstructed difference0
1
13/41
SVBRDF Representation
• Basis BRDFs and spatial blending weights
object
( )3, ;f i o x
( )2, ;f i o x
( ) ( )1
, ; ,B
b bb
f fw=
=åi o x i o
( )1, ;f i o x
( )( ), ; :SVBRDF
, : basis BRDF
: blending weight: # of basis BRDFs
b
f
f
Bw
i o x
i o
14/41
Problem Definition
, , ,minimize
bX N W F
• Objective function using photometric consistency
-2
photographs rendering
X: 3D verticesN: surface normals
W: blending weightsFb: basis BRDFs
15/41
Problem Definition
( )2, , , 1 1
minimize ( ; ) ( , ; ) ( ; )( )P K
p kL f L
= =
- - ×ååbX N W F
o x i o x i x n i
• Objective function using photometric consistency
for all images & for all 3D vertices
captured radiance reconstructed radiance
X: 3D verticesN: surface normals
W: blending weightsFb: basis BRDFs
16/41
Overview
• Iterative & alternating optimization of {X, N, W, Fb}
initialize update W update Fb update N update X
update SVBRDF
X: 3D verticesN: surface normals
W: blending weightsFb: basis BRDFs
17/41
Data Capture
• Any camera with a flash will work
• We test two types of cameras
– Mobile phone with built-in LED
– DSLR with built-in flash
• 100 - 200 images (still/video)
example images
18/41
Initialization
• Initial geometry from image-based 3D modeling COLMAP [Schönberger 2016]: SfM è MVS è meshing
point cloud high-res mesh low-res mesh(our initial geo.)
point cloud high-res mesh low-res mesh
19/41
Initialization
• Initial geometry from image-based 3D modeling COLMAP [Schönberger 2016]: SfM è MVS è meshing
point cloud high-res mesh low-res mesh(our initial geo.)
initial geo.point clouds
20/41
Initialization
• Initial geometry from image-based 3D modeling COLMAP [Schönberger 2016]: SfM è MVS è meshing
point cloud high-res mesh low-res mesh(our initial geo.)
point cloud high-res mesh low-res mesh
21/41
Initialization
• Initial geometry from image-based 3D modeling COLMAP [Schönberger 2016]: SfM è MVS è meshing
point cloud high-res mesh our result point cloud high-res mesh our result
22/41
Overview
• Iterative & alternating optimization of {X, N, W, Fb}
X: 3D verticesN: surface normals
W: blending weightsFb: basis BRDFs
initialize update W update Fb update N update X
update SVBRDF
23/41
2
, , , ,, 1 1 1
minimizeP K B
p k p k p k p b bp k b
v f w= = =
æ ö¢ -ç ÷è ø
åå åbW F
Φ f
Update SVBRDF
captured reflectance reconstructed reflectance
• Re-formulate the equation w.r.t. the reflectance
( )2, , , 1 1
minimize ( ; ) ( , ; ) ( ; )( )P K
p kL f L
= =
- - ×ååbX N W F
o x i o x i x n i
update W update Fb update N update X
update SVBRDF
W: blending weightsFb: basis BRDFs
24/41
• Per-vertex optimization
Update SVBRDF: Blending Weights
2,
1
1minimize s.t. 0, 12
B
b p bb
w w=
- > =åωQω r
update W update Fb update N update X
update SVBRDF
é ù é ùé ùê ú ê úê úê ú ê ú=ê úê ú ê úê úê ú ê úë û
ë û ë û
Q ω r
geometric factors& basis BRDFs
weights(unknowns)
observations
25/41
Update SVBRDF: Basis BRDFs
• Optimize for all observations (NOT per-vertex)
• Hold W fixed and solve for Fb• Sparse quadratic programming
2
, , , ,1 1 1
minimizeP K B
p k p k p k p b bp k b
v f w= = =
æ ö¢ -ç ÷è ø
åå åbF
Φ f
observation basis BRDFs(unknowns)
estimated weights
update W update Fb update N update Xupdate SVBRDF
26/41
Update Normals
• Per-vertex optimization
• Non-linear optimization è hard to solve!
( )21
minimize ( ; ) ( , ; , ) ( ; )( )K
kL f L
=
- - ×ån
o x i o x n i x n i
normal for BRDF normal for irradiance
update W update Fb update N update X
update SVBRDF
K: # of images
27/41
Update Normals
• Per-vertex optimization
• Linear regression è easy to solve!
geometric normal( , known)
shading normal( , unknown)
update W update Fb update N update X
update SVBRDF
( )21
minimize ( ; ) ( , ; , ) ( ; )( )K
kL f L
=
- - ×ån
o x i o x n i x n i!
K: # of images
n!n
( )21
minimize ( ; ) ( , ; , ) ( ; )( )K
kL f L
=
- - ×ån
o x i o x n i inx !
28/41
Update Geometry
• Screened Poisson surface reconstruction [Kazhdan 2015]
2 2minimize ( ) ( ) ( )p
p p p pdc
c a cÎ
-Ñ + åòx X
V x x x x
3D vector field fromshading normals
gradient of new geometry
squared distance ofcurrent and new geometry
: new geometry (unknown)c
update W update Fb update N
update SVBRDF
update X
29/41
Update Geometry
• Screened Poisson surface reconstruction [Kazhdan 2015]
shading normalinput
geometryestimated
shading normals Nupdate
geometry X
mesh vertex
mesh edge
GT surface
update W update Fb update N
update SVBRDF
update X
30/41
Iterative Optimization
• Iterate the whole process until RMSE of the test set starts to increase
geometric difference
0.00
0.01
0.02
0.03
1 3 5 7
Hau
sdor
ff di
stan
ce
iteration
photo consistency error
0.087
0.095
0.103
0.111
1 3 5 7
RM
SE
iteration
train test
update W update Fb update N update Xupdate SVBRDF
31/41
Results
32/41
Results
33/41
Results
34/41
Comparison with Previous Work
reference(NextEngine)
Xia SIGA'16(passive env. lighting)
Ours(active flashlight)
35/41
Evaluation: Geometry Refinement
high low ours
0.10
63
0.09
17
0.05
93
avg. geo. diff. [mm]
reference (NextEngine)
COLMAP (low res.)
oursCOLMAP (high res.)
36/41
Evaluation: Geometry Refinement
reference (NextEngine)
COLMAP (low res.)
oursCOLMAP (high res.)
high low ours
0.10
63
0.09
17
0.05
93
avg. geo. diff. [mm]
37/41
Discussion: DLSR vs. Mobile Phone
• DLSRs
– Higher sensor signal-to-noise ratio
– Brighter flashlight intensity
– Does not need a darkroom
experiment setup flash + indoor lightflash in a darkroom
SNR: 85.22 dB
38/41
Discussion: DLSR vs. Mobile Phone
• Mobile phones
– Better portability
– Record a video with the LED light on
39/41
Limitation
• Geometry– Complex geometries cannot be reconstructed accurately from
image-based 3D modeling.• Pinecones, hair strands, etc.
• Material Appearance– Inter-reflections
– Subsurface scattering– Transparency
40/41
Conclusion
• Capture images using commercial cameras with flash lights
• You will get high-quality 3D geometry & SVBRDF!
41/41