Variational Methods - in 3D reconstruction and light field analysis

Variational Methods - in 3D reconstruction and light field analysisBastian Goldlucke
1 ∫
1 Introduction
5 Summary
1 Introduction
5 Summary
∫ x 0 Bastian Goldlucke
Image labeling problems
Segmentation and Classification
3D Reconstruction
• Unknown object: Vector-valued function
• Problem solution: minimizer of an energy functional
argmin u∈V
6 ∫
• Unknown object: Vector-valued function
• Problem solution: minimizer of an energy functional
argmin u∈V
6 ∫
0 Seminar Oxford 2012 Variational Methods
Rockafellar 1993
“The great watershed in optimization is not between linearity and nonlinearity, but convexity and nonconvexity.”
non-convex energy convex energy 7 ∫
Introduction ∫ x 0 Convex vs. non-convex methods
Best of both worlds?
• Modeling with realistic non-convex energy E
• Relaxation to convex lower bound R • Optimality bound ε to guarantee quality of solution
8 ∫
E
R
• Modeling with realistic non-convex energy E • Relaxation to convex lower bound R
• Optimality bound ε to guarantee quality of solution
8 ∫
E
R
ε {
• Modeling with realistic non-convex energy E • Relaxation to convex lower bound R • Optimality bound ε to guarantee quality of solution
8 ∫
u = 1u = 0
F (u) =
9 ∫
u = 1u = 0
F (u) =
9 ∫
u = 1u = 0 argmin
• Space of binary functions u : → {0,1} not convex
• globally optimal solution by relaxation to u : → [0,1] and subsequent thresholding.
Chan, Esedoglu and Nikolova 2006
10 ∫
u = 1u = 0 argmin
cu dx }
• Space of binary functions u : → {0,1} not convex • globally optimal solution by relaxation to u : → [0,1] and
subsequent thresholding.
10 ∫
1 Introduction
5 Summary
The Multilabel Problem ∫ x
Find a labeling g : → Γ = {γ1, ..., γN} which minimizes total assignment costs∫

cg(x)(x) dx ,
defined by arbitrary local costs cγ(x), under certain assumtions on regularity of the solution.
Label γ1
Label γ2
Label γ3
Label γ4
Σ
γ2
γ1
times the length of the interface.
In this example d(γ1, γ2) · L(Σ)
Euclidean representation of the label distance: • Each label γ is represented by a point aγ ∈ Rk . • Label distance d(γ, µ) = |aγ − aµ|2 .
13 ∫
Σ
γ2
γ1
times the length of the interface.
In this example d(γ1, γ2) · L(Σ)
Euclidean representation of the label distance: • Each label γ is represented by a point aγ ∈ Rk . • Label distance d(γ, µ) = |aγ − aµ|2 .
13 ∫
Important special cases ∫ x
aγ = γ ∈ R
Ordered Labels • Example: depth reconstruction • Can be solved globally with functional lifting [Pock,
Schonemann, Graber, Bischof, Cremers ’08] • Continuous version of [Ishikawa ’03]
aγ = eγ ∈ RN
Potts model • Example: segmentation • No globally optimal solution possible if N > 2 • Continuous version of [Potts ’52]
14 ∫
Important special cases ∫ x
aγ = γ ∈ R
Ordered Labels • Example: depth reconstruction • Can be solved globally with functional lifting [Pock,
Schonemann, Graber, Bischof, Cremers ’08] • Continuous version of [Ishikawa ’03]
aγ = eγ ∈ RN
Potts model • Example: segmentation • No globally optimal solution possible if N > 2 • Continuous version of [Potts ’52]
14 ∫
Color input images I0, I1 : → R3:
Label each pixel in I0 with a flow vector in Γ ⊂ R2
Γ
x
y
Cost function compares e.g. pointwise pixel colors in the images:
cγ(x) = |I0(x)− I1(x + γ)|2
15 ∫
Indicator function uγ : → {0,1} assigned to each label γ:
u1 = 1, all others zero
∑ γ uγ must be one !
Problem relaxation
Indicator function uγ : → {0,1} assigned to each label γ:
∑ γ uγ must be one !
Problem relaxation
Different Multilabel Regularizers ∫ x
Zach, Gallup, Frahm, Niethammer ’08
J1(u) = 1 2
J2(u) =
∫
√∑ γ
√ vT AT Av
17 ∫
J1(u) = 1 2
J2(u) =
∫
√∑ γ
√ vT AT Av
17 ∫
J1(u) = 1 2
J2(u) =
∫
√∑ γ
√ vT AT Av
17 ∫
J1(u) = 1 2
J2(u) =
∫
√∑ γ
√ vT AT Av
17 ∫
• Stereo assignment cost cγ(x) = |Ileft(x)− Iright(x + γ)| • Linear discontinuity penalty⇒ globally optimal solution.
Images from UCSD lightfield repository
18 ∫
One of two input images Depth reconstruction (Courtesy of Microsoft Graz)
19 ∫
Labeling regions should have certain spatial relationships, i.e. heaven is always above ground.
20 ∫
Direction-Aware New Regularizer ∫ x
Strekalovskiy and Cremers, ICCV 2011
The labeling penalty may depend also on the normal n of the interface between two regions,
d : Γ× Γ× S→ R+.
New direction-aware regularizer:
∑ γ
∫
pγ ,∇uγ dx ,
with C = {(pγ : → Rn) : pµ − pγ ,n ≤ d(γ, µ,n) ∀γ, µ,n} .
21 ∫
Input Data term Potts Ordering
22 ∫
23 ∫
The optic flow label space is a product space ∫ x
Γ
x
y
Each red dot requires one indicator function - too many. Can we exploit the special structure of the label space?
24 ∫
1
λ2 .
25 ∫
1
λ2 .
25 ∫
The data term is now non-convex:
E(u1,u2) = ∑ γ∈Γ
R(u) = sup q1 λ1 +q2
λ2≤cγ
Optimal envelope relaxation of the data term
26 ∫
E(u1,u2) = ∑ γ∈Γ
λ2≤cγ
26 ∫
E(u1,u2) = ∑ γ∈Γ
λ2≤cγ
26 ∫
First image I0 Second image I1 Result
32× 32 labels, image resolution 320× 240, TV regularity 1.5 minutes runtime, within 2% of global optimum.
27 ∫
E(u, σ) = J(u, σ) + 1
2σ u − f2
28 ∫
E(u, σ) = J(u, σ) + 1
2σ u − f2
29 ∫
Color space segmentation ∫ x
Input RGB 6 × 6 × 6 L∗a∗b∗ 8 × 5 × 5
Segmentation using different three-dimensional spaces of equidistant color labels, L2 dataterm
30 ∫
VML α-EXP α-β-SWAP BP TRW-S
VML TRW-S
Potts regularizer 8 × 8 × 8
mem [MiB] time [s] bound [%]
VML 2173 94.59 1.03 α-EXP 2746 173.64 0.90 SWAP 2746 461.48 1.34 BP 8667 254.08 16.29 TRW-S 8667 287.30 1.95
32 ∫
1 Introduction
5 Summary
Given n images Ii : i → R3
with projections πi : R3 → R2
Find surface Σ ⊂ R3 with texture T : Σ→ R3 which optimally matches the input images.
34 ∫
Given n images Ii : i → R3
with projections πi : R3 → R2
Find surface Σ ⊂ R3 with texture T : Σ→ R3 which optimally matches the input images.
34 ∫
Variational 3D reconstruction ∫ x
Classical variational formulation (Faugeras and Keriven, 1998)
Find a surface Σ ⊂ R3 which minimizes the photo-consistency error,
argmin Σ
argmin Σ
different views • small value of ρ
35 ∫
argmin Σ
different views • large value of ρ
35 ∫
Convex functional minimizes photo-consistency error:
argmin u:→{0,1}
36 ∫
argmin u:→{0,1}
A ray through the silhouette must intersect the surface
Kolev, Klodt, Brox, Cremers IJCV’09
36 ∫
argmin u:→{0,1}
A ray through the background must miss the surface
Kolev, Klodt, Brox, Cremers IJCV’09
36 ∫
argmin u:→{0,1}
Silhouette constraints to avoid constant solutions
Relaxation to convex domain leads to convex problem with known optimality bound.
L2(, {0,1}) non-convex
⊂ L2(, [0,1]) convex
36 ∫
Kalin Kolev, Svetlana Matiouk 2010 for Akademisches Kunstmuseum Bonn
37 ∫
37 ∫
37 ∫
movie ”statue”
37 ∫
statue.mpeg
Improvement: normal optimization ∫ x
E(u) =
∫
√ vT v = 1
38 ∫
Superresolution texture maps ∫ x
Given
approximate surface Σ (assumed Lambertian)
39 ∫
Superresolution texture maps ∫ x
Given
approximate surface Σ (assumed Lambertian)
Find
and accurate geometry
blur kernel
downsamplingSensor element
• Each sensor element samples incoming light over its area • Sampling modeled by blur kernel b • Leads to image formation model
Iobserved (low-res) = b ∗ Iincoming (high-res)
40 ∫
Data term: squared difference between input images and downsampled high-resolution rendering
E(T ) := n∑
i=1
βi
πi
41 ∫
Conformal texture atlas ∫ x
T τ−→ Σ
42 ∫
Bird Beethoven Bunny
• 30 cameras, input image resolution 768× 576 • Initial geometry: Kolev and Cremers, ECCV 2008 • Texture resolution 2048× 2048
43 ∫
Rendered model
Input image
Rendered model
Input image
Additional dependance on displacement map D : T→ R,
E(T ,D) := n∑
)2 dx + Etv(T ,D).
• In T : energy is convex • In D: multilabel problem with convex regularizer,
global optimization in D possible
Geometry Estimated Texture
46 ∫
Textured bunny model ∫ x
movie ”bunny”
Variational camera calibration ∫ x
Idea: assume the projection parameters π are unknowns in the superresolution energy,
E(T , π) := N∑
b ∗ (T βi )− Ii dx .
Optimize alternatingly for texture and projection. In a way, this can be thought of as a continuous version of bundle adjustment.
Aubry, Goldluecke, Kolev, Cremers, ICCV 2011. 48
∫
1 Introduction
5 Summary
Stanford MCA HCI Gantry Raytrix plenoptic camera
Marc Levoy (2006)
“in 25 years, most consumer photographic cameras will be light field cameras.”
50 ∫
Stanford MCA HCI Gantry Raytrix plenoptic camera
Marc Levoy (2006)
“in 25 years, most consumer photographic cameras will be light field cameras.”
50 ∫
Light field structure ∫ x
A 2D horizontal cut (green) is called an epipolar plane image (EPI)
51 ∫
λj
λi
ni
Forbidden transition
Depth λi < λj , corresponding to direction ni ⇒ transitions only allowed orthogonal to ni
52 ∫
Typical epipolar plane image
Use structure tensor to compute local directions on the EPI which correspond to disparity or depth.
53 ∫
Noisy local depth estimate
Use structure tensor to compute local directions on the EPI which correspond to disparity or depth.
53 ∫
Use multilabel framework with ordering constraints to obtain globally consistent depth labeling on the EPI
53 ∫
53 ∫
Results on Stanford light fields ∫ x
Center view Convex stereo Our method Stanford Light Field Archive, 17× 17 views at 1280× 960
54 ∫
Center view Reference algorithm Our result (by manufacturer)
Only 9× 9 effective views, challenging metallic surfaces
55 ∫
Light field super-resolution ∫ x
Use overlapping views to infer additional information
• Super-resolution in both spatial as well as angular domain • Goal: obtain high-resolution view u from novel viewpoint • Model: explain low-res input views vi given high-res view u
• Requires subpixel-accurate matching (disparity maps) • Leads to (convex) inverse problem for u
Wanner and Goldlucke ECCV 2012
56 ∫
• Super-resolution in both spatial as well as angular domain • Goal: obtain high-resolution view u from novel viewpoint
• Model: explain low-res input views vi given high-res view u • Requires subpixel-accurate matching (disparity maps) • Leads to (convex) inverse problem for u
56 ∫
56 ∫
56 ∫
Warp map illustrations ∫ x
on i on Γ lo
w -r
es ol
ut io
Input view vi Disparity map di Forward warp vi βi
hi gh
-r es
ol ut
io n
Backward warp u τi Visibility mask mi Novel view u
57 ∫
on i on Γ lo
w -r
es ol
ut io
hi gh
-r es
ol ut
io n
57 ∫
on i on Γ lo
w -r
es ol
ut io
hi gh
-r es
ol ut
io n
57 ∫
Backward warp is downsampled to low-res input views
Exact model: vi = b ∗ (u τi )
Variational energy:
E(u) = σ2 ∫
58 ∫
Backward warp is downsampled to low-res input views
Exact model: vi = b ∗ (u τi )
Variational energy:
E(u) = σ2 ∫
58 ∫
59 ∫
59 ∫
Gaussian noise σ = 0.2 Single view denoising Light field denoising
PSNR 14.66 PSNR 22.61 PSNR 24.45
Solvers for inverse problems on ray space which take into account the light field structure
60 ∫
1 Introduction
5 Summary
• Vector-valued labeling problems • Labeling with spatial layout constraints
• 3D reconstruction • Super-resolved texture maps
• Consistent light field depth labeling
• Light field super-resolution

Documents

Variational Methods - in 3D reconstruction and light field analysis