24/01/2014

1

Shape-from-Template

Adrien BartoliALCoV-ISIT, Clermont-Ferrand

Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley

Computer Vision Mini Workshop

Toulouse, January 24, 2014

3D Reconstruction: to Recover Shape from Images

Active methods Passive methods

Image from [Crandall et al, CVPR’11]Using Shape-from-Motion

Image by Visnjic, 2010Using Kinect

Escaping Criticism by del Caso, 1874

Model Town by Matt West, 2006

Blur ShadingMotion OcclusionsStereoscopy Texture

[Gibson ~ 1960 ; Marr ~ 1970]

Passive Single-View Reconstruction: Visual Cues

Blur ShadingMotion OcclusionsStereoscopy

Unapplicableto single-view

Very weak hereUnapplicableto single-view

Weak in general

Very weak here Very weak here

Texture

+ database

[Hoeim et al, SIGGRAPH’05]

Passive Single-View Reconstruction: Shape Priors

Low dimensional shape space

3D Morphable Face Model[Blanz and Vetter, PAMI’03]

High dimensional shape space…

… but simple deformation model!

Ob

ject

leve

l

High dimensional shape space

Scen

ele

vel

Faces Newspapers

Detection Recognition

Shape-from-Template

ShapeImage

Shape-from-Template

MaterialShapeAppearance

Template

Scope: object-specific passive single-view reconstruction using simple physics-based deformation from a known reference shape and a matchable appearance

[Gumerov et al, ECCV’04 ; Salzmann et al, BMVC’05 ; Perriollat et al, BMVC’08]

Detection

Shape-from-Template

Template not found

Template not found

Shape-from-Template

Shape-from-Template

Shape-from-Template

Shape-from-Template

Shape-from-Template

Template not found

ShapeAppearance

Template

Material

24/01/2014

2

Shape-from-Template

Template not found

Template not found

Template not found

Shape-from-Template

ShapeAppearance

Template

Material

Template not found

Augmented Reality for Deformable Surfaces

Differential Geometric Setup

3D

2D

𝑢𝑣-map - Ω ⊂ ℝ2 Image

Shape

Cameraprojection Π

𝜑 ∈ 𝐶2 Ω,ℝ3

Π ∈ 𝐶∞ ℝ3, ℝ2

ShapeAppearance

Template

Material

∘ Π ∘ 𝜑=⇒

⇒ 𝜑 =

Image warp 𝜂

𝜂

Two-Step Approach

3D

2D

𝑢𝑣-map - Ω ⊂ ℝ2 Image

Image warp 𝜂

Cameraprojection Π

Shape

1 – Image registration

2 – Shape inference

Differential Problem Statement

ShapeAppearance

Template

Material

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.

𝜑 =

∘ Π ∘ 𝜑=

Shape-from-Template

Problem Relaxation

ShapeAppearance

Template

Material

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.

𝜑 =

∘ Π ∘ 𝜑=

Shape-from-Template

𝜼 ∈ ? ?𝜼

𝑇𝑉2𝐿2 𝜂 → min

24/01/2014

3

Image Registration

ShapeAppearance

Template

Material

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.

𝜑 =

∘ Π ∘ 𝜑=

Shape-from-Template

𝜼 ∈ ? ?𝜼

𝑇𝑉2𝐿2 𝜂 → min

Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2

1 – Image registration

Factoring the Warp into Embedding and Projection

ShapeAppearance

Template

Material

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.

𝜑 =

∘ Π ∘ 𝜑=

Shape-from-Template 𝜼 = 𝚷 ∘ 𝝋

Shape Inference

ShapeAppearance

Template

Material

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.

𝜑 =

∘ Π ∘ 𝜑=

Shape-from-Template 𝜼 = 𝚷 ∘ 𝝋

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

Shape-from-Template

Two-Step Approach

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

Template

Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2

1 – Image registration

Appearance Shape Material

Step 1: Image Registration, in a Nutshell

Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2

1 – Image registration

Putative keypoint matches [Lowe, IJCV’04]

Densification[Bookstein, PAMI’89]

Template not found

Local consistency [Pizarro et al, IJCV’12]

1

2

[Schmid et al, PAMI’97]

Step 1: Image Registration, in a Nutshell

Local consistency [Pizarro et al, IJCV’12]

Partial self-occlusion detection[Pizarro et al, IJCV’12]

Densification [Bookstein, PAMI’89]

Self-occlusion aware densification[Pizarro et al, IJCV’12]

Color-based refinement[Gay-Bellile et al, PAMI’10]

2

3

4

5

6

24/01/2014

4

Step 1: Image Registration, Results Step 1: Image Registration, Results

Shape-from-Template

Two-Step Approach

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

Template

Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2

1 – Image registration

Appearance Shape Material

Step 2: Shape Inference

3D

2D

𝑢𝑣-map - Ω ⊂ ℝ2 Image

Cameraprojection Π

Image warp 𝜂

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

Calibratedpinhole

Zeroth Order Methods

3D

2D

𝑢𝑣-map - Ω ⊂ ℝ2 Image

Cameraprojection Π

Image warp 𝜂

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

𝛻𝜑⊤𝛻𝜑 = I

Main result: shape-wise solution of a convex relaxation[Perriollat et al, IJCV’11; Salzmann et al, PAMI’11; Brunet et al, ACCV’10; Östlund et al, ECCV’12]

Zeroth orderreprojection

[Perriollat et al, IJCV’11; Salzmann et al, PAMI’11]

First Order Methods

3D

2D

𝑢𝑣-map - Ω ⊂ ℝ2 Image

Cameraprojection Π

Image warp 𝜂

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

𝛻𝜑⊤𝛻𝜑 = I

Zeroth orderreprojection

𝛻𝜂 = 𝛻Π ∘ 𝜑 𝛻𝜑

First orderreprojection

[Bartoli et al, CVPR’12]

24/01/2014

5

First Order Methods

Find s.t.

2 – Shape inference

Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3

Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =

𝜂 = Π ∘ 𝜑

𝛻𝜑⊤𝛻𝜑 = I

Zeroth orderreprojection

𝛻𝜂 = 𝛻Π ∘ 𝜑 𝛻𝜑

First orderreprojection

𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = I

Let 𝛾 ∈ 𝐶1 Ω, ℝ be the depth function

Shape inference is this first-order quadratic PDE in 𝜸

Main result: exact point-wise solution

[Bartoli et al, CVPR’12]

Intuition: neglect the dependency between 𝛾 and 𝛻𝛾

First Order Methods

𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = I

Isometric developable

Conformal

𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = 𝜈𝛻Δ⊤𝛻Δ

𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = 𝛻Δ⊤𝛻Δ

Isometric non-developable object

Isometric (infinitesimal) weak-perspective

𝛻𝛾 22 + 𝛾2 𝛻𝜂𝛻𝜂⊤ + 𝛼2𝛻𝛾⊤𝛻𝛾 = 𝜈𝛻Δ⊤𝛻Δ

Isometric, unknown focal length

𝑓2 𝛻𝛾 22 + 𝛾2 𝛻𝜂𝛻𝜂⊤ + 𝛼2𝛻𝛾⊤𝛻𝛾 = 𝜈𝛻Δ⊤𝛻Δ

‘true’ 𝑓 = 2040 pixels

3.8% relative error

Estimated 𝑓 = 2118 pixels

𝑓 = 5870 pixels

Non-flattenable Objects

3D

2D

Image warp 𝜂

Cameraprojection Π

Deformation Ψ

Conformal flattening Δ−1

𝑢𝑣-map - Ω ⊂ ℝ2 Image

Δ ∈ 𝐶2 Ω,ℝ3

Ψ = 𝜑 ∘ Δ−1

Step 2: Shape Inference, Results

Ground-truth(Rigid Shape-from-Motion)

Shape-from-Template

Step 2: Shape Inference, Results Step 2: Shape Inference, Results

24/01/2014

6

Some Extensions

ShapeAppearance

Template 1

Material ShapeAppearance

Template 2

Material

Multiobject shape-from-template [Alcantarilla et al, BMVC’12]

etc.

Linear elastic deformations [Malti et al, CVPR’13]

template image shape

Isowarp and Conwarp [Pizarro et al, BMVC’13]

Differential constraints on 𝜂 so that 𝜂 = Π ∘ 𝜑

Main Surface Types

Physical flattening Virtual flattening

Iso

met

ric

def

orm

atio

nEl

asti

cd

efo

rmat

ion

Gynecologic Surgery

Procedures• Benign conditions• Cancer• Infertility• Incontinence

Organs of the female reproductive system

Located in the pelvic cavity

Scope

Abdominal (open, laparotomy)

Hysteroscopic(through vagina)

Laparoscopic(small incisions)

Types

Preoperative Imaging

MRI US

• Diagnosis• Procedure• Type of surgery

Uterine Fibroids or Myomas

• Microscopic to extremely large size• Often several of them

Benign tumors from the myometrium

May be invisible in laparoscopy(and hysteroscopy)

Intramural myomas (type b)

Clearly visible in MRI

Laparoscopic Augmented Reality

3 Displaying

Augmenting

Augmentation data

2

Registration

1 Filming

24/01/2014

7

Registration is Challenging

Blur Occlusions

Bleeding Smoke

Augmentation data

Preoperative MRI Preparation

Axial Sagittal Coronal

Uterus surface First fibroid Second fibroid

Augmented Reality Framework

1 Registration

2 Augmentation

Current frame

Current frame

Registration Φ𝑖 :ℝ3 → ℝ2

Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic

Two-Step Registration

Registration Φ𝑖 :ℝ3 → ℝ2

1 Registration

Current frame

Reference frame

1b

Reference shape1a

Requirements: (R1), R3, R4

Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic

Relax requirements with Φ𝑖 = Γ𝑖 ∘ Γ0

Preoperative to intraoperative reference

Intraoperativereference to current frame

WBMTR (Wide-Baseline Multi-Texturemap Registration)[Collins et al, MIAR@MICCAI’13]

Registration Φ𝑖 :ℝ3 → ℝ2

1 Registration

Reference frame

Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic

Relax requirements with Φ𝑖 = Γ𝑖 ∘ Γ0

Current frame

Shape-from-Motion

Pose with keypoints from best reference frame

Reference frames

Reference shape

WBMTR Registration Results

24/01/2014

8

WBMTR Registration Results Generalizing Rigid Pose to Deformations

Registration Φ𝑖 :ℝ3 → ℝ2

1 Registration

Current frame

Reference shape

Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic

Relax requirements with Φ𝑖 = Γ𝑖 ∘ Γ0

Ψ𝑖 :ℝ3 → ℝ3

Π:ℝ3 → ℝ2

Deformation

Projection

Shape-from-TemplateTo recover Ψ𝑖 (and Π)

Uterine Shape-from-Template Results

First order, isometricZeroth order, isometric

[Salzmann et al, PAMI’09]First order, conformal

Shape-from-Template

Adrien BartoliALCoV-ISIT, Clermont-Ferrand

Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley

Computer Vision Mini Workshop

Toulouse, January 24, 2014