Vector Valued Image Regularization with PDE’s: A Common Framework for Different Applications

CVPR 2003 Best Student Paper Award

Definitions

Scalar images: Intensity images Vector valued images: RGB, HSV, YIQ… Regularization: Finding approximate solution

for ill-posed problems.

Let G be 2x2 matrix

dc

baG

2

1,)(

iiigtrace G

Divergence and Curl

Vectors are obtained from image gradients I

y

A

x

AAAdiv yx

)(

Divergence defines expansion or contraction per unit volume

),( yxA vector function

yyxxy

xII

I

IdivIdivI

)(

Curl of a vector field is defined by cross product between vectors

Structure Tensor

n

iyx

y

xn

i

Tii II

I

III

11

G

Structure Tensor

n

iyx

y

xn

i

Tii II

I

III

11

G

One dimensional image (gray level)

yyyx

yxxxgray IIII

IIIIG

)1,(),(

),1(),(

yxIyxII

yxIyxII

y

x

Structure Tensor

yyyx

yxxx

yyyx

yxxx

yyyx

yxxxcolor

TTTcolor

BBBB

BBBB

GGGG

GGGG

RRRR

RRRR

BBGGRR

G

G

reigenvecto

eigenvalue

color

2

1

2

1

φ

φλ

φ

φG

Eigenvalue and eigenvectors of G (spectral elements)

Can be computed using Matlab functions

x derivative y derivative

More Insight (hessian&tensor)

Hessian of image I

Laplacian of I

Tensor: Matrix of matrices. They abuse the notation: Symmetric semi-positive definite matrix

yyyx

xyxx

II

IIH

yyxx IItraceII )()div( H

xx derivative yy derivative

tracexx derivative yy derivative

Image Regularization Functional minimization: Euler-Lagrange equations

Divergence expression: Diffusion of pixel values from high to low concentration

Oriented Laplacians: Image smoothing in eigenvector directions weighted by corresponding eigenvalue.

dINnRI

))((min:

)( itI IDdiv

vvuutI IcIc 21

Variational Problem

Define variational problem:

dIIIIFIE yxyx ),,,()(

0

yx I

F

dy

d

I

F

dx

d

I

FEuler-Lagrange equation

Solution using classic iterative method:

)1()1()1()1()()(

00

tI

F

dy

dt

I

F

dx

dt

I

FtItI

t

tI

II

yx

t

Variational Problem Horn&Schunck Example

0 tyx fvfuf dtdy

dtdx vu

dxdyvvuufvfufvuvuE yxyxtyx22222),,,(

022 avguu

yyxxxtyx uuffvfuf

022

avgvv

yyxxytyx vvffvfuf

tyxxt

avgt fvfuf

fuu

1

yx u

E

yu

E

xu

E

yx v

E

yv

E

xv

E

tyxyt

avgt fvfuf

fvv

1

Defining Energy Functional

Functional: (minimization based)

dIERI

),(min)(3:

Euler-Lagrange is given by (relation to divergence based methods)

)div(div i

i

ii ID

I

I

t

I

y

x

TTD

22

increasing function)()(, afbfab

D Matrix

TTD

22 is 2x2.

Eigenvalues and eigenvectors of D are

22 21

21 uu

iTTi I

t

I

22div

Old School Laplacian Approach

TT ccT 222111 vvuu

vvuu IcIct

I21

Second order image derivatives in directions of eigenvectors of G at that point (structure tensor)(Edge preserving smoothing)

1 tt II

)( ii Ttracet

IH

Solution of this PDE can be given by:

),(

)0()(

tii GIItt

T

)4

exp(4

1)(

1),(

ttG t xxT

xT

Laplacian Example (constant T)

Constant T, such that direction (eigenvectors are same)

Laplacian Example

Laplacian Example (varying T)

T is not constant and but independent of image content. There are similarities with bilateral filtering.

Laplacian Example (varying T)

)( ii IDdivt

I

Relation Between T and D

cb

baD

)()()( DdivIDtraceIDdiv Tiii H

Homework Due 19 January 2005

Show the following: (Page 3 of the paper):

Relation Between T and D

)( ii Ttracet

IH

)()( DdivIDtracet

I Tii

i

H

Ti

Ti

Ti

Ti

Ti

Itrace

IGtrace

IGdivtraceIDdivtraceDdivI

)()()(

2

2

Relation Between T and D

3

1

3

1

3

1

3

12

2

2

2

)(

jjjjj

j jjjj

jjjj

jjj

jjj

j jjjjjj

jjjjjj

j jjj

jjj

III

IIII

IIII

III

III

IIIIII

IIIIII

III

IIIdivGdiv

yyyxyx

xyyxxx

yyxxy

yyxxx

xyxyxxyyy

xyyyyxxxx

yyx

yxx

H

Let’s just solve for div(G). Rest can be solved similarly:

yyxxy

xII

I

IdivIdivI

)(

Relation Between T and D

Using trace property:

B

AAAB

2)(

T

tracetrace

3

1

)()(j

jij

iji QDtraceIDdiv H

Kronecker func.

2

2

2

Tijijij

Tij

TT

Tij

TT

jTi

ij

II

II

II

PPQ

G

P

Hessian

TT ffD

2

2

2

1 ),(),(

Rewriting the Formula

3

1

)()(j

jij

iji QDtraceIDdiv H

)()(

)(3

1

AHtraceIDdiv

jtraceIDdivj

iji

HA

333231

232221

131211

AAA

AAA

AAA

A

3

2

1

H

H

H

H

super matrix notation

Unified Expression

)(AHtracet

I

iti

ti

iti

ti

TT

traceII

traceII

H

TH

**

**

****

1

1

1

1

1

1

)(

),(1 f

),(2 f

Numerical Implementation

Conventional approach Compute image Hessians and Gradients

Proposed method Use local filtering by Gaussians (2nd page)

),(

)0()()( t

iiii GIItracet

Itt

TTH

)4

exp(4

1)(

1),(

ttG t xxT

xT

Similar to bilateral filtering

Numerical Implementation

1. Compute local convolution mask defining local geometry by T.

2. Estimate trace(THi) in local neighborhood of x.

3. Apply filtering for each trace(AijHj) in the vector trace(AH)

),( tG T

)1,1(),1()1,1(

)1,(),()1,(

)1,1(),1()1,1(

)1,1(),1()1,1(

)1,(),()1,(

)1,1(),1()1,1(

),()(

yxGyxGyxG

yxGyxGyxG

yxGyxGyxG

yxiyxiyxi

yxiyxiyxi

yxiyxiyxi

yxtrace iTH

Comparison Between Two Implementations

Noise Removal and Image impainting

Reconstruction, VisualizationSuperscale Images