Siwei LyuComputer Science Department

University at Albany, SUNY, USA

Stefan RothComputer Science Department

Technische Universität Darmstadt, Germany

tutorial web page: http://www.gris.informatik.tu-darmstadt.de/teaching/iccv2009/index.en.htm

Modeling Natural ImageStatistics for Computer Vision

Part III - MRF Models in the Wavelet DomainLecturer: Siwei Lyu

09/27/2009Siwei Lyu and Stefan Roth

MRFs in wavelet domain

■ extend local statistical models for wavelet coefficients to a global extent

■ examples• tree based models [Ronberg et.al., 2001; Wainwright et.al., 2003]• field of GSM (FoGSM) [Lyu & Simoncelli, NIPS 2006; PAMI 2009]• implicit MRF model [Lyu, CVPR 2009]

■ we will focus on the latter two models in this tutorial

2

09/27/2009Siwei Lyu and Stefan Roth

x field

z field

3

field of GSM (FoGSM)

p(x)

x

x = u×√

zmarginal

GSM

blockGSM

field of GSM

x = u×√

z

x = u⊗√

z

single coefficient

coefficient block

one subband

09/27/2009Siwei Lyu and Stefan Roth

x u log z

decomposition & samples

4

09/27/2009Siwei Lyu and Stefan Roth

marginal

joint

model evaluation

5

09/27/2009Siwei Lyu and Stefan Roth

original image noisy image (! = 25) matlab wiener2 FoGSM

(14.15dB) (27.19dB) (30.02dB)

denoising with FoGSM

6

20.17

09/27/2009Siwei Lyu and Stefan Roth

pairwise conditional density

x1

x 2

p(x2|x1)

E(x2|x1)

E(x2|x1)+std (x2|x1)

E(x2|x1)-std (x2|x1)

“bow-tie”[Buccigrossi & Simoncelli, 1997]

7

09/27/2009Siwei Lyu and Stefan Roth

pairwise conditional density

x1

x 2

E(x2|x1)

E(x2|x1)+std (x2|x1)

E(x2|x1)-std (x2|x1)

E(x2|x1) ≈ ax1 var(x2|x1) ≈ b + cx21

8

09/27/2009Siwei Lyu and Stefan Roth

■ constraints

■ maximum entropic conditional density

■ known as the singleton conditional density

conditional density

µi = E(xi|xj,j∈N (i)) =�

j∈N(i)

ajxj

σ2i = var(xi|xj,j∈N (i)) = b +

�

j∈N(i)

cjx2j

p(xi|xj,j∈N (i)) =1�2πσ2

i

exp�− (xi − µi)2

2σ2i

�

9

09/27/2009Siwei Lyu and Stefan Roth

Brook’s Lemma

■ in an MRF, all singleton conditional densities ⇔ joint density

■ the joint density may not have closed form■ thus the resulting MRF is implicit

10

{p(xi|xj,j∈N (i))|∀i}⇔ p(x)

[Brook, 1964]

09/27/2009Siwei Lyu and Stefan Roth

implicit MRF model

■ defined by all singletons■ joint density (and clique potential) is implicit■ learning: maximum pseudo-likelihood

11

θMPL = argmaxθ

�

i

log p(xi|xj,j∈N (i); θ)

09/27/2009Siwei Lyu and Stefan Roth

ICM-MAP denoising

- set initial value for �x(0), and t = 1

- repeat until convergence

- repeat for all i

- compute the current estimation for xi, as

x(t)i = argmax

xi

log p(x(t)1 , · · · , x(t)

i−1,

xi, x(t−1)i+1 , · · · , x(t−1)

d |�y).

- t← t + 1

12

argmaxx

p(x|y) = argmaxx

p(y|x)p(x) = argmaxx

log p(y|x) + log p(x)

iterative conditional mode (ICM)

09/27/2009Siwei Lyu and Stefan Roth

ICM-MAP denoising

local adaptive and iterative Wiener filtering

xi =σ2

wσ2i

σ2w + σ2

i

yi

σ2w

+µi

σ2i

−�

i�=j

wij(xj − yj)

.

13

argmaxxi

log p(y|x) + log p(x)

= argmaxxi

log p(y|x) + log p(x1, · · · , xi−1, xi, xi+1, · · · , xn)

= argmaxxi

log p(y|x)� �� �can be further simplified

+ log p(xi|xj,j∈N(i))� �� �singleton conditional

+ ✭✭✭✭✭✭✭✭✭✭✭✭✭✭log p(xj,j∈N(i))� �� �constant w.r.t xi

.

09/27/2009Siwei Lyu and Stefan Roth

summary

imagerepresentation

statisticalobservations

computer visionapplications

mathematical model

14

09/27/2009Siwei Lyu and Stefan Roth

edge + texture?■ primal sketch model [Guo, Zhu & Wu 2005]

■ a unified statistical image model for texture and structures

15

09/27/2009Siwei Lyu and Stefan Roth

related challenges■ time -- natural videos■ 3D -- natural range images■ motion -- natural optical flows■ chromatics -- natural color images■ lighting - natural illuminations■ properties of specific image class

• medical images [Pineda et.al., SPIE 2008]• satellite images [Jager & Hellwich, IGRASS 2005]• face images [Liu et.al., IJCV 2004]• human bodies [Norouzi, CVPR 2009]

16

09/27/2009Siwei Lyu and Stefan Roth

big question marks■ what are natural images, anyway?

■ white noises are “natural” as they are the result of cosmic radiations

■ naturalness is in the eyes of the beholders

17

09/27/2009Siwei Lyu and Stefan Roth

big question marks■ what are natural images, anyway?

■ white noises are “natural” as they are the result of cosmic radiations

■ naturalness is in the eyes of the beholders

17

“unnatural” to a prehistoric human

09/27/2009Siwei Lyu and Stefan Roth18

natural!

image!

statistics

math

statisticsbiology

computer!

science

image!

processing

machine!

learning

computer!

vision

optimization

perception

neuro-!

science

signal!

processing

09/27/2009Siwei Lyu and Stefan Roth19

future directions

■ comprehensive model capturing all known statistical properties of natural images

■ efficient algorithms for learning and inference■ tighter connection to mid and high level computer

vision• principled framework to find effective feature types based on

image statistics• and many more … …

09/27/2009Siwei Lyu and Stefan Roth

resources■ D. L. Ruderman. The statistics of natural images. Network: Computation in

Neural Systems, 5:517–548, 1996. (good introduction)■ E. P. Simoncelli and B. Olshausen. Natural image statistics and neural

representation. Annual Review of Neuroscience, 24:1193–1216, 2001. (neural science perspective)

■ S.-C. Zhu. Statistical modeling and conceptualization of visual patterns. IEEE Trans PAMI, 25(6), 2003. (computer vision perspective)

■ A. Srivastava, A. B. Lee, E. P. Simoncelli, and S.-C. Zhu. On advances in statistical modeling of natural images. J. Math. Imaging and Vision, 18(1):17–33, 2003. (mathematical perspective)

■ E. P. Simoncelli. Statistical modeling of photographic images. In Handbook of Image and Video Processing, 431–441. Academic Press, 2005. (signal processing perspective)

■ A. Hyvärinen, J. Hurri, and P. O. Hoyer. Natural Image Statistics: A probabilistic approach to early computational vision. Springer, 2009. (statistical modeling and recent accounts)

20

09/27/2009Siwei Lyu and Stefan Roth21

thank you