Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The...

Estimating the Likelihood of Statistical Models

of Natural Image PatchesDaniel Zoran

ICNC – The Hebrew University of Jerusalem

Advisor: Yair Weiss

CifAR NCAP Summer School

Natural Image Statistics

• Natural scenes and images exhibit very distinctive statistics

• A lot of research has been made in this field since the 1950s

• Important in image processing, computer vision, computational neuroscience and more…

Natural Image Statistics Properties

• The space of all possible images is huge– For a 256 gray levels, NxN sized matrix, there are

possible images– Natural Images occupy a tiny fraction of this space

• Some statistical properties of natural images:– Translation invariance– Power law spectrum – – Scale invariance– Non-Gaussianity of marginal statistics - (more on

that later)

2

256N

2( ~A

Pw

Estimating the Likelihood of different statistical models• During the years, a lot of models for natural

image distributions have been proposed• It is hard to test the validity of such models,

especially when comparing one model to the other

• A step towards this – estimating the (log) likelihood of a given model and comparing the results with other models

Estimating the Likelihood of different models• Variable sized patches were extracted from

natural images• Different models assumed• A training set was used to estimate various

parameters of the model• Likelihood was calculated over a test set• 5000 patches in each set• Source images are mostly JPEGs from a Panasonic

digital camera, portraying outdoor scenes• Also tested on standard images (Lena, Boat and

Barbara – PNG format)

The models – 1D Gaussian• A 1D Gaussian distribution for every pixel

– Mean and Variance estimated directly from the sample

– The likelihood of an image x under this model is:

– Where:

• This model captures nothing about natural images

2222

2

1( | )

2

i i

i

x

i i

L e

x

1 pi i

p

xP

22 1

1p

i i ip

xP

Results – 1D Gaussian

Patch SizeTest Set 1 (JPG)Test Set 2

(JPG)Test Set 3 (PNG)Test Set 4

(Noise)

10x1020.3-19.618.4-26.9

12x1229.4-28.026.2-38.8

14x1441.4-38.435.8-54.5

16x1652.1-44.647.7-70.5

18x1868.3-54.760.5-89.1

20x2084.5-6772.9-106.4

The models – Multidimensional Gaussian with PCA• Using the covariance matrix, rotate the images in

the image space towards directions of maximum variance (PCA)

• A Multidimensional Gaussian distribution for the components:

• Where the covariance matrix is estimated from the training set:

• This captures the Power-Law spectrum property

/2 1/2

1( | 0 ) exp( )

2NL

T -1x Σ x Σ x

Σ

Txx

Results – Multidimensional Gaussian

Patch SizeTest Set 1 (JPG)Test Set 2

(JPG)Test Set 3 (PNG)Test Set 4

(Noise)

10x1043840046526946

12x1263157769640345

14x1488679294557808

16x1611771039128579020

18x18150213341661104777

20x20191617022007135830

The models – Gaussian Mixture Model with PCA• Using the same rotation scheme (PCA), now

assume a Gaussian Mixture Model for the marginal filter response distributions

• Under this model:

• Where W’s rows are the eigenvectors of the covariance matrix

• The GMM parameters were found using EM• This captures both the Power-Law spectrum and

the sparseness properties

2 2

1

( | 0 , ) ( | 0, )K

k i kki

L c N y

x σ c y = Wx

Results – GMM with PCA

Patch SizeTest Set 1 (JPG)Test Set 2

(JPG)Test Set 3 (PNG)Test Set 4

(Noise)

10x10200210181-4852

12x12285295244-8138

14x14376400312

16x16472511404

18x18595646488

20x20698767557

The models – Generalized Gaussian with PCA• Finally, instead of using a GMM, we now use a

Generalized Gaussian• This has the advantage of having less

parameters, while still capturing Sparseness:

• Parameters were obtained directly from the training set

( | 0 , ) ( , ) exp( )ii i

i i

L A

Wxx σ α

Results – Generalized Gaussian with PCA

Patch SizeTest Set 1 (JPG)Test Set 2

(JPG)Test Set 3 (PNG)Test Set 4

(Noise)

10x10208212191-1894

12x12291305273-3099

14x14386411356-5242

16x16497529456-7875

18x18615663543-11966

20x20739796668-16783

Mean Log Likelihood - 12x12 patches

-500

-300

-100

100

300

500

700

900

Noise Natural Patches

1D Gaussian

PCA-MDGaussian

PCA-GMM

PCA-GG

Mean Log Likelihood - 18x18 patches

-500

0

500

1000

1500

Noise Natural Patches

1D Gaussian

PCA-MDGaussian

PCA-GMM

PCA-GG

The GG shape parameter

• During the analysis of the data we have encountered a strange phenomena

• Marginal distributions get wider as we go measure higher frequency filter responses

• This is not due to increase in variance (which drops as we go to high frequencies)

• We modeled this using the shape parameter obtained from the samples

Shape parameter for test set 1

Shape parameter for test set 2

Shape parameter for test set 3

Shape parameter for test set 4 - PNG

Shape parameter for noise test set

Conclusion

• This is (very) early work, still in progress• A lot of things left to do:

– Try more models and filter (ICA is in progress)– Actually compare the different models– Try to make some sense out of the shape of the

distributions– Look into higher order dependencies and

correlations

• A lot more…

Thank you!

Questions?