Shrinkage/Thresholding Iterative Methods

1IPIM, IST, José Bioucas, 2007

Shrinkage/Thresholding Iterative Methods

• Nonquadratic regularizers• Total Variation• lp- norm• Wavelet orthogonal/redundant representations• sparse regression

• Majorization Minimization revisietd• IST- Iterative Shrinkage Thresolding Methods• TwIST-Two step IST


Linear Inverse Problems -LIPs


References

J. Bioucas-Dias and M. Figueiredo, "A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration“ Submitted to IEEE Transactions on Image processing, 2007.

M. Figueiredo, J. Bioucas-Dias, and R. Nowak, "Majorization-Minimization Algorithms for Wavelet-Based Image Deconvolution'', Submitted to IEEE Transactions on Image processing, 2006.


More References

M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE Trans. on Image Processing, vol. 12, no. 8, pp. 906–916, 2003.

J. Bioucas-Dias, “Bayesian wavelet-based image deconvolution: a GEM algorithm exploiting a class of heavy-tailed priors,” IEEE Trans. on Image Processing, vol. 15, pp. 937–951, 2006.

A. Chambolle, “An algorithm for total variation minimization andapplications,” Journal of Mathematical Imaging and Vision, vol. 20,pp. 89-97, 2004.

P. Combettes and V. Wajs, “Signal recovery by proximal forwardbackward splitting,” SIAM Journal on Multiscale Modeling & Simulation vol. 4, pp. 1168–1200, 2005

I. Daubechies, M. Defriese, and C. De Mol, “An iterative thresholding algorithm for linear inverse problems with a sparsity constraint”, Communications on Pure and Applied Mathematics, vol. LVII, pp. 1413-1457, 2004


Majorization Minorization (MM) Framework

Let

EM is an algorithm of this type.

Majorization Minorization algorithm:

....with equality if and only if

Easy to prove monotonicity:

Notes: should be easy to maximize


MM Algorithms for LIPs

IST Class: Majorize

IRS Class: Majorize

IST/IRS: Majorize and


MM Algorithms: IST class

Assume that



Majorizer:

Let:

IST Algorithm



Overrelaxed IST Algorithm

Convergence: [Combettes and V. Wajs, 2004]

• is convex •

• the set of minimizers, G, of is non-empty• 2 ]0,1]

• Then converges to a point in G


Denoising with convex regularizers

Denoising function also known as the Moreou proximal mapping

Classes of convex regularizers:

1- homogeneous (TV, lp-norm (p>1)) 2- p power of an lp norm


1-Homogeneous regularizers

Then

where is a closed convex set

and denotes the orthogonal projection on the convex set


Total variation regularization

Total variation [S. Osher, L. Rudin, and E. Fatemi, 1992]

is convex (although not strightly) and 1-homogeneous

Total variation is a discontinuity-preserving regularizer

have the same TV


Then

Total variation regularization

[Chambolle, 2004]


Total variation denoising


Total variation deconvolution

2000 IST iterations !!!


Weighted lp-norms

is convex (although not strightly) and 1-homogeneous

There is no closed form expression for excepts for some particular cases

Thus


Soft thresholding: p=1

Thus






Another way to look at it:

Since L is convex: the point is a global minimum of L iif

where is the subdifferential of L at f’


Example: Wavelet-based restoration

Wavelet basis

Wavelet coefficients Detail coefficients (h – high pass filter)

Approximation coefficients (g-low pass filter)

g,h – quadrature mirror filters

DWT, Harr, J=2


Example: Wavelet-based restoration

-500 -400 -300 -200 -100 0 100 200 300 400 5000

1

2

3

4

5

6

7

8

9

10

-150 -100 -50 0 50 100 1500

2000

4000

6000

8000

10000

12000

Histogram of coefficients - h Histogram of coefficients – log h

Documents

Shrinkage/Thresholding Iterative Methods