Lausanne, August 19, 2004

Dear Dr. Liebling,

I am pleased to inform you that you were selected to the receive the 2004 Research Award of the

Swiss Society of Biomedical Engineering for your thesis work “On Fresnelets, interference

fringes, and digital holography”. The award will be presented during the general assembly of the

SSBE, September 3, Zurich, Switzerland.

Please, lets us know if

1) you will be present to receive the award,

2) you would be willing to give a 10 minutes presentation of the work during the general

assembly.

The award comes with a cash prize of 1000.- CHF.

Would you please send your banking information to the treasurer of the SSBE, Uli Diermann

(Email:uli.diermann@bfh.ch), so that he can transfer the cash prize to your account ?

I congratulate you on your achievement.

With best regards,

Michael Unser, Professor

Chairman of the SSBE Award Committee

cc: Ralph Mueller, president of the SSBE; Uli Diermann, treasurer

Dr. Michael Liebling

Biological Imaging Center

California Inst. of Technology

Mail Code 139-74

Pasadena, CA 91125, USA

BIOMEDICAL IMAGING GROUP (BIG)

LABORATOIRE D’IMAGERIE BIOMEDICALE

EPFL LIB

Bât. BM 4.127

CH 1015 Lausanne

Switzerland

Téléphone :

Fax :

E-mail :

Site web :

+ 4121 693 51 85

+ 4121 693 37 01

Michael.Unser@epfl.ch

http://bigwww.epfl.ch

Sparse stochastic processes

Part I: Introduction

Michael UnserBiomedical Imaging GroupEPFL, Lausanne, Switzerland

Lausanne, August 19, 2004

Dear Dr. Liebling,

I am pleased to inform you that you were selected to the receive the 2004 Research Award of the

Swiss Society of Biomedical Engineering for your thesis work “On Fresnelets, interference

fringes, and digital holography”. The award will be presented during the general assembly of the

SSBE, September 3, Zurich, Switzerland.

Please, lets us know if

1) you will be present to receive the award,

2) you would be willing to give a 10 minutes presentation of the work during the general

assembly.

The award comes with a cash prize of 1000.- CHF.

Would you please send your banking information to the treasurer of the SSBE, Uli Diermann

(Email:uli.diermann@bfh.ch), so that he can transfer the cash prize to your account ?

I congratulate you on your achievement.

With best regards,

Michael Unser, Professor

Chairman of the SSBE Award Committee

cc: Ralph Mueller, president of the SSBE; Uli Diermann, treasurer

Dr. Michael Liebling

Biological Imaging Center

California Inst. of Technology

Mail Code 139-74

Pasadena, CA 91125, USA

BIOMEDICAL IMAGING GROUP (BIG)

LABORATOIRE D’IMAGERIE BIOMEDICALE

EPFL LIB

Bât. BM 4.127

CH 1015 Lausanne

Switzerland

Téléphone :

Fax :

E-mail :

Site web :

+ 4121 693 51 85

+ 4121 693 37 01

Michael.Unser@epfl.ch

http://bigwww.epfl.ch

Invited seminar, Korea Advanced Inst. of Science & Technology (KAIST), July 17-18, Seoul, Korea

Fig. 5. Rate versus distortion for various transforms for a second-order Gauss- Markov process ( p = 0.95, N = 256).

modulation the cross-modulation loss. improvement system can it conventional

20th century statistical signal processing

2

Karhunen-Loève transform (KLT) is optimal for compression

Hypothesis: Signal = stationary Gaussian process

(Pearl et al., IEEE Trans. Com 1972)

DCT asymptotically equivalent to KLT(Ahmed-Rao, 1975; U., 1984)

20th century statistical signal processing

3

Wiener filter is optimal for restoration/denoising

Hypothesis: Signal = Gaussian process

Signal covariance: Cs = E{s · sT }y = Hs+ n

sMAP = argmins1

�2ky �Hsk22

| {z }Data Log likelihood

+ kC�1/2s sk22| {z }

Gaussian prior likelihood

Wiener (LMMSE) solution = Gauss MMSE = Gauss MAP

Noise: i.i.d. Gaussian with variance �2

sLMMSE = CsHT�HCsH

T + �2I��1

y = FWiener y

, quadratic regularization (Tikhonov) �kLsk22

m L = C�1/2s : Whitening filter

Then came wavelets ...

4

Stéphane Mallat Ingrid Daubechies

Martin Vetterli

David Donoho

Alfred Haar

1910//

1982

1987-88

Sparsity Compressed sensing

1994

2006

Applications

Emmanuel Candès

and sparsity

Yves Meyer

Fact 1: Wavelets can outperform Wiener filter

5

�

2

w̃ = T�(w)

w

Fact 2: Wavelet coding can outperform jpeg

6

Wavelet transform

Inverse wavelet transform

Discarding “small coefficients”

f(x) =X

i,k

�i,k(x)wi,k

(Shapiro, IEEE-IP 1993)

Fact 3: l1 schemes can outperform l2 optimization

7

s? = argmin ky �Hsk22| {z }data consistency

+ �R(s)| {z }regularization

(Nowak et al., Daubechies et al. 2004)

(Rudin-Osher, 1992)`1 regularization (Total variation)

R(s) = kLsk`1 with L: gradient

(Candes-Romberg-Tao; Donoho, 2006)Compressed sensing/sampling

Wavelet-domain regularization

Wavelet expansion: s = Wv (typically, sparse)

Wavelet-domain sparsity-constraint: R(s) = kvk`1 with v = W�1s

SPARSE STOCHASTIC MODELS:                         The step beyond Gaussianity

8

Requirements for a comprehensive statistical framework

Backward compatibility

Continuous-domain formulation

piecewise-smooth signals, translation and scale-invariance, sampling . . .

Predictive power

Can wavelets really outperform sinusoidal transforms (KLT) ?

Statistical justification and refinement of current algorithms

Sparsity-promoting regularization, `1 norm minimization

1.3 FROM SPLINES TO STOCHASTIC PROCESSES

9

■ Splines and Legos revisited■ Higher-order polynomial splines■ Random splines, innovations, Lévy processes■ Wavelet analysis of Lévy processes■ Lévy’s synthesis of Brownian motion

Splines and Legos revisited

10

Ê

Ê ÊÊ

Ê

Ê

Ê

ÊÊ

Ê

Ê

0 2 4 6 8 10

Cardinal spline of degree 0: piecewise-contant

�0+(t) =

(1, for 0 t < 1

0, otherwise.

Notion of D-spline:

Df1(t) =X

k2Za1[k]�(t� k)

f1(t) =X

k2Zf1[k]�

0+(t� k)

B-spline and derivative operator

11

1 2 3 4 5

1

�0+(t) = +(t)� +(t� 1)

�0+(t) = DdD

�1�(t) = Dd +(t)

Finite difference operator

B-spline of degree 0

Derivative Df(t) =df(t)

dtD

F ! j!

Ddf(t) = f(t)� f(t� 1) DdF ! 1� e�j!

= (�0+ ⇤Df)(t)

�̂0+(!) =

1� e�j!

j!

l

Random splines and innovations

12

Ê

Ê ÊÊ

Ê

Ê

Ê

ÊÊ

Ê

Ê

0 2 4 6 8 10

0 2 4 6 8 10

Anti-derivative operators

Shift-invariant solution: D�1'(t) = ( + ⇤ ')(t) =Z t

�1'(⌧)d⌧

Scale-invariant solution: D�10 '(t) =

Z t

0'(⌧)d⌧

non-uniform

cardinal

Ds(t) =X

n

an�(t� tn) = w(t)

Random weights {an} i.i.d. and random knots {tn} (Poisson with rate �)

Impulse response

Translation invariance

Linearity

Innovation-based synthesis

13

L�1{·}

L�1{·}

L�1{·}

⇢ = L�1�

L = ddt = D ) L�1

: integrator

�(t)

�(t� t0)

X

n

an�(t� tn)

⇢(t� t0)

s(t) =X

n

an⇢(t� tn)

Compound Poisson process

14

0 2 4 6 8 10

Stochastic differential equation

Ds(t) = w(t)

with boundary condition s(0) = 0

Innovation: w(t) =X

n

an�(t� tn)

s(t) = D�10 w(t) =

X

n

anD�10 {�(·� tn)}(t)

=X

n

an�

+(t� tn)� +(�tn)�

(impose boundary condition)

Formal solution

Generalization: Lévy processes

15

(unstable SDE !)

0.0 0.2 0.4 0.6 0.8 1.0

0 0

0.0 0.2 0.4 0.6 0.8 1.00 0

0.0 0.2 0.4 0.6 0.8 1.0

0 0

Compound Poisson

Brownian motion

Integrator

Gaussian

Impulsive Z t

0d⌧

Lévy flight

s(t)w(t)

White noise (innovation) Lévy process

S↵S (Cauchy)

(Paul Lévy circa 1930)

(Wiener 1923)

Generalized innovations : white Lévy noise with E{w(t)w(t0)} = �2w�(t� t0)

Ds = w

s = D�10 w , 8' 2 S, h', si = hD�1⇤

0 ', wi

Decoupling Lévy processes: increments

16

u(t) = Dds(t) = DdD�10 w(t) = (�0

+ ⇤ w)(t).

Increment process is stationary with autocorrelation function

u[k] = s(k)� s(k � 1) = hw,�0+(·� k)i.

Discrete increments

h , iu[k] are i.i.d. because

{�0+(·� k)} are non-overlapping

w is independent at every point (white noise)

*⇤ ⇥� ⌅

Ru(⌧) = E{u(t)u(t+ ⌧)} =��0+ ⇤ (�0

+)_ ⇤Rw

�(y)

= �2w�

1+(⌧ � 1)

Increment process:

Wavelet analysis of Lévy processes

17

Haar wavelets

Haar(t) =

8><

>:

1, for 0 t < 12

�1, for 12 t < 1

0, otherwise.

i,k(t) = 2�i/2 Haar

✓t� 2ik

2i

◆

8><

>:

2,0

1,0

0,0 0,2

i = 0

i = 1

i = 2

8><

>:

(

Wavelets as multi-scale derivatives

18

i,k = 2i/2�1D�i,k

D�10 i,k = 2i/2�1�i,k.

Yi,k = hs, i,ki / hs,D�i,ki

/ hD⇤s,�i,ki = �hw,�i,ki

8><

>:

2,0

1,0�1,0

0,0 0,2 �0,2�0,0

i = 0

i = 1

i = 2

8><

>:

(

8><

>:

i = 0

i = 1

i = 2

8><

>:

(�2,0

Wavelet coefficients of Lévy process

10−3 10−2 10−1 100

10−15

10−10

10−5

100

Identity KLT Haar

10−3 10−2 10−1 100

10−10

10−5

100

Identity KLT Haar

10−3 10−2 10−1 100

10−10

10−5

100

Identity KLT Haar

Brownian motion

Compound Poisson

Lévy flight (Cauchy)

19

M-term approximation: wavelets vs. KLT

Gaussian

Finite rate of innovation

Even sparser ...

Wavelet-based synthesis of Brownian motion

20

White Gaussian noise

w =X

i2Z

X

k2ZZi,k i,k with Zi,k = hw, i,ki

s(t) = D�10 w

=X

i2Z

X

k2ZZi,kD

�10 i,k(t)

=X

i2Z

X

k2Z2i/2�1Zi,k�i,k(t)

Brownian motion

EDEE Course 21

http://www.sparseprocesses.orgebook: web preprint

One puff to follow, one puff to follow, one puff to follow, one puff to follow, one puff to follow, one puff to follow, one puff to follow…

AUTHOR NAME, affiliation

Providing a novel approach to sparse stocastic processes, this comprehen-sive book presents the theory of stochastic process that are ruled by stochastic differential equations, and that admit a parsimonious represen- tation in a matched wavelet-like basis. Two key themes are the statistical property of infinite divisibility, which leads to two distinct types of behavior – Gaussian and sparse – and the structural link between linear stochastic processes and spline functions, which is exploited to simplify the mathematical analysis. The core of the book is devoted to investigating sparse processes, including a complete description of their transform-domain statistics. The final part develops practical signal-processing algorithms that are based on these models, with special emphasis on biomedical image reconstruction. This is an ideal reference for graduate students and researchers with an interest in signal/image processing, compressed sensing, approximation theory, machine learning, or statistics.

MICHAEL UNSER is Professor and Director of EPFL’s Biomedical Imaging Group, Switzerland. He is a member of the Swiss Academy of Engineering Sciences, a fellow of EURASIP, and a fellow of the IEEE.

POUYA D. TAFTI is a researcher at Qlaym BmbH, Düsseldort, and a former member of the Biomedical Imaging Group at EPFL, where he conducted research on the theory and applications of probablilistic models for data.

Unser and Tafti

An Introduction to

Sparse Stochastic Processes

Cover illustration: to follow

An Introduction to Sparse Stochastic ProcessesMichael Unser and Pouya D. Tafti

Unser and Tafti. 9781107058545 PPC

. C M

Y K

Extension to fractional orders: B-splines

22

L = D�(fractional derivative) Fourier multiplier: (j!)�

Ld = ��+ (fractional differences) Fourier multiplier: (1� e�j!)�

�0+(r) = �+r

0+

F ! 1� e�j!

j!

�↵+(r) =

�↵+1+ r↵+

�(↵+ 1)F !

✓1� e�j!

j!

◆↵+1

One-sided power function: r↵+ =

(r↵, r � 0

0, r < 0

......

Degree ↵ = � � 1

Mandelbrot’s fractional Brownian motion

23

Hurst exponent: H = � � 12

Solution of fractional SDE (Blu-U., 2007)

D�s = w where w is Gaussian white noise

) s = I�2w

I�2 : scale-invariant, right-inverse of D�

Continuous-domain innovation model

24

Main outcome: non-Gaussian solutions are necessarily sparse (infinitely divisible)

w(x)

Generalized white noise Stochastic process

L{·}

Shaping filter

(appropriate boundary conditions)

Whitening operator

L�1{·}

s(x),x 2 Rd

Why? ... as will explained in next chapters ...(invoking powerful theorems in functional analysis:

Bochner-Minlos, Gelfand, Schoenberg & Lévy-Khinchine)

2.1 DECOUPLING OF SPARSE PROCESSES

25

s = L�1w , w = Ls

■ Discrete approximation of operator

■ Operator-like wavelet analysis

Decoupling: Linear combination of samples

26

Input: s(k),k 2 Zd (sampled values)

Discrete approximation of whitening operator: Ld

Ld�(x) =X

k2Zd

dL[k]�(x� k)

Generalized B-spline:

�L(x) = LdL�1�(x) A-to-D translator

Discrete increment process:

u[k] = Lds(x)|x=k

= (�L ⇤ w)(x)|x=k

= h�_L (·� k), wi| {z }

'

s = L�1w

Decoupling: Wavelet analysis

27

Generalized operator-like wavelets:

i(x) = L⇤�i(x)

Operator-like wavelet analysis of sparse process:

(Khalidov-U. 2006, Ward-U. JFAA 2013)

Ls = w

h i(·� x0), si =hL⇤�i(·� x0), si

=h�i(·� x0),Ls)i

=h�i(·� x0)| {z }'

, wi = (�_i ⇤ w)(x0)