Compressive Sampling a.k.a. Compressed Sensing...Compressive Sampling: Sub-Nyquist sampling maybe possible. Two principles of Compressive Sampling 1 Sparsity: Signals are sparse in

Zentrum furTechnomathematik

Fachbereich 03Mathematik/Informatik

Compressive Samplinga.k.a. Compressed Sensing

Dennis Trede

Center for Industrial Mathematics (ZeTeM),University of Bremen, Germany

September 2009

Working Group Seminar 2009, Alghero

D. Trede 1 / 25

http://www.math.uni-bremen.de/zetem

http://www.uni-bremen.de



Outline

1 Introduction and Motivation

2 Sparsity and Incoherence

3 Undersampling and Sparse Signal Recovery

4 Compressive Sampling in the Presence of Noise

5 Single Pixel Camera

6 Summary

D. Trede 2 / 25





Shannon Sampling Theorem

Nyquist sampling rate sufficient:We don’t loose information if we sampletwice the maximal frequency in signal.

Compressive Sampling:Sub-Nyquist sampling maybe possible.

Two principles of Compressive Sampling

1 Sparsity:Signals are sparse in some transform domain.

2 Incoherence:Signals are dense in domain in which they are acquired.

nonadaptive, efficient sampling is possible

D. Trede 3 / 25





Shannon Sampling Theorem

Nyquist sampling rate sufficient:We don’t loose information if we sampletwice the maximal frequency in signal.

Compressive Sampling:Sub-Nyquist sampling maybe possible.

Two principles of Compressive Sampling

1 Sparsity:Signals are sparse in some transform domain.

2 Incoherence:Signals are dense in domain in which they are acquired.

nonadaptive, efficient sampling is possible

D. Trede 3 / 25





The Sensing Problem

Obtain information about a signal f by recording bycorrelating with waveforms ϕk ,

yk = 〈f , ϕk〉.

Examples in 1D:

I Sampling in time domain: Dirac’s peaksI Sampling in Fourier domain: Sinusoids

D. Trede 4 / 25





The Sensing Problem


yk = 〈f , ϕk〉.Examples in 1D:

I Sampling in time domain: Dirac’s peaks

I Sampling in Fourier domain: Sinusoids

D. Trede 4 / 25





The Sensing Problem


yk = 〈f , ϕk〉.Examples in 1D:

I Sampling in time domain: Dirac’s peaksI Sampling in Fourier domain: Sinusoids

D. Trede 4 / 25





The Discrete Sensing Problem

We restrict to discrete signals f ∈ Rn.(theory for continuous signals is sparsely developed)

Undersampling means

number m of measurements � dimension n of signal f

Reasons for undersampling: I number of sensors limitedI measurements extremely expensiveI sensing process slow

Questions:

Reconstruction of f from m� n samples?

Design of {ϕk}mk=1 to capture almost all information of f ?

How to approximate f from this information?

D. Trede 5 / 25







Undersampling means



Questions:




D. Trede 5 / 25







Undersampling means



Questions:




D. Trede 5 / 25





Linear Algebra Formulation

Solve underdetermined linear system of equations:

Measurements: correlations with waveforms ϕk ,

yk = 〈f , ϕk〉, k = 1, . . . ,m.

Let Φ denote m × n sensing matrix, Φ = [ϕ∗1, . . . , ϕ∗m].

Aim: recover f from

y = Φf ∈ Rm.

If m < n: infinitely many solutions.

Way out: The two principles of compressive sampling:

sparsity and incoherence.

D. Trede 6 / 25








yk = 〈f , ϕk〉, k = 1, . . . ,m.


Aim: recover f from

y = Φf ∈ Rm.




D. Trede 6 / 25








yk = 〈f , ϕk〉, k = 1, . . . ,m.


Aim: recover f from

y = Φf ∈ Rm.




D. Trede 6 / 25





Outline






6 Summary

D. Trede 7 / 25





Sparsity

Assume that f has sparse representation in ONB {ψi}ni=1, i.e.

f =n∑

i=1

xi ψi ,

with sparsely supported coefficients x = 〈f , ψi 〉ni=1, ‖x‖`0 � n.

Denote the n × n matrix with columns ψi by Ψ, i.e.

f = Ψx .

D. Trede 8 / 25





Sparsity

Assume that f has sparse representation in ONB {ψi}ni=1, i.e.

f =n∑

i=1

xi ψi ,

with sparsely supported coefficients x = 〈f , ψi 〉ni=1, ‖x‖`0 � n.Denote the n × n matrix with columns ψi by Ψ, i.e.

f = Ψx .

D. Trede 8 / 25





IncoherenceSuppose a pair (Φ,Ψ) of orthonormal bases of Rn,

Φ sensing basis, i.e. yk = 〈f , ϕk〉, ϕk ∈ Φ,Ψ representation basis, i.e. f =

∑ni=1 xi ψi , ψi ∈ Ψ.

Definition.

The coherence between the sensing basis Φ and the representationbasis Ψ is defined as

µ(Φ,Ψ) :=√n max1≤k, i≤n

|〈ϕk , ψi 〉|.

measures maximal correlation of elements of Φ and Ψµ(Φ,Ψ) ∈

[1,√n]

CS: Choose (Φ,Ψ) such that coherence is small

D. Trede 9 / 25





IncoherenceSuppose a pair (Φ,Ψ) of orthonormal bases of Rn,

Φ sensing basis, i.e. yk = 〈f , ϕk〉, ϕk ∈ Φ,Ψ representation basis, i.e. f =

∑ni=1 xi ψi , ψi ∈ Ψ.

Definition.

The coherence between the sensing basis Φ and the representationbasis Ψ is defined as

µ(Φ,Ψ) :=√n max1≤k, i≤n

|〈ϕk , ψi 〉|.

measures maximal correlation of elements of Φ and Ψµ(Φ,Ψ) ∈

[1,√n]

CS: Choose (Φ,Ψ) such that coherence is smallD. Trede 9 / 25





Examples for Low Coherence Pairs

1 I spike basis Φ ={ϕk(t)

}={δ(t − k)

}I Fourier basis Ψ =

{ψk(t)

}={n−1/2 exp(2πikt/n)

}yields to µ(Φ,Ψ) = 1, i.e. maximal incoherence.

2 I random matrix ΦI any fixed basis Ψyields with a high probability to µ(Φ,Ψ) ≈

√2 log n

D. Trede 10 / 25





Examples for Low Coherence Pairs

1 I spike basis Φ ={ϕk(t)

}={δ(t − k)

}I Fourier basis Ψ =

{ψk(t)

}={n−1/2 exp(2πikt/n)

}yields to µ(Φ,Ψ) = 1, i.e. maximal incoherence.

2 I random matrix ΦI any fixed basis Ψyields with a high probability to µ(Φ,Ψ) ≈

√2 log n

D. Trede 10 / 25





Outline






6 Summary

D. Trede 11 / 25





Undersampling and Sparse Signal Recovery

Just observe a subset M of correlations, M ⊂ {1, . . . , n},yk = 〈f , ϕk〉, k ∈ M.

|M| =: m < n =⇒ underdetermined system of equations

Pick up minimum-`1-solution, i.e.

minx∈Rn‖x‖`1 subject to yk = 〈Ψx , ϕk〉, ∀ k ∈ M. (1)

Theorem (Candes, Romberg [2]).

Let f ∈ Rn, f = Ψx , ‖x‖`0 = S . Suppose that

m ≥ C µ2(Φ,Ψ)S log n,

for some constant C > 0, then the solution of (1) is exact withoverwhelming probability. Y

D. Trede 12 / 25












for some constant C > 0, then the solution of (1) is exact withoverwhelming probability. Y

D. Trede 12 / 25












for some constant C > 0, then the solution of (1) is exact withoverwhelming probability.

Y

D. Trede 12 / 25












for some constant C > 0, then the solution of (1) is exact withoverwhelming probability. YD. Trede 12 / 25





Overwhelming Probability?

Introducing a probability model:I select subset M of measurements uniformly at randomI choose sign sequence z with supp(z) = supp(x) u.a.r.

Theorem: Suppose that

m ≥ C µ2(Φ,Ψ)S log(n/δ),

with constant C = C (M, z) > 0, then with probabilityexceeding 1− δ the signal can be recovered.

Weak uncertainty principle for general ONBs:one cannot be sparse on sets in Ψ and Φ simultaneously.

What does that mean for praxis?

D. Trede 13 / 25








m ≥ C µ2(Φ,Ψ) S log(n/δ),




D. Trede 13 / 25












D. Trede 13 / 25












D. Trede 13 / 25





Example: Noiselet Measurement (from [2])

1, 024× 1, 024 = 1, 048, 576 pixels,S = 25, 000 Haar wavelet coefficients,

m = 70, 000 randomly chosen noiselet coefficients observed, Yfor Haar wavelets and noiselets it holds: µ = 1,m ≈ 4S exact recovery.

D. Trede 14 / 25





Outline






6 Summary

D. Trede 15 / 25





Compressive Sampling in the Presence of NoiseRecovery x ∈ Rn from y ∈ Rm with

y = Ax + z ,

where A ∈ Rm×n is the sensing matrix, and z is error termwith bounded noise ‖z‖ ≤ ε.

Setup coincides with noiseless case for z = 0:

with sensing matrix Φ = [ϕ∗1, . . . , ϕ∗n].

`1-minimization with relaxed constraints (basis pursuit)

minx∈Rn‖x‖`1 subject to ‖Ax − y‖`2 ≤ ε.

D. Trede 16 / 25






y = Ax + z ,

where A ∈ Rm×n is the sensing matrix, and z is error termwith bounded noise ‖z‖ ≤ ε.Setup coincides with noiseless case for z = 0:

yk = 〈Ψx , ϕk〉, ∀ k ∈ M.




D. Trede 16 / 25






y = Ax + z ,


y = PM Φ Ψ x ,




D. Trede 16 / 25






y = Ax + z ,


y = PM Φ Ψ x ,




D. Trede 16 / 25





Restricted Isometry Property(RIP)

For S ∈ N, define the isometry constant δS as the smallestnumber such that for all S-sparse vectors x ,

(1− δS)‖x‖2`2 ≤ ‖Ax‖2`2 ≤ (1 + δS)‖x‖2`2 .

Sparsity and the RIP yields to linear convergence rates:

Theorem (Candes, Romberg, Tao [3]).

Assume that δ3S + 3δ4S < 2. Then for x , ‖x‖`0 ≤ S , and z ,‖z‖ ≤ ε, the solution x∗ of

min ‖x‖`1 subject to ‖Ax − y‖`2 ≤ ε

obeys‖x∗ − x‖`2 = O(ε).

D. Trede 17 / 25







(1− δS)‖x‖2`2 ≤ ‖Ax‖2`2 ≤ (1 + δS)‖x‖2`2 .





obeys‖x∗ − x‖`2 = O(ε).

D. Trede 17 / 25







(1− δS)‖x‖2`2 ≤ ‖Ax‖2`2 ≤ (1 + δS)‖x‖2`2 .





obeys‖x∗ − x‖`2 = O(ε).

D. Trede 17 / 25





Outline






6 Summary

D. Trede 18 / 25





Traditional vs. Compressive Sampling

traditional digital data acquisition:I sample and compress (e.g. JPEG & JPEG-2000)

sample → compress → transmit/store → decompress →

compressive sampling data acquisition:I measure linear projections onto incoherent basis (e.g. Haar)

project → transmit/store → reconstruct →

D. Trede 19 / 25










D. Trede 19 / 25










D. Trede 19 / 25





Single Pixel Camera

new camera architecture that works with thetheory and algorithms of compressive sampling

single detector camera (measures 〈f , ϕk〉)digital mirror device (represents inner products 〈f , ϕk〉) optical computation of projections onto incoherent basis

from [5] from

[6]

Φ

D. Trede 20 / 25





Single Pixel Camera

new camera architecture that works with thetheory and algorithms of compressive samplingsingle detector camera (measures 〈f , ϕk〉)digital mirror device (represents inner products 〈f , ϕk〉) optical computation of projections onto incoherent basis

from [5] from

[6]

Φ

D. Trede 20 / 25





Single Pixel Camera


from [5]

from

[6]

Φ

D. Trede 20 / 25





Single Pixel Camera


from [5] from

[6]

Φ

D. Trede 20 / 25





Imaging Results with the Single Pixel Camera

ideal image

400 wavelets 675 wavelets

1600 measurements 2700 measurements

from

[6]

64× 64 pixelS-term Haar

wavelets approx.768× 768

DMD pixelm ≈ 4S

D. Trede 21 / 25






ideal image 400 wavelets

675 wavelets

1600 measurements

2700 measurements

from

[6]



DMD pixelm ≈ 4S

D. Trede 21 / 25






ideal image 400 wavelets 675 wavelets

1600 measurements 2700 measurements

from

[6]



DMD pixelm ≈ 4S

D. Trede 21 / 25





Outline






6 Summary

D. Trede 22 / 25





Summary

Compressive Sensing makes Sub-Nyquist sampling possible,with two principles sparsity and incoherence.

Theory in finite dimensions ensuresprobabilistic statements for exact data,linear convergence rates for noisy data.

Theory in infinite dimensions is sparsely developed.

Connection to inverse problems:Basis pursuit is related to `1-penalized Tikhonov regularization,

min ‖x‖`1 s.t. ‖Ax − y‖`2 ≤ ε ⇐⇒ min ‖Ax − y‖2`2 +α‖x‖`1 .RIP constant δS for infinite dimensional ill-posed operatorstypically big,

(1− δS)‖x‖2`2 ≤ ‖Ax‖2`2 ≤ (1 + δS)‖x‖2`2 .

Hot topic!

D. Trede 23 / 25





Summary



Theory in infinite dimensions is sparsely developed.Connection to inverse problems:

Basis pursuit is related to `1-penalized Tikhonov regularization,


(1− δS)‖x‖2`2 ≤ ‖Ax‖2`2 ≤ (1 + δS)‖x‖2`2 .

Hot topic!

D. Trede 23 / 25





Summary



Theory in infinite dimensions is sparsely developed.Connection to inverse problems:

Basis pursuit is related to `1-penalized Tikhonov regularization,


(1− δS)‖x‖2`2 ≤ ‖Ax‖2`2 ≤ (1 + δS)‖x‖2`2 .

Hot topic!D. Trede 23 / 25





Bibliography[1] Candes and Wakin. An introduction to compressive sampling. IEEE Signal

Processing Magazine, 25(2):21–30, 2008.

[2] Candes and Romberg. Sparsity and incoherence in compressive sampling.Inverse Problems, 23(3):969–985, 2007.

[3] Candes, Romberg and Tao. Stable Signal Recovery from Incomplete andInaccurate Measurements. Communications on Pure and AppliedMathematics, 59(8):1207–1223, 2006.

[4] Romberg. Imaging via compressive sampling. IEEE Signal ProcessingMagazine, 25(2):14–20, 2008.

[5] Takhar, Laska, Wakin, Duarte, Baron, Sarvotham, Kelly, Baraniuk. A newcompressive imaging camera architecture using optical-domain compression.In Proc. Computational Imaging IV, vol. 6065, pp. 43–52, 2006.

[6] Duarte, Davenport, Takhar, Laska, Sun, Kelly, Baraniuk. Single-pixelimaging via compressive sampling. IEEE Signal Processing Magazine,25(2):83–91, 2008.

D. Trede 24 / 25





Thank you for your interest!

Dennis [email protected]

D. Trede 25 / 25





Noiselets

from

http://laurent-duval.blogspot.co

m

ˇD. Trede 25 / 25



Documents

Compressive Sampling a.k.a. Compressed Sensing...Compressive Sampling: Sub-Nyquist sampling maybe possible. Two principles of Compressive Sampling 1 Sparsity: Signals are sparse in