Compressive Sampling(of Analog Signals)

Moshe Mishali Yonina C. Eldar

Technion – Israel Institute of Technology

http://www.technion.ac.il/~moshikomoshiko@tx.technion.ac.ilhttp://www.ee.technion.ac.il/people/YoninaEldar yonina@ee.technion.ac.il

Advanced topics in sampling (Course 049029)Seminar talk – November 2008

2

Context - Sampling

Digital worldAnalog world

Continuous signal

ReconstructionD2A

SamplingA2D

3

Compression

Original 2500 KB100%

Compressed 950 KB38%

Compressed 392 KB15%

Compressed 148 KB6%

“Can we not just directly measure the part that will not end up being thrown away ?”

Donoho

4

Outline

• Mathematical background

• From discrete to analog

• Uncertainty principles for analog signals

• Discussion

5

References• M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering

Arbitrary Sets of Jointly Sparse Vectors," IEEE Trans. on Signal Processing, vol. 56, no. 10, pp. 4692-4702, Oct. 2008.

• M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals," CCIT Report #639, Sep. 2007, EE Dept., Technion.

• Y. C. Eldar, "Compressed Sensing of Analog Signals", submitted to IEEE Trans. on Signal Processing, June 2008.

• Y. C. Eldar and M. Mishali, "Robust Recovery of Signals From a Union of Subspaces", arXiv.org 0807.4581, submitted to IEEE Trans. Inform. Theory, July 2008. #

• Y. C. Eldar, "Uncertainty Relations for Analog Signals", submitted to IEEE Trans. Inform. Theory, Sept. 2008.

6

Mathematical background

• Basic ideas of compressed sensing• Single measurement model (SMV)• Multiple- and Infinite- measurement models (MMV, IMV)• The “Continuous to finite” block (CTF)

7

Compressed Sensing

“Can we not just directly measure the part that will not end up being thrown away ?”

Donoho

“sensing … as a way of extracting information about an object from a small number of randomly selected observations”Candès et. al.

Nyquist rateSampling

AnalogAudioSignal

Compression(e.g. MP3)

High-rate Low-rateCompressedSensing

8

Concept

Goal: Identify the bucket with fake coins.

Nyquist: Weigh a coinfrom each bucket

CompressionBucket #

numbers 1 number

Compressed Sensing: Bucket #

1 number

Weigh a linear combinationof coins from all buckets

9

Mathematical Tools

y A x

non-zero entries at least measurements

Recovery: brute-force, convex optimization, greedy algorithms, and more…

10

CS theory – on 2 slides

Compressed sensing (2003/4 and on) – Main results

Maximal cardinality of linearly independent column subsets

Hard to compute !

is uniquely determined by

Donoho and Elad, 2003

11

is uniquely determined by

is random with high probabilityDonoho, 2006 and Candès et. al., 2006

NP-hard

Convex and tractable

Greedy algorithms: OMP, FOCUSS, etc.

Donoho, 2006 and Candès et. al., 2006

Tropp, Cotter et. al. Chen et. al. and many other

Compressed sensing (2003/4 and on) – Main results

CS theory – on 2 slides

Donoho and Elad, 2003

12

IMV = Infinite Measurement Vectors (countable or uncountable)with joint sparsity prior

How can be found ?

Sparsity models

measurements unknowns

SMV MMV Joint sparsity

Infinite many variables

Exploit prior Reduce problem dimensions

Infinite many constraints

13

Reduction Framework

Find a frame for

Solve MMV

Mishali and Eldar (2008)

Theorem

IMV MMV

Deterministicreduction

Infinite structure allowsCS for analog signals

14

From discrete to analog

• Naïve extension• The basic ingredients of sampling theorem

• Sparse multiband model• Rate requirements• Multicoset sampling and unique representation• Practical recovery with the CTF block

• Sparse union of shift-invariant model• Design of sampling operator• Reconstruction algorithm

15

Naïve Extension to Analog Domain

Standard CSDiscrete Framework

Analog Domain

Sparsity prior what is a sparse analog signal ?

Generalized sampling

Finite dimensional elements Infinitesequence

Continuoussignal

Operator

Random is stable w.h.p Stability Randomness Infinitely manyNeed structure for efficient implementation

Finite program, well-studied Undefined program over a continuous signal

Reconstruction

16

Random is stable w.h.p

Naïve Extension to Analog Domain

Sparsity prior what is a sparse analog signal ?

Generalized sampling

Finite dimensional elements Infinitesequence

Continuoussignal

Operator

Stability Randomness Infinitely manyNeed structure for efficient implementation

Finite program, well-studied Undefined program over a continuous signal

Reconstruction

Questions:

1. What is the definition of analog sparsity ?

2. How to select a sampling operator ?

3. Can we introduce stucture in sampling and still preserve stability ?

4. How to solve infinite dimensional recovery problems ?

Standard CSDiscrete Framework

Analog Domain

17

A step backward

Nyquist1928

Shannon1949

Kotelnikov1933

“Success has many fathers …”

Whittaker1915

Every bandlimited signal ( Hertz)

can be perfectly reconstructed from uniform sampling

if the sampling rate is greater than

18

A step backward

Every bandlimited signal ( Hertz)

can be perfectly reconstructed from uniform sampling

if the sampling rate is greater than

• A signal model• A minimal rate requirement• Explicit sampling and reconstruction stages

Fundamental ingredients of a sampling theorm

19

Discrete Compressed Sensing

Analog Compressive Sampling

20

Analog Compressed Sensing

A signal with a multiband structure in some basis

no more than N bands, max width B, bandlimited to

(Mishali and Eldar 2007)

1. Each band has an uncountable number of non-zero elements

2. Band locations lie on an infinite grid

3. Band locations are unknown in advance

What is the definition of analog sparsity ?

21

Multi-Band Sensing: Goals

bands

Sampling Reconstruction

Goal: Perfect reconstruction

Constraints:

1. Minimal sampling rate

2. Fully blind system

Analog Infinite Analog

What is the minimal rate ?What is the sensing mechanism ?

How to reconstruct from infinite sequences ?

22

Rate Requirement

Average sampling rate

Theorem (non-blind recovery)

Subspace scenarios:Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98) Half blind system (Herley and Wong 99, Venkataramani and Bresler 00)

Landau (1967)

1. The minimal rate is doubled.2. For , the rate requirement is samples/sec (on average).

23

Sampling

Analog signal

In each block of samples, only are kept, as described by

Point-wise samples

02

30

0

2

2

3

3

Multi-Coset: Periodic Non-uniform on the Nyquist grid

Bresler et. al. (96,98,00,01)

24

The Sampler

DTFTof sampling sequences

Constant

matrixknown

in vector form

unknowns

Length .knownProblems:

1. Undetermined system – non unique solution

2. Continuous set of linear systems

is jointly sparse and unique under appropriate parameter selection ( )

is sparse

Observation:

25

Paradigm

Solve finiteproblem

Reconstruct

0

1

2

3

4

5

6

S = non-zero rows

26

CTF block

Solve finiteproblem

Reconstruct

MMV

Continuous to Finite

Continuous

Finite

span a finite spaceAny basis preserves the sparsity

27

Algorithm

CTF

Continuous-to-finite block: Compressed sensing for analog signals Perfect reconstruction at minimal rate

Blind system: band locations are unkown

Can be applied to CS of general analog signals

Works with other sampling techniques

28

Blind reconstruction flow

Spectrum-blindSampling

No

Yes

Spectrum-blindReconstruction

Uniform at

Multi-coset with

Universal

Ideal low-pass filter

SBR4

CTF

SBR2

CTFBi-section

Yes

No

29Bresler et. al. (96,00)

Final reconstruction (non-blind)

30

Framework: Analog Compressed Sensing

Sampling signals from a union of shift-invariant spaces (SI)

generators

Subspace

31

Framework: Analog Compressed Sensing

What happen if only K<<N sequences are not zero ?

There is no prior knowledge on the exact indices in the sum

Not a subspace !

Only k sequences are non-zero

32

Framework: Analog Compressed Sensing

Only k sequences are non-zero

CTF

Step 1: Compress the sampling sequencesStep 2: “Push” all operators to analog domain

System AHigh sampling rate = m/T

Post-compression

33

Framework: Analog Compressed Sensing

CTF

Eldar (2008)

Theorem

System BLow sampling rate = p/T

Pre-compression

34

Does it work ?

35

Simulations

Brute-Force

M-OMP

5 10 150

0.2

0.4

0.6

0.8

1

r

Em

pir

ica

l s

uc

ce

ss

ra

te

SBR4SBR2

5 10 150

0.2

0.4

0.6

0.8

1

r

Em

pir

ica

l s

uc

ce

ss

ra

te

SBR4SBR2

Sampling rate Sampling rate

Minimal rate Minimal rate

36

Simulations (2)

SBR4 SBR2

r

SN

R

5 10 15

10

15

20

0.2

0.4

0.6

0.8

1

rS

NR

5 10 15

10

15

20

0.2

0.4

0.6

0.8

1

r

SN

R

5 10 15

10

15

20

50 100 150 200 250

0.2 0.4 0.6 0.8 1

0.2 0.4 0.6 0.8 1

Empirical recovery rate

Sampling rate Sampling rate

0% Recovery 100% Recovery 0% Recovery 100% Recovery

Noise-free

37

Simulations (3)

0 5 10 15 20 25

1

2

3

4

5

x 104

Time (nano secs)

-50 0 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

n

0 5 10 15 20 25

1

2

3

4

5

x 104

Time (nano secs)

Signal Reconstruction filter

Output

Time (nSecs)

Time (nSecs)

Am

plitu

de

Am

plitu

de

38

Break(10 min. please)

39

Uncertainty principles

• Coherence and the discrete uncertainty principle• Analog coherence and principles• Achieving the lower coherence bound• Uncertainty principles and sparse representations

40

The discrete uncertainty principle

Uncertainty principle

41

Discrete coherence

Which bases achieve the lowest coherence ?

42

Discrete coherence

Which signal achieves the uncertainty bound ?

Spikes Fourier

43

Discrete to analog

• Shift invariant spaces

• Sparse representations

Questions:• What is the analog uncertainty principle ?• Which bases has the lowest coherence ?• Which signal achieves the lower uncertainty bound ?

44

Analog uncertainty principle

Eldar (2008)

Theorem

45

Bases with minimal coherenceIn the DFT domain

Spikes Fourier

What are the analog counterparts ?

• Constant magnitude• Modulation

• “Single” component• Shifts

46

Bases with minimal coherence

In the frequency domain

47

Tightness

48

Sparse representations

• In discrete setting

49

Sparse representations

• Analog counterparts

Undefined program !

But, can be transformed into an IMV model

50

Discussion

• IMV model as a fundamental tool for treating sparse analog signals

• Should quantify the DSP complexity of the CTF block

• Compare approach with the “analog” model

• Building blocks of analog CS framework.

51

Thank you