Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Iterative Hard Thresholding: Theory and Practice
TH
E
U N I V E R S
I TY
OF
ED I N B U
RG
H
T. BlumensathInstitute for Digital Communications
Joint Research Institute for Signal and Image ProcessingThe University of Edinburgh
February, 2009
home · prev · next · page
IV) ResultsUp there with the best
III) Stability Beyond the Grave (RIP)Operating beyond the limits
II) Theory and PracticeNice theory, bad attitude
I) Iterative Hard Thresholding for Sparse Inverse ProblemsWhat’s the probelm?
y = Φx + e
What to Expect
home · prev · next · page 1 of 20
y = Φx + e
PART 1
Iterative Hard Thresholding for Sparse InverseProblems
home · prev · next · page 2 of 20
THE SOLUTION: The Iterative Hard Thresholding (IHT) algorithm uses theiteration
xn+1 = PK(xn + ΦT (y −Φxn)),
where PK is a hard thresholding operator that keeps the largest (in magnitude)K elements of a vector (Or, more generally, a projector onto the closest elementin the model).
THE PROBLEM: Giveny = Φx + e,
where y ∈ RM , x ∈ RN , Φ ∈ RM×N and e is observation noise, estimate xgiven y and Φ when M << N but x is approximately K-sparse.
y = Φx + e
The Problem and Solution
home · prev · next · page 3 of 20
y = Φx + e
PART 2
Theory and Practice
home · prev · next · page 4 of 20
RECOVERY: If δ3K(Φ) ≤ 1/√
32, then after at most⌈log2
(‖xK‖2
ε̃K
)⌉iterations,
the IHT approximation x? satisfies
‖x? − x‖2 ≤ 7ε̃K.
where ε̃K = ‖x− xK‖2 + ‖x−xK‖1√K
+ ‖e‖2 and where δK(Φ) is the smallestconstant for which
(1− δK(Φ))‖x‖22 ≤ ‖Φx‖22 ≤ (1 + δK(Φ))‖x‖22
holds for all K sparse x.
CONVERGENCE: IHT is guaranteed to converge to a local minimum of‖y −Φx‖22 s. t. ‖x‖0 ≤ K whenever ‖Φ‖2 ≤ 1.
y = Φx + e
Convergence and Recovery Performance
home · prev · next · page 5 of 20
0 0.1 0.2 0.3 0.4 0.50
0.2
0.4
0.6
0.8
1
sparsity K/M
prob
abili
ty o
f exa
ct re
cove
rty
l1||!||2<1
||!||2 " RIP
y = Φx + e
But
home · prev · next · page 6 of 20
y = Φx + e
PART 3
Stability Beyond RIP
home · prev · next · page 7 of 20
The Normalised Iterative Hard Thresholding (NIHT) algorithm uses the iteration
xn+1 = PK(xn + µnΦT (y −Φxn)),
where PK is a hard thresholding operator that keeps the largest (in magnitude)K elements of a vector (or, more generally, a projector onto the closest elementin the model) and µn is a step-size.
y = Φx + e
The Normalised Iterative Hard Thresholding Algorithm
home · prev · next · page 8 of 20
y = Φx + e
Calculating the step size
Assume the support of xn = Γn and that the support of xn+1 = Γn+1 = Γn,then the optimal step-size is (in terms of reduction in squared approximationerror)
µn =gT
ΓngΓn
gTΓnΦT
ΓnΦΓngΓn,
where g = ΦT (y −Φxn).However, if Γn+1 6= Γn, this step-size might not be optimal and does notguarantee convergence. For guaranteed convergence we require that:
µ ≤ (1− c)‖xn+1 − xn‖22‖Φ(xn+1 − xn)‖22
,
for some c > 0.
Hence, if Γn+1 6= Γn, calculate ω = (1− c) ‖xn+1−xn‖22
‖Φ(xn+1−xn)‖22
and, if µn > ω, set
µn ← µn/(κ(1− c)) or, alternatively, set µn ← ω.
home · prev · next · page 9 of 20
RECOVERY: If Φ satisfies 0 < α2K ≤ ‖Φx‖2‖x‖2
≤ β2K for all x : ‖x‖0 ≤ 2K. Given
a noisy observation y = Φx + e, where x is an arbitrary vector, let xK be thebest K-term approximation to x.
If γ2K = max{1− α22K
κβ22K
,β2
2K
α22K− 1} < 1/8, then after at most
n? =⌈log2
(‖xK‖2/ε̃K
)⌉iterations, IHTK estimates x with accuracy given by
‖x− xn?‖2 ≤ 9ε̃K, (1)
where
ε̃K = ‖x− xK‖2 +‖x− xK‖1√
K+
1α2K‖e‖2. (2)
CONVERGENCE: NIHT is guaranteed to converge to a local minimum of‖y −Φx‖22 s. t. ‖x‖0 ≤ K.
y = Φx + e
Why the Hassle?
home · prev · next · page 10 of 20
y = Φx + e
PART 4
Results
home · prev · next · page 11 of 20
y = Φx + e
Before and After
0 0.1 0.2 0.3 0.4 0.50
0.2
0.4
0.6
0.8
1
sparsity K/M
prob
abili
ty o
f exa
ct re
cove
rty
home · prev · next · page 12 of 20
y = Φx + e
Comparison to other Algorithms
0 0.1 0.2 0.3 0.4 0.50
0.2
0.4
0.6
0.8
1
sparsity K/M
prob
abili
ty o
f exa
ct re
cove
rty
home · prev · next · page 13 of 20
y = Φx + e
Comparison to other Algorithms
0 0.1 0.2 0.3 0.4 0.50
0.2
0.4
0.6
0.8
1
sparsity K/M
prob
abili
ty o
f exa
ct re
cove
rty
home · prev · next · page 14 of 20
y = Φx + e
Speed Comparison
0 0.05 0.1 0.15 0.2 0.25 0.30
0.01
0.02
0.03
0.04
0.05
0.06
0.07
sparsity K/M
com
puta
tion
time
home · prev · next · page 15 of 20
0 0.1 0.2 0.3 0.4 0.5!10
0
10
200 dB SNR
sparsity K/M
estim
atio
n SN
R in
dB
0 0.1 0.2 0.3 0.4 0.50
10
20
3010 dB SNR
sparsity K/M
estim
atio
n SN
R in
dB
0 0.1 0.2 0.3 0.4 0.50
10
20
30
4020 dB SNR
estim
atio
n SN
R in
dB
0 0.1 0.2 0.3 0.4 0.50
1020304050
30 dB SNR
sparsity K/M
estim
atio
n SN
R in
dB
y = Φx + e
Robustness to Noise
home · prev · next · page 16 of 20
0 0.1 0.2 0.3 0.4 0.50
5
10
15
20xn = n!1
estim
atio
n SN
R in
dB
0 0.1 0.2 0.3 0.4 0.510
20
30
40
50
60xn = n!2
estim
atio
n SN
R in
dB
0 0.1 0.2 0.3 0.4 0.5
406080
100120140
xn = n!4
sparsity K/M
estim
atio
n SN
R in
dB
y = Φx + e
Robustness to Non-Exact-Sparsesness
home · prev · next · page 17 of 20
y = Φx + e
Larger Problems
Original / Reconstruction Haar Wavelet Transform
Frequency Domain Observation
home · prev · next · page 18 of 20
Who want’s my job?
Larger Problems
0.1 0.12 0.14 0.16 0.18 0.220
40
60
80
100
120
140
ratio of measurments to signal dimension
PSN
R in
dB
IHTOMPGPL1
19 of 20
Salary scale: 28,290 - 33,779 pa. (pounds sterling)Informal enquiries to Prof. Mike Davies ([email protected])
For further details please go to: http://www.jobs.ed.ac.uk/
Sparse representations for the processing of large scale dataCompressed sensing theory and algorithms with particular application to dynamic MRI
School of Engineering and Electronics, The University of Edinburgh36 months, starting as soon as possible after 1st June 2009.
Who want’s my job?
20 of 20