Frame-Based Image Deblurring withBalanced-Compound Regularization

Shoulie Xie and Susanto RahardjaSignal Processing Department, Institute for Infocomm Research, Singapore

Email: {slxie, rsusanto}@i2r.a-star.edu.sg

Abstract—This paper presents a novel balanced-compound reg-ularization approach for solving the frame-based image deblur-ring. The proposed balanced-compound regularization employstwo different frames as synthesis and analysis operators, and itis formulated as a minimization problem involving an �2 data-fidelity term, an �1 regularizer on sparsity of synthesis frame co-efficients, an �1 regularizer on sparsity of analysis frame operator,and a penalty on distance of sparse synthesis frame coefficients tothe range of the frame operator. Thus the proposed regulariza-tion consists of a synthesis-analysis compound regularizer anda balanced regularizer. Then the balanced-compound optimalproblem is solved based on a variable splitting strategy and theclassical alternating direction method of multiplier (ADMM). Nu-merical simulations show that the proposed balanced-compoundapproach can achieve less coefficient estimated error than thehybrid synthesis-analysis approach under comparable qualitiesin image deblurring problem. This improvement is due to theadded balanced term. Moreover, by exploiting the related fasttight Parseval frames and the special structure of the observationmatrix, the regularized Hessian matrix can perform efficiently forthe frame-based image deblurring.

I. INTRODUCTION

Image deblurring is a classical and important research topicin image processing, which is often formulated as an inverseproblem [1]. The goal is to recover an unknown true imageu ∈ R

n from a noisy measurement y ∈ Rm that is often

modeled asy = Bu+ n (1)

where B is a convolution operator and n is a white Gaussiannoise with variance σ2.

In frame-based image deblurring, a balanced regularizationapproach utilizing sparseness of the frame coefficients wasstudied in [3]–[5], [16] recently and its formulation is

minx

1

2‖BWx− y‖22 +

γ

2‖(I −WTW )x‖22 + λT |x|1 (2)

where γ > 0 and λ are given nonnegative weight vectors,W is the redundant and tight frame (Parseval frame) withWWT = I . Thus u = W (WTu) for every vector u ∈ R

n, andthe components of the vector WTu are called the canonicalcoefficients representing u. ‖ · ‖2 denotes the l2-norm (i.e.,‖z‖22 =

∑n

j=1z2j for any z ∈ R

n) and |z|1 denotes the vectorobtained from z by taking absolute values of its elements.The first term denotes penalty on the data fidelity, the lastterm penalizes the sparsity of coefficient vector, the secondterm penalizes the distance between the frame coefficients xand the range of WT , i.e., the distance to the canonical frame

coefficients of u. The larger γ makes the frame coefficients xcloser to the range of WT , that is to say, the frame coefficientsx is closer to the canonical frame coefficients of u for thelarger γ.

It can be seen that when γ = 0, the problem (2) isreduced to the synthesis-based approach, and when γ = ∞,it becomes analysis-based approach. In [3]–[5], [16], it hasbeen shown that the balanced regularization approach (2)bridges the synthesis-based and analysis-based approaches andbalances the fidelity, sparsity and smoothness of the solution.Hence it is called the balanced approach.

It should be noted that the balanced regularization approach(2) requires the analysis and synthesis operators (WT and W )to be of the same frame, thus it is less general. A related com-pound method (called as hybrid analysis-synthesis approach)was recently proposed in [6], [7] and was formulated as

minx

1

2‖BWx− y‖22 + λT

1 |x|1 + λT2 |PWx|1 (3)

where two different tight frames W and P were employedfor the synthesis and analysis operators, respectively, and theregularizer was a line combination of two l1 norms in frame-based image restoration. In [6], [7], simulations showed thatthis hybrid approach yields the best speed/ISNR (improvementin SNR) trade-off for image reconstructions.

In this paper, we propose a novel balanced-compoundregularization approach by marrying the balanced and hybridformulations, which can be expressed by

minx

1

2‖BWx−y‖22+

γ

2‖(I−WTW )x‖22+λT

1 |x|1+λT2 |PWx|1

(4)where WWT = I and PTP = I , γ > 0, λ1 and λ2 aregiven nonnegative weight vectors. Obviously, when γ = 0,the problem (4) is reduced to the hybrid analysis-synthesisapproach, and when λ2 = 0, it is reduced to the balancedapproach. As the second term is added in balanced-compoundapproach (4), the estimated coefficients x will be close to thetrue canonical frame coefficients WTu. While this term is notadded in the hybrid approach, the estimated coefficients x maybe far away from the true canonical frame coefficients WTu.This claim will be verified in simulations of this paper.

In this paper, we solve the proposed balanced-compoundoptimization problem by combining a variable splitting and theclassical alternating direction method of multipliers (ADMM),which was used for solving the analysis-based and synthesis-based problems in image restoration [9], [10]. ADMM was

978-1-4799-0434-1/13/$31.00 ©2013 IEEE ICICS 2013

originally proposed in the mid-1970s by Gabay and Mercier[11] and Glowinski and Marrocco [12]. The algorithm wasstudied throughout the 1980s and by the mid-1990s, its con-vergence was established by Eckstein and Bertsekas [13], [14].

Furthermore, we show in this paper that the applicationof ADMM to the balanced-compound regularization probleminvolves a regularized version of the Hessian of the datafidelity term, which contains the penalty on the distance termof the sparse coefficients to the canonical frame coefficientsand the penalties on analysis- and synthesis-related terms.This seems like an unsurmountable obstacle on large size ofimage representations, but we show that it is not the casein the standard image deblurring problem. The regularizedHessian matrix and its inverses can be computed efficientlyby exploiting the special structure of the observation matrixand tight Parseval frames (W and P ) with fast computationalalgorithms. Therefore the results of this paper show that thebalanced-compound regularization problem in the standardframe-based image deblurring can be solved efficiently by us-ing ADMM algorithm, and especially, the proposed balanced-compound regularization can ensure that the estimated framecoefficients x will be close to the true canonical framecoefficients WTu, as compared with the hybrid approach.

II. STANDARD ALTERNATING DIRECTION METHOD

Consider an unconstrained optimization problem of the form

minu∈Rn

f(u) + g(Gu) (5)

where f(·) and g(·) are closed, proper convex functions, andG ∈ R

d×n. Variable splitting consists in creating a newvariable, say v, to serve as the argument of g, under theconstraint that Gu = v. This leads to the constrained problem:

minu∈Rn

f(u) + g(v), subject to Gu = v (6)

which is clearly equivalent to the unconstrained problem (5).A simple and intuitive way to address (6) is the so-calledalternating direction method of multipliers (ADMM) [9], [13].

Algorithm ADMM:

1) Set k = 0, choose μ > 0, v0 and d0.2) repeat3) uk+1 ∈ argminu f(u) +

μ

2‖Gu− vk − dk‖

22.

4) vk+1 ∈ argminv g(v) +μ

2‖Guk+1 − v − dk‖

22.

5) dk+1 = dk − (Guk+1 − vk+1).6) k ← k + 1.7) until stopping criterion is satisfied.

The convergence of ADMM is guaranteed by the theoremin [13] if f and g are closed, proper convex functions, andG ∈ R

d×n has full column rank.

III. ADMM-BASED ALGORITHM FOR

BALANCED-COMPOUND REGULARIZATION

By variable splitting, the balanced-compound regularizationproblem (4) can be rewritten as

minx,v∈Rn

f1(x) + f2(v) + f3(w) (7)

subject to v = x, w = PWx

where

f1(x) =1

2‖BWx− y‖22 +

γ

2‖(I −WTW )x‖22, (8)

f2(v) = λT1 |v|1, (9)

f3(w) = λT2 |w|1. (10)

Hence Problem (7) can be written in the form (6) using thefollowing definitions:

f(x) = f1(x), g(v, w) = f2(v) + f3(w), (11)

[vT wT ]T = Gx with G = [I (PW )T ]T . (12)

If the ADMM is applied to solve the above constrainedoptimization problem (7), the steps 3) – 5) in AlgorithmADMM should be replaced with

3a) xk+1 = argminx f1(x) + μ1

2‖x − vk − dvk‖

22 +

μ2

2‖PWx− wk − dwk ‖

22.

4a) vk+1 = argminv f2(v) +μ1

2‖xk+1 − v − dvk‖

22.

4b) wk+1 = argminw f3(w)+μ2

2‖PWxk+1−w−dwk ‖

22.

5a) dvk+1= dvk − (xk+1 − vk+1).

5b) dwk+1= dwk − (PWxk+1 − wk+1).

Note that the step 3a) is a strictly convex quadratic mini-mization problem with respect to x, hence it can be reducedto the following linear system:

xk+1 = A−1(WTBT y + μ1ξvk + μ2W

TPT ξwk ) (13)

where ξvk = vk + dvk , ξwk = wk + dwk and

A = WTBTBW + γ(I −WTW )T (I −WTW )

+μ1I + μ2(PW )T (PW ). (14)

The matrix A can be seen as a regularized version of theHessian of 1/2‖BWx− y‖22 by adding other three terms. Ingeneral, the computations of this matrix and its inverse arenot affordable for large-size matrices B and W , we may justtake a steepest descent step instead, and that leads to the linearsystem being solved inexactly. However, in the standard imagedeblurring problem, the matrix B represents a convolution,the matrix-vector products can be performed with the help offast Fourier transform (FFT). Moreover since W and P aretight wavelet frames, any matrix-vector multiplications can beperformed by fast transform algorithms [2]. Hence these factshave motivated that we can exploit these special structures tosolve the linear system (13) fast and exactly, in which theoperations involving matrices are only matrix-vector productswith fast algorithms. However, since the last three terms areadded into A, it is not straightforward to obtain the inverse ofA such that the fast computations can be employed explicitly.

We can solve this computational issue. Along the linesof [16], and using the Sherman-Morrison-Woodbury matrixinversion lemma and WWT = I and PTP = I , we canobtain the following formula.

Proposition 3.1:

A−1 =1

μ1 + μ2

[αI + (1− α)WTW −WTFW ](15)

where

α =μ1 + μ2

μ1 + γ(16)

F = BT((μ1 + μ2)I +BBT

)−1B. (17)

Proof: The proof of this proposition is shown in the ap-pendix. �

Proposition 3.1 shows that the inverse of regularized matrixA involves the tight frame W and its transpose which can beefficiently calculated by fast algorithms. It is also noted thatin our solution, the proposed algorithm uses the second-orderinformation of the data-fidelity function, not like gradient-based algorithms that only use the first-order information.

The minimization problems in the steps 4a) and 4b) withrespect to v and w can be solved by the soft thresholdingmethod [8] which has a closed form:

vk+1 = soft(v′

k,λ1

μ1

), wk+1 = soft(w′

k,λ2

μ2

), (18)

where v′

k = xk+1−dvk , w′

k = PWxk+1−dwk and soft(x, τ) =sign(x) � max{|x| − τ, 0}with � denoting the component-wise product, i.e., (x� y)i = xiyi and sign being the signumfunction.

In view of (18), (17), (15) and (13), we can obtain thefollowing algorithm to solve the balanced regularization opti-mization problem in the frame-based image restoration.

Algorithm ADMM for balanced-compound regularization(ADMM-BC):

1) Set k = 0, choose μ1 > 0, μ2 > 0, v0, w0, dv0 and dw0 .2) repeat3) ξvk = vk + dvk, ξwk = wk + dwk .4) rk = WTBT y + μ1ξ

vk + μ2W

TPT ξwk .5) xk+1 = 1

μ1+μ2

(αrk + (1− α)WTWrk −WTFWrk).6) vk+1 = soft(xk+1 − dvk,

λ1

μ1

).7) wk+1 = soft(PWxk+1 − dwk ,

λ2

μ2

).8) dvk+1

= dvk − (xk+1 − vk+1).9) dwk+1

= dwk − (PWxk+1 − wk+1).10) k ← k + 1.11) until stopping criterion is satisfied.

It is noted that WTBT y does not change during thealgorithm and can be precomputed. Since G = [I (PW )T ]T ,the convergence of the proposed ADMM-BC algorithms canbe guaranteed by the existing ADMM theory [13].

Computing F : In the standard image deblurring, the matrixB represents a periodic convolution, thus F can be computedin Fourier domain, which has a fast computational algorithm.In fact, B can be factorized as

B = UTDU (19)

where U represents the 2-D discrete Fourier transform (DFT)with UT = U−1, D is a diagonal matrix containing the DFT

coefficients of B. Thus we have

F = BT (μI +BBT )−1B = UTD∗(|D|2 + μI)−1DU (20)

where μ = μ1 + μ2, D∗ denotes complex conjugate and |D|2

the squared absolute values of the entries of D. Since allthe matrices in D∗(|D|2 + μI)−1D are diagonal, it can becomputed with O(n) cost, while the products by U and UT

can be computed with O(n logn) cost using FFT. Thus theproducts by the matrix F have O(n log n) cost. Hence, xk+1

can be computed with O(n log n) cost using the fast tightframe W and the above fast algorithm of F .

IV. EXPERIMENTAL RESULTS

In this section, numerical results are reported to illustratethe performance of our proposed balanced-compound regu-larization approach in the frame-based image deblurring. In[6], [7], the hybrid approach yielded better speed/ISNR trade-off than the analysis- and synthesis-based approaches. Hence,we only need to compare our proposed balanced-compoundapproach with hybrid approach. Here the stopping criterionis chosen as the relative variation of the objective function,i.e., |obj(k+1)− obj(k)|/obj(k) ≤ Tol, where obj(k) is thevalue of the objective function at kth iteration, and Tol is amoderately small tolerance. The improvement in SNR (ISNR)is computed by 10 log10(

∑k ‖u−yk‖

2/∑

k ‖u− u‖2), whereu is the original image, yk is the observed image at the kthiteration, and uk is the corresponding estimated image. Oursimulations are written in MATLAB and performed on a Dellcomputer with Intel Xeon CPU 2.66GHz and 4GB of RAMand Windows XP.

Furthermore, each blurred image is generated by applyinga 9 × 9 uniform blurring kernel to the original image first,and then followed by an additive Gaussian white noise withzero mean and standard variance σ having SNR of 40 dB.Meanwhile the blur operator B is applied via FFT to theoriginal images, and W is a redundant 4-level Haar waveletframe and F is a 4-level redundant Daubechies frame. In oursimulations, we empirically choose γ = 0.005, λ1 = 0.0025,λ2 = 0.005, μ1 = 0.1λ1, μ2 = 0.1λ2 and Tol = 0.001.

The deblurring results on the well-known Cameraman imagewith sized 256×256 pixels are reported in Table I. Clearly, thethe speed/ISNR is comparable for both approaches to imagedeblurring.

TABLE ICOMPARISON OF THE DEBLURRED IMAGES ON THE CAMERAMAN

Approach Iters CPU time ISNRBalanced-Compound 12 10.1 8.73

Hybrid 9 7.61 8.61

Moreover, the MSE (i.e., ‖x − WTu‖21/N ) evolutions areshown in Fig. 1, where x is the estimated frame coefficient,WTu is the true frame coefficient of image u and N isimage size. One can see that in our proposed balanced-compound approach, the estimated coefficient x is closed

to the true wavelet frame coefficient WTu, while in hybridapproach the estimated coefficient x is far away from the truewavelet frame coefficient WTu. This property owes to theadded second term in the balanced-compound regularizationapproach (4). Finally, the deblurred images produced by theseapproach are shown in Fig. 2. The simulation results showthat our proposed balanced-compound regularization approachcan keep the estimated coefficient error less than the hybridapproach and achieve comparable qualities of the deblurredimages.

0 2 4 6 8 10 1210

0

101

102

103

104

(a)

0 1 2 3 4 5 6 7 810

1

102

103

104

(b)

Fig. 1. Estimated frame coefficient MSE vs CPU time (seconds): (a) Ourbalanced-compound approach (4). (b) Hybrid approach (3).

V. CONCLUSIONS

A novel balanced-compound regularization approach forthe frame-based image restoration has been presented.The balanced-compound regularization approach ensures lessframe coefficient error and equalizes the analysis- andsynthesis-based approaches in image deblurring. The proposedADMM-based algorithm (ADMM-BC) can be used to solvethis optimization problem efficiently by exploiting the fasttight frame transform algorithms and the special structure ofobservation matrix in the standard image deblurring prob-lem. Theoretical and experimental results have shown thatthe proposed balanced-compound approach can achieve lessestimated coefficient error than the state-of-the-art hybridapproach with comparable qualities on the standard imagedeblurring problem.

50 100 150 200 250

50

100

150

200

250

(a)

50 100 150 200 250

50

100

150

200

250

(b)

50 100 150 200 250

50

100

150

200

250

(c)

Fig. 2. Image deblurring: (a) Image blurred with noise. (b) Estimated imagewith our proposed balanced-Compound approach. (c) Estimated image withhybrid approach.

APPENDIX APROOF OF PROPOSITION 3.1

In this appendix, by exploiting the tight Parseval framestructure and the Sherman-Morrison-Woodbury matrix inver-sion lemma and along the lines of [9], we show that althoughthe regularized Hessian matrix consists of three terms, itsinverse can be expressed in terms of very simple and intu-itive mathematical formula in the standard frame-based imagerestoration with balanced regularization.

In view of WWT = I and PTP = I , the matrix A in (14)

can be rewritten as

A = WTBTBW + γ(I −WTW ) + μ1I + μ2WTW (21)

Define

C = γ(I −WTW ) + μ1I + μ2WTW

= (γ + μ1)I + (μ2 − γ)WTW. (22)

Using the Sherman-Morrison-Woodbury matrix inversionlemma and WWT = I , we can obtain

C−1 = [(γ + μ1)I + (μ2 − γ)WTW ]−1

=1

μ1 + γI −

1

(μ1 + γ)2WT ·

·

(1

μ2 − γI +

1

μ1 + γWWT

)−1

W

=1

μ1 + γI +

γ − μ2

(μ1 + γ)(μ1 + μ2)WTW. (23)

Obviously this includes the special case μ2 = γ. In this case,

C−1 =1

μ1 + γI.

Thus, using WWT = I again, we have

I +BWC−1WTBT

= I +BW ·

·

(1

μ1 + γI +

γ − μ2

(μ1 + γ)(μ1 + μ2)WTW

)WTBT

= I +1

μ1 + μ2

BBT

=1

μ1 + μ2

((μ1 + μ2)I +BBT

). (24)

Using the Sherman-Morrison-Woodbury matrix inversionformula and WWT = I again, it follows that

A−1 = [WTBTBW + C]−1

= C−1 − C−1WTBT (I +BWC−1WTBT )−1BWC−1

= C−1 − (μ1 + μ2)C−1WTBT ·

·((μ1 + μ2)I +BBT

)−1BWC−1

= C−1 − (μ1 + μ2)C−1WTFWC−1 (25)

where

F = BT((μ1 + μ2)I +BBT

)−1B. (26)

In view of Equation (23) and WWT = I , we have

C−1WTFWC−1

=

(1

μ1 + γI +

γ − μ2

(μ1 + γ)(μ1 + μ2)WTW

)WTFW ·

·

(1

μ1 + γI +

γ − μ2

(μ1 + γ)(μ1 + μ2)WTW

)

=1

(μ1 + μ2)2WTFW. (27)

Hence we have

A−1 = C−1 −1

(μ1 + μ2)WTFW

=1

μ1 + γI +

γ − μ2

(μ1 + γ)(μ1 + μ2)WTW

−1

μ1 + μ2

WTFW

=1

μ1 + μ2

[αI + (1 − α)WTW −WTFW ](28)

whereα =

μ1 + μ2

μ1 + γ.

The proof of Proposition 3.1 is completed. �

REFERENCES

[1] H. Andrews and B. Hunt, Digital Image Restoration. Upper SaddleRiver, NJ: Prentice-Hall, 1977.

[2] S. Mallat, A Wavelet Tour of Signal Processing. New York: Academic,2009.

[3] J.-F. Cai, R. H. Chan, L. Shen, and Z. Shen, “Restoration of choppedand nodded images by framelets,” SIAM J. Sci. Comput., vol. 30, pp.1205–1227, 2008.

[4] J.-F. Cai and Z. Shen, “Framelet based deconvolution,” J. ComputationalMathematics, vol. 28, no. 3, pp. 289–308, 2010.

[5] Z. Shen, K.-C. Toh, and S. Yun, “An accelerated proximal gradientalgorithm for frame-based image restoration via the balanced approach,”SIAM J. Imaging Sci., vol. 4, no. 2, pp. 573–596, 2011.

[6] M. Afonso, J. Bioucas-Dias, and M. Figueiredo, “An augmented La-grangian approach to linear inverse problems with compound regular-ization,” in Proc. IEEE Int. Conf. Image Processing, Hong Kong, 2010,pp. 4169–4172.

[7] M. Afonso, J. Bioucas-Dias, and M. Figueiredo, “Hybrid synthesis-analysis frame-based regularization: a criterion and an algorithm”, inSignal Processing with Adaptive Sparse Structured Representations -SPARS11, Edinburgh, Scotland, UK, 2011, pp. 43–43.

[8] P. Combettes and V. Wajs, “Signal recovery by proximal forward-backward splitting,” SIAM J. Multiscale Model. Sim., vol. 4, pp. 1168–1200, 2005.

[9] M. Afonso, J. Bioucas-Dias, and M. Figueiredo, “Fast image recoveryusing variable splitting and constrained optimization,” IEEE Trans.Image Process., vol. 19, no. 9, pp. 2345–2356, 2010.

[10] ——, “An augmented Lagrangian approach to the constrained opti-mization formulation of imaging inverse problems,” IEEE Trans. ImageProcess., vol. 20, no. 3, pp. 681–695, 2011.

[11] D. Gabay and B. Mercier, “A dual algorithm for the solution of nonlinearvariational problems via finite-element approximations,” Comput. Math.Appl., vol. 2, pp. 17–40, 1976.

[12] R. Glowinski and A. Marroco, “Sur l’approximation, par elements finisd’ordre un, et la resolution, par penalisation-dualite d’une classe deproblemes de dirichlet non lineares,” Rev. Francaise d’Automat. Inf.Recherche Operationelle, vol. 9, pp. 41–76, 1975.

[13] J. Eckstein and D. Bertsekas, “On the Douglas-Rachford splittingmethod and the proximal point algorithm for maximal monotone op-erators,” Math. Program., vol. 5, pp. 293–318, 1992.

[14] E. Esser, “Applications of Lagrangian-based alternating direction meth-ods and connections to split-Bregman,” Computat. Appl. Math., Univ.California, Los Angeles,” Technical Report CAM09-31, 2009.

[15] J. Douglas and H. Rachford, “On the numerical solution of heatcondution problems in two and three variables,” Trans. Amer. Math.Soc., vol. 82, no. 2, pp. 42–439, 1956.

[16] S. Xie and S. Rahardja, “An Alternating Direction Method for BalancedImage Restoration,” IEEE Trans. Image Process., vol. 21, no. 11, pp.4557–4567, 2012.