6270 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 12, DECEMBER 2012 Message ...big · 2012-11-23 · 6270 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 12, DECEMBER

6270 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 12, DECEMBER 2012

Message-Passing De-Quantization With Applicationsto Compressed Sensing

Ulugbek S. Kamilov, Student Member, IEEE, Vivek K Goyal, Senior Member, IEEE, andSundeep Rangan, Member, IEEE

Abstract—Estimation of a vector from quantized linear mea-surements is a common problem forwhich simple linear techniquesare suboptimal—sometimes greatly so. This paper develops mes-sage-passing de-quantization (MPDQ) algorithms for minimummean-squared error estimation of a random vector from quan-tized linear measurements, notably allowing the linear expansionto be overcomplete or undercomplete and the scalar quantizationto be regular or non-regular. The algorithm is based on general-ized approximate message passing (GAMP), a recently-developedGaussian approximation of loopy belief propagation for estimationwith linear transforms and nonlinear componentwise-separableoutput channels. ForMPDQ, scalar quantization of measurementsis incorporated into the output channel formalism, leading to thefirst tractable and effective method for high-dimensional esti-mation problems involving non-regular scalar quantization. Thealgorithm is computationally simple and can incorporate arbitraryseparable priors on the input vector including sparsity-inducingpriors that arise in the context of compressed sensing. More-over, under the assumption of a Gaussian measurement matrixwith i.i.d. entries, the asymptotic error performance of MPDQcan be accurately predicted and tracked through a simple setof scalar state evolution equations. We additionally use stateevolution to design MSE-optimal scalar quantizers for MPDQsignal reconstruction and empirically demonstrate the superiorerror performance of the resulting quantizers. In particular, ourresults show that non-regular quantization can greatly improverate-distortion performance in some problems with oversamplingor with undersampling combined with a sparsity-inducing prior.

Index Terms—Analog-to-digital conversion, approximate mes-sage passing, belief propagation, compressed sensing, frames,non-regular quantizers, overcomplete representations, quantiza-tion, Slepian-Wolf coding, Wyner-Ziv coding.

Manuscript received November 20, 2011; revised May 08, 2012; acceptedAugust 19, 2012. Date of publication September 07, 2012; date of currentversion November 20, 2012. The associate editor coordinating the review ofthis manuscript and approving it for publication was Prof. Konstantinos I. Dia-mantaras. This material is based upon work supported by the National ScienceFoundation under Grants 0729069 and 1116589 and by the DARPA InPhoprogram through the US Army Research Office award W911-NF-10-1-0404.The material in this paper was presented in part at the IEEE InternationalSymposium on Information Theory, St. Petersburg, Russia, July–August 2011,and the Fourth IEEE International Workshop on Computational Advances inMulti-Sensor Adaptive Processing, San Juan, Puerto Rico, December 2011.U. S. Kamilov was with Research Laboratory of Electronics, Massachusetts

Institute of Technology, Cambridge, MA 02139 USA. He is now with theBiomedical Imaging Group, École Polytechnique Fédérale de Lausanne,CH-1015 Lausanne Switzerland (e-mail: [email protected]).V. K Goyal is with the Research Laboratory of Electronics, Massachusetts

Institute of Technology, Cambridge, MA 02139 USA (e-mail: [email protected]).S. Rangan is with the Department of Electrical and Computer Engineering,

Polytechnic Institute of New York University, Brooklyn NY 11201 USA(e-mail: [email protected]).Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TSP.2012.2217334

I. INTRODUCTION

E STIMATION of a signal from quantized samples is a fun-damental problem in signal processing. It arises both from

the discretization in digital acquisition devices and the quanti-zation performed for lossy compression.This paper considers estimation of an i.i.d. vector from

quantized transformed samples of the form whereis a linear transform of and is a scalar (componen-

twise separable) quantization operator. Due to the transform, the components of may be correlated. Even though the

traditional transform coding paradigm demonstrates the advan-tages of expressing the signal with independent componentsprior to coding [1], quantization of vectors with correlated com-ponents nevertheless arises in a range of circumstances. Forexample, to model oversampled analog-to-digital conversion(ADC), we may write a vector of time-domain samples as, where the entries of the vector are statistically indepen-

dent Fourier components and is an oversampled inverse dis-crete Fourier transform. The oversampled ADC quantizes thecorrelated time-domain samples , as opposed to the Fouriercoefficients . Distributed sensing also necessitates quantiza-tion of components that are not independent since decorrelatingtransforms may not be possible prior to the quantization. Morerecently, compressed sensing has become a motivation to con-sider quantization of randomly linearly mixed information, andseveral sophisticated reconstruction approaches have been pro-posed [2]–[4].Estimation of a vector from quantized samples of the form

is challenging because the quantization function isnonlinear and the transform couples, or “mixes,” the compo-nents of , thus necessitating joint estimation. Although recon-struction from quantized samples is typically linear, more so-phisticated, nonlinear techniques can offer significant improve-ments in the case of quantized transformed data. A key exampleis ADC, where the improvement from replacing conventionallinear estimation with nonlinear estimation increases with theoversampling factor [5]–[13].We propose a simple reconstruction algorithm called mes-

sage-passing de-quantization (MPDQ) that improves upon thestate of the art. The algorithm is based on a recently-devel-oped Gaussian-approximated belief propagation (BP) algorithmcalled generalized approximate message passing (GAMP) [14]or relaxed belief propagation [15], [16], which extends earliermethods [17]–[19] to nonlinear output channels. The applica-tion of GAMP to de-quantization was first introduced in a con-ference version of this paper [20] for regular quantization in a

1053-587X/$31.00 © 2012 IEEE

KAMILOV et al.: MESSAGE-PASSING DE-QUANTIZATION WITH APPLICATIONS TO COMPRESSED SENSING 6271

compressive acquisition setting; the present paper provides ex-tensive explanations and simulations for both overcomplete andundercomplete settings, with both regular and non-regular quan-tization.MPDQ is the first computationally-tractablemethod forsettings with non-regular quantization.

A. Contributions

Gaussian approximations of loopy BP have previously beenshown to be effective in several other applications [16]–[19],[21], [22]; for our application to estimation from quantized sam-ples, the extension to general output channels [14], [16] is es-sential. Using this extension to nonlinear output channels, weshow that MPDQ estimation offers several key benefits:• General quantizers: The MPDQ algorithm permits essen-tially arbitrary quantization functions including non-uniform and even non-regular quantizers (i.e., quantizerswith cells composed of unions of disjoint intervals) used,for example, in Wyner-Ziv coding [23] and multiple de-scription coding [24]. In Section VI, we will demonstratethat a non-regular modulo quantizer can provide perfor-mance improvements for correlated data. We believe thatthe MPDQ algorithm provides the first tractable estimationmethod that can exploit such quantizers.

• General priors:MPDQ estimation can incorporate a largeclass of priors on the components of , provided that thecomponents are independent. For example, in Section VI,we will demonstrate the algorithm on recovery of vectorswith sparse priors arising in quantized compressed sensing[2]–[4].

• Exact characterization with random transforms: In thecase of certain large random transforms , the componen-twise performance of MPDQ can be precisely predictedby a so-called state evolution (SE) analysis presented inSection V-A. From the SE analysis, one can precisely eval-uate any componentwise performance metric, includingmean-squared error (MSE). In contrast, works such as[5]–[13] mentioned above have only obtained bounds orscaling laws.

• Performance: Our simulations indicate significantly-im-proved performance over traditional methods for esti-mating from quantized samples in a range of scenarios.

• Computational simplicity: The MPDQ algorithm is com-putationally extremely fast. Our simulation and SE anal-ysis indicate good performance with a small number of it-erations (10 to 20 in our experience), with the dominantcomputational cost per iteration simply being multiplica-tion by and .

• Applications to optimal quantizer design:When quantizeroutputs are used as inputs to a nonlinear estimation algo-rithm, minimizing the MSE between quantizer inputs andoutputs is generally not equivalent to minimizing the MSEof the final reconstruction [25]. To optimize the quantizerfor the MPDQ algorithm, we use the fact that the MSEunder large random mixing matrices can be predictedaccurately from a set of simple SE equations [14], [15].We use the SE formalism to optimize the quantizer to min-imize the asymptotic distortion after the reconstruction byMPDQ. Note that our use of random is for rigor of the

SE formalism; the effectiveness of MPDQ does not dependon this.

B. Outline

The remainder of the paper is organized as follows. Section IIprovides basic background material on quantization andcompressed sensing. Section III introduces the problem ofestimating a random vector from quantized linear transformcoefficients. It concentrates on geometric insights for both theoversampled and undersampled settings. Section IV presentstheMPDQ algorithm by formulating the reconstruction problemin Bayesian terms. Note that this Bayesian formulation doesnot require sparsity of the signal nor specify undersampling oroversampling. Section V describes the use of SE to optimizethe quantizers for MPDQ reconstruction. Experimental resultsare presented in Section VI. Section VII concludes the paper.

Notation

Vectors and matrices will be written in boldface typeto distinguish from scalars written in normal

weight . Random and non-random quantities (orrandom variables and their realizations) are not distinguishedtypographically since the use of capital letters for random vari-ables would conflict with the convention of using capital lettersfor matrices (or in the case of quantization, an operator on avector rather than a scalar). The probability density function(p.d.f.) of random vector is denoted , and the conditionalp.d.f. of given is denoted . When these densities areseparable and identical across components, we use for thescalar p.d.f. and for the scalar conditional p.d.f. Writing

indicates that is a Gaussian random variablewith mean and variance . The resulting p.d.f. is written as

.

II. BACKGROUND

This section establishes concepts and notations central to thepaper. For a comprehensive tutorial history of quantization, werecommend [26]; and for an introduction to compressed sensing,[27].

A. Scalar Quantization

A -level scalar quantizer is defined by its outputlevels or reproduction points and (partition) cells

. It can be decomposed into a composition of twomappings where is the(lossy) encoder and is the decoder. Theboundaries of the cells are called decision thresholds. One mayallow to denote that is countably infinite.A quantizer is called regular when each cell is a convex set,

i.e., a single interval. Each cell of a regular scalar quantizer thushas a boundary of one point (if the cell is unbounded) or twopoints (if the cell is bounded). When the input to a quantizeris a continuous random variable, it suffices to specify the cellsof a -point regular scalar quantizer by its decision thresholds

, with and ; the encoder satisfies

and the output for boundary points can be safely ignored.


The lossy encoder of a non-regular quantizer can be decom-posed into the lossy encoder of a regular quantizer followedby a many-to-one integer-to-integer mapping. Suppose -levelnon-regular scalar quantizer has decision thresholds ,and let be the lossy encoder of a regular quantizer with thesedecision thresholds. Since is not regular, . Let

denote the lossy encoder of . Then, where

is called a binning function, labeling function, or index assign-ment. The binning function is not invertible.The distortion of a quantizer applied to scalar random vari-

able is typically measured by the MSE

A quantizer is called optimal at fixed rate when itminimizes distortion among all -level quantizers. To opti-mize scalar quantizers under MSE distortion, it suffices to con-sider only regular quantizers; a non-regular quantizer will neverperform strictly better.While regular quantizers are optimal for the standard lossy

compression problem, non-regular quantizers are sometimesuseful when some information aside from is availablewhen estimating . Two key examples are Wyner-Ziv coding[23] and multiple description coding [24]. One method forWyner-Ziv coding is to apply Slepian-Wolf coding across ablock of samples after regular scalar quantization [28]; theSlepian-Wolf coding is binning, but across a block rather thanfor a single scalar. In multiple description scalar quantization[29], two binning functions are used that together are invertiblebut individually are not. In these uses of non-regular quantizers,side information aids in recovering with resolution commen-surate with while the rate is only commensurate with ,with .A quantizer is called a scalar quantizer when

it is the Cartesian product of scalar quantizers . Inthis paper, always represents a scalar quantizer with identicalcomponent quantizers .

B. Compressed Sensing

An important value of the proposedMPDQ framework is thatit can exploit non-Gaussian priors on the input vector . To il-lustrate this feature, wewill apply theMPDQ algorithm to quan-tization problems in compressed sensing (CS) [30]–[32], whichconsiders the reconstruction of sparse vectors through ran-domized linear transforms.A vector is sparse if it has a relatively small number of

nonzero components. A central principle of CS is that sparsevectors can be reconstructed from certain underdeterminedlinear transforms , where and . Suchlinear transforms can thus be used as a form of “compression”on , reducing the vector’s dimension from to a smaller value. Since many signals are naturally sparse in some domain,

there are now a large number of works advocating CS methodsin analog front ends prior to quantization to reduce the overall

acquisition bit rate. However, properly understanding therate-distortion performance of such approaches requires thatwe analyze and design CS reconstruction methods preciselyaccounting for the effects of quantization on the transform-do-main measurements [33].Analysis and design of CS reconstruction algorithms is chal-

lenging, even in the absence of quantization. Most approachesare based on either greedy heuristics (matching pursuit [34] andits variants with orthogonalization [35]–[37] and iterative re-finement [38], [39]) and convex relaxations (basis pursuit [40],LASSO [41], Dantzig selector [42], and others). These methodsare all nonlinear and their performance can be difficult to pre-cisely characterize, particularly with quantization. Some ini-tial performance bounds for CS reconstruction with quantiza-tion can be found in [33], [43]. In [44], high-resolution func-tional scalar quantization theory was used to design quantizersfor LASSO estimation. The papers [2]–[4] consider alternate re-construction algorithms that use the partition cells of the quan-tizers that compose . Analyses of these methods produce per-formance bounds that are not generally tight. Moreover, the re-sults are generally limited to specific sparsity priors as well asregular quantizers.We will show here that the MPDQ framework enables CS

reconstruction for a large class of sparse priors and essentiallyarbitrary quantization functions. Moreover, the method is com-putationally simple and, for certain large random transforms,admits an exact performance analysis.

III. QUANTIZED LINEAR EXPANSIONS

This paper focuses on the general quantized measurement ab-straction of

(1)

where is a signal of interest, is a linearmixing matrix, and is a scalar quantizer. We willbe primarily interested in (per-component) MSE

for various estimators that depend on , , and . Thecases of and are both of interest. We sometimesuse to simplify expressions.

A. Overcomplete Expansions

Let have rank . Then is a frame in ,where is row of . Rank can occur only with , so

is called an overcomplete expansion of . In some cases ofinterest, the frame may be uniform, meaning for each.Commonly-used linear reconstruction forms estimate

(2)

where is the pseudoinverse of . Linearreconstruction generally has MSE inversely proportional to .For example, suppose the frame is uniform, ,and is an unknown deterministic quantity. By modeling scalarquantization as the addition of zero-mean white noise, one cancompute the MSE to be [45].


Fig. 1. Visualizing the information present in a quantized overcomplete expansion of when is a regular quantizer (a) A single hyperplane wave partitionwith one single-sample consistent set shaded (b) Partition boundaries from two hyperplane waves; is specified to the intersection of two single-sample consistentsets, which is a bounded convex cell (c) Partition from part (b) in dashed lines with a third hyperplane wave added in solid lines.

Even when an additive white noise model is accurate [46], thelinear reconstruction (2) may be far from optimal. A nonlinearestimate may exploit the boundedness of the single-sample con-sistent sets

Assuming for now that scalar quantizer is regular and its cellsare bounded, the boundary of is two parallel hyperplanes.The full set of hyperplanes obtained for one index by varyingover the output levels of is called a hyperplane wave parti-

tion [47], as illustrated for a uniform quantizer in Fig. 1(a). Theset enclosed by two neighboring hyperplanes in a hyperplanewave partition is called a slab; one slab is shaded in Fig. 1(a).Intersecting for distinct indexes specifies an -dimen-sional parallelotope as illustrated in Fig. 1(b). Using more thanof these single-sample consistent sets restricts to a finer par-

tition, as illustrated in Fig. 1(c) for .The intersection

is called the consistent set. Since each is convex, one mayreach asymptotically through a sequence of projectionsonto using each infinitely often [5], [6].In a variety of settings, nonlinear estimates achieve MSE in-

versely proportional to , which is the best possible depen-dence on [47]. The first result of this sort was in [5]. Whenis an oversampled discrete Fourier transform matrix and

is a uniform quantizer, represents uniformly quantizedsamples above Nyquist rate of a periodic bandlimited signal.For this case, it was proven in [5] that any has

MSE, under a mild assumption on . This was ex-tended empirically to arbitrary uniform frames in [7], whereit was also shown that consistent estimates can be computedthrough a linear program. The techniques of alternating projec-tions and linear programming suffer from high computationalcomplexity; yet, since they generally find a corner of the con-sistent set (rather than the centroid), the MSE performance issuboptimal.

Full consistency is not necessary for optimal MSE depen-dence on . It was shown in [8] that MSE is guar-anteed for a simple algorithm that uses each only once,recursively, under mild conditions on randomized selection of

. These results were strengthened and extended to de-terministic frames in [13].Quantized overcomplete expansions arise naturally in acqui-

sition subsystems such as ADCs, where represents over-sampling factor relative to Nyquist rate. In such systems, highoversampling factor may be motivated by a trade-off betweenMSE and power consumption or manufacturing cost: within cer-tain bounds, faster sampling is cheaper than a higher number ofquantization bits per sample [48]. However, high oversamplingdoes not give a good trade-off between MSE and raw number ofbits produced by the acquisition system: combining the propor-tionality of bit rate to number of samples with the best-case

MSE, we obtain MSE; this is poor comparedto the exponential decrease of MSE with obtained with scalarquantization of Nyquist-rate samples.Ordinarily, the bit-rate inefficiency of the raw output is made

irrelevant by recoding, at or near Nyquist rate, soon after acqui-sition or within the ADC. An alternative explored in this paperis to combat this bit-rate inefficiency through the use of non-reg-ular quantization.

B. Non-Regular Quantization

The bit-rate inefficiency of the raw output with regular quan-tization is easily understood with reference to Fig. 1(c). Afterand are fixed, is known to lie in the intersection of the

shaded strips. Only four values of are possible (i.e., the solidhyperplane wave breaks into four cells), and bitsare wasted if this is not exploited in the representation of .Recall the discussion of generating a non-regular quantizer

by using a binning function in Section II-A. Binning does notchange the boundaries of the single-sample consistent sets, but itmakes these sets unions of slabs that may not even be connected.Thus, while binning reduces the quantization rate, in the absenceof side information that specifies which slab contains (at least


Fig. 2. Visualizing the information present in a quantized overcomplete expansion of when using non-regular (binned) quantizers (a) A single hyperplanewave partition with one single-sample consistent set shaded. Note that binning makes the shaded set not connected (b) Partition boundaries from two hyperplanewaves; is specified to the intersection of two single-sample consistent sets, which is now the union of four convex cells (c) A third sample now specifies towithin a consistent set that is convex.

with moderately high probability), it increases distortion signif-icantly. The increase in distortion is due to ambiguity amongslabs. Taking quantized samples together may pro-vide adequate information to disambiguate among slabs, thusremoving the distortion penalty.The key concepts in the use of non-regular quantization are

illustrated in Fig. 2. Suppose one quantized sample spec-ifies a single-sample consistent set composed of twoslabs, such as the shaded region in Fig. 2(a). A second quan-tized sample will not disambiguate between the two slabs.In the example shown in Fig. 2(b), is composed of twoslabs, and is the union of four connected sets. Athird quantized sample may now completely disambiguate;the particular example of shown in Fig. 2(c) makes

a single convex set.When the quantized samples together completely disam-

biguate the slabs as in the example, the rate reduction frombinning comes with no increase in distortion. The price to paycomes in complexity of estimation.The use of binned quantization of linear expansions was in-

troduced in [49], where the only reconstruction method pro-posed is intractable in high dimensions because it is combinato-rial over the binning functions. Specifically, using the notationfrom Section II-A, let the quantizer forming be defined by

. Then will be a set of possible valuesof specified by . One can try every combination, i.e.,element of

(3)

to seek a consistent estimate. If the binning is effective, mostcombinations yield an empty consistent set; if the slabs are dis-ambiguated, exactly one combination yields a nonempty set,which is then the consistent set . This technique has com-plexity exponential in (assuming non-trivial binning). Therecent paper [50] provides bounds on reconstruction error forconsistent estimation with binned quantization; it does not ad-dress algorithms for reconstruction.

This paper provides a tractable and effective method for re-construction from a quantized linear expansion with non-regularquantizers. To the best of our knowledge, this is the first suchmethod.

C. Undercomplete Expansions

Maintaining the quantizedmeasurement model (1), let us turnto the case of . Since the rank of is less than ,is a many-to-one mapping. Thus, even without quantization,

one cannot recover from . Rather, specifies a propersubspace of containing ; when is in general position,the subspace is of dimension . Quantization increasesthe ambiguity in the value of , yielding consist sets similarto those depicted in Figs. 1(a) and 2(a). However, as describedin Section II-B, knowledge that is sparse or approximatelysparse could be exploited to enable accurate estimation offrom .For ease of explanation, consider only the case where is

known to be -sparse with . Let bethe support (sparsity pattern) of , with . The product

is equal to , where denotes the restriction of thedomain of to and is the submatrix of con-taining the -indexed columns. Assuming has rank (i.e.,full rank), is a quantized overcompleteexpansion of . All discussion of estimation of from theprevious subsections thus applies, assuming is known.The key remaining issue is that may or may not pro-

vide enough information to infer . In an overcomplete repre-sentation, most vectors of quantizer outputs cannot occur; thisredundancy was used to enable binning in Fig. 2, and it can beused to show that certain subsets are inconsistent with thesparse signal model. In principle, one may enumerate the setsof size and apply a consistent reconstruction method for

each . If only one candidate yields a nonempty consistentset, then is determined. This is intractable except for smallproblem sizes because there are candidates for .The key concepts are illustrated in Fig. 3. To have an inter-

pretable diagram with , we letand draw the space of unquantized measurements . (This


Fig. 3. Visualizing the information present in a quantized undercomplete expansion of a 1-sparse signal when . The depicted 2-di-mensional plane represents the vector of measurements . Since is 1-sparse, the measurement lies in a union of 1-dimensional subspaces (the angledsolid lines); since is 3 dimensional, there are three such subspaces (a) Scalar quantization of divides the plane of possible values for into vertical strips. Oneparticular value of does not specify which entry of is nonzero since the shaded strip intersects all the angled solid lines. For each possible support,the value of the nonzero entry is specified to an interval (b) Scalar quantization of both components of specifies to a rectangular cell. In most cases, includingthe one highlighted, the quantized values specify which entry of is nonzero because only one angled solid line intersects the cell. The value of the nonzero entryis specified to an interval (c) In many cases, including the one highlighted, the quantizers can be non-regular (binned) and yet still uniquely specify which entryof is nonzero.

contrasts with Figs. 1 and 2where the space of is drawn.)The vector has one of possible supports .Thus, lies in one of 3 subspaces of dimension 1, which are de-picted by the angled heavy lines. Scalar quantization of cor-responds to separable partitioning of with cell boundariesaligned with coordinate axes, as shown with lighter solid lines.Only one quantized measurement is not adequate to

specify , as shown in Fig. 3(a) by the fact that a single shadedcell intersects all the subspaces.1 Two quantized measurementstogether will usually specify , as shown in Fig. 3(b) by thefact that only one subspace intersects the specified square cell;for fixed scalar quantizers, ambiguity becomes less likely asdecreases, increases, increases, or increases. Fig. 3(c)shows a case where non-regular (binned) quantization stillallows unambiguous determination of .The naïve reconstruction method implied by Fig. 3(c) is to

search combinatorially over both and the combinations in (3);this is extremely complex. While the use of binning for quan-tized undercomplete expansions of sparse signals has appearedin the literature, first in [49] and later in [50], to the best of ourknowledge this paper is the first to provide a tractable and ef-fective reconstruction method.

IV. ESTIMATION FROM QUANTIZED SAMPLES

In this section, we provide the Bayesian formulation of thereconstruction problem from quantized measurements and in-troduce the MPDQ algorithm as a low-complexity alternativeto belief propagation.

A. Bayesian Formulation

We now specify more explicitly the class of problems forwhich we derive new estimation algorithms. Generalizing (1),let

(4)

1Intersections with two subspaces are shown within the range of the diagram.

as depicted in Fig. 4. The input vector is randomwith i.i.d. entries with prior p.d.f. . The linear mixing matrix

is random with i.i.d. entries . The(pre-quantization) additive noise is random with i.i.d.entries . The quantizer is a scalar quantizerwith identical component quantizers and has output levels.Note that, the mapping from to is a separable probabilisticmapping with identical marginals. Specifically, quantized mea-surement indicates , so each component outputchannel can be characterized as

where is the Gaussian p.d.f. We then construct the followingconditional probability distribution over random vector giventhe measurements

(5)

where denotes identity after normalization to unity and. The posterior distribution (5) of the signal provides a

complete statistical characterization of the problem. In partic-ular, we wish to obtain a tractable approximation to the MMSEestimator of specified by

(6)

B. Loopy Belief Propagation

Loopy BP [51], [52] is a popular computational method to ap-proximate theMMSE estimator iteratively. The methodis based on computation of marginal probability distributionsof . To apply loopy BP to the quantization reconstructionproblem, construct a bipartite factor graph where


Fig. 4. Quantized linear measurement model considered in this work. Vectorwith an i.i.d. prior is estimated from scalar quantized measurements. The quantizer input is the sum of and an i.i.d.

Gaussian noise vector .

denotes the set of variable or input nodes associated withtransform inputs , and is the set of factoror output nodes associated with the transform outputs

. The set of (undirected) edges consist of the pairssuch that . Loopy BP passes the following mes-

sages along the edges of the graph:

(7a)

(7b)

where integration is over all the elements of except . Werefer to messages as variable updates and tomessages as factor updates. BP is initialized bysetting . The approximate marginal distri-bution is computed as

(8)

Finally, the component of the estimate is computed as

(9)

When the graph induced by the matrix is cycle free,the BP outputs will converge to the true marginals of the pos-terior density . However, for general , loopy BP is onlyapproximate—the reader is referred to the references above fora general discussion on the performance of loopy BP. We willdiscuss the performance of the specific variant of loopy BP usedin MPDQ in detail in Section V-A. What is important here is thecomputational complexity of loopy BP: Direct implementationof loopy BP is impractical for the de-quantization problem un-less is very sparse. For dense , the algorithm must computethe marginal of a high-dimensional distribution at each mea-surement node; i.e., the integration in (7b) is over many vari-ables. Furthermore, integration must be approximated throughsome discrete quadrature rule.

C. Message-Passing De-Quantization

To overcome the computational complexity of loopy BP, theproposed MPDQ algorithm uses a Gaussian approximation.Gaussian approximations of loopy BP have been used success-fully in CDMA multiuser detection [15], [17], [18] and, morerecently, in compressed sensing [14], [16], [22]. We apply thespecific generalized approximate message passing (GAMP)method in [14], which allows for nonlinear output channels.The approximations are based on a Central Limit Theorem

and other second-order approximations at the measurementnodes. Details can be found in [14]. Here, we simply restate thealgorithm as applied to the specific de-quantization problem.Given the measurements , the measurement matrix

, the noise variance , the mapping of the scalarquantizer, and the prior , the MPDQ estimation proceeds asfollows:1) Initialization: Set and evaluate

(10a)

(10b)

(10c)

where the expected value and the variance are with respectto the prior .

2) Factor update: First, compute the linear step

(11a)

(11b)

where denotes the Hadamard product (i.e., component-wise multiplication). Then, evaluate the nonlinear step

(12a)

(12b)

where is an all-ones vector. The scalar functions andare applied component-wise and given by

(13a)

(13b)

The expected value and the variance are evaluated withrespect to .

3) Variable update: First, compute the linear step

(14a)

(14b)

Then, evaluate the nonlinear step

(15a)

(15b)

where the scalar functions and are applied compo-nent-wise and given by

(16a)

(16b)

The expected value and the variance are evaluated withrespect to . This is essen-tially a scalar AWGN denoising problem with noise

.4) Set and proceed to step 2).For each iteration , the proposed update rulesproduce estimates of the true signal . Thus the algorithm


reduces the intractable high-dimensional integration to a se-quence of matrix-vector products and scalar nonlinearities.Note that scalar inequalities (13a) and (13b) are easy to eval-uate since they admit closed-form expressions in terms of

. Depending on the prior distribu-tion , the scalar inequalities (16a) and (16b) either admitclosed-form expressions or can be implemented as a look-uptable.

V. QUANTIZER OPTIMIZATION

A remarkable fact about MPDQ is that, under large randomtransforms, the MPDQ performance can be precisely predictedby a scalar state evolution (SE) analysis presented in this sec-tion. One can then apply the SE analysis to the design of MSEoptimal quantizers under MPDQ reconstruction. Superior re-construction performance of MPDQ with such quantizers is nu-merically confirmed in the Section VI.

A. State Evolution for MPDQ

The computations (10)–(16) are easy to implement, howeverthey provide us no insight into the performance of the algorithm.The goal of the SE equation is to describe the asymptotic be-havior of MPDQ under large random measurement matrices .For , it is defined as a recursion

(17)

where the scalar function implicitly depends on ,the prior distribution , the mapping of the scalar quantizer,and AWGN variance ; it is given by

(18a)

(18b)

(18c)

where and are defined in (13b) and (16b), respectively.The recursion is initialized by setting , with. The expectation in (18b) is taken over and

, where covariance matrix is given by

(19)

Similarly, the expectation in (18c) is taken over the scalarrandom variable , with and .One of the main results of [14], which is an extension of

the analysis in [19], was to demonstrate the convergence of theerror performance of the GAMP algorithm to the SE equations.Specifically, these works consider the case where is an i.i.d.Gaussian matrix, is i.i.d. with a prior and with

. Then, under some further technical conditions, it isshown that for any fixed iteration number , the empirical jointdistribution of the components of the unknown vectorand its estimate converges to a simple scalar equivalent

model parameterized by the outputs of the SE equations. Fromthe scalar equivalent model, one can compute any asymptoticcomponentwise performance metric. It can be shown, in partic-ular, that the asymptotic MSE is given simply by . That is,

(20)

Thus, can be used as a metric for the design and analysisof the quantizer, although other non-squared error distortionscould also be considered. Although our simulations will con-sider dense transforms , similar SE equations can be derivedfor certain large sparse matrices [15]–[18]. In this case, whenthe fixed points of the SE equations are unique, it can be shownthat the approximate message passing method is mean-squarederror optimal.To conclude, despite the fact that the prior on may be non-

Gaussian and the quantizer function is nonlinear, one can pre-cisely characterize the exact asymptotic behavior of MPDQ atleast for large random transforms.

B. Optimization

Ordinarily, quantizer designs depend on the distribution of thequantizer input, with an implicit aim of minimizing theMSE be-tween the quantizer input and output. Often, only uniform quan-tizers are considered, in which case the “design” is to choosethe loading factor of the quantizer. When quantized data is usedas an input to a nonlinear function, overall system performancemay be improved by adjusting the quantizer designs appropri-ately [25]. In the present setting, conventional quantizer designminimizes , but minimizing is de-sired instead.The SE description of MPDQ performance facilitates the

desired optimization. By implementing the SE equations forMPDQ, we can make use of the convergence result (20) torecast our optimization problem to

(21)

where minimization is done over -level scalar quantizers.Based on (20), the optimization is equivalent to finding thequantizer that minimizes the asymptotic MSE. In the optimiza-tion (21), we have considered the limit in the iterations, .One can also consider the optimization with a finite , althoughour simulations exhibit close to the limiting performance witha relatively small number of iterations.It is important to note that the SE recursion behaves well

under quantizer optimization. This is due to the fact that SE is in-dependent of actual output levels and small changes in the quan-tizer boundaries result in onlyminor change in the recursion (see(18b)). Although closed-form expressions for the derivatives offor large ’s are difficult to obtain, we can approximate them

by using finite difference methods. Finally, the recursion itselfis fast to evaluate, which makes the scheme in (21) practicallyrealizable under standard optimization methods.


Fig. 5. Performance comparison for oversampled observation of a jointlyGaussian signal vector (no sparsity). MPDQ outperforms linear MMSE andMAP estimators.

VI. EXPERIMENTAL RESULTS

A. Overcomplete Expansions

Consider overcomplete expansion of as discussed inSection III-A. We generate the signal with i.i.d. elementsfrom the standard Gaussian distribution . Weform the measurement matrix from i.i.d. zero-mean Gaussianrandom variables. To concentrate on the degradation due toquantization we assume noiseless measurement model (1); i.e.,

in (4).Fig. 5 presents squared-error performance of three estima-

tion algorithms while varying the oversampling ratio andholding . To generate the plot we considered estima-tion from measurements discretized by a 16-level regular uni-form quantizer. We set the granular region of the quantizer to

, where is the variance of the measure-ments. For each value of , 200 random realizations of theproblem were generated; the curves show the median-squarederror performance over these 200 Monte Carlo trials. We com-pare error performance of MPDQ against two other commonreconstruction methods: linear MMSE and maximum a poste-riori probability (MAP). The MAP estimator was implementedusing quadratic programming (QP).The MAP estimation is a type of consistent reconstruction

method proposed in [5]–[13]; since the prior is a decreasingfunction of , the MAP estimate is the vector consistentwith of minimum Euclidean norm. In the earlier works,it is argued that consistent reconstruction methods offer im-proved performance over linear estimation, particularly at highoversampling factors.We see in Fig. 5 thatMAP estimation doesindeed outperform linear MMSE at high oversampling. How-ever, MPDQ offers significantly better performance than bothLMMSE andMAP, with more than 5 dB improvement for manyvalues of . In particular, this reinforces that MAP is subop-timal because it finds a corner of the consistent set, rather thanthe centroid. Moreover, the MPDQ method is actually compu-tationally simpler than MAP, which requires the solution to aquadratic program.With Fig. 6 we turn to a comparison among quantizers, all

with MPDQ reconstruction, , , and anddistributed as above. To demonstrate the improvement in rate-

Fig. 6. Performance comparison of MPDQ with optimal uniform quantizersunder Gaussian prior for regular and binned quantizers.

distortion performance that is possible with non-regular quan-tizers, we consider simple uniform modulo quantizers

(22)

where is the size of the quantization cells. These quantizersmap the entire real line to the set in a peri-odic fashion.We compare three types of quantizers: those optimized for

MSE of the measurements (not the overall reconstruction MSE)using Lloyd’s algorithm [26], regular uniform quantizers withloading factors optimized for reconstruction MSE using SEanalysis, and (non-regular) uniform modulo quantizers withoptimized for reconstruction MSE using SE analysis. The lasttwo quantizers were obtained by solving (21) via the standardSQP method found in MATLAB. The uniform modulo quan-tizer achieves the best rate-distortion performance, while theperformance of the quantizer designed with Lloyd’s algorithmis comparatively poor. The stark suboptimality of the latter isdue to the fact that it optimizes the MSE only between quantizerinputs and outputs, ignoring the nonlinear estimation algorithmfollowing the quantizer.It is important to point out that, without methods such as

MPDQ, estimation with a modulo quantizer such as (22) is noteven computationally possible in works such as [5]–[13], sincethe consistent set is non-convex and consists of a disjoint unionof convex sets. Beyond the performance improvements, we be-lieve that MPDQ provides the first computationally-tractableand systematic method for such non-convex quantization recon-struction problems.

B. Compressive Sensing With Quantized Measurements

We next consider estimation of an -dimensional sparsesignal from random measurements—a problem con-sidered in quantized compressed sensing [2]–[4]. We assumethat the signal is generated with i.i.d. elements from theGauss-Bernoulli distribution

with probability

with probability(23)

where is the sparsity ratio that represents the average fractionof nonzero components of . In the following experiments we


Fig. 7. Performance comparison of MPDQ with LMMSE, BPDN, and BPDQ(with moment ) for estimation from compressive measurements.

assume . Similarly to the overcomplete case, we formthe measurement matrix from i.i.d. Gaussian random vari-ables and we assume no additive noise ( in (4)).Fig. 7 compares MSE performance of MPDQ with three

other standard reconstruction methods. In particular, we con-sider linear MMSE and the Basis Pursuit DeNoise (BPDN)program [53]

where and is the parameter representing thenoise power. In the same figure, we additionally plot the errorperformance of the Basis Pursuit DeQuantizer (BPDQ)2 of mo-ment , proposed in [3], which solves the problem above for

. It has been argued in [3] that BPDQ offers better errorperformance compared to the standard BPDN as the number ofsamples increases with respect to the sparsity of the signal.We obtain the curves by varying the ratio and holding

. We perform estimation from measurements obtainedfrom a 16-level regular uniform quantizer with granular regionof length centered at the origin.The figure plots the median of the squared error from 1000

Monte Carlo trials for each value of . For basis pursuitmethods we optimize the parameter for the best squared errorperformance; in practice this oracle-aided performance wouldnot be achieved. The top curve (worst performance) is for linearMMSE estimation; and middle curves are for the basis pursuitestimators BPDN and BPDQ with moment . As expected,BPDQ achieves a notable 2 dB reduction in MSE compared toBPDN for high values of , however MPDQ significantly out-performs both methods over the whole range of . Note alsothat MPDQ is significantly faster than both BPDN and BPDQ.For example, in Fig. 7 the average reconstruction times—acrossall realizations and undersampling rates—were 7.45, 19.95, and4.52 seconds for BPDN, BPDQ, and MPDQ, respectively.In Fig. 8, we compare the performance of MPDQ under

three quantizers consider before: those optimized for MSE ofthe measurements using Lloyd’s algorithm, and regular andnon-regular quantizers optimized for reconstruction MSE using

2The source codes for the BPDQ algorithm can be downloaded from http://wiki.epfl.ch/bpdq

Fig. 8. Performance comparison of MPDQ with optimal uniform quantizersunder Gauss-Bernoulli prior for regular and binned quantizers.

SE analysis. Note that MPDQ is the first tractable reconstruc-tion method for compressive sensing that handles non-regularquantizers. We assume the same and distributions as above.We plot MSE of the reconstruction against the rate measured inbits per component of . For each rate and for each quantizer,we vary the ratio for the best possible performance. Wesee that, in comparison to regular quantizers, binned quantizerswith MPDQ estimation achieve much lower distortions forthe same rates. This indicates that binning can be an effectivestrategy to favorably shift rate-distortion performance of theestimation.

VII. CONCLUSIONS

We have presented message-passing de-quantization as an ef-fective and efficient algorithm for estimation from quantizedlinear measurements. The proposed methodology is general, al-lowing essentially arbitrary priors and quantization functions.In particular, MPDQ is the first tractable and effective methodfor high-dimensional estimation problems involving non-reg-ular scalar quantization. In addition, the algorithm is computa-tionally extremely simple and, in the case of large random trans-forms, admits a precise performance characterization using astate evolution analysis.The problem formulation is Bayesian, with an i.i.d. prior over

the components of the signal of interest ; the prior may or maynot induce sparsity of . Also, the number of measurements maybe more or less than the dimension of , and the quantizers ap-plied to the linear measurements may be regular or not. Exper-iments show significant performance improvement over tradi-tional reconstruction schemes, some of which have higher com-putational complexity. Moreover, using extensions of GAMPsuch as hybrid approximate message passing [54], [55], onemayalso in the future be able to consider quantization of more gen-eral classes of signals described by general graphical models.MATLAB code for experiments with GAMP is available online[56].Despite the improvements demonstrated here, we are not ad-

vocating quantized linear expansions as a compression tech-nique—for the oversampled case or the undersampled sparsecase; thus, comparisons to rate-distortion bounds would obscurethe contribution. For regular quantizers and some fixed over-sampling , the MSE decay with increasing rateis , worse than the distortion-rate bound. For


a discussion of achieving exponential decay of MSE with in-creasing oversampling, while the quantization step size is heldconstant, see [57]. For the undersampled sparse case, [33] dis-cusses the difficulty of recovering the support from quantizedsamples and the consequent difficulty of obtaining near-optimalrate-distortion performance. Performance loss rooted in the useof a random transformation is discussed in [58].

REFERENCES[1] V. K Goyal, “Theoretical foundations of transform coding,” IEEE

Signal Process. Mag., vol. 18, no. 5, pp. 9–21, Sep. 2001.[2] A. Zymnis, S. Boyd, and E. Candès, “Compressed sensing with quan-

tized measurements,” IEEE Signal Process. Lett., vol. 17, no. 2, pp.149–152, Feb. 2010.

[3] L. Jacques, D. K. Hammond, and J. M. Fadili, “Dequantizing com-pressed sensing: When oversampling and non-Gaussian constraintscombine,” IEEE Trans. Inf. Theory, vol. 57, no. 1, pp. 559–571, Jan.2011.

[4] J. N. Laska, P. T. Boufounos, M. A. Davenport, and R. G. Baraniuk,“Democracy in action: Quantization, saturation, and compressivesensing,” Appl. Comput. Harmon. Anal., vol. 31, no. 3, pp. 429–443,Nov. 2011.

[5] N. T. Thao and M. Vetterli, “Reduction of the MSE in -times over-sampled A/D conversion from to ,” IEEE Trans.Signal Process., vol. 42, no. 1, pp. 200–203, Jan. 1994.

[6] N. T. Thao and M. Vetterli, “Deterministic analysis of oversampledA/D conversion and decoding improvement based on consistent esti-mates,” IEEE Trans. Signal Process., vol. 42, no. 3, pp. 519–531, Mar.1994.

[7] V. K Goyal, M. Vetterli, and N. T. Thao, “Quantized overcompleteexpansions in : Analysis, synthesis, and algorithms,” IEEE Trans.Inf. Theory, vol. 44, no. 1, pp. 16–31, Jan. 1998.

[8] S. Rangan and V. K Goyal, “Recursive consistent estimation withbounded noise,” IEEE Trans. Inf. Theory, vol. 47, no. 1, pp. 457–464,Jan. 2001.

[9] Z. Cvetkovic, “Resilience properties of redundant expansions underadditive noise and quantization,” IEEE Trans. Inf. Theory, vol. 49, no.3, pp. 644–656, Mar. 2003.

[10] J. J. Benedetto, A. M. Powell, and Ö. Yilmaz, “Sigma–deltaquantization and finite frames,” IEEE Trans. Inf. Theory, vol. 52, no.5, pp. 1990–2005, May 2006.

[11] B. G. Bodmann and V. I. Paulsen, “Frame paths and error bounds forsigma-delta quantization,” Appl. Comput. Harmon. Anal., vol. 22, no.2, pp. 176–197, Mar. 2007.

[12] B. G. Bodmann and S. P. Lipshitz, “Randomly dithered quantizationand sigma-delta noise shaping for finite frames,” Appl. Comput.Harmon. Anal., vol. 25, no. 3, pp. 367–380, Nov. 2008.

[13] A. M. Powell, “Mean squared error bounds for the Rangan-Goyal softthresholding algorithm,” Appl. Comput. Harmon. Anal., vol. 29, no. 3,pp. 251–271, Nov. 2010.

[14] S. Rangan, “Generalized approximate message passing for estimationwith random linear mixing,” 2010 [Online]. Available: http://arxiv.org/abs/1010.5141, ArXiv:1010.5141v1 (cs.IT).

[15] D. Guo and C.-C. Wang, “Random sparse linear systems observed viaarbitrary channels: A decoupling principle,” in Proc. IEEE Int. Symp.Inf. Theory, Nice, France, Jun. 2007, pp. 946–950.

[16] S. Rangan, “Estimation with random linear mixing, belief propagationand compressed sensing,” in Proc. Conf. Inf. Sci. Syst., Princeton, NJ,Mar. 2010, pp. 1–6.

[17] J. Boutros and G. Caire, “Iterative multiuser joint decoding: Unifiedframework and asymptotic analysis,” IEEE Trans. Inf. Theory, vol. 48,no. 7, pp. 1772–1793, Jul. 2002.

[18] D. Guo and C.-C. Wang, “Asymptotic mean-square optimality of be-lief propagation for sparse linear systems,” in Proc. IEEE Inf. TheoryWorkshop, Chengdu, China, Oct. 2006, pp. 194–198.

[19] M. Bayati and A. Montanari, “The dynamics of message passing ondense graphs, with applications to compressed sensing,” IEEE Trans.Inf. Theory, vol. 57, no. 2, pp. 764–785, Feb. 2011.

[20] U. Kamilov, V. K Goyal, and S. Rangan, “Optimal quantization forcompressive sensing under message passing reconstruction,” in Proc.IEEE Int. Symp. Inf. Theory, St. Petersburg, Russia, Jul.–Aug. 2011,pp. 390–394.

[21] T. Tanaka and M. Okada, “Approximate belief propagation, densityevolution, and neurodynamics for CDMA multiuser detection,” IEEETrans. Inf. Theory, vol. 51, no. 2, pp. 700–706, Feb. 2005.

[22] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algo-rithms for compressed sensing,” Proc. Nat. Acad. Sci., vol. 106, no. 45,pp. 18914–18919, Nov. 2009.

[23] A. D. Wyner and J. Ziv, “The rate-distortion function for source codingwith side information at the decoder,” IEEE Trans. Inf. Theory, vol.IT-22, no. 1, pp. 1–10, Jan. 1976.

[24] V. K Goyal, “Multiple description coding: Compression meets the net-work,” IEEE Signal Process. Mag., vol. 18, no. 5, pp. 74–93, Sep. 2001.

[25] V. Misra, V. K Goyal, and L. R. Varshney, “Distributed scalar quanti-zation for computing: High-resolution analysis and extensions,” IEEETrans. Inf. Theory, vol. 57, no. 8, pp. 5298–5325, Aug. 2011.

[26] R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE Trans. Inf.Theory, vol. 44, no. 6, pp. 2325–2383, Oct. 1998.

[27] E. J. Candès and M. B. Wakin, “An introduction to compressive sam-pling,” IEEE Signal Process. Mag., vol. 25, no. 2, pp. 21–30, Mar.2008.

[28] Z. Liu, S. Cheng, A. D. Liveris, and Z. Xiong, “Slepian–Wolf codednested lattice quantization for Wyner–Ziv coding: High-rate perfor-mance analysis and code design,” IEEE Trans. Inf. Theory, vol. 52,no. 10, pp. 4358–4379, Oct. 2006.

[29] V. A. Vaishampayan, “Design of multiple description scalar quan-tizers,” IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 821–834, May1993.

[30] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles:Exact signal reconstruction from highly incomplete frequency infor-mation,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb.2006.

[31] E. J. Candès and T. Tao, “Near-optimal signal recovery from randomprojections: Universal encoding strategies?,” IEEE Trans. Inf. Theory,vol. 52, no. 12, pp. 5406–5425, Dec. 2006.

[32] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol.52, no. 4, pp. 1289–1306, Apr. 2006.

[33] V. K Goyal, A. K. Fletcher, and S. Rangan, “Compressive samplingand lossy compression,” IEEE Signal Process. Mag., vol. 25, no. 2,pp. 48–56, Mar. 2008.

[34] S. G. Mallat and Z. Zhang, “Matching pursuits with time-frequencydictionaries,” IEEE Trans. Signal Process., vol. 41, no. 12, pp.3397–3415, Dec. 1993.

[35] S. Chen, S. A. Billings, andW. Luo, “Orthogonal least squares methodsand their application to non-linear system identification,” Int. J. Con-trol, vol. 50, no. 5, pp. 1873–1896, Nov. 1989.

[36] Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, “Orthogonalmatching pursuit: Recursive function approximation with applicationsto wavelet decomposition,” in Conf. Rec. 27th Asilomar Conf. Signals,Syst., Comput., Pacific Grove, CA, Nov. 1993, vol. 1, pp. 40–44.

[37] G. Davis, S. Mallat, and Z. Zhang, “Adaptive time-frequency decom-position,” Opt. Eng., vol. 33, no. 7, pp. 2183–2191, Jul. 1994.

[38] D. Needell and J. A. Tropp, “CoSaMP: Iterative signal recovery fromincomplete and inaccurate samples,” Appl. Comput. Harmon. Anal.,vol. 26, no. 3, pp. 301–321, May 2009.

[39] W. Dai and O. Milenkovic, “Subspace pursuit for compressive sensingsignal reconstruction,” IEEE Trans. Inf. Theory, vol. 55, no. 5, pp.2230–2249, May 2009.

[40] S. S. Chen, D. L. Donoho, andM.A. Saunders, “Atomic decompositionby basis pursuit,” SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33–61, 1999.

[41] R. Tibshirani, “Regression shrinkage and selection via the lasso,” J.Roy. Stat. Soc., Ser. B, vol. 58, no. 1, pp. 267–288, 1996.

[42] E. J. Candès and T. Tao, “The Dantzig selector: Statistical estima-tion when is much larger than ,” Ann. Stat., vol. 35, no. 6, pp.2313–2351, Dec. 2007.

[43] E. J. Candès and J. Romberg, “Encoding the ball from limited mea-surements,” in Proc. IEEE Data Compression Conf., Snowbird, UT,Mar. 2006, pp. 33–42.

[44] J. Z. Sun and V. K Goyal, “Optimal quantization of random measure-ments in compressed sensing,” in Proc. IEEE Int. Symp. Inf. Theory,Seoul, Korea, Jun.–Jul. 2009, pp. 6–10.

[45] V. K Goyal, J. Kovacevic, and J. A. Kelner, “Quantized frame expan-sions with erasures,” Appl. Comput. Harmon. Anal., vol. 10, no. 3, pp.203–233, May 2001.

[46] H. Viswanathan and R. Zamir, “On the whiteness of high-resolutionquantization errors,” IEEE Trans. Inf. Theory, vol. 47, no. 5, pp.2029–2038, Jul. 2001.

[47] N. T. Thao andM. Vetterli, “Lower bound on the mean-squared error inoversampled quantization of periodic signals using vector quantizationanalysis,” IEEE Trans. Inf. Theory, vol. 42, no. 2, pp. 469–479, Mar.1996.

[48] R. H.Walden, “Analog-to-digital converter survey and analysis,” IEEEJ. Sel. Areas Commun., vol. 17, no. 4, pp. 539–550, Apr. 1999.


[49] R. J. Pai, “Nonadaptive Lossy Encoding of Sparse Signals,” Master’sthesis, Massachusetts Inst. of Technol., Cambridge, MA, 2006.

[50] P. T. Boufounos, “Universal rate-efficient scalar quantization,” IEEETrans. Inf. Theory, vol. 58, no. 3, pp. 1861–1872, Mar. 2012.

[51] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks ofPlausible Inference. San Mateo, CA: Morgan Kaufmann, 1988.

[52] C. M. Bishop, Pattern Recognition and Machine Learning, ser. Infor-mation Science and Statistics. New York, NY: Springer, 2006.

[53] E. J. Candès, J. K. Romberg, and T. Tao, “Stable signal recoveryfrom incomplete and inaccurate measurements,” Commun. Pure Appl.Math., vol. 59, no. 8, pp. 1207–1223, Aug. 2006.

[54] S. Rangan, A. K. Fletcher, V. K Goyal, and P. Schniter, “Hybrid ap-proximate message passing with applications to structured sparsity,”2011 [Online]. Available: http://arxiv.org/abs/1111.2581, ArXiv:1111.2581 (cs.IT).

[55] S. Rangan, A. K. Fletcher, V. K Goyal, and P. Schniter, “Hybrid gen-eralized approximate message passing with applications to structuredsparsity,” in Proc. IEEE Int. Symp. Inf. Theory, Cambridge, MA, Jul.2012, pp. 1241–1245.

[56] S. Rangan et al., “Generalized Approximate Message Passing, Source-Forge.net Project Gampmatlab,” [Online]. Available: http://gamp-matlab.sourceforge.net/.

[57] Z. Cvetkovic and M. Vetterli, “Error-rate characteristics of oversam-pled analog-to-digital conversion,” IEEE Trans. Inf. Theory, vol. 44,no. 5, pp. 1961–1964, Sep. 1998.

[58] A. K. Fletcher, S. Rangan, and V. K. Goyal, “On the rate-distor-tion performance of compressed sensing,” in Proc. IEEE Int. Conf.Acoust., Speech, Signal Process., Honolulu, HI, Apr. 2007, vol. III,pp. 885–888.

Ulugbek S. Kamilov (S’11) received the M.Sc.degree in communications systems from the ÉcolePolytechnique Fédérale de Lausanne (EPFL), Lau-sanne, Switzerland, in 2011.In 2007–08, he was an exchange student in elec-

trical and computer engineering at Carnegie MellonUniversity, Pittsburgh, PA. In 2008, he was a researchintern at the Telecommunications Research Center,Vienna, Austria. In 2009, he worked as a softwareengineering intern at Microsoft. In 2010–11, he wasa visiting student at the Massachusetts Institute of

Technology. In 2011, he joined the Biomedical Imaging Group at EPFL, wherehe is currently working toward the Ph.D. degree. His research interests includemessage-passing algorithms and the application of signal processing techniquesto various biomedical problems.

Vivek K Goyal (S’92–M’98–SM’03) received theB.S. degree in mathematics and the B.S.E. degree inelectrical engineering from the University of Iowa,Iowa City, and the M.S. and Ph.D. degrees in elec-trical engineering from the University of California,Berkeley.He was a Member of Technical Staff in the Math-

ematics of Communications Research Department ofBell Laboratories, Lucent Technologies from 1998to 2001 and a Senior Research Engineer for DigitalFountain, Inc., from 2001 to 2003. Since 2004, he has

been with theMassachusetts Institute of Technology, Cambridge. He is coauthorof the forthcoming textbooks Foundations of Signal Processing and Fourierand Wavelet Signal Processing (both from Cambridge University Press). Hisresearch interests include computational imaging, sampling, quantization, andsource coding theory.Dr. Goyal is a member of Phi Beta Kappa, Tau Beta Pi, Sigma Xi, Eta Kappa

Nu, and SIAM. While at the University of Iowa, he received the John BriggsMemorial Award for the top undergraduate across all colleges, and while atthe University of California, Berkeley, he received the Eliahu Jury Award foroutstanding achievement in systems, communications, control, or signal pro-cessing. He was awarded the 2002 IEEE Signal Processing Society MagazineAward and an NSF CAREER Award. As a research supervisor, he has beencoauthor of papers that won Student Best Paper Awards at the IEEE Data Com-pression Conference in 2006 and 2011 and the IEEE Sensor Array and Multi-channel Signal Processing Workshop in 2012. He served on the IEEE SignalProcessing Society’s Image and Multiple Dimensional Signal Processing Tech-nical Committee from 2003 to 2009. He is a Technical Program CommitteeCo-Chair of the IEEE ICIP 2016 and a permanent Conference Co-Chair of theSPIE Wavelets and Sparsity conference series.

Sundeep Rangan (M’02) received the B.A.Sc. de-gree from the University of Waterloo, ON, Canada,and the M.S. and Ph.D. degrees from the Universityof California, Berkeley, all in electrical engineering.He held postdoctoral appointments at the Univer-

sity of Michigan, Ann Arbor, and Bell Laboratories.In 2000, he co-founded (with four others) FlarionTechnologies, a spin-off of Bell Laboratories, thatdeveloped Flash OFDM, one of the first cellularOFDM data systems. In 2006, Flarion was ac-quired by Qualcomm Technologies, where he was

a Director of Engineering involved in OFDM infrastructure products. In2010, he joined the Department of Electrical and Computer Engineering atthe Polytechnic Institute of New York University, where he is currently anAssociate Professor. His research interests are in wireless communications,signal processing, information theory, and control theory.

Documents

6270 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 12, DECEMBER 2012 Message ...big · 2012-11-23 · 6270 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 12, DECEMBER