Embed Size (px)
Citation preview
Abstract—In this paper, we propose an image block loss restoration method based on the notion of sparse
representation. The sparsity pattern is exploited as side information to efficiently restore block losses, by
iteratively imposing the constraints of spatial and transform domains on the corrupted image. Two novel
features, including a pre-interpolation and a criterion for stopping the iterations, are added for performance
improvement. Besides, a scheme is presented for no-reference quality estimation of the restored image with
respect to the original and sparse images. To the best of our knowledge, this is the first attempt to estimate
the image quality in the restoration methods. Also, to deal with practical applications, we develop a technique
to transmit the side information along with the image. In this technique, we first compress the side
information and then embed its LDPC coded version in the least significant bits of the image pixels. This
technique ensures the error-free transmission of the side information, while causing only a small perturbation
on the transmitted image. Mathematical analysis and extensive simulations are performed to evaluate the
method and investigate the efficiency of the proposed techniques. The results verify that the suggested
method outperforms its counterparts for image block loss restoration.
Index Terms—Image Restoration, Block Loss, Image Sparsity, Sparsity Pattern, Side Information, Pre-
interpolation, PSNR Estimation, Data Embedding.
On Image Block Loss Restoration
Using the Sparsity Pattern as Side Information
H. Hosseini*, A. Goli, N. B. Marvasti, M. Azghani and F. Marvasti
Advanced Communication Research Institute (ACRI)
Department of Electrical Engineering, Sharif University of Technology
I. INTRODUCTION
Image restoration is intended to compensate for the loss occurred during transmission or storage. In many
applications, image pixels are sampled in a non-uniform manner or it may frequently happen that some
pixels are lost or unavailable. A common missing pattern is block loss. Block loss may occur due to the
packet loss in image transmission through networks or it may appear in the distorted biomedical signals,
corrupted archived images, and erroneously received block-based codes. In this paper, we address the
problem of image block loss restoration.
Most of the natural signals, including images, possess sparsity property, i.e., most of their coefficients
are zero in the transform domains such as Discrete Fourier Transform (DFT), Discrete Cosine Transform
(DCT), and Discrete Wavelet Transform (DWT). This property has found wide applications in different
fields including signal compression [1], classification [2], pattern recognition [3], and denoising [4].
One straightforward application of the sparsity property is sampling and compression. The sparsity
based sampling techniques can be divided into two groups: 1) compressive sampling and 2) random
sampling. Compressive Sampling (CS) refers to taking a small number of linear measurements of the
signal entries with rates significantly lower than the Shannon-Nyquist rate [5-8]. The original signal is
recovered from its compressive measurements using CS recovery techniques, which are divided into three
groups: Basis Pursuit (BP) algorithms, greedy techniques, and thresholding methods. The BP algorithms
which are based on convex optimization offer better recovery at the cost of higher computational
complexity [9]. The greedy techniques such as Orthogonal Matching Pursuit (OMP) [10-11], regularized
Orthogonal Matching Pursuit (ROMP) [12-13], and Compressive Sampling Matching Pursuit (CoSaMP)
[14] offer relatively lesser computational complexity and lower recovery performance. The iterative
thresholding methods lie between these two groups in both recovery performance and computational
complexity. Some of the well-known iterative thresholding methods are Iterative Hard Thresholding
(IHT) [15-16] and Iterative Method with Adaptive Thresholding for Compressed Sensing (IMATCS)
[18].
In random sampling techniques, a subset of the signal entries is selected randomly. The random
sampling recovery techniques such as Iterative Method with Adaptive Thresholding (IMAT) [17] and
Iterative Method with Adaptive Thresholding and Interpolation (IMATI) [18] exploit the sparsity property
to interpolate the non-uniform pixels. We can employ random sampling techniques for missing pixels
restoration.
Also, there are methods which use other features for image restoration. In [19], Guleryuz developed
adaptive linear estimators for non-stationary signals, which can be used for robust estimation of image
missing regions. In [20], Takeda et al. presented an adaptive kernel regression method for image
interpolation and proposed an iterative scheme to further improve its performance.
In this paper, we propose an image block loss restoration method, assuming that the recipient is aware
of the sparsity pattern as side information of the transmitted image. The corrupted image is restored by
consecutively imposing the spatial and transform constraints on the image until a stopping criterion is
satisfied. Before the iterations, an image interpolation method is employed to speed up the convergence.
The pre-interpolation and the suggested stopping criterion can be used for improving the performance of
other iterative methods. Moreover, a scheme is presented for no-reference quality estimation of the
restored image. The quality is evaluated in terms of the Peak Signal-to-Noise Ratio (PSNR) value of the
restored image with respect to the original and sparse images.
In some applications, the side information may not be available in the receiver. To deal with such
situations, we develop a technique to embed the sparsity pattern into the sparse image. In this technique,
we first compress the sparsity pattern, by a binary image compression algorithm, and then embed its
coded version in the least significant bits of the image pixels. This technique ensures the error-free
transmission of the sparsity pattern, while causing only a small perturbation on the transmitted image. We
discuss the effect of using inaccurate spatial and transform information, resulting from data embedding,
on the performance of the iterative method.
Extensive simulations are performed to evaluate the method and investigate the efficiency of the
introduced techniques. Simulation results verify that the proposed method outperforms other algorithms
of image block loss restoration.
The rest of this paper is organized as follows. Section II provides the definitions and the essential
background for the paper. The proposed method is explained in section III. Section IV presents a scheme
for image quality estimation. Section V develops a technique for embedding the sparsity pattern in the
sparse image. The experimental results are provided in Section VI, and Section VII concludes the paper.
II. PRELIMINARIES
In this section, we provide the notations, definitions, and the essential background for the paper.
2-1- Notations:
Images are represented by -dimensional vectors and the transform is implemented using an
matrix , denoting the two-dimensional DCT. DCT is chosen because it offers superior energy
compaction properties for images and has been employed as a standard tool in many signal and image
processing applications [21].
2-2- Definitions:
a- Sparsity Projector:
An image is called sparse if a large portion of its transform coefficients is zero. The binary map of
zero and nonzero coefficients is the sparsity pattern and the sparsity rate is the rate of zero coefficients.
The Sparsity Projector ( ) is defined as follows:
( ) ( ) (1)
where is the sparsity mask, which replaces the pre-specified transform coefficients with zero.
Equation (1) implies that takes an image and yields the corresponding sparse image with the pre-
determined sparsity pattern. With a fixed sparsity pattern, the sparsity projection is a linear operation.
Therefore, we can rewrite (1) as:
(2)
where and are the original and sparse images, respectively, and is an matrix, representing
the sparsity projector.
b- Data Projector:
In missing pixel model, either a pixel is delivered flawlessly or it is corrupted entirely, respectively
named as known and missing pixels. Therefore, we have:
(3)
where and are the original and corrupted images, respectively, and is the sampling mask.
The sampling mask is an binary matrix, where some of its elements on the main diagonal,
corresponding to the known pixels, are one.
The Data Projector ( ) of the image with respect to the image , which is masked by , is defined
as follows:
( ) (4)
where ( ), and is the identity matrix.
The effect of the data projector on the image can be computed as:
( )
( )
( ) (5)
Equations (4-5) indicate that substitutes pixels with their original values, if they are in the
corresponding positions of the known pixels, and leaves other pixels unchanged.
2-3- Binary Erasure Channel:
Binary Erasure Channel (BEC) is a common communication channel model. In this model, the
recipient either receives the transmitted bit or a message that it was not received („erased‟). Therefore, we
can model the missing pixels as the BEC. Figure 1 illustrates a BEC with error probability .
2-4- Low-Density Parity-Check Coding:
A Low-Density Parity-Check (LDPC) code is a linear error correcting code used for protecting
messages during transmission over noisy channels [22-23], and is constructed using a sparse bipartite
graph [24]. LDPC codes are capacity-approaching codes, which means that practical construction
methods exist that allow the noise threshold to be set very close (or even arbitrarily close for the BEC) to
the theoretical maximum (the Shannon limit) for a symmetric memory-less channel. The noise threshold
defines an upper bound for the channel noise, where the loss probability can be arbitrarily small.
X Y
0 0
1 1
?
1-ɛ
1-ɛ
ɛ
ɛ
Fig. 1. The Binary Erasure Channel with error probability .
III. THE PROPOSED METHOD
In this section, we propose an image block loss restoration method, assuming that the sparsity pattern
of the transmitted image is available as side information.
3-1- The Method:
For image restoration, we use the concept of error correction in channel coding theory. It is known
that, to achieve error correction, one can add a kind of redundancy to the message which is exploited in
the decoder to recover data faithfully. In our case, the sparsity projector provides linear equations which
can be used for error correction.
Suppose that the sparse image is transmitted via a channel which causes block loss and the knowledge
of the sparsity pattern is available at the receiver. The recipient restores the corrupted image using the
constraints of spatial and transform domains, which are the positions and values of the known pixels and
the positions of transform zeros, respectively. If the original image is sparse enough, the consecutive
projections between these two domains yield the transmitted sparse image.
Let be the corrupted image. The iterative method is formulated as follows:
( ( )) (6)
where is the iteration number.
According to (2) and (4), we can rewrite (6) as follows:
( ) ( ) (7)
The recursive relation of (7) yields the following solution:
( ) ( )
( ) ( )( ) ( )
( ) ( ( )
) ( )
. (8)
Since and , we have:
( ) ( ) (9)
Therefore:
( ) ( ( )
) ( )
( )
( ) ( ( )
)
( ) ( ) (10)
Form (10), it can be seen that the deviation of the outcome of the -th iteration from the transmitted
sparse image is ( ) ( ). Since the term ( ) is constant, the convergence rate is related to
the matrix ( ). It implies that the capability of the block loss restoration depends on the missing
probability and sparsity rate of the transmitted image; that is for an image with more sparsity, the
recipient can restore larger losses.
However, it should be noted that although higher sparsity rate allows faster convergence to the sparse
image, it tends to blur the original image. Since our aim is the faithful transmission of the original image,
we should decide on the trade-off of the capability of image restoration at the expense of blurring it. The
knowledge of the channel parameters is the key for deciding this trade-off.
3-2- Improving the Performance of the Method:
Two features, including a pre-interpolation and a criterion for stopping the iterations, are introduced to
improve the performance of the proposed method. These features are applicable to other iterative
methods. The features are described in the following.
a- Pre-Interpolation:
Due to the absence of a portion of the received image in the spatial domain, the major coefficients of
the transform domain, including the DC coefficient, does not reflect their true values. Thus, for each
pixel, it takes several iterations to reach a proper value, particularly in larger losses. By interpolating the
irregular pixels, we can skip these iterations and significantly reduce the complexity. Also, the pre-
interpolation helps the method to converge when the sparsity rate is not enough to restore the block
losses. The impact of pre-interpolation can be observed in (10); the restored image reaches in fewer
iterations if provides a better approximation of .
For the pre-interpolation, we need a fast method which could efficiently estimate the original values of
the missing pixels. For this purpose, we employ the Adaptive Iterative Mean filter (AIM) introduced in
[25]; this filter is adaptive in terms of the number of iterations, which is different for each missing pixel,
according to their Euclidean distance from the nearest uncorrupted pixel.
b- Stopping Criterion for Iterations:
We suggest an efficient approach to stop the iterations. The approach is to identify when more
iterations bring no improvement.
Suppose ( ). According to (5), the effect of the data projector at iteration is as follows:
( ) (11)
Since the known pixels of have integer values, if the absolute values of the elements of lie in
), the data projector is equivalent to rounding the pixel values. Therefore, it becomes independent
of , and hence, proceeding the iterations does not make further improvement. In this case, in the
corresponding position of known pixels, the absolute value of elements of has a uniform distribution
with the mean value of . Thus, we define the Stopping Criterion for Iterations (SCI) as follows:
Stop the iterations, if the data projector causes the average change of lower than in the
corresponding position of known pixels.
Generally, as the sparsity rate increases, the SCI is satisfied in fewer iterations.
IV. PSNR ESTIMATION
In the receiver, it is important to estimate the quality of the restored image. Several quality estimation
methods have been proposed for image compression algorithms [26-28]; but none of the image restoration
applications provide a tool for no-reference image quality estimation.
For quantitative evaluation of the proposed method, we use the PSNR. The PSNR value between two
images and is computed as follows:
( ) (
( )) (12)
where ( ) is defined as follows:
( ) ‖ ‖
∑ ( )
(13)
where ‖ ‖ represents the Euclidean norm.
For simplicity, the PSNR value of the restored image, with respect to the sparse and original images
are denoted by PSNR-RS and PSNR-RO, respectively, and the PSNR value of the sparse image with
respect to the original image is represented by PSNR-SO.
We present schemes to estimate the PSNR-RS and PSNR-RO, with the assumption that the restored
image has been converged to the sparse image. The details are described in the following.
4-1- Estimating the PSNR-RS:
The PSNR-RS value is directly proportional to the sparsity rate. The following lemma helps to
estimate the PSNR-RS value.
Lemma 1: For images with similar missing patterns and the same sparsity rate, the PSNR-RS value
does not depend on the image.
Proof: According to (10) and (13), we have:
( ) ‖ ‖
‖( ) ( )‖
‖ ‖ ‖ ‖ (14)
With the use of pre-interpolation, the term ‖ ‖ becomes smaller for smoother images because
the AIM filter exploits the correlation among neighboring pixels in the spatial domain, hence it produces
better results for more correlated images. However, to ensure convergence, we should have ‖ ‖ ,
thus, the term ‖ ‖
would be dominant in (14). This term is related to the missing pattern and the
sparsity rate, and is independent of the image. Therefore, the PSNR-RS value is almost equal for different
images, which have the same sparsity rate and are corrupted by similar missing patterns.
As a result, for estimating the PSNR-RS value, it is enough to simulate the whole process of the
sparsity projector, to apply the sampling mask and finally the restoration stage to another image, and then
to compute the PSNR-RS value.
4-2- Estimating the PSNR-RO:
By following the SCI and with sufficient sparsity rate, the restored image is nearly equivalent to the
transmitted sparse image. Thus, the PSNR-RO value is almost the same as the PSNR-SO value.
Due to the Parseval‟s identity [29] and the linearity of DCT, the MSE is equal to:
( )
∑ (( ) ( ) )
(15)
where is the matrix representation of the two-dimensional DCT, and and are the original and
restored images, respectively.
The two vectors and are similar, except for the corresponding positions of the zero coefficients
of the sparse image, which are near zero for . As a result, to estimate the PSNR-RO value, we should
compute the energy discarded by the sparsity projector.
It is known that the histogram of the DCT coefficients of natural images can be accurately modeled
using a generalized Gaussian distribution [30]. Figures 2 and 3 show and , which are the typical
histograms of the DCT coefficients of an image and its corresponding sparse image, respectively. Note
that, for , the pick at zero has been removed. To obtain from , we need to interpolate . For the
sake of simplicity, the linear interpolation is used. Figure 4 shows the interpolated histogram. The lines
are drawn according to the following measures:
1- The zero duration of , which is ( ).
2- The amplitude of in points – and , which are denoted by and
, respectively.
3- And the number of zero coefficients .
Let the two lines (of the left and right hand sides of zero) have slopes and and suppose that the
coefficients are located at integer points. The slopes are obtained as follows:
( (
)
(
)
) (16)
( (
)
(
)
) (17)
Thus, the estimated MSE ( ) is computed as:
[∑(
| |)( )
∑( ( )| |)
]
[(
)∑
(| | | |) ∑ ( ) ] (18)
Using the following two equalities:
∑
( )( ) (19)
∑ ( )
( ) (20)
we can rewrite the estimated MSE as follows:
*
( )( )(
)
( ) + (21)
Fig. 2. A typical histogram of the DCT coefficients of a natural image.
Fig. 3. A typical histogram of the DCT coefficients of a sparse image, ignoring the peak at zero.
Fig. 4. The interpolated histogram of the DCT coefficients of a sparse image.
V. TRANSMITTING THE SPARSITY PATTERN ALONG WITH THE SPARSE IMAGE
The availability of the sparsity pattern at the receiver may not be a realistic assumption in many
applications. In this section, we develop a technique for transmitting the sparsity pattern along with the
sparse image and discuss its effect on the performance of the iterative method.
5-1- Using LDPC Coding to Protect the Sparsity Pattern:
It is known that the channel capacity of the BEC, with error probability , is [24]. Therefore, to
transmit the sparsity pattern of an -dimensional image, a number of
bits should be embedded into
the sparse image. To minimize the distortion introduced to the sparse image, we use pixel LSBs to embed
these extra bits. This means that, on average,
LSBs of each pixel will be devoted to the sparsity
pattern.
To map the bits of the sparsity pattern to
bits, we use LDPC coding because, as mentioned in
subsection 2-4, theses codes are known to be capacity-approaching for the BEC. The resultant is
interleaved to overcome the burst error caused by block losses.
Since the data embedding affects the values of the sparse image pixels, we have to modify the data
projector in the restoration algorithm. Previously, the data projector forced known pixels to hold their
actual values in each iteration; now it replaces just those bits that are not affected by embedding the
sparsity pattern. In other words, the data projector replaces all bits except those LSBs that were devoted to
the sparsity pattern.
However, although the SCI is closely related to the data projector, its definition does not change. That
is we stop the iterations if the modified data projector causes the average change of lower than in the
corresponding position of known pixels.
5-2- Compressing the Sparsity Pattern:
Due to the smoothness of natural images, higher frequencies are much more probable to be replaced
with zero by the sparsity projector. As a result, the sparsity pattern is not a random binary matrix and,
thus, is compressible with the aid of efficient binary image compression algorithms.
Assume that we can achieve a compression rate of . Therefore, the sparsity pattern of an -
dimensional image can be represented by bits, and consequently a number of
bits should be
embedded into the sparse image. This means that, on average,
LSBs of each pixel will be devoted to
the sparsity pattern, which reduces the distortion of the transmitted image caused by data embedding.
5-3- The Effect of Data Embedding on the Restoration Stage:
Embedding the sparsity pattern in the sparse image degrades the quality of the transmitted sparse
image, and as a result, the spatial information, used in iterations, would be inaccurate. Also, there might
be some errors in the decoded sparsity pattern. These effects can potentially harm the performance of the
iterative method. In the following, we discuss these unwanted consequences of the data embedding in the
sparse image.
a- Inaccurate Spatial Information:
Assume that a number of LSBs of the image pixels are devoted to the sparsity pattern. The values of
both the original and modified LSBs are discrete random variables with a uniform distribution function,
say , and elements in ); thus, the change in the image LSBs is a random variable with the
distribution function ( ), where the symbol is the convolution operation. This random variable
has the mean value of zero and the variance of ( )
.
As a result, the modified sparse image is and we can rewrite (9) as follows:
( ) ( ) (22)
Therefore:
( ) ( ) ( ( )
) ( )
(23)
Thus, the error caused by altering the image LSBs is equal to ( ( ) ) ( )
. Assuming
that ‖ ‖ , this error is approximately equal to ( )
. Therefore, the method converges to
an image which is contaminated by a zero mean noise with a variance of ( ).
This error is the trade-off for not to consider the sparsity pattern as shared information between the
transmitter and the receiver. Although this error, generally, degrades the quality of the restored image,
simulation results confirm that the degradation is quite negligible.
b- Inaccurate Sparsity Pattern:
For inaccurate sparsity pattern, we consider the cases of false negative (undetected zero coefficients)
and false positive (coefficients identified as zero in error). Assume that and are matrix
representations of the sparsity projector, corresponding to sparsity patterns with false negative and false
positive, respectively. We analyze these cases in the following.
1- False Negative:
In this case, we have: . Therefore, Equations (9-10) hold and the method converges,
although at a slower rate.
2- False Positive:
In this case, we have . Assume that , where is the matrix representation of
the sparsity projector, corresponding to wrongly zero-detected transform coefficients.
We can rewrite (9) as follows:
( ) ( ) ( ) (24)
Therefore, Equation (10) will be modified as:
( ) ( ) ( ( )
) ( )
(25)
Thus, the error caused by the false positive is equal to ( ( ) ) ( )
. Assuming
that ‖ ‖ , this error is approximately equal to ( )
. This term does not fade with
proceeding the iterations and introduces an error floor for the restored image.
This effect emphasizes that the method is very sensitive to the wrong-detection of non-zero
coefficients as zero. Therefore, to attain a reliable restoration, the rate of LDPC coding should be chosen
carefully to ensure error-free transmission of the sparsity pattern.
VI. SIMULATION RESULTS
Extensive simulations are performed to investigate the performance of the proposed method and
evaluate the efficiency of the introduced techniques. Simulations are carried out on several 8-bit grey-
scale images. The results are illustrated for three test images Lena, Boat and Baboon. We consider
missing patterns as block losses with block sizes of 1, 2, 4, 8 and 16. The block loss with block size of
one is also called the random loss. For better comparison, the missing probability has been fixed at 50%.
In the following, we provide the simulation results.
6-1- The Effect of Pre-interpolation:
The pre-interpolation causes the method to converge, on average, three times faster. Also, it slightly
improves the PSNR-RO and PSNR-RS values. More importantly, the method is more likely to converge
with the use of the pre-interpolation, especially when the sparsity rate is not enough to restore the block
losses.
Figure 5 illustrates the effect of pre-interpolation on block loss restoration for sparse images Lena and
Baboon with different sparsity rates. While without the pre-interpolation, there are strict requirements for
the minimum sparsity rates to satisfy the SCI, the method always converges using the pre-interpolation.
Also, for the 95% sparsity rate, in which the method converges for all cases, the required number of
iterations is mentioned in the figure to indicate the effect of pre-interpolation on the convergence speed.
6-2- Evaluating the PSNR Estimation Schemes:
In this subsection, we assess the accuracy of the schemes presented in Section 4 for estimating the
PSNR-RS and PSNR-RO values.
a- Evaluating the PSNR-RS Estimation Scheme:
Figure 6 shows the results for estimating the PSNR-RS value. The results are obtained for restoring
the three sparse images with different sparsity rates, corrupted by random loss. The simulations verify the
conclusion of the subsection 4-1, which is the PSNR-RS value is directly proportional to the sparsity rate
and for images with similar missing patterns, does not depend on the image itself.
b- Evaluating the PSNR-RO Estimation Scheme:
Figure 7 shows the results for estimating the PSNR-RO value. The results are obtained for restoring
the sparse images Lena and Baboon with different sparsity rates, corrupted by random loss. It can be seen
that the PSNR-RO value is inversely proportional to the sparsity rate and is almost equal to the PSNR-SO
value. Also, the PSNR estimator works quite well in all cases.
6-3- Evaluating the Effect of Data Embedding:
As stated in subsection 5-2, the sparsity pattern can be compressed using efficient binary image
compression algorithms. With the use of the algorithm introduced in [31], on average, we can achieve the
compression rate of
. This means that a single LSB can bear the LDPC coded version of the
compressed sparsity pattern, in the case of the 50% missing probability. This will reduce the distortion of
the transmitted image, caused by the data embedding. In the following, we investigate the effect of the
data embedding on the restoration stage through simulation.
a- Evaluating the Effect of Inaccurate Spatial Information:
Figure 8 demonstrates the effect of using inaccurate spatial information on the restoration stage. It can
be seen that by altering the image LSBs, the restoration quality only slightly degrades; therefore, the
method is quite stable with non-exact spatial information.
b- Evaluating the Effect of Inaccurate Sparsity Pattern:
Table I investigates the effect of using non-exact sparsity pattern on the restoration stage. Let the
modified sparsity pattern be the sparsity pattern which is used in iterations. It can be seen that if the
modified sparsity pattern is a subset of the original sparsity pattern, the restoration will be done, although
in more iterations. Thus, false negative is tolerable. However, if the original sparsity pattern is a subset of
the modified sparsity pattern, the method will not converge.
Table I clarifies how much the method is sensitive to the wrongly detected zero coefficients.
Therefore, as stated in subsection 5-3, it is necessary to choose a sufficient coding rate to avoid the false
positive case in the sparsity pattern during the restoration.
Table I
PSNR Values of the Restored Image Lena Corrupted by Random Loss, using Non-exact Sparsity Patterns
original sparsity rate 75% 76%
sparsity rate used in iterations 75% 76% 75% 76%
PSNR-RO (dB) 39.2 No Convergence 38.9 39.0
6-4- The Ultimate Method Including All Features:
In this subsection, we evaluate the performance of the ultimate method, which includes all features,
and compare the results with other methods. Comparisons are done in terms of the PSNR-RO value and
the visual quality.
Figures 9-11 provide the results of the proposed method for restoring the three sparse images with
different sparsity rates, corrupted by block loss with various block sizes. It can be seen that for each
image and missing pattern, there is an optimum sparsity rate which yields the best outcome. Generally,
the optimum sparsity rate increases as the block sizes increase. Besides, the results illustrate that, as
expected, larger losses are more difficult to restore.
Also, the proposed method is compared with the methods IMAT [17] and IMATI [18], which exploit
the image sparsity property, and the algorithms introduced by Guleryuz [19] and Takeda et al. [20]. None
of the IMAT and IMATI methods provide a stopping rule; therefore, we adopt our technique for stopping
their iterations. Also, for better comparisons, the AIM interpolator is used in IMATI, which produces
better results than the original interpolator.
Since in our method, the recipient has some knowledge about the transmitted image, the comparison
might seem unfair; however we do the comparisons to justify our thesis that the quality of the transmitted
image can be sacrificed for a better missing pixels restoration. Certainly, there is a trade-off between the
amount of the image degradation, resulted from sparsifying and data embedding, and the capability of
missing pixels restoration. Simulation results verify that by choosing an appropriate sparsity rate, which is
possible by well-estimating the missing pattern, we can achieve the best trade-off, and therefore, ensure
much higher degree of the overall fidelity to the original image.
Figures 12-14 compare different methods for restoring the three images corrupted by block loss with
various block sizes. It can be seen that the proposed method has superior performance in all cases,
especially for large block losses.
For visual comparison of different methods, we consider three cases, which are restoring the images
Lena, Boat and Baboon corrupted by block losses with block sizes of 4, 8 and 16, respectively. Figures
15-17 depict the missing patterns and restored images. Also, Figure 18 demonstrates the sparsity patterns
which are used in our proposed method for restoring the three images in Figures 15-17, respectively. In
the sparsity patterns, we can see a strong inclination of non-zero coefficients toward low frequency
coefficients. This feature has been used to compress the sparsity pattern before embedding it in LSB bits.
Fig. 5- The effect of pre-interpolation on block loss restoration. The missing pattern is the 50% block loss with block
sizes of 8. The required number of iterations to satisfy SCI is specified for the case of the 95% sparsity rate. In cases
where the method does not converge, the results are shown for 1000 iterations.
Fig 6- Estimating the PSNR-RS value. The results are shown for the three sparse images corrupted by random loss.
70 75 80 85 90 9510
15
20
25
30
35
40
45
50
55
Sparsity Rate (%)
PS
NR
-RS
(dB
)
Lena, without pre-interpolation
Lena, with pre-interpolation
Baboon, without pre-interpolation
Baboon, with pre-interpolation
202
234
953
80
65 70 75 80 85 90 9546
48
50
52
54
56
58
60
62
Sparsity Rate (%)
PS
NR
-RS
(dB
)
Lena
Boat
Baboon
Fig. 7- Estimating the PSNR-RO value. The results are shown for sparse images corrupted by random loss.
Fig. 8- The effect of using inaccurate spatial information on the restoration stage. The results are shown for the three
sparse images corrupted by random loss.
70 72 74 76 78 80 82 84 86 88 90
25
30
35
40
45
Sparsity Rate (%)
PS
NR
(dB
)
Lena, Estimated PSNR-RO
Lena, PSNR-SO
Lena, PSNR-RO
Baboon, Estimated PSNR-RO
Baboon, PSNR-SO
Baboon, PSNR-RO
65 70 75 80 85 90 95
25
30
35
40
45
Sparsity Rate (%)
PS
NR
-RO
(dB
)
Lena, without data embedding
Lena, with data embedding
Boat, without data embedding
Boat, with data embedding
Baboon, without data embedding
Baboon, with data embedding
Fig. 9- The results for restoring the sparse image Lena with different sparsity rates, corrupted by various block
losses. The numbers indicate the size of missing blocks.
Fig. 10- The results for restoring the sparse image Boat with different sparsity rates, corrupted by various block
losses. The numbers indicate the size of missing blocks.
35 40 45 50 55 60 65 70 75 80 85 90 95
24
26
28
30
32
34
36
38
40
42
Sparsity Rate (%)
PS
NR
-RO
(dB
)
1
2
4
8
16
35 40 45 50 55 60 65 70 75 80 85 90 9522
24
26
28
30
32
34
36
38
40
Sparsity Rate (%)
PS
NR
-RO
(dB
)
1
2
4
8
16
Fig. 11- The results for restoring the sparse image Baboon with different sparsity rates, corrupted by various block
losses. The numbers indicate the size of missing blocks.
Fig. 12- Comparing different methods for restoring the image Lena corrupted by various block losses. The numbers
indicate the sparsity rate of the transmitted sparse image for the proposed method.
35 40 45 50 55 60 65 70 75 80 85 90 95
22
24
26
28
30
32
Sparsity Rate (%)
PS
NR
-RO
(dB
)
1
2
4
8
16
1 2 4 8 1610
15
20
25
30
35
40
Block Size
PS
NR
(dB
)
The Proposed Method
IMATI [18]
IMAT [17]
Takeda et al. [20]
Guleryuz [19]
65%
75%
85%
95%
95%
Fig. 13- Comparing different methods for restoring the image Boat corrupted by various block losses. The numbers
indicate the sparsity rate of the transmitted sparse image for the proposed method.
Fig. 14- Comparing different methods for restoring the image Baboon corrupted by various block losses. The
numbers indicate the sparsity rate of the transmitted sparse image for the proposed method.
1 2 4 8 16
10
15
20
25
30
35
40
Block Size
PS
NR
(dB
)
The Proposed Method
IMATI [18]
IMAT [17]
Takeda et al. [20]
Guleryuz [19]
65%
75%
85%
90%
95%
1 2 4 8 16
10
15
20
25
30
35
Block Size
PS
NR
(dB
)
The Proposed Method
IMATI [18]
IMAT [17]
Takeda et al. [20]
Guleryuz [19]
55%
60%
70%
80%
85%
(a) (b) (c)
(d) (e) (f)
Fig. 15. The results for restoring the image Lena corrupted by the 50% block loss with block size of 4. (a) Missing
pattern, (b) Guleryuz‟s method [19] (29.2 dB), (c) Takeda et al.‟s method [20] (29.8 dB), (d) IMAT [17] (14.5 dB),
(e) IMATI [18] (29.9 dB), and (f) The proposed method (85% sparsity rate, 34.1 dB).
(a) (b) (c)
(d) (e) (f)
Fig. 16. The results for restoring the image Boat corrupted by the 50% block loss with block size of 8. (a) Missing
pattern, (b) Guleryuz‟s method [19] (25.2 dB), (c) Takeda et al.‟s method [20] (16.6 dB), (d) IMAT [17] (10.4 dB),
(e) IMATI [18] (24.9 dB), and (f) The proposed method (90% sparsity rate, 29.5 dB).
(a) (b) (c)
(d) (e) (f)
Fig. 17. The results for restoring the image Baboon corrupted by the 50% block loss with block size of 16. (a)
Missing pattern, (b) Guleryuz‟s method [19] (20.7 dB), (c) Takeda et al.‟s method [20] (10.8 dB), (d) IMAT [17]
(10.0 dB), (e) IMATI [18] (19.7 dB), and (f) The proposed method (85% sparsity rate, 23.9 dB).
(a) (b) (c)
Fig. 18. The sparsity patterns which are used in the proposed method for restoring the three images in Figures 15-17.
(a) Lena (85% sparsity rate), (b) Boat (90% sparsity rate), and (c) Baboon (85% sparsity rate). Sparsity pattern is a
binary matrix and the black portion represents zero coefficients.
VII- CONCLUSION AND FUTURE WORKS
In this paper, we used side information for image block loss restoration. In the proposed iterative
method, the sparsity projector adds a redundancy to the image by discarding the small coefficients of the
DCT domain. The resultant sparsity pattern is used as side information, which enables the recipient to
estimate the intensity values of the missing pixels. Two novel features, including a pre-interpolation and a
criterion for stopping the iterations, are employed for performance improvement. The method can restore
large missing regions at the expense of blurring the transmitted image. Besides, for the first time in image
restoration applications, a scheme is presented for no-reference quality estimation of the restored image.
Also, to deal with practical applications, a technique is developed to transmit the side information along
with the image. This technique first compresses the side information and then embeds its LDPC coded
version in the least significant bits of the image pixels, to ensure the error-free transmission of the side
information by causing only a small perturbation to the transmitted image. Mathematical analysis is
performed to support the efficiency of the introduced techniques. The experimental results verify that the
proposed method outperforms the existing algorithms for image block loss restoration.
As mentioned, there is a trade-off between the amount of image blurring, resulted from sparsifying,
and the capability of block loss restoration. In simulations, for each image and missing pattern, the
optimum sparsity rate is specified. However, mathematically determining the optimum sparsity rate for an
image and a missing pattern is an open problem.
The proposed method can be employed to efficiently combat the block loss in the compression
algorithms. In transform-based compression methods like JPEG, we suggest to combine the sparsity
projector and the compression function to achieve more effectiveness. Also, we can use the proposed
method for video block loss restoration. Since consecutive frames are roughly similar, we can assign one
sparsity pattern for a series of frames, and therefore, transmit less side information. The set of new zero
coefficients should be chosen as the intersection of the corresponding sets of consecutive frames because,
as stated in the paper, unlike the false positive, false negative does not affect the convergence.
ACKNOWLEDGMENT
The authors wish to thank Mohammad Mahdi Kamani for helping with some parts of the simulations.
REFERENCES
[1] S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way, 3rd ed., Academic Press, Orlando, 2008.
[2] E. Elhamifar and R. Vidal, “Sparse subspace clustering,” IEEE Conf. Computer Vision and Pattern
Recognition, 2790-2797, 2009.
[3] J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. Huang and S. Yan, “Sparse representation for computer vision and
pattern recognition,” IEEE Proc., 98(6), 1031-1044, 2010.
[4] A.K. Fletcher, S. Rangan, V.K. Goyal and K. Ramchandran, “Denoising by sparse approximation: error bounds
based on rate-distortion theory,” EURASIP J. Applied Signal Processing, 2006.
[5] E. Cand`es, J. Romberg and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly
incomplete frequency information,” IEEE Trans. Information Theory, 52(2), 489–509, 2006.
[6] E. Cand`es, J. Romberg and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,”
Communications on pure and applied mathematics, 59(8), 1207–1223, 2006.
[7] E. Cand`es and J. Romberg, “Quantitative robust uncertainty principles and optimally sparse decompositions,”
Foundations of Computational Mathematics, 6(2), 227 – 254, 2006.
[8] D. Donoho, “Compressed sensing,” IEEE Trans. Information Theory, 52(4), 1289–1306, 2006.
[9] S.S. Chen, D. Donoho and M.A. Saunders, “Atomic decomposition by basis pursuit,” SIAM journal on
scientific computing, 20(1), 33-61, 1998.
[10] S. Mallat, G. Davis and Z. Zhang, “Adaptive time-frequency decompositions,” SPIE Journal of Optical
Engineering, 33, 2183–2191, 1994.
[11] J.A. Tropp and A.C. Gilbert, “Signal recovery from partial information via orthogonal matching pursuit,” 2006.
[12] D. Needell and R. Vershynin, “Uniform uncertainty principle and signal recovery via regularized orthogonal
matching pursuit,” Foundations of computational mathematics, 9(3), 317-334, 2009.
[13] D. Needell and R. Vershynin, “Signal recovery from incomplete and inaccurate measurements via regularized
orthogonal matching pursuit,” IEEE J. Selected Topics in Signal Processing, 4(2), 310-316, 2010.
[14] D. Needell and J. Tropp, “COSAMP: Iterative signal recovery from incomplete and inaccurate samples,”
Applied and Computational Harmonic Analysis, 26(3), 301-321, 2009.
[15] T. Blumensath and M.E. Davies, “Iterative thresholding for sparse approximations,” J. Fourier Analysis and
Applications, 14(5), 2008.
[16] M. Fornasier and H. Rauhut, “Iterative Thresholding Algorithms,” J. Fourier Analysis and Applications, 14(5),
2008.
[17] F. Marvasti, A. Amini, F. Haddadi, M. Soltanolkotabi, B. Hossein Khalaj, A. Aldroubi, S. Sanei and J.
Chambers, “A Unified Approach to Sparse Signal Processing,” EURASIP J. Advances in Signal Processing,
2012.
[18] M. Azghani and F. Marvasti, “Iterative methods for random sampling and compressed sensing recovery,”
SampTA, 2013.
[19] O.G. Guleryuz, “Nonlinear approximation based image recovery using adaptive sparse reconstructions and
iterated denoising-part I: theory,” IEEE Trans. Image Processing, 15(3), 539-554, 2006.
[20] H. Takeda, S. Farsiu and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Trans.
Image Processing, 16(2), 349-366, 2007.
[21] K.R. Rao, P. Yip and V. Britanak, Discrete Cosine Transform: Algorithms, Advantages, Applications,
Academic Press, Orlando, 2007.
[22] D.J. MacKay, Information Theory, Inference and Learning Algorithms, Cambridge University Press, 2003.
[23] T.K. Moon, Error Correction Coding: Mathematical Methods and Algorithms, Wiley. Com, 2005.
[24] A. Shokrollahi, “LDPC codes: An introduction,” Digital Fountain Inc., 2003.
[25] H. Hosseini and F. Marvasti, “Fast restoration of natural images corrupted by high-density impulse noise,”
EURASIP J. Image and Video Processing, 1, 1-7, 2013.
[26] Z. Wang, H.R. Sheikh and A.C. Bovik, “No-reference perceptual quality assessment of JPEG compressed
images,” IEEE Int. Conf. Image Processing, 1, 2002.
[27] D.S. Turaga, Y. Chen and J. Caviedes, “No reference PSNR estimation for compressed pictures,” Signal
Processing: Image Communication, 19(2), 173-184, 2004.
[28] H.R. Sheikh, A.C. Bovik and L. Cormack, “No-reference quality assessment using natural scene statistics:
JPEG2000,” IEEE Trans. Image Processing, 14(11), 1918-1927, 2005.
[29] A.V. Oppenheim, A.S. Willsky and S.H. Nawab, Signals and systems, Englewood Cliffs, NJ: Prentice-Hall,
1983.
[30] E.Y. Lam and J.W. Goodman, “A mathematical analysis of the DCT coefficients distributions for images,”
IEEE Trans. Image Processing, 9(10), 1661–1666, 2000.
[31] M. Kafashan, H. Hosseini, S. Beygiharchegani, P. Pad and F. Marvasti, “New rectangular partitioning methods
for lossless binary image compression,” IEEE Int. Conf. Signal Processing, 694-697, 2010.
