Blind Deconvolution for Retinal Image Enhancement

Uvais Qidwai Computer Science & Engineering Department Qatar University, P.O. Box 2713, Doha, Qatar.

Email: uqidwai@qu.edu.qa

Umair Qidwai Department of Ophthalmology

Isra Postgraduate institute of ophthalmology Karachi, Pakistan.

Abstract- In this paper, a new technique is presented to enhance the blurred images obtained from retinal imaging. One of the main steps in inspecting the eye (especially the deeper image of retina) is to look into the eye using a slit-lamp apparatus that shines a monochromatic light on to the retinal surface and captures the reflection in the camera as the retinal image. While most of the cases, the image produced is quite clean and easily used by the ophthalmologists, there are many cases in which these images come out to be very blurred due to the disease in the eye such a cataract etc… in such cases, having an enhanced image can enable the doctors to start the appropriate treatment for the underlying disease. The proposed technique utilizes the Blind Deconvolution approach using Maximum Likelihood Estimation approach. Further post-processing steps have been proposed as well to extract specific regions from the image automatically to assist the doctors in visualizing these regions related to very specific diseases. The post-processing steps include Image color space conversions, thresholding, Region Growing, and Edge detection.

I. INTRODUCTION While the clinical technology related to ophthalmology is very advanced and easy to use, there are many areas that are still open for further enhancement. One of the key examination procedures is called retinopathy in which a specific type of image is obtained for the retinal layer in eye. Figure1 shows one such image with areas of interest.

Figure 1. Typical retinal image with areas of interest.

When we see the inner portions of the eye (retina or fundus and optic disc) light needs to pass through different parts of the eye. Light rays first pass through the cornea, than it passes the aqueous humour after that it pass through the lens and in the end it passes through the gel like substance called vitreous to reach the inner most parts of the eye such as retina/fundus and optic disc. Cornea is the outer most transparent layer of the eye, any abnormality in cornea such as corneal edema can lead to hazy view of the inner parts of the eye. The cavity between cornea and the lens is filled with a fluid of specific concentration called aqueous humor. If even cornea is clear but there are opacities in the aqueous humour such as inflammatory cells in case of a condition called uveitis, than again the view of inner parts of the eye will be difficult due to haziness. Cataract is the post common cause of hazy view of the fundus and optic disc. Cataract is opacification in the lens. vitreous is a gel like substance that fills the space between lens and the retina, and any opacification here such as vitritis, vitreous hemorrhage, and asteroid hyalosis can again lead to hazy view of fundus and optic disc [1]. Many times in ophthalmologic examination/diagnosis it is necessary to see the inner parts of the eye including retina/fundus and the optic disc. Diabetic retinopathy is a very highly prevalent condition all over the world and it can only be diagnosed when fundus is thoroughly viewed. similarly many other retinal vascular disorders such as retinal vein obstruction, hypertensive retinopathies etc are all diagnosed by looking into the fundus of the patients eyes. Also, glaucoma is a sight threatening condition and in order to diagnose it and to see its progression or efficacy of treatment again we need to look at the optic disc. In order to diagnose above mentioned diseases of the eye and many more, an ophthalmologist either require a clarity in all the components of the eye from where light passes through or treatment of the cause of the haziness first than to look for the other diseases, for example if a patient has cataract and diabetic retinopathy, but his fundus cannot be seen due to cataract so his diabetic retinopathy cannot be diagnosed nor treated. it can only be treated and diagnosed when cataract is removed surgically than reexamined for diabetic retinopathy. Every day delay in the diagnosis of diabetic retinopathy is leads to advancement of the disease until it reaches the point of no return of lost vision. There has been an extensive application of various techniques from the Image Processing domain on this application area. In fact, when visiting the literature to review the status of

2010 IEEE EMBS Conference on Biomedical Engineering & Sciences (IECBES 2010), Kuala Lumpur, Malaysia, 30th November - 2nd December 2010.

978-1-4244-7600-8/10/$26.00 ©2010 IEEE 20

existing techniques in this area, one feels overwhelmed by the activity found and the diversity of techniques that fills up a large proportion of the known scientific knowledge. Researchers have applied almost every known technique during the past five years to improve and automate the ophthalmologic inspection procedures. Some of these techniques are summarized in the following. It has been observed that the information found in different layers of the RGB image has specific information contents that can be used for isolating subject-specific information [2]. Adaptive Equalization of the gray-scale has been used extensively as well for improving the image of vessels and optical disc [2,3]. Several classical computational approaches are also used such as the Principal Component Analysis [3], Wavelet decomposition [4], Otsu’s Maximum Entropy method [5] and sharpening filters [6,7]. In addition to these methods, all possible morphological operations from the Image processing domain have been used for post-processing the images. These include Erosion, Dilation, opening, and closing operators [3, 5], as well as adaptive thresholding for appropriate conversion to binary masks [5,8], and selective subtraction [9]. A very comprehensive survey of the techniques used in enhancing the retinal images has been given in [10]. However, one marked commonality is the use of very clean images as the starting point. Most of the reported techniques are focused on automatic detection of the various areas of interest in the retinal images and hence they assume that the image is quite clean and nothing is occluded. Usually a more invasive technique, called the Fundus Fluorescence Angiography, follows that uses a dye which is injected in the eye to obtain a better localization of blood vessels and the doctors thus try to make more informed decisions. While it is a commonly available technique in the developed countries, the cost is a major impediment in using this test often in the developing regions of the world. The proposed solution attempts to utilize the basic examination data from the ophthalmologists’ standard clinical procedures and improves the quality of the same to an extent that the doctors can make a better informed decision without having to resort to more expensive and invasive techniques.

II. THE DECONVOLUTION MODEL

As a general notation, the retinal images are considered as the matrix representation in x and y coordinates and usually are colored in the RGB color space hence producing an N�M�3 image. Thus, the retinal image of interest f(m, n) is a matrix of M�N order that is resulted after 2-D sampling of the real image. A very general Retinal imaging system is shown in Figure 2. The same can be represented more formally as a standard block diagram representation as shown in Figure 3. Hence, the observed image g(m, n) is given by:

���� �� � ���� �� ��� �� � ���� �� (1)

Figure 2. Basic ray diagram to represent the image deconvolution model.

Figure 3. Block diagram for the image deconvolution model.

Where h��� �� represents the degradation model and ⊗ represents the 2-D convolution operation. The additive term v��� �� represents the noise added in the degraded image, further adding to the distortion of image. A. Restoration of Images When the degradation due to blur is to be removed then the process is not as simple as filtering out of the noise. Since the blurring of the image is essentially a convolution process of the clean image with the Point Spread Function (PSF) of the blur, the principal procedure in any restoration scheme is finding out an inverse procedure to cancel out the effect of the blur PSF. This is also called deconvolution or inverse filtering. Given the image model of (1) without noise, assuming F = f(m, n), G = g(m, n), and H = h(m, n), then

FHGFHG ˆˆˆ =⇔⊗= (2)

where G , H , and F are the Fourier Transforms for G, H, and F respectively. Therefore by inverse filtering or deconvolution operation, the original image can be retrieved as follows:

( ) GDFGDF ⊗ℑ=⇔= −−− 111 ˆˆˆ (3) Where 1−ℑ represents the inverse Fourier Transform. This seemingly simple problem is not so simple in reality. In many

21

practical cases, it is useless or even impossible to apply it as given by (2) and (3). This is mostly due to the reason that the PSF is usually unknown and is often zero in wide ranges. Hence D-1 will be infinite in this case. If the additive noise, v(m, n), is also considered for the inverse filtering problem, then the end result will be similar to (3) but with an additional term of D-1v(m, n). This means that even if

D is nonzero, the noise will be amplified by a constant factor of D-1. In effect, the signal to noise ratio in this case is not improved but stays the same due to the fact that noise and the degraded system are amplified by the same factor. When the blurring function h(m, n) is not accurately known, then h(m ,n) must be estimated prior to inverse filtering. Since the attempt is to deconvolve the degraded image g(m, n) without any prior knowledge of the cause that is producing the degradation, such a procedure is called Blind Deconvolution, as it is blind to the source of the degradation. B. Problem Formulation Given a degraded image g(m, n) (Figure 1), with the characteristics of both h(m, n) and v(m, n) being unknown, the problem is to recover f(m, n) by deconvolving the degraded image g(m, n). Optimal H2 filters (or Kalman filters) are suitable solutions to the filtering and estimation problem when the power spectral density of the noise is precisely known. In general least square estimation problems, one minimizes the integral of the power spectral density estimation error. This minimization of the average error power or error variance might result in a relatively large error power in some frequency range. In many practical situations, however, there is significant uncertainty in the power spectral density of the noise.

C. Maximum Likelihood Deconvolution

Maximum likelihood deconvolution is an improved subset of Iterative Constrained optimization algorithms [10]. The iteration is designed based upon a probability model. The mathematics of this algorithm is based upon the behavior of quantum photon emissions and diffraction. Among all known approaches, the Maximum Likelihood approach has proved to provide the best quality images [11, 12]. Usually, in MLE, it is known that the function H belongs to a certain family of distributions{Jh(·|θ), θ ∈ ΘJ}, called the parametric model, so that H = h(m, n|θ0). The value θ0 is unknown and is referred to as the “true value” of the parameter. It is desirable to find some ���(the estimator) which would be as close to the true value θ0 as possible. To use the method of maximum likelihood, one first specifies the joint density function for all observations. For iid sample this joint density function will be

����� ��� � � ����� � ������� � �������� ������� (4) Where x1 to xk represent the k-sized window of data extracted and worked upon by the iterative algorithm. Then, the extended density can be considered as a function of the parameter θ. This extended density is the likelihood function of the parameter [13]:

������� ��� � � ��� � ����� ��� � � ����� � � ���������� (5)

In practice it is often more convenient to work with the logarithm of the likelihood function, ln ϕ, called the log-likelihood, or its scaled version, called the average log-likelihood:

� �������� ��� � � ��� � ! � ��������� �" ��

�� �φ�

��� (6) where �" is the expected log-likelihood of a single observation in the model. The method of maximum likelihood estimates θ by finding a value of θi that maximizes �"�����:

��#$% �&'���&�� ∈ Θ ��"������ ��� � � ��� (7)

D. Blind Deconvolution

Blind deconvolution is a subset of Iterative Constrained algorithms which produce an estimate of h(m, n) concurrently with f(m, n). It does not need the PSF h(m, n) to be measured. Blind deconvolution was first introduced to the imaging community, outside of light microscopy by Ayers and Dainty [14]. It is a blind deconvolution only if the algorithm is producing the PSF from information within the data set g(m, n). This is done by first assuming a hi(m, n), then estimating which f(m, n) could have caused g(m, n). This calculation is followed by estimating which h(m, n) could have caused g(m, n) from the estimated f(m, n), and then these steps are repeated again and again. It is believable that the PSF information is in the data because one often sees the light spreading from fine point or line structures in the data, and this spreading makes up the PSF.

III. THE RETINAL IMAGES The images we are working with are obtained as part of the ophthalmologic procedure called Fundoscopy in which the image of the retina and its surrounding is obtained using a standard Fundus photographic camera. The images used belong to patients having diabetes mellitus and Cataract. The purpose of taking these images is to look into the changes seen in patients having diabetic retinopathy. In contrast to most of the presented work (a sample of which is shown in Figure 5) the analyzed case is extremely difficult to be tackled by using the previously reported techniques such as contrast stretching, histogram equalization, filtering, and Region of Interest (ROI) techniques [2-9]. Also, most of the available databanks (e.g., [15]) containing these images also have the cleaner images.

22

Figure 4. The actual blurred image used in this work.

Figure 5. Type of images used in general for enhancement.

IV. PROPOSED ALGORITHM For a black-box problem, as the one in hand, the inverse approach to getting the actual image from its blurred measurement is called blind deconvolution. The idea is to initialize an iterative algorithm with a pre-selected structure (state-space or transfer function) with random or flat initialization. Using this assumed structure, a new image estimate is then obtained. A tuning parameter is then adjusted based on the error between this estimate and the actual image. Since nothing is known for the true image as well, some statistical assumptions are needed to decide on the correction. Usually, some form of covariance matrix is used to adjust the underlying model and then the iterative procedure is continued until a suitable convergence is reached [11]. The general algorithm proposed in this work is shown in Figure 6. Once the deconvolution loop (Loop t) finishes, the resulting

image f is quite improved in terms of its visual appearance. For manual inspection purposes this image is sufficiently improved for most of the ophthalmologic parameters. However, a post processing step is also proposed here to identify the three most important components in a retinal image that corresponds to the diagnosis of almost all the eye-related diseases. These components are (as listed in Figure 1) Optical Disc, Major Vessels, and Macula.

The post-processing steps are related to the Region of Interest (ROI) processing and are outlined as follows:

Figure 6. The blind deconvolution algorithm used in this work.

Locating Disc:

1. The deconvolved RGB image is first converted to YCbCr format [16, 17].

2. Each layer of the resulting image is equalized for gray-scale histogram.

3. The resulting image is then multiplied by a very large number to make the disc portion prominent.

4. A binary image is then obtained by simple 50% thresholding.

5. Further removal of artifacts is done using Morphological operators “Dilate” and “Remove” successively [16, 17]

6. The ROI (disc) is then isolated and further contrast enhancement is done to enhance the RGB image area corresponding to the isolated binary mask.

Major Vessels:

1. Each layer of the deconvolved RGB image is individually equalized for the gray-scale histogram.

2. The RGB image is then displayed with equalized layers and shows the major vessels much more clearly.

3. Further clarity is obtained in the 2nd layer in the RGB image is viewed separately.

Macula:

1. Isolate the 1st layer of the deconvolved and equalized RGB image and convert it to binary at 90% thresholding.

2. Further removal of artifacts is done using Morphological operators “Erode” and “Remove” successively [16, 17]

3. The ROI (macula) is then isolated and further contrast enhancement is done to enhance the RGB image area corresponding to the isolated binary mask.

Figure 7 is a flow diagram that represents the complete algorithmic implementation.

Start

LoadImage (g)

Initialize h=f(θ,m,n), fi

Loop tUntil error

< γ

DisplayImages

End

Loop (m,n) Till M, N

Calculate Iteratively

Convolve and fi to get

2fg −=γ

Update fi to

ROI Post Processing:1. Color-space

Transformation2. Histogram

Equalization3. Edge Detection

23

Figure 7. Flow diagram showing the implementation of the algorithm to extract various components of the retinal image.

V. RESULTS AND DISCUSSION

The algorithm is implemented in MATLAB using the Image Processing toolbox and the results are shown Using the blurred image from Figure 4, the first step was to resize the image to 25% in order to speed up the processing. However, no interpolation is done later on to retrieve the same size after processing the images. Figure 8 shows thimage after the application of the blind deconvolution.

Figure 8. The deconvolved image. The process of deconvolution was initiated with a general purpose Gaussian kernel as the estimated blurring function. Figure 9 shows the initial and the final blurring kernel resulted as a result of the deconvolution process.

(a) (b)Figure 9. Blurring functions, (a) initial estimate of a standard Gaussian

function, and (b) final estimate at which the deconvolution iterations were stopped.

Figure 7. Flow diagram showing the implementation of the algorithm to

extract various components of the retinal image.

V. RESULTS AND DISCUSSION

The algorithm is implemented in MATLAB using the Image in the following.

, the first step was to resize the image to 25% in order to speed up the processing. However, no interpolation is done later on to retrieve the same

Figure 8 shows the resulting image after the application of the blind deconvolution.

The process of deconvolution was initiated with a general purpose Gaussian kernel as the estimated blurring function.

final blurring kernel resulted

(b)

Figure 9. Blurring functions, (a) initial estimate of a standard Gaussian function, and (b) final estimate at which the deconvolution iterations were

In an iterative scheme such as the one presented here, several compromises are already associated by assuming the system to be linear and that a valid initial guess actually exists for the blurring function. The deconvolution can fail as well due to incorrect initial start, in which case the quality of the image can further deteriorate. While the deconvolved image is already a lot enhanced in terms of visual quality, further clarity is achieved through post-processing steps. First of all, only second RGB image was displayed and was found to be much more clearing terms of its details related to identifying the vessels and the related hemorrhages. This is shown as labeled image in Figure 10.

Figure 10. Post processed image to display the

In addition to the correct spotting of the clear detection of Optical disc and Macula is done.artifacts are also pointed out but these are ruled out as artifacts due to their location. These artifacts are a errors and are usually negligible. Once located, the area under the binary mask with some percentage extension, can be further enhanced by using the contrast enhancement and histogram equalization. Figure 12 shows the same procedure for localization of Macula.

(a)

Figure 11. Detecting the location of the optical disc, (a) binary mask, and (b) mask superimposed on the actual deconvolved image.

In an iterative scheme such as the one presented here, several compromises are already associated by assuming the system to be linear and that a valid initial guess actually exists for the blurring function. The deconvolution can fail as well due to

ect initial start, in which case the quality of the image

While the deconvolved image is already a lot enhanced in terms of visual quality, further clarity is achieved through

First of all, only second slice of the RGB image was displayed and was found to be much more

terms of its details related to identifying the vessels and the related hemorrhages. This is shown as labeled image

Figure 10. Post processed image to display the hemorrhaged regions.

In addition to the correct spotting of the hemorrhages, the detection of Optical disc and Macula is done. Some

artifacts are also pointed out but these are ruled out as artifacts due to their location. These artifacts are a result of imaging

Once located, the area under the binary mask with some percentage extension, can be further enhanced by using the contrast enhancement and histogram equalization. Figure 12

localization of Macula.

(b)

Figure 11. Detecting the location of the optical disc, (a) binary mask, and (b) mask superimposed on the actual deconvolved image.

24

Figure 12. Detecting the location of the macula, (a) binary mask, and (b)

mask superimposed on the actual deconvolved image.

VI. CONCLUSION

In this paper, we have presented a new technique to restore the distorted retinal image occluded by a specific blur similar to the out-of-focus blur. The proposed deconvolution approach has shown promising results and will be further explored for an ultimate clinical tool development. Such a tool will be very useful for the ophthalmological experts in examining the eye in a better way and without risking un-necessary medical and chemical procedures on the eye in order to obtain a better picture of the things. The post processing steps enable the usage of the system on another level where specific areas of the eye can by automatically identified and further enhanced. The main importance of the work is the extraction of these important areas from a very bad quality image. Getting such an image is usually not very common, but when it is obtained, it leaves the doctors guessing what to do next. Usually they will start some type of treatment based on their understanding of the disease which may or may not be correct. With our proposed technique, they can get a more inside knowledge and that will help them in making a better and more informed decision regarding the illness. Further work is underway to make the proposed algorithm more reliable and more robust to other naturally occurring factors. Cataract has been the most prevalent cause of blindness in developing world, diabetic retinopathy is right now the fastest growing disease that leads to blindness. one stage of diabetic retinopathy is clinically significant macular edema, which has to be treated otherwise it will lead to significant loss of vision. Many times ophthalmologist face certain circumstances in which patients are diabetic but their fundus cannot be viewed due to cataract. Many of such patients have diabetic macular edema as well. In order to look inside the eye, the ophthalmologist first removes cataract by cataract surgery than examines the fundus for diabetic changes. In patients who have diabetic macular edema, cataract surgery leads to increase in edema with further loss of vision. That’s why it is recommended that in patients having diabetic macular edema and cataract, edema should be treated first than cataract should be extracted. But in case of dense opacities this practice cannot be done practically. Our technique will help ophthalmologists in such circumstances, instead of removing cataract by cataract surgery, they can treat edema first and then later on with improvement in edema can plan cataract surgery thus preventing the exaggeration of macular edema

that would otherwise occur if cataract surgery would have been performed.

REFERENCES

[1]. Jack, J. K., “Fundus photograph of diabetic retinopathy patients”, Clinical Ophthalmology, 6th Edition, Elsevier Publishing Company, 2007.

[2]. Xu, Z., Guo, X., Hu, X., Cheng, X., and Wang, Z., “The blood vessel recognition of ocular fundus”, 7th Argentinean Symposium on Artificial Intelligence, 2005, pp. 183-190.

[3]. Sagar, A. V., Balasubramanian, S., and Chandrasekaran, V., “Automatic Detection of anatomical structures in digital fundus retinal images”, IAPR Conference on Machine Vision Applications, 2007, pp. 483-486.

[4]. Mengko, T. R., Handayani, A., Valindria, V. V., Hadi, S., and Sovani, I., “Image processing in retinal Angiography: Extracting angiographical features without the requirement of contrast agents”, IAPR Conference on Machine Vision Applications, 2007, pp. 451-454.

[5]. Youssif, A., Ghalwash, A., and Ghoneim, A., “Comparative study of contrast enhancement and illumination equalization methods for retinal vasculature segmentation”, Cairo International Biomedical Engineering Conference, 2006, pp. 1-5.

[6]. Walter, T., Klein, J., Massin, P., and Erginay, A., “A contribution of image processing to the diagnosis of diabetic retinopathy – detection of exudates in color fundus images of human retina”, IEEE Transactions on Medical Imaging, Vol. 21, No. 10, 2002, pp. 1236-1243.

[7]. Li, H., and Chutatape, O., “Fundus image feature extraction”, IEEE 22nd Annual EMBS International Conference, 2000, pp. 3071-3073.

[8]. Hani, A., Izhar, L., and Nugroho, H., “Analysis of Foveal Avascular zone in color fundus image for grading of diabetic retinopathy”, International Journal of Recent Trends in Engineering, Vol. 2, No. 6, 2009, pp. 101-104.

[9]. Iqbal, M., Aibinu, A., Gubbal, N., and Khan, A., “Automatic diagnosis of diabetic retinopathy using fundus images”, Master thesis, Belkinge Institute of Technology, Sweden, 2006.

[10]. Holmes, T. J., “Background of Deconvolution”, Media Cybernetics Application Note, August 2006, http://www.mediacy.com/pdfs/Applications/BackgroundofDeconvolution.pdf

[11]. Verveer, P., Computational and Optical Methods for Improving Resolution and Signal Quality in Fluorescence Microscopy, PhD Thesis, Tecnische Universiteit Delft, 1998.

[12]. Holmes, T., Bhattacharyya, S., Cooper, J., Hanzel, D., Krishnamurthi, V., Lin, W., Roysam, B., Szarowski, D., Turner, J., Light Microscopic Images Reconstructed by Maximum Likelihood Deconvolution, Ch. 24, Handbook of Biological Confocal Microscopy, J. Pawley, Plenum, 1995.

[13]. Le Cam, Lucien; Lo Yang, Grace (2000). Asymptotics in statistics: some basic concepts (Second Ed.). Springer. ISBN 0-387-95036-2.

[14]. Ayers, G.R., Dainty, J.C., Iterative Blind Deconvolution Method and Its Applications,

[15]. The Stare database: http://www.ces.clemson.edu/~ahoover/stare/

[16]. Qidwai, U., and Chen, C.H., Digital Image Processing: An Algorithmic approach with MATLAB, CRC Publishing Company, November 2009.

[17]. Image processing toolbox in MATLAB.

25