10
A. Taşyapı et al.: Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation 119 Contributed Paper Manuscript received 12/31/14 Current version published 03/30/15 Electronic version published 03/30/15. 0098 3063/15/$20.00 © 2015 IEEE Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation Aysun Taşyapı ÇELEBİ, Ramazan DUVAR, and Oğuzhan URHAN, Member, IEEE Abstract In this work, a high dynamic range (HDR) image generation method using a single input image is presented. The proposed approach generates over- and under-exposed images by making use of a novel adaptive histogram separation scheme. Thus, it becomes possible to eliminate ghosting effects which generally occur when several input image containing camera/object motion are utilized in HDR imaging. Additionally, it is proposed to utilize a fuzzy logic based approach at the fusion stage which takes visibility of the inputs pixels into account. Since the proposed approach is computationally light-weight, it is possible to implement it on mobile devices such as smart phones and compact cameras. Experimental results show that the proposed approach is able to provide ghost-free and improved HDR performance compared to the existing methods 1 . Index Terms — High dynamic range, HDR, Histogram separation, Ghost-free. I. INTRODUCTION Traditional imaging devices can capture limited range of intensities. A conventional consumer grade camera has typically 24-bit depth per pixel and 256 discrete intensity levels for each color channel of the image. However, light intensity in the real world has much larger dynamic range compared to that can be shown on digital images or computer screens. In order to overcome dynamic range limitation of the typical low dynamic range (LDR) images, High Dynamic Range (HDR) imaging techniques have been utilized in recent years as an alternative approach for digital imaging [1]. HDR images generally have higher quality and powerful appearance. In HDR imaging, usually a set of differently exposed low dynamic range (LDR) images are fused into a single image to overcome the dynamic range limitation of existing imaging sensors. As a result, the fused image may have higher amount of information and enhanced visual quality because of its extended dynamic range. Recently, various approaches have been proposed in the literature to produce HDR images with extended dynamic range using LDR images. In these techniques, generally multiple images of the scene are captured at different exposure times via a single camera and fused to form the HDR image 1 A.T. Celebi is with the Electronics and Telecom. Eng. Dept., University of Kocaeli, Umuttepe Campus, 41380, Izmit/Kocaeli, Turkey (e-mail: [email protected]) R. Duvar and O. Urhan are with the Kocaeli University Laboratory of Embedded and Vision Systems (KULE), Electronics and Telecom. Eng. Dept., Umuttepe Campus, 41380, İzmit/Kocaeli, Turkey (e-mail: [email protected], [email protected]). [2], [3]. Shen et al [4] proposed a novel algorithm based on extended Retinex model to fuse multi-exposure images. Instead of composing the input image intensities, this approach decomposes them into the luminance and reflectance components which are composed independently before being merged together to produce the final result. Gu et al [5] proposed an algorithm to fuse multi-exposure images in the gradient field. Fused gradient field was derived from the structure tensor of inputs based on multi-dimensional Riemannian geometry with a Euclidean metric. However, these approaches are mostly suitable for static scenes only. In case of dynamic scenes with camera motion and moving objects such as car, people etc., various artifacts such as ghosting cannot be avoided due to the sequential capture of differently exposed images. There are various methods in which the camera and local motion are compensated to eliminate ghosting effect. Jacobs et al [6] proposed a ghost detection approach based on entropy computation. The local entropy of each LDR image is extracted and then added to the difference of local entropy values with weighting factors computed between LDR images in this method. Khan et al [7] presented an iterative ghost effect removal approach. This method generates HDR image by weighting the probability of the pixels belonging to the background or the moving objects. Gallo et al [8] removed the ghost using a patch based method where the Poisson blending is performed for removing the boundary discontinuity. Im et al [9] proposed LDR image registration where the camera motion is compensated to align multiple LDR images using elastic registration method for removing ghost effect. Zheng et al [10] proposed a hybrid patching algorithm to prevent ghosting effect where a new optimization problem is formulated to correct the motion regions of differently exposed images by considering both spatial and temporal consistencies. Zhang et al [11] proposed to use gradient direction changes among the different images for object motion detection. The main drawback of this approach is that it estimates the moving objects without considering the local properties of ghost regions. In addition, this approach assumes that the moving object occupies only a small portion of the image. Zhang et al [12] proposed a new de-ghosting algorithm to overcome these shortcomings of the method presented by Zhang et al [11] by taking not only the temporal consistency but also spatial consistency. Thus, this approach generalizes the previous assumption and allows higher amount of local motion. However, if an inappropriate exposed image is selected as the reference view, this approach may fail to produce acceptable HDR images. Lee et al [13] proposed to obtain a ghost-free

ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

Embed Size (px)

Citation preview

Page 1: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

A. Taşyapı et al.: Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation 119

Contributed Paper Manuscript received 12/31/14 Current version published 03/30/15 Electronic version published 03/30/15. 0098 3063/15/$20.00 © 2015 IEEE

Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation

Aysun Taşyapı ÇELEBİ, Ramazan DUVAR, and Oğuzhan URHAN, Member, IEEE

Abstract — In this work, a high dynamic range (HDR) image

generation method using a single input image is presented. The proposed approach generates over- and under-exposed images by making use of a novel adaptive histogram separation scheme. Thus, it becomes possible to eliminate ghosting effects which generally occur when several input image containing camera/object motion are utilized in HDR imaging. Additionally, it is proposed to utilize a fuzzy logic based approach at the fusion stage which takes visibility of the inputs pixels into account. Since the proposed approach is computationally light-weight, it is possible to implement it on mobile devices such as smart phones and compact cameras. Experimental results show that the proposed approach is able to provide ghost-free and improved HDR performance compared to the existing methods1.

Index Terms — High dynamic range, HDR, Histogram separation, Ghost-free.

I. INTRODUCTION

Traditional imaging devices can capture limited range of intensities. A conventional consumer grade camera has typically 24-bit depth per pixel and 256 discrete intensity levels for each color channel of the image. However, light intensity in the real world has much larger dynamic range compared to that can be shown on digital images or computer screens. In order to overcome dynamic range limitation of the typical low dynamic range (LDR) images, High Dynamic Range (HDR) imaging techniques have been utilized in recent years as an alternative approach for digital imaging [1]. HDR images generally have higher quality and powerful appearance. In HDR imaging, usually a set of differently exposed low dynamic range (LDR) images are fused into a single image to overcome the dynamic range limitation of existing imaging sensors. As a result, the fused image may have higher amount of information and enhanced visual quality because of its extended dynamic range.

Recently, various approaches have been proposed in the literature to produce HDR images with extended dynamic range using LDR images. In these techniques, generally multiple images of the scene are captured at different exposure times via a single camera and fused to form the HDR image

1 A.T. Celebi is with the Electronics and Telecom. Eng. Dept., University

of Kocaeli, Umuttepe Campus, 41380, Izmit/Kocaeli, Turkey (e-mail: [email protected])

R. Duvar and O. Urhan are with the Kocaeli University Laboratory of Embedded and Vision Systems (KULE), Electronics and Telecom. Eng. Dept., Umuttepe Campus, 41380, İzmit/Kocaeli, Turkey (e-mail: [email protected], [email protected]).

[2], [3]. Shen et al [4] proposed a novel algorithm based on extended Retinex model to fuse multi-exposure images. Instead of composing the input image intensities, this approach decomposes them into the luminance and reflectance components which are composed independently before being merged together to produce the final result. Gu et al [5] proposed an algorithm to fuse multi-exposure images in the gradient field. Fused gradient field was derived from the structure tensor of inputs based on multi-dimensional Riemannian geometry with a Euclidean metric. However, these approaches are mostly suitable for static scenes only. In case of dynamic scenes with camera motion and moving objects such as car, people etc., various artifacts such as ghosting cannot be avoided due to the sequential capture of differently exposed images.

There are various methods in which the camera and local motion are compensated to eliminate ghosting effect. Jacobs et al [6] proposed a ghost detection approach based on entropy computation. The local entropy of each LDR image is extracted and then added to the difference of local entropy values with weighting factors computed between LDR images in this method. Khan et al [7] presented an iterative ghost effect removal approach. This method generates HDR image by weighting the probability of the pixels belonging to the background or the moving objects. Gallo et al [8] removed the ghost using a patch based method where the Poisson blending is performed for removing the boundary discontinuity. Im et al [9] proposed LDR image registration where the camera motion is compensated to align multiple LDR images using elastic registration method for removing ghost effect. Zheng et al [10] proposed a hybrid patching algorithm to prevent ghosting effect where a new optimization problem is formulated to correct the motion regions of differently exposed images by considering both spatial and temporal consistencies. Zhang et al [11] proposed to use gradient direction changes among the different images for object motion detection. The main drawback of this approach is that it estimates the moving objects without considering the local properties of ghost regions. In addition, this approach assumes that the moving object occupies only a small portion of the image. Zhang et al [12] proposed a new de-ghosting algorithm to overcome these shortcomings of the method presented by Zhang et al [11] by taking not only the temporal consistency but also spatial consistency. Thus, this approach generalizes the previous assumption and allows higher amount of local motion. However, if an inappropriate exposed image is selected as the reference view, this approach may fail to produce acceptable HDR images. Lee et al [13] proposed to obtain a ghost-free

Page 2: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

120 IEEE Transactions on Consumer Electronics, Vol. 61, No. 1, February 2015

Input Image

RGB/HSVAdaptive

WHS

LS

LS

CLAHE

CLAHE

Fuzz

y B

ased

Fusio

n

V0

V1

N-Exposed Image

VOE

VUE

V

O-Exposed Image

HSV/ RGB

U-Exposed Image

HSV / RGB

INE

IOE

IUE

S

H

I HDRImage

V0_LS

V1_LS

Fig. 1. The flowchart of proposed method.

high dynamic range image based on a low-rank matrix completion. The background and moving objects are represented as a low-rank matrix and sparse matrix, respectively. Next, the ghost region detection is formulated as the low-rank matrix completion problem with multiple physical constraints on the properties of the ghost regions.

However, these kind of approaches have considerably higher computational complexity which may not be suitable for mobile/consumer level cameras and real-time applications. Recently, Im et al [14] proposed to generate high dynamic range (HDR) images using a single image instead of acquiring differently exposed images in order to avoid ghosting effect. This method utilizes weighted histogram separation (WHS) to estimate the threshold for histogram separation. Then it divides the histogram into two sub-histograms and generates differently exposed (over and under) LDR images from a single image. This method is not affected by ghosting effect inherently since it generates differently exposed images from a single image. However, this method utilizes a fixed weighting factor for histogram separation which is not suitable for images displaying different characteristics. Im et al [15] proposed a similar approach for generating ghost-free HDR images. This method utilizes histogram quantization based spatially adaptive histogram equalization approach and generates two LDR images using single input image by performing two sub-histogram equalizations. Finally, it fuses these LDR images with the input image to produce the HDR image by making use of a fixed weight for all input images.

In this paper, it is proposed to produce HDR image using a single image similar to [14]. However, the proposed approach computes an adaptive weight for each image to improve HDR performance. Additionally, contrast limited adaptive histogram equalization (CLAHE) is utilized to improve overall appearance of the HDR image in local dark and bright regions. As a final contribution, fuzzy logic based fusion of LDR images is presented which takes pixel visibility into account. Thus, the proposed single image based HDR approach produces an image which has increased dynamic range and more local details.

II. PROPOSED ADAPTIVE HDR APPROACH The proposed single image based HDR approach has three

main steps. The first step is to generate LDR images from a single original image by making use of an adaptive approach. At the second step, these LDR images are enhanced using a hybrid enhancement approach which includes global Linear Stretching (LS) and local CLAHE. Finally, the LDR images are fused employing a fuzzy logic based approach to obtain output HDR image. The block diagram of the proposed method is shown in Fig. 1. The novel contributions of this work are highlighted in this figure.

The proposed method utilizes adaptive weighted histogram separation firstly in order to perform single image based HDR. The weighted histogram separation (WHS) is presented by Pei et al [16]. However, a fixed weight at the WHS process is not suitable for wide range of images. Thus, it is proposed to obtain adaptive weights specific to input images. It is important to note that only the V (Value) channel is used along the HDR process.

The WHS method with the proposed adaptive weight computation is applied to the V channel to obtain initial under- and over-exposed images. LS and CLAHE are executed on these images to obtain final under- and over-exposed images. It is important to note that, by the introduction of the CLAHE, local details in the input images are extracted efficiently. Finally, the HDR image is constructed from these three differently exposed LDR images by making use of a fuzzy based fusion approach which takes pixel visibility into account.

A. Adaptive Weighted Histogram Separation Histogram separation stage is one of the most important part

for single image based HDR methods because it directly affects the generation of LDR images and subsequent stages in HDR process. In the proposed approach, two differently exposed LDR images are generated from the V channel of the input image. The WHS as explained by Pei et al [16] is used to divide the input image into two separate images by making use of an adaptively computed weight. WHS can be carried

Page 3: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

A. Taşyapı et al.: Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation 121

out using so-called data separation units (DSUs) which separate an image dataset into two subsets. While in practice it is possible to further separate each subset into subsequent subsets using a DSU hierarchy, it has been observed that using only one level is sufficient in terms of image quality and computational complexity.

Note that VH is used to denote the histogram of the V

channel of the input image, that is the dataset used in WHS, and ( )VH l denotes the number of pixels with gray level l in V,

where l ranges from 0 to 255 for an 8-bit image. The following steps describe the WHS. Step 1: Compute the threshold for separating V:

0 0

1argmin ( )

t

Vt V l

w H ln

(1)

where t denotes a gray-level value, and n is the total number of pixels in the dataset, w is an experimentally determined weighting factor controlling the size of the two separated subsets. Im et al [14] sets this weighting factor to a fixed number. Thus, this approach is not able to separate histograms efficiently for images with different characteristics.

In this paper, it is proposed to utilize an adaptive weighting scheme to separate input image histograms in order to obtain LDR images for a better HDR image. The experiments performed show that a relatively small weighting factor results in detail loss in the HDR image whereas a higher weighting factor is not able to improve dynamic range of the input image. On the other hand, a mid-range fixed weight cannot improve the image quality for relatively brighter and darker images. Thus, it is very important to decide a proper weight for different kind of input images. Thus, in this paper, it is proposed to utilize an adaptive weight based histogram separation approach. The experiments carried out reveal that there is a nearly linear relation between the amount of the pixels in relatively dark region and the optimal weights. The ratio of dark region to overall pixel intensities is defined as

( )

( )

b

Vm a

Vl

H mr

H l

(2)

where a and b are fixed intensity levels that correspond to the dark region in the input image. Fig. 2 shows optimal weights obtained by means of visual examination versus r for twelve input images having different characteristics. Note that a and b are set to 0 and 64, respectively for this examination.

As seen from Fig. 2, optimal weights and r values have relatively linear relationship with some exceptions. In order to estimate linear model parameters, two approaches are examined: least squares and RANSAC (RANdom SAmple Consensus) [17]. Since the least squares approach takes all

Fig. 2. Optimal weights versus r values and their linear relation estimated by least squares fitting and RANSAC methods.

input data into account, it suffers from outliers. On the other hand, RANSAC eliminates outliers by making use of an iterative approach. Because of this improvement, RANSAC is able to fit a better model between optimal weight and r as seen from Fig. 2.

The adaptive weight w in the proposed approach is determined as follow: w r (3)

where and are the estimated model parameters by

RANSAC. Step 2: After estimating the adaptive weight factor (w) and computing the threshold ( ), divide the input histogram into two sub histograms based on the threshold value:

0

,

0 ,V

V

H l if lH l

otherwise

(4)

1

,

0,V

V

H l if lH l

otherwise

(5)

Two images are generated by dividing the histogram into two subsets making use of the adaptive WHS approach. While the first image referred to as 0V contains intensity levels from 0 to

the threshold , the second image referred to as 1V contains

intensity levels from the threshold to 255, for an 8-bit depth image. Note that, 0V and 1V is used to obtain the over- and

under-exposed images, respectively.

B. Linear Stretching

The linear stretching enables efficient utilization of whole dynamic range and is applied to 0V and 1V . The linear

stretching can be formulated as

0 0

0 _

0 0

, min ,, 255

max , min ,LS

V x y V x yV x y

V x y V x y

(6)

Page 4: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

122 IEEE Transactions on Consumer Electronics, Vol. 61, No. 1, February 2015

where 0 ,V x y denotes the images obtained by

applying WHS, and 0 _ ,LSV x y shows the over-exposed

image obtained after the linear stretching. The under-exposed image 1_ ,LSV x y is obtained by processing 1V in the same

way.

C. Contrast Limited Adaptive Histogram Equalization

Im et al [14] propose to apply only linear stretching to 0V and 1V for obtaining differently exposed images. In this paper,

it is proposed to apply CLAHE [18] after linear stretching in order to improve appearance of the local details in HDR image. Note that a recently presented approach by Çelebi [19] proposes to utilize CLAHE together with linear stretching specifically for dynamic range improvement on underwater images.

Adaptive histogram equalization (AHE) is a method for local contrast enhancement and it is an extension to the traditional histogram equalization technique. AHE simply partitions the image into non-overlapping regions and applies histogram equalization to each sub-region in order to redefine the pixel values of the image. As a result, contrast of each region is enhanced by improving local details. One of the important problem with the AHE is the over amplified noise in relatively homogeneous regions. CLAHE is an improved version of AHE which prevents over amplification of noise in homogenous regions by limiting the contrast. In CLAHE, each sub-histogram is partially flattened by clipping the values which are larger than a given threshold, and distributing them to other bins in the histogram. By selecting the clipping level of the histogram, undesired noise amplification can be reduced.

In the proposed HDR approach, CLAHE is applied to the

0 _ LSV and 1_ LSV separately to enable enhancement of local

details. Thanks to CLAHE, a higher dynamic range is achieved on local regions by virtue of local pixel statistics.

D. Fuzzy Logic Based Exposure Fusion

The final stage of the proposed method is the fusion of over-, under- and normally-exposed images to generate the HDR image. Firstly, the over-exposed V channel ( OEV ) and

under-exposed V channel ( UEV ) are combined individually

with the unprocessed H and S channels and transformed back into the RGB color space to obtain the over-exposed and under-exposed images for fusion process. Fig. 3 shows an example of under-exposed ( UEI ) and over-exposed ( OEI )

images together with the normally-exposed image ( NEI ).

The idea of exposure fusion is to exploit the best parts of the differently exposed images and fuse them to generate a high quality image. There are several quality metrics for this purpose, such as well-exposedness, saturation, contrast, or variance.

In this paper, it is proposed to utilize pixel visibility criterion to determine contribution of LDR images (i.e. UEI , OEI and

NEI ) on the final HDR image. The pixel visibility can be

(a) (b) (c) Fig. 3. (a) The normally-exposed image (

NEI ), (b) the over-exposed image,

(c) the under-exposed image (UEI ).

(a) The normalized difference between local max and min values

(b) Normalized pixel intensity

(c) Output

Fig. 4. Membership functions of inputs and output of the fuzzy fusion method.

assessed by taking only the pixel values into account as in the method presented by Im et al [14]. However, the detailed experiments reveal that it might be possible to improve image quality when local properties of the pixels are also considered. Based on this observation, it is proposed to analyze the amount of the local details surrounding each pixel in LDR images. Local variance computation for each pixel provides important information about the amount of details in the local region. However, its computational cost is not suitable for devices having limited processing capabilities. Thus, it is proposed to capture same information by making use of local min-max operations with a fixed window length. The difference between local extreme values provides evidence about the amount of local details within the examined region. Another important information about the pixel visibility is its intensity value.

In this paper, it is proposed to utilize these two important evidences about the pixel visibility by making use of a fuzzy approach to decide contribution level of each pixel in LDR images to the HDR image. The smoothed and normalized difference between the local extremes is assigned as the first input of the fuzzy system whereas the normalized pixel value of the V channel is considered as the second input. The smoothing is required to eliminate unwanted local noise in the fuzzy inputs. The membership functions for the inputs and output are shown in Fig. 4.

Page 5: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

A. Taşyapı et al.: Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation 123

TABLE I. RULE BASE OF THE PROPOSED FUZZY FUSION APPROACH

Rule Number

I1 I2 Output

1 mf1 mf1 mf1 2 mf1 mf2 mf1 3 mf1 mf3 mf1 4 mf2 mf1 mf3 5 mf2 mf2 mf2 6 mf2 mf3 mf1 7 mf3 mf1 mf3 8 mf3 mf2 mf2 9 mf3 mf3 mf1

The rule base for the proposed fuzzy fusion approach is

given in Table I. As seen from the rule base, when the difference between local extremes around a pixel is low then the contribution of this pixel on the HDR image will be low even though the pixel intensity is relatively high. These rules eliminate erroneous contribution of the pixels located at the homogenous region. It is important to note that the contribution of the pixels having mid-level values are higher compared to that of relatively low-level and high-level pixel values (see Fig. 4(b)).

Once the contribution level of each pixel in the LDR images is determined the final HDR image is constructed as

, ,

, ,

, ,

, ,,

, ,

ck k

k OE UE NEcHDR

kk OE UE NE

I x y FuzzyWeight V x y

I x yFuzzyWeight V x y

for c R G B

(7)

where c

kI and c

HDRI are denote LDR images and the output

HDR image, respectively.

III. EXPERIMENTAL RESULTS

The proposed method is tested using various input images that have different characteristics. Fig. 5 shows comparative results of the approach presented by Im et al [14] and proposed approach for different input images. As described in previous section, Im et al [14] proposed to generate HDR image using histogram separation with a fixed weight for all input images.

(a) (b) (c)

Fig. 5. Performance comparison. (a) Original images, (b) HDR images obtained by Im et al [14], (c) HDR images obtained by the proposed method.

Page 6: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

124 IEEE Transactions on Consumer Electronics, Vol. 61, No. 1, February 2015

However, in this paper, the weight is estimated adaptively for each input image. The results shown in Fig. 5 for the approach presented by Im et al [14] are obtained for a fixed weight of 0.3 which is determined experimentally as the average optimum weight.

These results show that the fixed weight does not perform well for different type of images. But the proposed method obtains better results thanks to adaptive weight. The adaptive weights of these images are computed as 0.44, 0.16 and 0.18, respectively by the proposed approach. It is seen from these results that there is considerable improvement in the dynamic range and visual quality of the images. In addition, HDR images generated by the proposed method have more details, because the regions are properly selected from the LDR images.

When the first row of the Fig. 5 is evaluated, it is obvious that the proposed method provides superior results compared to the approach presented by Im et al [14]. Especially, the details on the sand are extracted efficiently compared to the

approach proposed by Im et al [14]. At the second row of the Fig. 5, the proposed method gives more details on the floor (i.e. paving stone) and shadow region. Additionally, the color of the trees is more appealing compared to the original image and HDR image produced by Im et al [14]. At the third row of the Fig. 5, the clouds on the sky provide good visual quality compared to the original image and the method presented by Im et al [14]. Furthermore, the output image of the proposed method is more brilliant and crisp.

As described in the previous section, it is proposed to apply CLAHE after linear stretching in this paper. In Fig. 6, images are shown together with the zoomed area from the highlighted portions of the images. In this figure, the first column shows the original image, whereas second and third column denote the HDR result of only LS and LS+CLAHE cases, respectively. As seen from these images, the proposed hybrid scheme enhances the details, particularly in the shadow and bright regions.

(a) (b) (c)

Fig. 6. The contribution of the CLAHE. (a) Original images, (b) Images obtained after applying LS only for generating HDR, (c) Images obtained after applying the proposed method (LS+CLAHE).

Page 7: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

A. Taşyapı et al.: Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation 125

In order to evaluate the contribution of fuzzy based fusion of LDR images introduced in this work, appearance based fusion approach presented by Im et al [14] is integrated into the proposed method. The visual results are shown in Fig. 7 for comparison. As seen from this figure, the proposed fuzzy logic based fusion approach not only improves the general HDR performance but also enables efficient HDR image generation for extremely dark regions. In general, these experiments show that the proposed method is able to improve visual quality of the images having different characteristics compared to original images and a recently proposed single image based HDR technique proposed by Im et al [14].

In addition to the method presented by Im et al [14], the HDR performance of the proposed approach is compared against to some recent techniques [11]-[13] where multiple input images are required to generate HDR image. In Fig. 8 and Fig. 9, original differently exposed input images and HDR images generated by the methods presented by Zhang et al [11], Zhang et al [12], Lee et al [13], Im et al [14] and the proposed method are shown for visual assessment. As seen from these figures, the methods presented by Zhang et al [12] and Lee et al [13] still suffer from ghost effect even though these approaches have anti-ghost schemes.

(a) (b) (c)

Fig. 7. The contribution of the fuzzy based fusion. (a) Original image, (b) HDR image obtained after applying exposure fusion by Im et al [14], (c) Image obtained after applying the proposed method (Fuzzy based fusion).

(a) (b) (c) (d) (e)

(f) (g) (h) (i) (j)

Fig. 8. (a-e) The differently exposed images (Image1 sequence), HDR image obtained after applying the approach by (f) Zhang et al [11], (g) Zhang et al [12], (h) Lee et al [13], (i) Im et al [14], (j) proposed method.

Page 8: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

126 IEEE Transactions on Consumer Electronics, Vol. 61, No. 1, February 2015

(a) (b) (c) (d)

(e) (f) (g) (h) (i)

Fig. 9. (a-d) The differently exposed images (Image2 sequence), HDR image obtained after applying the approach by (e) Zhang et al [11], (f) Zhang et al [12], (g) Lee et al [13], (h) Im et al [14], (i) proposed method.

TABLE II. OBJECTIVE IMAGE QUALITY ASSESSMENT

Image 1 Image 2 Entropy CPBD Entropy CPBD

Zhang et al [11] 7.752 0.758 7.171 0.768 Zhang et al [12] 7.385 0.775 7.423 0.764

Lee et al [13] 7.623 0.773 7.629 0.715 Im et al [14] 7.358 0.774 7.637 0.784

Proposed Method 7.648 0.743 7.644 0.789

When the visual HDR performance of the other compared methods is evaluated, the proposed method has better visualization results compared to the methods presented by Zhang et al [11] and Im et al [14]. The HDR performance of the proposed method is also examined by making use of some objective blind image quality criteria such as entropy and Cumulative Probability of Blur Detection (CPBD) [20]. The entropy measures the richness of information in HDR image. Hence, the higher entropy simply means better performance. The CPBD is based on the just noticeable blur (JNB) and is able to predict the relative amount of blurriness in images with different content. A higher value of this metric depicts a sharper image. As the blurriness in an image increases the value of this metric is expected to decrease. Table II shows the entropy and CPBD values of the proposed approach and other methods. As seen from this table the proposed approach provides good performance in this objective evaluation as well.

The approach presented by Im et al [14] and the proposed approach are implemented on a smart phone with a 1.9GHz CPU using C++ language at the same software optimization level. Table III shows average execution times in milliseconds for full HD (1920×1080 pixel) input images. As seen from this table, the computational complexity before the fusion stage is similar. The additional processing time at this stage for the proposed method originates from the implementation of CLAHE introduced in this paper. At the fusion stage, the approach presented by Im et al [14] is implemented by making use of a 1-dimensional Look-Up Table (LUT) based weighting factor computation in addition to color space conversion. However, in the proposed approach, additional local extreme computation and filtering is required to decide contribution level of each pixel in the LDR images to final HDR image.

TABLE III. AVERAGE EXECUTION TIMES ON A SMART PHONE

Im et al [14] Proposed Approach Before fusion stage 814 ms 943 ms

Fusion stage 527 ms 1684 ms Total 1341 ms 2627 ms

These extra computations consume around additional 1300ms compared to the method presented by Im et al [14]. For the implementation of the fuzzy fusion stage in the proposed method, a 2-dimensional LUT is generated by sampling inputs at 0.001 resolution. Thus, the size of LUT is 1000×1000 which requires an additional 8MB of memory. Compared to the 1-dimensional LUT utilized for the implementation of the method presented by Im et al [14], the proposed fuzzy based fusion requires an additional 100ms processing time. Thus, the proposed approach requires approximately two times higher processing time in total for HDR image generation process. However, it is important to note that the proposed approach provides significantly better HDR performance compared to the method presented by Im et al [14]. Additionally, it might be possible to improve computation time of the local extreme calculations by making use of some software optimizations.

IV. CONCLUSION

In this paper, a novel method to generate HDR image using a single input image is presented. This method utilizes an adaptive histogram separation approach to produce over- and under-exposed images. The CLAHE based image enhancement enables efficient extraction of local details. Finally, the proposed fuzzy logic based fusion approach facilitates to construct an attractive HDR image with the efficient combination of the LDR images obtained in the previous stages.

The proposed method generates ghost-free HDR images inherently because it uses a single image instead of using multiple differently exposed images. Hence, this method is not affected by the moving objects and camera movement. Additionally, the low computational complexity of the proposed approach makes it suitable for smart phones and compact consumer cameras.

Page 9: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

A. Taşyapı et al.: Fuzzy Fusion Based High Dynamic Range Imaging using Adaptive Histogram Separation 127

REFERENCES [1] G. Ward, E. Reinhard, S. Pattanaik, and P. Debevec, High dynamic

range imaging: acquisition, display, and image-based lighting, 2nd ed., Morgan Kaufmann Publisher, 2005.

[2] P. Debevec, and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. ACM SIGGRAPH, Los Angeles, USA, pp. 369-378, Aug. 1997.

[3] Y. Bandoh, G. Qiu, M. Okuda, S. Daly, T. Aach, and O. Au, “Recent advances in high dynamic range imaging technology,” in Proc. IEEE Int. Conf. on Image Processing, Hong Kong, pp. 3125-3128, Sep. 2010.

[4] F. Shen, Y. Zhao, X. Jiang, M. Suwa, “Recovering high dynamic range by Multi-Exposure Retinex,” J. Vis. Commun. Image Represent., vol. 20, no. 8, pp. 521-531, Nov. 2009.

[5] B. Gu, W. Li, J. Wong, M. Zhu, and M. Wang, “Gradient field multi-exposure images fusion for high dynamic range image visualization,” J. Vis. Commun. Image Represent., vol. 23, no. 4, pp. 604-610, May 2012.

[6] K. Jacobs, C. Loscos, and G. Ward, “Automatic high dynamic range image generation for dynamic scenes,” IEEE Comput. Graph. Appl., vol. 28, no. 2, pp. 84–93, Mar. 2008.

[7] E. A. Khan, A. O. Akyuz, and E. Reinhard, “Robust generation of high dynamic range images,” in Proc. IEEE Int. Conf. on Image Processing, pp. 2005–2008, Atlanta, GA, USA, Oct. 2006.

[8] O. Gallo, N. Gelfand, W. Chen, M. Tico, and K. Pulli, “Artifact-free high dynamic range imaging,” in Proc. Int. Conf. Computational Photography, San Francisco, USA, pp. Apr. 2009.

[9] J. Im, S. Lee, and J. Paik, “Improved elastic registration for ghost artifact free high dynamic range imaging,” IEEE Trans. Consum. Electron., vol. 57, May 2011.

[10] J. Zheng, Z. Li, Z. Zhu, S. Wu, and S. Rahardja, “Hybrid patching for a sequence of differently exposed images with moving objects,” IEEE Trans. Image Process., vol. 22, no. 12, pp. 5190–5201, Dec. 2013.

[11] W. Zhang and W-K. Cham, “Gradient-directed multi-exposure composition,” IEEE Trans. on Image Process., vol. 21, no. 4,pp. 2318-2323, Apr. 2012.

[12] W. Zhang and W-K. Cham, “Reference-guided exposure fusion in dynamic scenes,” J. Vis. Commun. Image Represent., vol. 23, no. 3, pp. 467-475, Apr. 2012.

[13] C. Lee, Y. Li, and V. Monga, “Ghost-free high dynamic range imaging via rank minimization,” IEEE Signal Process. Lett., vol. 21, no. 9, Sep. 2014.

[14] J. Im, J. Jeon, M. Hayes, and J. Paik, “Single image-based ghost-free high dynamic range imaging using local histogram stretching and spatially-adaptive denoising,” IEEE Trans. Consum. Electron., vol. 57, no. 4, pp.1478-1484, Nov. 2011.

[15] J. Im, and J. Paik, “Spatially adaptive histogram equalization for single image-based ghost-free high dynamic range imaging,” J. Arts, Imaging Science, vol. 1, no. 1, pp. 6-13, Feb. 2014.

[16] S. Pei, Y. Zeng, and J. Ding, “Color images enhancement using weighted histogram separation,” in Proc. IEEE Int. Conf. on Image Processing, Atlanta, GA, USA, pp. 2889-2892, Oct. 2006.

[17] M.A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Comm. of the ACM vol. 24, no. 6, pp. 381–395, 1981.

[18] S. M. Pizer, “Adaptive histogram equalization and its variations,” Comput. Vision, Graph. Image Process, vol. 39, pp. 335-368, 1987.

[19] A.T. Çelebi, “Image Enhancement and Target Detection in Underwater Images,” PhD Dissertation, Electronics and Telecom. Engineering Dept. of University of Kocaeli, 2011.

[20] N.D. Narvekar, L.J. Karam, “A no-reference image blur metric based on the cumulative probability of blur detection (CPBD) ,” IEEE Trans. on Image Process., vol. 20, no. 9, pp. 2678-2683, Mar. 2011.

BIOGRAPHIES Aysun Taşyapı Çelebi was born in Çanakkale, Turkey. She received her B.Sc., M.Sc. and Ph.D. degrees in Electronics and Telecommunication engineering from Kocaeli University, Kocaeli, Turkey, in 2002, 2008, and 2012, respectively. Since 2013 she has been with the Department of Electronics and Telecommunications Engineering, University of Kocaeli, Turkey, where she is

currently working as an Assistant Professor. Her research interests include digital signal and image/video processing.

Ramazan Duvar received his B.Sc., M.Sc., degrees in Electronics and Telecommunication engineering from the University of Kocaeli, Kocaeli, Turkey, in 2010, 2012, respectively. Currently, he is pursing towards to Ph.D. degree. Since 2011, he has been with the Department of Electronics and Telecommunications Engineering, University of Kocaeli, Turkey as a research assistant. His

research interests include image processing and embedded systems.

Oğuzhan Urhan (S’02-M’06) received his B.Sc., M.Sc., and Ph.D. degrees in Electronics and Telecommunication engineering from the University of Kocaeli, Kocaeli, Turkey, in 2001, 2003, and 2006, respectively. Since 2001, he has been with the Department of Electronics and Telecommunications Engineering, University of Kocaeli, Turkey, where he is currently associate

professor. He was a visiting professor at Chung-Ang University, South Korea, from 2006 to 2007. He is the director of Kocaeli University Laboratory of Embedded and Vision Systems (KULE). His research interests include digital signal, image/video processing and embedded systems.

Page 10: ş Fuzzy Fusion Based High Dynamic Range Imaging using ...download.xuebalib.com/xuebalib.com.35003.pdf · it is proposed to utilize a fuzzy logic based approach at the fusion

本文献由“学霸图书馆-文献云下载”收集自网络,仅供学习交流使用。

学霸图书馆(www.xuebalib.com)是一个“整合众多图书馆数据库资源,

提供一站式文献检索和下载服务”的24 小时在线不限IP

图书馆。

图书馆致力于便利、促进学习与科研,提供最强文献下载服务。

图书馆导航:

图书馆首页 文献云下载 图书馆入口 外文数据库大全 疑难文献辅助工具