PERCEPTUALLY FINE DETAIL EXTRACTION FROM A VECTOR FIELD

Jinghong Zheng, Zhengguo Li, Shiqian Wu, Zijian Zhu, Wei Yao, and Susanto Rahardja

Signal Processing Department, Institute for Infocomm Research, Singapore.{jzheng, ezgli, shiqian, zhuzj, wyao, rsusanto}@i2r.a-star.edu.sg

ABSTRACT

In this paper, a perceptually weighted quadratic optimizationproblem is formulated for the extraction of fine details from avector field. To achieve better edge-preserving performance,a weighted sum square error (SSE) is adopted in the proposedoptimization problem. The weighted SSE is used as data termin the proposed quadratic optimization problem to refine theextraction strength according to local intensity variance. Ex-perimental results show that the perceptually weighted opti-mization problem can minimize sharp edge information in theextracted details.

Index Terms— Detail extraction, weighted sum squareerror, quadratic optimization, vector field.

1. INTRODUCTION

Detail extraction is a desired feature of many applications inthe fields of image processing and computer photography,such as detail enhancement, image editing and de-noising.Typically, the detail extraction is formulated as an edge-preserving image smoothing problem wherein small scalevariations of input image are smoothed while the sharp edgesare preserved. Then a detail layer can be extracted from thedifference between the input image and the coarsened image.

Many edge-preserving image smoothing operators havebeen proposed for detail extraction. For example, nonlinearimage filter, such as bilateral filter, is one of de facto tools foredge-preserving image smoothing. In [1], a multi-scale bi-lateral filter is proposed for the detail enhancement of multi-lighting images. The detail layer is extracted from each imageseparately, and then merged together to form a comprehensivedetail layer. To facilitate the detail extraction from multipleimages, Li et al. [2] extended the spatial-domain bilateral fil-ter to gradient domain. The detail information from multipleimages can be extracted simultaneously. Besides the nonlin-ear filter, edge-preserving image smoothing can also be for-mulated as optimization problems [3] [4] [5]. These schemescan achieve good performance in edge-preserving, but theircomputational cost makes them inefficient for multiple imageprocessing. A more flexible scheme was proposed by Li et al.in [7]. A quadratic optimization problem is built up with theinput as a vector field. Through composing the detail infor-

mation from different images into a vector field, the vector-field optimization problem can directly extract a detail layerfrom multiple images simultaneously without generating thecoarsened images.

The optimization problems used for detail extraction havetwo terms, one is a regularization term that adjusts smooth-ing strength according to the magnitude of intensity variation,and the other is a data term that measures similarity betweeninput image and the coarsened image. These two terms arebalanced by a trade-off factor. The data term in existing op-timization problems [3] [4] [5] are always defined as the sumsquared error (SSE) between the input image and the coars-ened image. As such, distortions on sharp edges and smalldetails are treated equally by the data term. It is obvious thatthese SSE-based data terms are not optimized for the edge-preserving image smoothing. In this paper, we proposed aperceptually weighted quadratic optimization problem witha vector field as input. Inspired by the idea of SSIM [6], aperceptual weight is computed from local intensity varianceto represent structure information. Then a weighted SSE isintroduced as the data term of a quadratic optimization prob-lem. Complex areas with large local variation will have largeweight in the data term. Then at the complex areas, the coars-ened image will resemble more to the input image, and lessdetail information is extracted. As such, sharp edges can bekept without blurring. On the other hand, small weight isgiven to a flat area in the data term, and more details fromthe flat area are extracted by the optimization problem. Thismatches the characteristics of HVS more, i.e., the details inflat areas should be better preserved. Compared to the vector-field SSE optimization problem [7], the data term in the pro-posed weighted SSE optimization problem is adaptive to thelocal variance, and therefore has better performance on edge-preserving.

The rest of this paper is organized as follows. In Section2, the proposed perceptually weighted quadratic optimizationproblem is presented. Then in Section 3 the proposed opti-mization problem is applied to an exposure fusion scheme fordetail enhancement. Experimental results are given in Sec-tion 4, to show the performance of the proposed optimizationproblem.

2. PERCEPTUALLY WEIGHTED QUADRATICOPTIMIZATION PROBLEM

An image Z can be considered as the product of a base layerZb and a detail layer Zd. Then in log domain, an image canbe written as:

L(p) = Lb(p) + Ld(p), (1)

where p indicates the pixel position. L, Lb and Ld denoteZ, Zb and Zd in log domain, respectively. In [5], an edge-preserving quadratic optimization problem used for detail ex-traction is defined as:

arg minLb

{∑p

[‖L(p)− Lb(p)‖2

+λ(‖∂Lb(p)∂x

ψ(∂L(p)∂x )‖2 + ‖

∂Lb(p)∂y

ψ(∂L(p)∂y )‖2)]}, (2)

where ψ(·) is smooth weighting computed as ψ(z) =√|z|α + ε. The default values of α and ε are 1.2 and 0.0001

respectively.There are two terms in Equation (2), one is a data term

which measures the objective fidelity of Lb with respect to theinput image L, and the other is a regularization term which ison the smoothness of Lb(p). λ is a trade factor between thetwo terms. In the SSE data term of Equation (2), the distor-tions of sharp edges and small details are summed with thesame weight. Thus the data term cannot protect the sharpedges from extraction. To improve the edge-preserving per-formance of the optimization problem, we introduce a percep-tually weighted data term. The local variance at each pixel po-sition is used as indicator of perceptual complexity. Let σZ,ρbe the local variance of Z(p) in the following neighboring ofZ(p)(p = (x, y)):

{Z(p′)|p′ = (x′, y′), |x− x′| ≤ ρ, |y − y′| ≤ ρ}, (3)

where value of ρ is 4. A pixel with large local variance be-longs to complex area, and a pixel with small local variance iswithin flat area. The perceptual weight of a pixel at positionp is then computed as:

w(p) =1

M

M∑p′=1

2σ2Z,ρ(p) + (cΨ)2

2σ2Z,ρ(p

′) + (cΨ)2, (4)

where M is the total number of pixels. c is a constant thatequals 0.03, and Ψ is the dynamic range of input image.

With the proposed perceptual weight, an edge-preservingimage smoothing problem is formulated as:

arg minLb

{∑p

[w(p)‖L(p)− Lb(p)‖2

+λ(‖∂Lb(p)∂x

ψ(∂L(p)∂x )‖2 + ‖

∂Lb(p)∂y

ψ(∂L(p)∂y )‖2)]}. (5)

The fidelity measurement between base layer and input imageis changed from the SSE to a perceptually weighted SSE.

When a pixel p is in a flat area, the value of w(p) is usu-ally less than 1. The fidelity of Lb(p) with respect to L(p) islower, and the regularization term dominates the performancecriterion. As a result, fine details are extracted to the detaillayer. On the other hand, when a pixel p is in a complex re-gion, the value of w(p) is larger than 1. The fidelity of Lb(p)with respect to L(p) is higher, and the data term dominatesthe performance criterion. The sharp edges are preserved bet-ter in the base layer. Thus the data term and the regularizationterm are adaptively balanced by w(p) according to the localvariance.

As Ld(p) = L(p) − Lb(p), the optimization problem (5)can be rewritten as:

arg minLd

{∑p

[w(p)‖Ld‖2

+λ(‖∂L(p)∂x −

∂Ld(p)∂x

ψ(∂L(p)∂x )‖2 + ‖

∂L(p)∂y −

∂Ld(p)∂y

ψ(∂L(p)∂y )‖2)]}.(6)

To extend the flexibility, we replace (∂L(p)∂x , ∂L(p)∂y ) inEquation (6) by a general vector field (V1(p), V2(p)). Then aperceptually weighted quadratic optimization problem in vec-tor field can be defined as:

arg minLd

{∑p

[w(p)‖Ld(p)‖2

+λ(‖V1(p)− ∂Ld(p)

∂x

ψ(V1(p))‖2 + ‖

V2(p)− ∂Ld(p)∂y

ψ(V2(p))‖2)]}.(7)

The perceptually weighted optimization problem in Equa-tion (7) can be considered as an extension of the vector-fieldoptimization problem in [7] by introducing the weighted SSEas the data term. There are two major components in the pro-posed problem formulation: one is the generation of the inputvector field (V1(p), V2(p)), and the other is the computationof the perceptual weight w(p). In the following section, fu-sion of differently exposed images will be taken as an exam-ple to illustrate them.

3. PERCEPTUALLY WEIGHTEDDETAIL-ENHANCED EXPOSURE FUSION

In this section, the proposed perceptually weighted quadraticoptimization problem is applied on exposure fusion to en-hance the fine details. Exposure fusion is to synthesize aset of images, which capture the same scene with differ-ent exposures, into a content-enriched image. In the expo-sure fusion scheme [8], three weights Ck, Sk and Ek, whichmeasure contrast, color saturation and well-exposedness, arecomputed for each input image Zk respectively. The prod-uct of three weights is denoted as Wk. L{Zlk} and G{W l

k}

are Laplacian pyramid of image Zk and Gaussian pyramid ofweight map Wk, respectively. Pixel intensities in the differentpyramid levels are blended according to the equation:

L{Z(p)}l =

N∑k=1

[L{Zk(p)}lG{Wk(p)}l].

Then the pyramid L{Z(p)}l is collapsed to produce a fusedimage Zef .

This Laplacian-pyramid based blending scheme canseamlessly fuse the differently exposed images together. Butthe smoothing effect of Gaussian pyramid makes some detailsinvisible after fusion. Thus a detail enhanced exposure fusionis desired to improve the visibility of fine details.

3.1. Generation of the vector field

The vector field (V1(p), V2(p)) in Equation (7) is computedfrom intensity gradients of input images, since the details arerepresented by intensity variation. Generally, the gradientthat has largest absolute value among collocated pixels corre-sponds to the most visible detail. But there is a likelihood thatthe maximum gradient is noise. To alleviate the noise effect,the vector field (V1(p), V2(p)) is calculated as a weighted sumof gradients along differently exposed images.

Let Zk be one of the differently exposed images, andYk be the luminance component of Zk. The gradient field(∇Yk,1(p),∇Yk,2(p)) is defined as (Yk(p+1)−Yk(p), Yk(p+H)−Yk(p)). H is the image width in pixels. Then the vectorfield (V1(p), V2(p)) is computed as:

V1(p) =

N∑k=1

we(Yk(p+ 1))we(Yk(p))∇ log Yk,1(p)

N∑k=1

we(Yk(p+ 1))we(Yk(p))

;

V2(p) =

N∑k=1

we(Yk(p+H))we(Yk(p))∇ log Yk,2(p)

N∑k=1

we(Yk(p+H))we(Yk(p))

,

where N is the number of differently exposed images, andwe(·) is a weighting factor defined as:

we(z) =

{z + 1; if z ≤ 127256− z; otherwise .

3.2. Computation of the perceptual weight

The perceptual weight w(p) is computed by using all lumi-nance components Yk(1 ≤ k ≤ N). Due to different expo-sure, a pixel well exposed in one image could be under/overexposed in another image. This implies that the value of

σ2Yk,ρ

(p) is distinct for different k. On the other hand, vari-ance magnitude becomes larger when a pixel gets better ex-posed, and it decreases as the pixel becomes under/over ex-posed. Thus the largest value of σ2

Yk,ρ(p) along all k’s is cho-

sen to represent the overall local variance of pixel p. Thevalue of perceptual weight is given as

w(p) =1

M

M∑p′=1

2σ2Y,ρ(p) + (cΨ)2

2σ2Y,ρ(p

′) + (cΨ)2,

where σY,ρ(p)(= max1≤k≤N

{σYk,ρ(p)}) is the maximum local

variance of the collocated pixels among input images.Through solving the optimization problem (7), a detail

layer Ld can be extracted from the vector filed (V1(p), V2(p))in log domain. Then a detail-enhanced exposure fusion imageZef can be computed as:

Zef (p) = Zef (p)× exp(θ · Ld(p)),

where θ is an exponential on detail layer to control the detailenhancement, and its default value is 1. The details can befurther amplified by increasing the value of θ.

4. EXPERIMENTAL RESULTS

In this section, the perceptually weighted optimization prob-lem is compared with the vector-field optimization problemin [7] for the detail enhancement of exposure fusion.

By using the exposure fusion scheme in [8], an image thatcontains the information of differently exposed images can begenerated, as shown in Fig.1(a). The detail layers extracted bythe vector-field optimization problem in [7] and the proposedoptimization problem are displayed in Figs.1(b) and 1(c) re-spectively. The value of λ is set as 2 in both optimizationproblems. Obvious sharp edges can be observed in the de-tail layer extracted by the scheme of [7] in Fig.1(b), such asthe contours of lamp and books. Then in the detail enhancedimage Fig.1(d), which is generated by adding detail layer inFig.1(b) to Fig.1(a), artifacts are presented at correspondingareas. For example, the lamp in Fig.1(d) has artificial blackcontour. That is because the sharp edges are extracted andenhanced as well as fine details by [7]. On the other hand,these sharp edges are minimized in the detail layer extractedby the proposed quadratic optimization problem, as shown inFig.1(c). Therefore, no artifact is displayed in Fig.1(e) whilethe enhancement of fine details in Fig.1(e) is visually as goodas the images generated by [7]. Another two examples aregiven in Figs.1(f) and 1(k) respectively. These images haveproven that the proposed optimization problem has better per-formance on preventing sharp edges in detail extraction due tothe adoption of perceptual weight. It is noted that sharper im-age not always results in better visual quality. The proposedoptimization problem is therefore to provide a tool for detailextraction and manipulation.

(a) (b) (c) (d) (e)

(f) (g) (h) (i) (j)

(k) (l) (m) (n) (o)

Fig. 1. Comparison between the scheme in [7] and the proposed optimization problem. (a) (f) (k) images fused by [8]; (b) (g)(l) detail layers extracted by [7] (θ = 2); (c) (h) (m) detail layers extracted by the proposed optimization problem (θ = 2); (d)(i) (n) detail enhanced images by [7]; (e) (j) (o) detail enhanced images by the proposed optimization problem.

5. CONCLUSION

In this paper, a perceptually weighted optimization problem isproposed for fine detail extraction. A perceptually weightedSSE is used as the data term of the proposed optimizationproblem. As such the detail extraction by the proposed op-timization problem is adaptive to the local intensity vari-ance. The proposed optimization problem can better matchthe characteristics of HVS in detail extraction. Details in flatareas are extracted to the detail layer while the sharp edgesare minimized in the detail layer.

6. REFERENCES

[1] R. Fattal, M. Agrawala and S. Rusinkiewicz, “Multiscale shapeand detail enhancement for multi-light image collection”, ACMTrans. Graphics, vol. 26, no.3, pp.51:1-51:10, August 2007.

[2] Z. G. Li, J. H. Zheng, Z. J. Zhu, S. Q. Wu, and S. Rahardja, “ABilateral Filter in Gradient Domain”, ICASSP, pp. 1113-1116,Kyoto, Japan, March, 2012.

[3] P. Charbonnier, L. B. Feraud, G. Aubert and M. Barlaud, “De-

terministic edge-preserving regularization in computed imag-ing”, IEEE Trans. Image Processing, vol.6, no.2, pp.298-311,February 1997.

[4] L. Xu, C. W. Lu, Y. Xu, and J. Jia, “Image smoothing via L0graident minimization”, ACM Trans. Graphics, vol. 30, no. 6,Article 174, December 2012.

[5] Z. Farbman, R. Fattal, D. Lischinshi and R. Szeliski, “Edge-preserving decompositions for multi-scale tone and details ma-nipulation”, ACM Trans. Graphics, vol. 27, no.3, pp.249-256,August 2008.

[6] Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Im-age quality assessment: from error visibility to structural simi-larity”, IEEE Trans. Image Processing, vol. 13, no. 4, pp. 660-612, April 2004.

[7] Z. G. Li, J. H. Zheng, and S. Rahardja, “Detail-enhanced Ex-posure Fusion”, IEEE Trans. Image Processing, vol. 21, no. 11,pp. 4672-4676, Noverber 2012.

[8] T. Mertens, J. Kautz, and F. V. Reeth, “Exposure fusion: a sim-ple and practical alternative to high dynamic range photogra-phy,” CGF, vol. 28, no. 1, pp. 161-171, 2009.