FAST MOVEMENT DETECTION FOR HIGH DYNAMIC RANGE IMAGING

Zhengguo Li, Zijian Zhu and Susanto Rahardja

Signal Processing Department, Institute for Infocomm Research, 1 Fusionopolis Way, Singapore

ABSTRACT

When a high dynamic range image is synthesized by using a setof differently exposed low dynamic range (LDR) images, it is im-portant to detect moving objects so as to remove ghosting fromthe final HDR image. A pixel level movement detection schemewas recently proposed in [8]. It included a pixel level similarityindex for differently exposed LDR images, an adaptive thresholdfor the classification of pixels and an approach that utilizes inten-sity mapping (IMF) function for patching invalid regions. In thispaper, we first propose a new adaptive threshold and a new patch-ing approach to improve the scheme in [8]. Then, a sub-samplingmethod is introduced to simplify the improved movement detec-tion scheme. Experimental results show that the improved move-ment detection scheme indeed outperforms the scheme in [8]. Inaddition, the speed is significantly improved by the proposed fastmovement detection scheme.

1. INTRODUCTION.

There are many real scenes which have higher dynamic ranges thanthose that can be captured by digital cameras. In such scenes, a sin-gle shot low dynamic range (LDR) image usually turns out to beunderexposed and/or overexposed in certain regions of the image.Because of this reason, a single shot does not have full dynamicrange and one way to overcome this is to capture a set of differ-ently exposed LDR images [1, 2]. There are two alternatives tointegrate the desired information of all input images into one im-age. One is to synthesize a high dynamic range (HDR) image [3].The HDR image is then compressed via a tone mapping such thatit can be displayed by conventional digital devices [4]. Both HDRand LDR images are produced and it is applicable to devices withdifferent dynamic ranges. The other is to fuse the LDR imagesdirectly into an LDR image which is much simpler [5]. The al-ternative is more suitable for real time applications, especially forthose handheld devices with limited computational resource, suchas mobile phones or digital cameras.

When an HDR/LDR image is synthesized for an outdoor sceneby using multiple differently exposed LDR images, one of chal-lenging problems is to detect moving objects in order to removeghosting artifacts from the final HDR/LDR image. As with thepopular HDR acquisition approach in [3], all input LDR imagesare assumed to be perfectly aligned, possibly using the registra-tion algorithm in [6]. To remove ghosting artifacts due to movingobjects in the scene, the pixels of all LDR images are required tobe properly classified into valid or invalid, and only valid pixelsare used to generate the HDR/LDR image. Recently, a pixel levelmovement detection scheme was proposed in [8]. This scheme wasshown to remove ghosting artifacts significantly better than exist-ing commercial software which is also demonstrated in Fig. 1.The paper has three major contributions. The first contribution is

a new pixel level similarity index for differently exposed LDR im-ages. The second is an adaptive threshold for the classification ofpixels into valid or invalid. The threshold is adaptive to the valuesof a pixel to be detected and its reference pixel and the exposuretimes of two images. The third is an intensity mapping function(IMF) based approach for patching invalid pixels in a detected im-age by using the co-located pixels in the reference image. Sincethe IMFs are computed by using all pixels of these two images, theIMF based patching approach can be regarded as a global one. Allpixels are exhaustively detected by using the scheme in [8] whichmakes the scheme complex, especially for real time applicationson digital cameras and mobile phones. It is thus desirable to re-duce the complexity of the scheme in [8].

Fig. 1. Comparison of different movement detection schemes.

In this paper, we first improve the movement detection schemein [8], particularly on the latter two contributions. Besides be-ing adaptive to the values of two co-located pixels and the expo-sure times of two images, the new threshold is also adaptive tothe ISO value and the average exposure value of all LDR images.The patching scheme is also improved by involving local informa-tion of under-exposed/saturated pixels in the reference image andintroducing a cross-image smoothing method for invalid pixels.Experimental results show that by using the improved movementdetection scheme, the performance is indeed improved. We thenpropose a sub-sampling based method to simplify the improvedmovement detection scheme. The proposed method is based onan observation that small portions of differently exposed imagesbelong to moving objects. Instead of detecting moving objects bychecking all pixels as in [8], a sub-sampling based method is pre-sented to detect moving objects by only checking part of pixels.If a pixel is detected as a background one, its neighboring pixelsare skipped. Otherwise, its neighboring pixels are further checkedby using a sub-sampling method. Experimental results verify thatthe complexity of the movement detection scheme in [8] can besignificantly reduced.

The rest of this paper is organized as follows. An improved

2011 18th IEEE International Conference on Image Processing

978-1-4577-1303-3/11/$26.00 ©2011 IEEE 365

movement detection scheme is provided in Section 2. A sub-sampling based method is proposed in Section 3 to simplify the im-proved movement detection method. Experimental results are pro-vided in Section 4 to show the efficiency of the proposed scheme.Concluding remarks are given in Section 5.

2. AN IMPROVED MOVEMENT DETECTION SCHEME

Let Zk,l(p) denote the intensity of the lth color channel at positionp when the kth LDR image is captured, i.e., p is a spatial position,l indexes over color channels of red, green and blue, and k indexesover exposure time ∆tk. Such a set of LDR images is knownas a Wyckoff set [1]. Let Zk be the reference image of Zk forthe classification of pixels in Zk as valid or invalid. An image isselected as the initial reference image for the movement detection.For simplicity, the image is denoted as k0 and its value is set as themiddle one. All pixels in Zk0 are marked as valid. Let p denote acoordinate (x, y). A pixel Zk(p)(k 6= k0) is marked as valid if itis similar to its co-located pixel Zk(p) [8], i.e.,

S(Zk(p), Zk(p)) > Thrk(p). (1)

Otherwise, it is marked as invalid.The function S(Zk(p), Zk(p)) in Equation (1) is given by [8]

S(Zk(p), Zk(p)) =

3∑l=1

2Φk,l(p)Ψk,l(p) + 1

3∑l=1

[Φ2k,l(p) + Ψ2

k,l(p)] + 1

, (2)

where Φk,l(p) and Ψk,l(p) are constructed by using a bi-directionalmapping method as [7]

Φk,l(p) =

Λl,π(k),k(Zk,l(p)); if w(Zk,l(p)) ≤ w(Zk,l(p))Zk,l(p); otherwise

,

Ψk,l(p) =

Λl,k,π(k)(Zk,l(p)); if w(Zk,l(p)) > w(Zk,l(p))

Zk,l(p); otherwise.

Here, the weighting function w(z) is defined as [3], and π(k)

corresponds to the exposure time of image Zk [8], Λl,k,π(k) andΛl,π(k),k are two IMFs [10], Λl,k,π(k) maps intensity values inZk,l into Zk,l and Λl,π(k),k vice versa.

2.1. An Improved Adaptive Threshold

According to the definition of S(Zk(p), Zk′(p)) in Equation (2),the threshold Thrk,π(k)(p) is computed as

Thrk,π(k)(p) =

2(1− max1≤l≤3

µ(Zk,l(p)), µ(Zπ(k),l(p)))

1 + (1− max1≤l≤3

µ(Zk,l(p)), µ(Zπ(k),l(p)))2, (3)

where the function µ(z) is defined as

µ(z) =

132

+ 532(1+exp(4z−45))

; if z < 128132

; otherwise. (4)

If the values of ∆tk, ∆tπ(k), and the ISO settingG for captur-ing the images are available, µ(Zk,l(p)) can then be fine tuned by

multiplying a factor of h(∆tMa

∆tmk,π(k)

,∆tMk,π(k)

∆tmk,π(k)

, EV ,G). Here, ∆tMa

is the maximal value of all exposure times. ∆tMi,j and ∆tmi,j are themaximal and minimal values of two exposure times ∆ti and ∆tj ,respectively. EV is the average exposure value of all LDR imagesand is defined as log2(100ω2/(G∆t)). ω and ∆t are the aper-ture value and the geometrical mean value of all exposure times,respectively. The function h(x1, x2, x3, x4) is selected as

h(x1, x2, x3, x4) = x1321 max1, 1

log2(x2)( 10

x3)18 (

x4

100)

132 . (5)

2.2. An Improved Patching Scheme

As the correlation among two successive images is the strongest,Zk0+1 and Zk0−1 are chosen as Zk0 . Zk(|k−k0| > 1) is updatedby

Zk(p) =

~Zπ(k)(p); if Zπ(k)(p) is invalidZπ(k)(p); otherwise

.

It is very important to properly select the values of ~Zπ(k)(p)

to fill in the invalid regions of Zπ(k). A new pixel, ~Zπ(k)(p), issynthesized by using Zπ(k),l(p) and Λπ(π(k)),π(k),l(z)(1 ≤ l ≤3) as

~Zπ(k),l(p) = Λl,π(π(k)),π(k)(Zπ(k),l(p)) ; 1 ≤ l ≤ 3. (6)

If none of Zπ(k),l(p)(1 ≤ l ≤ 3) is saturated or under-exposed,~Zπ(k)(p) is then used to replace Zπ(k)(p). Otherwise, ~Zπ(k)(p) isfirst filtered by a locally weighted averaging filter as

~Zπ(k),l(p) =

~Zπ(k),l(p) +∑

p′∈Ω(p,ρ)

gπ(k),l(p′)Zk,l(p

′)

1 +∑

p′∈Ω(p,ρ)

gπ(k),l(p′), (7)

where Ω(p, ρ) = p′ = (x′, y′)||x− x′| ≤ ρ, |y − y′| ≤ ρ withρ being a predefined parameter [9]. gπ(k),l(p

′) is the weightingfactor of Zk,l(p). Since a pixel p′ nearby a moving region hasa high possibility to belong to a moving object, gπ(k),l(p

′) is amonotonically increasing function of the distance between pixelsp and p′. Meanwhile, it is a monotonically decreasing function ofthe absolute difference between Zπ(k),l(p

′) and Zπ(k),l(p). In thispaper, the value of ρ is selected as 7, and gπ(k),l(p

′) is chosen as

gπ(k),l(p′) = ‖p′ − p‖2 exp−4(Zπ(k),l(p

′)−Zπ(k),l(p))2

Vπ(k)(p′),

Vπ(k)(p′) =

1; if Zπ(k)(p

′) is valid or updated0; otherwise .

~Zπ(k)(p) is then used to replace Zπ(k)(p). After all pixels areupdated, all invalid pixels are finally smoothed by using its eightneighboring pixels as

Zk,l(p) =

γπ(k),l(p)Zk,l(p) +∑p′

Γπ(k),l(p′)Zk,l(p

′)

γπ(k),l(p) +∑p′

Γπ(k),l(p′), (8)

2011 18th IEEE International Conference on Image Processing

366

Γπ(k),l(p′) = exp

−(16(Zπ(k),l(p

′)−Zπ(k),l(p))

2blog2(Zπ(k),l(p)+1)+0.5c

)2

, (9)

where bac is the largest integer less than a, and the value of γπ(k),l(p)is

γπ(k),l(p) =

1; if Zπ(k)(p) is under-exposed or saturated8; otherwise

.

(10)It is shown from Equations (8) and (9) that the weighting fac-

tor Γπ(k)(p′) is determined by Zπ(k)(p

′) instead of ~Zπ(k)(p′).

This is based on an observation that ~Zπ(k)(p′) is less reliable than

Zπ(k)(p′). The proposed smoothing method can thus be called “a

cross-image smoothing method”.

3. A SUB-SAMPLING BASED FAST MOVEMENTDETECTION SCHEME

The improved movement detection scheme is simplified by usinga sub-sampling based method in this section.

Fig. 2. A sub-sampling based movement detection scheme.

An example with a sub-sampling factor being selected as 4(=23−1) is adopted to illustrate the proposed fast movement detec-tion scheme. The example is shown in Fig. 2. The pixels that arelabeled with 1 are checked at the first round, and they are markedby the blur color if they are invalid. The neighboring pixels of ablue pixel labeled with 1 are then checked by using a sub-samplingmethod. In other words, all neighboring pixels that are labeled by2 are checked at the second round. They are also marked by theblur color if they are invalid. Finally, the eight neighboring pixelsof a blue pixel labeled by 2, i.e., those pixels are labeled by 3, arechecked in the final round. Only 1/16 of all pixels are detectedat the first round. Since only a small portion of pixels in an LDRimage belongs to moving objects, the second and third rounds ofdetections are only conducted for a small amount of pixels in theLDR image. As such, the complexity of the improved movementdetection scheme can be significantly reduced. Meanwhile, sincemany pixels are in the neighborhoods of two pixels, a flag is at-tached to each pixel so as to indicate whether it has been detected.With the flag, each pixel will only be detected once.

Besides simplifying the proposed movement detection scheme,another important issue is to design a parallel simplified method.This can be achieved by selecting the sub-sampling factor as apower of 3. An example is demonstrated in Fig. 3 with the sub-sampling factor being chosen as 9. All pixels that are marked by1 are detected at the first round. The neighboring pixels of a blur

pixel labeled by 1 are further detected by using a sub-samplingmethod. It is demonstrated in Fig. 3 that the sub-sampling methodcan run in parallel.

Fig. 3. A parallel sub-sampling based movement detectionscheme.

4. EXPERIMENTAL RESULTS

In this section, we first compare the improved movement detectionscheme with the one in [8] by testing two sets of differently ex-posed images. The first set is composed of 3 differently exposedimages as demonstrated in Fig. 4. The second set is composed offive images as illustrated in Fig. 5. The initial reference imagesare selected as the second image and the first image for the firstset and the second set, respectively. It is shown in Figs. 6 and 7that the improved scheme can be adopted to produce better HDRimages.

Fig. 4. A set of three differently exposed LDR images with wavingleafs.

We then test the proposed fast movement detection schemewith the sub-sampling factor being selected as 4. Both image se-quences in Figs. 4 and 5 are tested. 47.07% and 81.86% of pix-els are detected for two input images of the former, respectively.23.19%, 34.61%, 50.71% and 55.04% of pixels are checked forfour input images of the latter, respectively. Besides these two se-quences, another LDR image sequence that is shown in Figs. 8is also tested. 39.71%, 39.61%, 48.94%, 51.88% and 75.22% ofpixels are detected for five images of the sequence in Fig. 8, re-spectively. Overall, the number of detected pixels is reduced byup to 59.11%. It should be noted that the speed could be furtherimproved by choosing a larger sub-sampling factor such as 8 or 9.

2011 18th IEEE International Conference on Image Processing

367

Fig. 5. A set of five differently exposed LDR images with a mov-ing subject.

Fig. 6. The final LDR images for the first set.

5. CONCLUSION

An improved movement detection scheme has been provided byintroducing a new adaptive threshold and a new patching approach.A sub-sampling based method has also been proposed to simplifythe improved movement detection scheme. By using the move-ment detection scheme introduced in this paper, the quality of thefinal high dynamic range images is improved. The speed is alsosignificantly increased by the proposed fast movement detectionscheme.

References[1] C. W. Wyckoff, “An experimental extended response film,”

SPIE Newslett., pp.16-20, June-July 1962.

[2] S. Mann, “Comparametric equations with practical applica-

Fig. 7. The final LDR images for the second set.

Fig. 8. A set of six differently exposed LDR images with a movingsubject.

tions in quantigraphic image processing,” IEEE Trans. on Im-age Processing, vol. 9, no. 8, pp. 1389-1406. Aug. 2000.

[3] P. Debevec and J. Malik, “Recovering high dynamic range ra-diance maps from photograph,” In Proceedings SIGGRAPH1997, pp.369-378, 1997.

[4] Z. G. Li, S. Rahardja, S. S. Yao, J. H. Zheng, and W. Yao,“High dynamic range compression by half quadratic regular-ization,” In 2009 IEEE International Conference on ImageProcessing, pp. 3169-3172, Cairo, Egypt, Nov. 2009.

[5] T. Mertens, J. Kautz, and F. V. Reeth, “Exposure fusion,” InPacific Graphics 2007, pp. 382-390, Maui, USA, Oct. 2007.

[6] K. Jacobs, C. Loscos, and G. Ward, “Automatic high dynamicrange image generation for dynamic scenes,” IEEE ComputerGraphics and Applications, Vol. 128, No.2, pp.84-93, Feb.2008.

[7] Z. G. Li, S. Rahardja, S. Q. Wu, Z. J. Zhu, and S. L. Xie, “Ro-bust movement detection based on a new similarity index forHDR imaging,” In 2010 SIGGRAPH, California, USA, July,2010.

[8] Z. G. Li, S. Rahardja, Z. J. Zhu, S. L. Xie, and S. Q. Wu,“Movement detection for the synthesis of high dynamic rangeimages,” In IEEE International Conference on Image Process-ing, pp.3133-3136, September 2010, Hongkong, China.

[9] Z. G. Li, Z. J. Zhu, S. Q. Wu, and S. Rahardja, “Fast patchingof moving regions for high dynamic range imaging,” In ACMSIGGRAPH ASIA 2010, Korea, December 2010.

[10] M. Grossberg and S. Nayar, “Determining the camera re-sponse from images: what is knowable?” IEEE Transactionson pattern analysis and machine intelligence, pp. 1455-1467,Vol. 25, No. 11, Nov. 2003.

2011 18th IEEE International Conference on Image Processing

368