Technical ReportNumber 712

Computer Laboratory

UCAM-CL-TR-712ISSN 1476-2986

Flash-exposure highdynamic range imaging:virtual photography anddepth-compensating flash

Christian Richardt

March 2008

15 JJ Thomson AvenueCambridge CB3 0FDUnited Kingdomphone +44 1223 763500

http://www.cl.cam.ac.uk/

c© 2008 Christian Richardt

Technical reports published by the University of CambridgeComputer Laboratory are freely available via the Internet:

http://www.cl.cam.ac.uk/techreports/

ISSN 1476-2986

Flash-Exposure High Dynamic Range Imaging:Virtual Photography and Depth-Compensating Flash

Christian Richardt

Abstract

I present a revised approach to flash-exposurehigh dynamic range (HDR) imaging anddemonstrate two applications of this imagerepresentation. The first application enablesthe creation of realistic ‘virtual photographs’for arbitrary flash-exposure settings, based ona single flash-exposure HDR image. The sec-ond application is a novel tone mapping op-erator for flash-exposure HDR images basedon the idea of an ‘intelligent flash’. It compen-sates for the depth-related brightness fall-offoccurring in flash photographs by taking theambient illumination into account.

1 Introduction

The widespread availability of consumer dig-ital cameras has transformed the consumerphotography paradigm. From taking one ortwo carefully posed analogue photographs,we changed to taking hundreds or even thou-sands of digital photographs on a single oc-casion.While this is widely regarded as a signif-

icant improvement, in at least one way thistransition changed things for the worse: dy-namic range. Dynamic range, in this con-text, is the ratio between the brightest and thedarkest regions in an image.Digital photographs only have about one

tenth of the dynamic range of film-based pho-tographs, which are approximately on a parwith the simultaneous dynamic range of thehuman eye. Particularly in outdoor scenes,this dynamic range is easily exceeded, whichmakes dynamic range a precious resource indigital photography.

Modern digital cameras come with sophis-ticated exposure control algorithms which es-timate camera settings for fitting a particularscene into the available dynamic range. De-spite this, it is not uncommon for the lim-ited dynamic range of the imaging sensor toproduce under- or over-exposed photographs.This is particularly likely when using theflash.

Choosing the ‘right’ exposure and flashsettings is a hard problem which usuallyinvolves taking a lot of very similar pho-tographs with only slightly varying settings.This has to be done at the time of shooting,and the added flexibility is limited to choos-ing one of the taken photographs.

I introduce a novel technique (section 3)that effectively allows selected settings – theshutter speed and the flash intensity – to bevaried freely after shooting the initial pho-tographs. Based on a flash-exposure highdynamic range (HDR) image, which is con-structed from the initial photographs, mytechnique creates ‘virtual photographs’ for ar-bitrary flash-exposure settings.

Another common issue in consumer pho-tography is that the flash often leads to avery non-uniform scene illumination. Nearbyobjects are over-exposed and far objects areunder-exposed.

I propose a novel approach (section 4) forcompensating for the brightness fall-off withdepth, to create a more uniformly illuminatedimage. This ‘depth-compensating flash’ oper-ates on a flash-exposure HDR image and pro-duces enhanced flash images.

3

Flash HDR Image (unit flash intensity)

Combined Exposure

= Δ ⋅ Ambient HDR Image (unit exposure time)

+ ⋅

Figure 1: Visualisation of (1) showing how ambient and flash HDR images are combined.

2 Flash-Exposure HDR Imaging

Early work [4, 2] considered how to combine anumber of photographs, taken in an exposuresequence, into a single HDR image. Based onthis traditional approach, Agrawal et al. [1]first proposed flash-exposure HDR imaging.Flash-exposure HDR imaging is a step to-

wards computational photography [6] – a con-cept in stark contrast to conventional photog-raphy. It captures more aspects of a scenein such a way as to allow the photographerto choose selected camera settings after tak-ing the ‘photograph’. This would enable thephotographer to experiment freely with thesesettings in an offline post-processing step af-ter the scene has been captured.HDR images can be considered to be one-

dimensional in terms of the radiance asso-ciated with each pixel in the image. To bemore precise, the value stored for each pixelis the ambient (no-flash) radiance in unit ex-posure time. Flash-exposure HDR imagingintroduces a second value for flash intensity:the radiance due to unit flash intensity.The flash-exposure HDR techniques pre-

sented in this paper can be used as the ba-sis for a variety of applications, such as vir-tual photography and a depth-compensatingflash, which are presented in sections 3 and 4respectively.

2.1 Separability

The key observation that enables flash-exposure HDR imaging is that ambient andflash illumination in a scene are independentand can hence be separated.

Let F be the flash HDR image, or the imagestrictly due to a unit flash intensity. The flash-only exposure F ′ is then F scaled by the flashintensity p. Similarly, the ambient exposureA′ is the ambient HDR image A – the imagecaptured for unit exposure time – scaled bythe exposure time ∆t. The total exposure I isnow given by

I = A′ + F ′ = ∆t ·A+ p · F, (1)

because the total amount of light incident atthe image sensor is the sum of the ambientand the flash-only contributions in the scene(see figure 1).

2.2 Solution Process

In traditional HDR imaging [2], the camera re-sponse function gmaps pixel values Z (x, y) tologarithmic exposure g (Z (x, y)). In the flash-exposure case, the exposure is given by I in(1) which holds for any exposure time ∆ti andflash intensity pj :

exp (g (Zij (x, y))) =∆ti ·A (x, y) + pj · F (x, y) . (2)

To obtain a least squares error estimate forthe ambient and flash HDR images A and Ffrom (2), a minimum of two photographs isrequired. However, to increase numerical sta-bility of the solution, a total of nA × nF pho-tographs are taken at all possible combina-tions of nA exposure times and nF flash in-tensities (as in figure 2).Solving the equation system derived from

(2) directly would result in unacceptablecolour inaccuracies and saturation artefacts in

4

Figure 2: Input photographs for recovering a flash-exposure HDR image from three exposuretimes (vertical) and three flash intensities (horizontal).

regions where the input photographs are sat-urated. To avoid this, Agrawal et al. [1] sug-gested weighting equations with the signal-to-noise function of the underlying camera.However, for general applications, a bet-

ter weighted least squared error estimate canbe obtained by weighting each equation byw (Zij (x, y)), where w (x) = sin (πx) is a win-dowing function that reduces the effects ofunder- or over-exposed pixels. This consid-erably improves colour reproduction and re-duces the aforementioned artefacts.

3 Virtual photography

Virtual photography mimics the imaging pro-cess described by a camera response curve g.This is achieved by inverting g and using it tomap exposure values I∆t,p back to pixel val-

ues Z∆t,p. This process can be thought of asexposing the underlying HDR image virtually.However, contrary to conventional photog-

raphy, both exposure time ∆t and the flashintensity p can be varied freely and may alsolie between the flash-exposure settings physi-cally supported by the camera used to createthe HDR image. Additionally, the responsecharacteristics of any imaging process – andtherefore any digital camera – can be mod-elled by using an appropriate response curve.In more detail, the ambient image A and

the flash image F are recombined to the ex-posure I∆t,p = ∆t · A + p · F using (1), forthe exposure time ∆t and the flash intensityp. This exposure is converted to pixel valuesZ∆t,p by rearranging (2) as

Z∆t,p (x, y) =

g−1 (ln (∆t ·A (x, y) + p · F (x, y))) . (3)

5

(a) Real photographs (b) Virtual photographs

Figure 3: Comparison of (a) real and (b) virtual photographs.

Examples of virtual photographs are shownin figure 3, in which the original input pho-tographs (a) are compared with the virtualphotographs (b) obtained for the same cameraparameters.Virtual photographs can in principle be

produced for arbitrary exposure time andflash intensity settings. However, if the set-tings lie far outside the range of parametersused in taking the original input photographs,the virtual photograph loses fidelity. In theworst case, this leads to saturation and otherartefacts such as extreme amplification ofresidual camera noise. In order to reducethese kind of artefacts, the parameters shouldideally lie within the range used in the inputphotograph sequence.The combined approach of flash-exposure

HDR recovery and virtual photography de-couples the processes of taking the pho-

tographs and experimenting with the camerasettings in a post-processing step.

4 Depth-Compensating Flash

The depth-compensating flash is a noveltone mapping operator for flash-exposureHDR images which addresses the issue thatflash-induced brightness suffers a fall-off.Objects nearer to the camera tend to beover-illuminated and distant objects under-illuminated.The algorithm presented in this section

counteracts the depth fall-off using the depth-orientation map β [1] which results in a moreuniform scene illumination.The top-level design of the depth-

compensating flash is outlined in figure 5. Itcomprises the following five stages:

6

Figure 4: Comparison of virtual photographs with (right) and without (left) depth-compensating flash, for flash-only image (top) and combined image (bottom) respectively.

1. Extract Luminance and ChromaticitySince the flash predominantly affectsbrightness, the input images A and F areconverted to the Yxy colour space to onlyoperate on the luminance componentsAY and FY while leaving the chromatic-ity components Axy and Fxy unchanged.

2. Calculate Depth-Orientation MapThe depth-orientation map β = FY /AY iscalculated.

3. Truncate & FilterThe depth-orientation map β is truncatedto the range [βmin, βmax] to limit the am-plification and attenuation applied to theflash image in the next step. Additionally,β is filtered using a two-dimensional me-dian filter to reduce the amount of noisecontained, resulting in β′.

4. Scale Flash LuminanceThe flash luminance is divided by β′, tocompensate for the depth fall-off associ-ated with the flash. This results in F ′Y .

5. Recombine to Colour ImageFinally, the flash chromaticity Fxy is com-bined with the new flash luminance F ′Yand converted to the HDR image F ′ – thedepth-compensated flash image.

The results of applying the depth-compensating flash are shown in figure 4.Note the improved illumination visible in thebackground areas on the right-hand side.I expect that work on flash/no-flash pho-

tography [3, 5] could also be extended toflash-exposure HDR images. However, theseapproaches are not considered here.

7

5 Summary and Future Work

I presented a revised approach to flash-exposure HDR imaging and two applicationsbased on it. Virtual photography is a tech-nique that allows the user to generate pho-tographs for freely chosen exposure times andflash intensities. The depth-compensatingflash compensates for the brightness fall-offwith depth, to create a more uniformly illu-minated image.Potential future work includes an embed-

ded implementation of these applications ona digital camera which would calculate theflash-exposure HDR image on-board. Thiswould lead to a lot of post-processing poten-tial, in a similar way to ‘raw’ photographs.Taking the full set of input photographs

is clearly wasteful, because it comprises allpossible combinations of exposure times andflash intensities. An adaptive flash-exposuresampling strategy, e.g. based on the one pre-sented by Agrawal et al. [1], would reducethe number of required photographs consid-erably.

Acknowledgements

I would like to thank my proof-readers as wellas Rahul Vohra, Jon Fielder and Neil Dodgsonfor their support and suggestions.

References

[1] A. Agrawal, R. Raskar, S. K. Nayar,and Y. Li. Removing photography arti-facts using gradient projection and flash-exposure sampling. ACM TOG (Proc. SIG-GRAPH), 24(3):828–835, 2005.

[2] P. E. Debevec and J. Malik. Recoveringhigh dynamic range radiance maps fromphotographs. In Proc. SIGGRAPH, pages369–378, 1997.

[3] E. Eisemann and F. Durand. Flash pho-tography enhancement via intrinsic re-lighting. ACM TOG (Proc. SIGGRAPH),23(3):673–678, 2004.

[4] S. Mann and R. W. Picard. On being‘undigital’ with digital cameras: Extend-ing dynamic range by combining differ-ently exposed pictures. In Proc. IS&T An-nual Conference, pages 422–428, 1995.

[5] G. Petschnigg, R. Szeliski, M. Agrawala,M. Cohen, H. Hoppe, and K. Toyama.Digital photography with flash and no-flash image pairs. ACM TOG (Proc. SIG-GRAPH), 23(3):664–672, 2004.

[6] R. Raskar, J. Tumblin, A. Mohan,A. Agrawal, and Y. Li. Computationalphotography. In Eurographics Tutorials,pages 557–624, 2007.

8

′

′

′

Extract Luminance

Extract Luminance

Extract Chromaticity

Truncate & Filter

Recombine to Colour Image

Compute Depth-Orientation Map

Scale Flash Luminance

Flash Image

Ambient Image

Ambient Luminance

Flash Luminance

Flash Chromaticity

Depth-Orientation Map

Filtered Depth-Orientation Map

Enhanced Flash

Luminance

Enhanced Flash

Image

Figure 5: Top-level design of the depth-compensating flash.

9