High dynamic images between devices and vision limits

Le immagini ad alta dinamica tra i limiti dei dispositivi e quelli

della visioneAlessandro Rizzi

Dipartimento di Informatica e ComunicazioneUniversità degli Studi di Milano

Friday, June 10, 2011

Outline

HDR imaging

HDR in practice: measuring the limits

Using HDR


The dynamic range



Define HDR ?

do we need a threshold number ?


Define HDR ?

do we need a threshold number ?

NO


Define HDR

A rendition of a scene with greater dynamic range than

the reproduction media


That is ?



Annibale Carracci (1560-‐1609) PaesaggioFriday, June 10, 2011

Photo: C. OleariFriday, June 10, 2011

Photo: C. OleariFriday, June 10, 2011

Annibale Carracci (1560-‐1609) PaesaggioFriday, June 10, 2011

Source/lamp Average Luminance cd/m2

Xenon short arc 200 000 ÷ 5 000 000 000Sun 1 600 000 000Metal halide 10 000 000 ÷ 60 000 000Incandescent 20 000 000 ÷ 26 000 000compact Fluorescent 20 000 ÷ 70 000Fluorescent 5 000 ÷ 30 000Sunlit clouds 10 000Candle 7 500blue sky 5 000Preferred values for indoor lighIng

50 ÷ 500

White paper at sun 10 000White paper at 500 lx 100White paper at 5 lx 1

Courtesy: C. Oleari

Light levels


Dynamic ranges


Dynamic ranges

?Friday, June 10, 2011

Range limits and quantization: the ‘salame’ metaphor



Range compression from incorrect pixel perspective





Very wide range obtained with isolated stimuliimpossible to obtain in an image


The “salame” metaphor

Dynamic range Quantization


The “salame” metaphor

Dynamic range Quantization

More bits do not mean wider rangeLess bits do not mean shorter range


SceneDR

SensorDR

16 bit

28=2562-3 log unit

216=65536 4-5 log unit

8 bit


SceneDR

SensorDR

16 bit

28=2562-3 log unit

216=65536 4-5 log unit

8 bit

NO


SceneDR

SensorDR

8 bit

2-3 log unit

4-5 log unit

8 bit

SceneDR

SensorDR

16 bit

16 bit


SceneDR

SensorDR

8 bit 16 bit

SceneDR

SensorDR


SceneDR

SensorDR

8 bit 16 bit

SceneDR

SensorDR


The HDR idea

http://www.adolfo.trinca.name/public/2010/11/ahdrdiagram.jpg






The HDR ideaHow ?

general solution ?rendering intent ?







http://www.digitalcameratracker.com/how-to-create-high-definition-range-hdr-photos/






Two sides of the coin

• Objective data: recording/displaying physical light colorimetric distribution

• Subjective data: reproducing appearance (or different rendering intent)


Mapping the world: the characteristic curve


H & D curve


H & D curve


H & D curve


H & D curve


http://www.dpreview.com/reviews/olympuse3/page21.asp

Olympus E-3




Exposure problem




History of HDR imaging


HDR 1858H.P. Robinson “Fading Away


Mees (1920) 2 negative print

“The Fundamentals of Photography”


Ansel Adams


ISCC 11/05-McCann

Ansel Adams - Zone System


Jones and Condit, 1941Measurements of dynamic range of real scenes

0.0 3.01.5log range

REFLECTANCE RANGE OF PRINTS

Maximum

Average of 126 outdoor scenes

Minimum

SCENE RANGE OF WORLD


L.A.Jones & H.R.Condit, JOSA,1941


Retinex starting idea

Green record

5588

ratio = 0.62

146

ratio = 0.62

230

digit ~ luminance 119 119

Ratios are constant in sun and shadeFriday, June 10, 2011

1980Friday, June 10, 2011

Retinex cameraFriday, June 10, 2011

Capturing and reproducing the scene



Sensors dynamic range

Limited !


Is HDR a technological problem ?


Expanding sensors dynamic range• Sensors that compress their response to light due to their

logarithmic transfer function;• Multimode sensors that have a linear and a logarithmic

response at dark and bright illumination levels, (switches between linear and logarithmic modes of operation);

• Sensors with a capacity well adjustment method;• Frequency-based sensors, sensor output is converted into

pulse frequency; • Time-to-saturation [(TTS); time-to-first spike] sensors,

signal is the time the to saturated pixel; • Sensors with global control over the integration time; • Sensors with autonomous control over the integration time,

where each pixel has control over its own exposure.Spivak A, Belenky A, Fish A & Yadid-Pecht O (2009) Wide dynamic-range CMOS image sensors:

A comparative performance analysis, IEEE Trans. on Electron Devices, 56, 2446-2461.Friday, June 10, 2011



The HDR idea







The HDR ideaHow ?







Multiple image acquisition


•Multiple Exposures

•Use Multiple Times

•Recover scene radiances at all pixelsfrom camera digits

€

CameraDigit = radiance* time( )

New goal: Accurately measure radiances


Multiple Exposures

Flux = Luminance * time

Scene Luminance = Flux / time

Scene Luminance = Camera Digit / time


Multiple Exposures

One Spot (ScaleD)

0

50

100

150

200

250

0.0001 0.0010 0.0100 0.1000 1.0000 10.0000 100.0000 1000.0000Exposure Flux [(cd/m2) * sec]

Cam

era

Dig

it

1/8 sec1/4 sec1/2 sec1 sec2 sec4 sec8 sec16 sec32 sec64 secFIT

Flux = Luminance * time

Camera Digit


HDR file formats

Source: Reinhard et al., High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics)


HDR file formats

Source: Reinhard et al., High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics)


Acquisition limits



The glare problem


The glare problem



Effect of illumination

Assumes 0.0 glare

1.0 refl * 1.0 illum = 1.0 cd/m2

0.2 refl *0.01 illum = 0.002 cd/m2


Glare is image dependent

Assumes 0.001 glare

1.0 refl * 1.0 illum = 1.0 cd/m2

0.2 refl *0.01 illum = 0.002 cd/m2

0.002 cd/m2 *0.001 = 0.000002

1.0 cd/m2 *0.001 = 0.001

0.001

0.001


Ratio Signal/Glare

Assumes 0.001 glare

1.0 cd/m2)/(0.000002) = 5*10^5

( 0.002 cd/m2)) / (0.001) = 2


Sowerby, “Dictionary of Photography”, 1956


Parasitic Images


Camera limits•Glare•Unwanted scattered light in camera

•air - glass reflections •lens (number of elements)•aperture •angle off optical axis

•camera wall reflections•sensor surface reflections

•We must measure actual veiling glare limit


Measuring overall camera glare



HDR Test Setup


Text

Synthetic HDR(High-Dynamic Range)

Images

18,619:1

digit 255 = 2094.2 cd/m2

= 18,619

digit 0 = 0.11 cd/m2

2094.2 cd/m2

0.11 cd/m2

Goal Image


Targets

18,619:1

20:1


16 sec exposure - Target 1scaleBlackFriday, June 10, 2011



16 sec exposure

TextTarget 1B

Target 4B

Target 4W

Text


Constant Luminance - Variable SurroundFriday, June 10, 2011

Minimum Glare


Mild Glare


Maximum Glare




Measure In-camera Accuracy

SceneScene

Dynamic Range

In-cameraAccurate

Range

MaximumError

(% radiance)

1scaleB 20:1 20:1 0

4scaleB 18,619:1 3,000:1 300% Min

4scaleW 18,619:1 100:1 10,000% Max

4.3 log10 scene ----> 3.0 log10 image

1


Side Dupe FilmFriday, June 10, 2011

Slide Dupe FilmFriday, June 10, 2011

One Negative Capture4scale Black - Single Negative

1.50

1.70

1.90

2.10

2.30

2.50

-1.00 -0.50 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50Log Cd/m2

Log

dig

it

3.5 Log10 units


Dynamic Range (OD)Friday, June 10, 2011

HDR from cameras

• Range of usable captured information

• Range of accurate luminance information

(much smaller)

• Scene dependent


Courtesy: M. Fairchild


Glare insertion

Gregory Ward Larson, Holly Rushmeier, and Christine Piatko, “A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes”, IEEE Trans on VISUALIZATION AND COMPUTER GRAPHICS, VOL. 3, NO. 4, oct-dec 1997


Display: measuring the human limits



Magnitude estimates (100-1)Friday, June 10, 2011

•Luminance does not correlate uniquely with appearance

•No global tone scale can render the appearance


Magnitude Estimation of AppearanceChange Surrounds

0102030405060708090

100

0.10 1.00 10.00 100.00 1000.00 10000.00Log Luminance (cd/m2)

Mag

nit

ud

e Es

tim

atio

n

Min [0 cd/m2] Max [2094 cd/m2]


We need a new range target

•White surround•adds glare•changes surround

(simultaneous contrast)

•Vary dynamic range with•constant glare•contrast surround


Center/Surround Basic Unit

Fixed contrast surround 88%

Gray test areas 12%(small differences)


90o rotation



Testing different glares% of white surround

100%

50%0%


Single =

Double =(superimposed)

5.4 log10 range

2.7 log10 range

Single & Double Density Transparencies


5.4 & 2.7 log10 Ranges Constant Glare & Surround


0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

0123456relative optical density

mag

nitu

de e

stim

atio

n

50% Single Density

50% whitesurround

White[100] = 0.0 rOD - Black [1] = 2.89 rOD


0

10

20

30

40

50

60

70

80

90

100


mag

nitu

de e

stim

atio

n

50% Double Density 50% Single Density

50% whitesurround

2.3 log10 units


0

10

20

30

40

50

60

70

80

90

100


mag

nitu

de e

stim

atio

n

White Double Density White Single Density

2.0 log10 units

100% whitesurround


0

10

20

30

40

50

60

70

80

90

100


mag

nitu

de e

stim

atio

n

Black Double Density Black Single Density

0% whitesurround

5.0 log10 units


0

10

20

30

40

50

60

70

80

90

100


mag

nitu

de e

stim

atio

n

Black Double Density Black Single Density

0% whitesurround

5.0 log10 units

Over 20 not big improvement


Measurements of apparent range

(depends on area of white)

•100% = 2.0 log10 units

• 50% = 2.3 log10 units

• 8% = 2.9 log10 units


DD DD DD DDFriday, June 10, 2011

Test summary

•Double transmission contrast

•Double dynamic range

•very small change in appearance range

•Visual limit ~ area of white surround

•area of white controls glare


What is on the retina: calculated retinal luminance



What comes to the retina is different from the image

High glare Low glare


Glare vs. Contrast

Veiling glare increases gray luminance

Contrast decreases gray appearance

Contrast offsets glare


Discussion

• Glare lowers the physical contrast • Spatial comparisons increase the

contrast of appearance.

• The two act in opposition. • Change with distance are different and

the cancellation is far from exact.


1 Vos, J.J. and van den Berg, T.J.T.P, CIE Research note 135/1, “Disability Glare”, ISBN 3900734976 (1999).

PIGMENTBlue eyed Caucasian 1.21Blue green Caucasian 1.02Mean over all Caucasian 1.00Brown eyed Caucasian 0.50Non Caucasian with pigmented skin and dark brown eyes 0.00

Glare Spread Function


Glare Spread Function

Plotted in log scale


False-color LookUpTable (LUT)

Dynamic Range = 5.4 ODor 251,189:1


Visualize HDR targets

Same LUT applied to SD & DD


Retinal image



Visualize Retinal ImagesFriday, June 10, 2011


Change LUT for Retinal ImagesFriday, June 10, 2011

Change LUT for Retinal ImagesFriday, June 10, 2011

Two scene-dependent spatial mechanisms:glare and contrast

Glare masks the strength of spatial contrast

Scene Retina Appearance 1,000,000:1 100:1 1,000:1

SpatialGlare

SpatialContrast


Ranges


Tone-rendering problem and spatial comparisons



Choosing a rendering intent


124


124





Land experiment


Land experiment


Land experiment

Projector


Land experiment

Projector


Land experiment

Projector

ES=100 EL=100EM=100


Land experiment

Projector Colorimeter

ES=100 EL=100EM=100


Land experiment

LS=255 LL=255LM=115


ES=100 EL=100EM=100


Land experiment

LS=255 LL=255LM=115


ES=100 EL=100EM=100

Observer


Land experiment

LS=255 LL=255LM=115


ES=100 EL=100EM=100

Observer

PINK



Observer

Land experiment



ES=50 EL=50EM=111

Observer

Land experiment



ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land experiment


PINK


ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land experiment


PINKGRAY


ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land experiment


visual sensation


HVS: local compression of range


HVS: local compression of range


Tone mapping vs Tone rendering

No tone mapping operator (global) can mimic vision

We need an image dependent tone renderer operator (local)


Black and White Mondrian


HP 945 Images without “Frames of Reference”Friday, June 10, 2011

Some examples



Bob Sobol, HP

R. Sobol, “ Improving the Retinex algorithm for rendering

wide dynamic range photographs”, in Human Vision and Electronic

Imaging VII, B. E. Rogowitz and T. N. Pappas, ed., Proc. SPIE 4662-41, 341-348,

2002.



Original ACEOriginal ACE

ACE


STRESS Tone Rendering


Judging the results


Beauty contest

C. Gatta, A. Rizzi, D. Marini, “Perceptually inspired HDR images tone mapping with color correction”, Journal of Imaging Systems and Technology, Volume 17 Issue 5, pp. 285-294 (2007).


HDR is in the middle

Imagein CPUmemory

Scene DisplaySpatialImagein CPU

SpatialAlgorithm

GlareSensor

Pre-LUT

Post-LUTgraphics

card


Summary


• To understand HDR we need a new perspective!

1. Veiling glare limits the range on the retina 2. Neural processing (spatial) determines appearance 3. Neural is stronger than it appears [neural cancels glare] 4. General Solution requires spatial process [mimic vision] 5. Tone-Scale is limited, we need Tone-rendering [scene dependent]


Take home points

• HDR limits are not (only) technological

• Glare limits both acquisition and vision

• Glare is scene dependent

• Human vision use spatial comparison to overcome this limit

• Tone renderer operator can use the same approach


Take home points

HDR works very well

• because preserves image information

• not because are more accurate (not possible)


References• J. J. McCann, A. Rizzi, “Camera and visual veiling glare in HDR images”

Journal of the Society for Information Display 15/9, 721–730 (2007).

• J. J. McCann, “Art, Science and Appearance in HDR” Journal of the Society for Information Display 15/9, 709–719 (2007).

• A. Rizzi, J. J. McCann, “Glare-limited Appearances in HDR Images”, Journal of the Society for Information Display, 17/1, pp. 3-12, (2009).

• J. J. McCann, A. Rizzi, “Retinal HDR Images: Intraocular Glare and Object Size” Journal of the Society for Information Display, 17/11, pp. 913-920, (2009).


The art and science of HDR imagingJ.J. McCann, A. Rizzi

(expected publication date autumn 2011)


Thank you

[email protected]


mailto:[email protected]

mailto:[email protected]