Computational Optical Imaging - Optique Numerique --...

Ivo Ihrke / Winter 2013

Computational Optical Imaging -Optique Numerique

-- Noise, Dynamic Range and Color --

Winter 2013

Ivo Ihrke

Organizational Issues

I received your email addresses

Course announcements will be send via email

Course webpage at http://giana.mmci.uni-saarland.de/

Teaching -> Computational Optical Imaging

Sources of noise

Photon shot noise

Dark current shot noise

Fixed pattern noise

Readout noise

[Janesick97]

Noise Sources

[Reibel2003]

read noisephoton shot noise

dark current noise

Photon shot noise

Variance in number of photons that are counted

they arrive in a Poisson random process

Standard deviation is square root of signal

relative noise decreases with signal

Fundamental limit on photodetector precision!

Can be reduced by averaging multiple exposures.

Fixed pattern noise

Caused by variations in component values

Big problem for CMOS sensors

An amp at every pixel, and one for every column

Gain variation (proportional to signal PRNU)

Bias variation (independent of signal – dark current)

Can be partially canceled by correlated double sampling (CDS)

CCD’s transfer all charge to a single output amplifier

Dark current

Things besides photons can knock electrons loose in the silicon. These are collected, too.

Highly temperature dependent

doubles every 5-8 degrees C

May be reduced by cooling the sensor.

Proportional to exposure time

Limits exposure durations—eventually, the dark current fills your well capacity.

Dark Current Noise

Dark current has fixed pattern noise.

Dark current varies because of irregularities in the silicon.

Dark current has shot noise, too!

dominates in dark areas for long exposures

Mean dark current may be subtracted

but subtracting frames increases shot noise

subtract the average dark current

Dark current is why astronomers chill their image sensors.

Peltier Cooling of CMOS chip

[Gary Honis]

Thermal Noise

Generated by thermally induced motion of electrons in resistive regions (resistors, transistor channels in strong inversion…)

What does it mean?

Independent of the signal.

Zero mean, white (flat, wide bandwidth)

Another problem for CMOS, not CCD imagers

Dominates at low signal levels

Can limit dynamic range

Dark Current Noise – Removal

cooling the chip

noise removal techniques to separate image data from noise

e.g. median filtering

25 s exposure time

uncooled cooled

Noise, noise, noise…

Reset (kTC) noise

thermal noise when “resetting” the CMOS photodetector—a big deal, actually.

can be corrected with CDS

Amplifier noise

thermal

spatially non-uniform

1/f noise

non-linearities

Quantization noise

“truncate” analog value to N bits

Analog/Digital Conversion

Correlated Double Sampling

reduce noise by comparing against a reference charge

Combined Noise Model [Reibel2003]

- fixed pattern noise

- readout noise

- thermal dark current shot noise

- photon shot noise

- photo response non-uniformity

- non-linear effects

NLPRNUPSNPSNDSNRFPNN CTOT

2222222

Combined Noise Model [Reibel2003]

- fixed pattern noise (can be calibrated)

- readout noise (CDS)

- thermal dark current shot noise (cooling)

- photon shot noise (multiple exposures)

- photo response non-uniformity (per-pixel gain)

- non-linear effects (can also be calibrated for)

NLPRNUPSNPSNDSNRFPNN CTOT

2222222

Noise Distribution

[Reibel2003]

ADU = Analog-Digital Unit, e.g. 1 ADU = 0.5 e-

Digital Images

Images are now numbers (corrupted by noise)

Digital

Sensor noise

Dynamic Range

Tone Curve

Recording “Medium”

Monochromatic

Optical

Distortions

Aberrations

Digital Images - Limitations

Dynamic Range

max output swing

noise in the dark

Saturation level – dark current

Dark shot noise + readout noise=

“noise in the dark” is random noise sources that

cannot be corrected with circuit tricks

Photon shot noise and read noise

Dynamic Range of Standard Sensors

13.5 EVs or f-stops = contrast 11,000:1 = color negative

Dependency of Dynamic Range on ISO

“Unity gain” is where 1 digital unit (ADU) equals 1 electron (e-)

This happens at different ISO settings for different sensors

Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise)

“digital gain” multiplies the digital values

All gain settings beyond unity gain reducedynamic range

What is High Dynamic Range (HDR)?

http://en.wikipedia.org/wiki/High-dynamic-range_imaging

HDR Acquisition – Exposure Brackets

[Debevec & Malik 97]

Radiance Map Tonemapped HDR Image

Exposure Sequence

Shutter Speed

F/stop (aperture)

Neutral Density (ND) Filters

Gain / ISO / Film Speed

(noise)

Ways to vary the exposure

Combining the image

dttxlxI ),()(

• scene constant over exposure time (or ND-filter)

radiance

),()( xltxI

• assumes linear response (radiometric calibration!)

have several measurements with different t

Combining the image

• introduce a weighting function for the pixels:

• centered at the sensor mean value,

e.g. Gaussian (image data in [0,1])

• compute final image as

)5.0)((

finalxIw

txIxIw

xI))((

/)())((

• Important for many vision and graphics algorithms

EIg :1

• Use a color chart with precisely known reflectances.

• Use more camera exposures to fill up the curve.

• Method assumes constant lighting on all patches

• Works best when source is far away (example sunlight).

• Inverse exists (g is monotonic and smooth for all cameras)

Irradiance = const * Reflectance

3.1%9.0%19.8%36.2%59.1%90%

Radiometric Calibration

Response Curve - Practice

Measurement: ColorCalibrationToolbox http://giana.mmci.uni-saarland.de/website-template/software.php

Example: 29 exposures of Gretag-Macbeth color checker

(uses EXIF info - exiftool)

Color Calibration Toolbox

Zoom-in

Mark the patch rectangle

Make sure the patches are properly extracted

Verify response curve – the example is for jpg on the Canon 5D mark II

Make sure the samples are fit well

Response curves

(R,G,B)

Inverse response curves

samples

Check HDR image

How is the curve estimated ?

Variant of Mitsunaga and Nayar, “Radiometric Self Calibration”, CVPR 1999

Polynomial fit to data samples

Variations:

─ enforce monotonicity (derivative > 0)

Prevents “wiggling”

─ enforce passing of curve through (0,0) and (1,1)

map range to range

─ Perform a weighted fit

accounts for sample non-uniformity

Response Curve – Take Home Points

Usually linear for RAW images

Don’t rely on it – verify

Usually non-linear for jpg or other compressed/processed formats

Estimation from random images may be unstable

Use well defined target (color checker)

Prefer continuous-curve algorithms, especially for high bit depths

Applications - HDR Display

• 47” TFT LCD, LED backlight

• aspect ratio 16:9

• resolution 1920 x 1080

• contrast >1,000,000:1

• brightness 4,000 cd/m2

Images courtesy Dolby

Applications - Image Based Lighting

EDR and HDR Cameras

Grass Valley Viper

(10 bits log)

Panavision Genesis

(10 bits log)

Spheron

- Scanning

- 26 f-stops

Extended range High dynamic range

Super CCD (Fuji)

octagonal grid

elements with differentsensitivity

extended DR

better in low light

Used in consumer products (Finepix)

http://www.henner.info/super_ccd.htm

HDRC – Log Encoding

CMOS – pixel amplifier output is logarithmic

U - logarithmic

Per-Pixel Exposure Time Controlwww.pixim.com

no pixim with pixim no pixim with pixim

Adaptive Dynamic Range Imaging

[Nayar and Branzoi 03]

Programmable Imaging

[Nayar

Textbook

• HDR image / video encoding

• capture, display, tone reproduction

• visible difference predictors

• image based lighting, etc.

Sensing color

Eye has 3 types of color receptors

Therefore we need 3 different spectral sensitivities

Ways to sense color

Field-sequential color

simplest to implement

only still scenes

Proudkin-Gorskii, 1911

(Library of Congress exhibition)

Ways to sense color

only still scenes

Ways to sense color

only still scenes

Ways to sense color

only still scenes

Examples - Prokudin-Gorskij

Self-portrait 1915

Photograph 1910, Emir of Bukhara , Prokudin-Gorskii

Examples - Prokudin-Gorskij

Examples - Lew Tolstoy

1910 photograph, Sergey Prokudin-Gorskii1887 painting, Ilya Repin

Color Wheel

one color channel is captured at one shot

3 times the acquisition time

static images only

Liquid Crystal Tunable Filter (LCTF)

Computer controllable spectral filter

VariSpec LCTF

Spectral Response of Lyot Stage

)(cos)( 2

Relative

Transmissivity:

Waveplate

parameters

(birefringence,

thickness)

Electrically Tunable Birefringence

is implemented by liquid crystal

in Lyot stage

Spectral Response of 7 Differently tuned Lyot Stages

using several stages in sequence:

product = result

VariSpec spectral curves

Ways to sense color – 3-Chip Camera

dichroic mirrors divide light into wavelength bands

does not remove light: excellent quality but expensive

interacts with lens design

problem with polarization

image: Theuwissen

Foveon Technology

3 layers capture RGB at the same location

takes advantage of silicon’s wavelength selectivity

light decays at different rates for different wavelengths

multilayer CMOS sensor gets3 different spectral sensitivities

don’t get to choose the curves

Ways to sense color

Color filter array

paint each sensor with an individual filter

requires just one chip but loses some spatial resolution

“demosaicing” requires tricky image processing

primary secondary

Demosaicing

bilinear interpolation

sampling theory

edge-directed/pattern-based interpolation

correlation-based

Demosaicing

Original image Bilinear interpolationRon Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html

Demosaicing

Ron Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html

Bilinear interpolation Edge-weighted interpolation

Bilinear Interpolation

perform interpolation for each color channel separately

3432141223

343233

set all non-measured values to zero then convolve

BRF 4/

Problem: Aliasing

[Alleysson & Suesstrunk05]

Problem: Aliasing

Fourier Space

Excessive Blurring

Grid Effect

Bleak colors

Color Aliasing

optimize r1 and r2 to gain best separation

Low-pass filter luminance

High-pass filter chrominance (orthogonal filter)

Demultiplex chrominance

Interpolate opponent colors

Add luminance and interpolated colors

Gradient-based (dcraw)

1.Calculate gradients in 5x5 region

2.Select subset of gradients (below threshold)

3.Average color differences in the determined regions

[Chuang et al. 99]

Gradients

“Gradient” S = |1323|3/|818| RRGG

2/|717|2/|919| BBBB

3/|1222|3/|1424| GGGG

Regions

selection: “gradient” < 1.5*Min+0.5*(Max-Min)

e.g. {S,W,NE,SE}

S: R = (R13+R23)/2, G = G18, B = (B17+B19)/2

NE: R = (R13+R5)/2, G = (G4+G8+G10+G14)/4, B = B9

Average

Rsum = (Rs + Rw + Rne + Rse)/4

Gsum = (Gs + Gw + Gne + Gse)/4

Bsum = (Bs + Bw + Bne + Bse)/4

average of color differences

G13 = R13 + (Gsum-Rsum); B13 = R13 + (Bsum-Rsum)

Demosaicing – Take-home-points

2/3 of your image are just made up!

color resolution is less than image resolution

be careful with spiky BRDFs

combining multiple video frames might help

98% of all demosaicing algorithms are ad-hoc

─ smoothing based on constant hue assumption afterwards

White Balance

daylight

flashflourescent

tungsten

capture the spectral

characteristics of

the light source to

assure correct

color reproduction

White Balance

Human perception adapts to illumination condition

Practice: division of RGB values

White Balance

Camera built-in function

derive scale from white point

incandescent

tungsten

infraredred green blue

ultra violet

wavelength

White Balance

ultra violet

wavelength

White Balance

ultra violet

wavelength

White Balance

Human perception adapts to illumination condition

Practice: division of RGB values

Theory: achieve a “neutral” spectrum

(only works for broad band sources and

broad band reflectance)

Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma

Spectrum to Image

do not have spectral display

not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision)

cones 6-7 million

rods ~120 million

rods more sensitive

Color Vision

color vision by cones

significant overlap of the response functions

L = long

M = mid

S = short

Color Vision

L ~63%, M ~31%, S ~6% of cones

eye least sensitive to blue, most sensitive to yellowish-green

spectral region outside of support of the response functions cannot be perceived

Spectral response of human eye

reproducing color is tricky

color matching experiments

─ use light source with known spectral distribution

(i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter

─ additionally, use monochromatic sources

@ 700,546,435 nm

─ let human observers adjust apparent brightness of one of the sources to match the other

Color matching functions

Color Spaces

─ RGB matching functions

negative !

XYZ space

The CIE (1931) standard observer

How to compute a tristimulus image from a spectral representation ?

We have to integrate with the spectrum with the appropriate color matching function

dxlfxIXX ),(ˆ)()(

dxlfxIYY ),(ˆ)()(

dxlfxIZZ ),(ˆ)()(

Now to RGB

convert XYZ to RGB

Horseshoe Diagram

White Point for Different Color Temperatures

Planckian

Locus:

- convert black body

temperature to XYZ and

put intohorseshoe diagram

L_\lambda = spectral radiance [W/m^2/m]

lambda = wavelength [m]

h = Planck’s constant [J.s]

k = Boltzmann constant [J/K]

c = speed of light [m/s]

T = temperature of black body [K]

Display Gamut

white point

Bibliography

Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical

Engineering Press, Bellingham, Washington, 1998.

Theuwissen, A. Solid-State Imaging with Charge-Coupled Devices. Kluwer Academic Publishers, Boston, 1995.

Curless, CSE558 lecture notes (UW, Spring 01).

El Gamal et al., EE392b lecture notes (Spring 01).

Several Kodak Application Notes at http://www.kodak.com/global/en/digital/ccd/publications/a

pplicationNotes.jhtml

Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003

Bibliography

D. Alleysson, S. Suesstrunk: Linear Demosaicinginspired by the Human Visual System, IEEE Trans. on Image Processing, 14(4), 2005.

B. K. Gunturk, Y. Altunbasak, R. M. Mersereau: Color Plane Interpolation Using Alternating Projections, IEEE Trans. on Image Processing, 11(9), 2002.

E. Chang, S. Cheung, D.Y. Pan: Color filter array recovery using a threshold-based variable number of gradients. Proc. SPIE, vol. 3650, pp. 36-43, 1999.

http://www-ise.stanford.edu/~tingchen/algodep/vargra.html

Bibliography

Y. Takahashi, K. Hiraki, H. Kikuchi, S. Muaramtsu: Color Demosaicing Using Asymmetric Directional Interpolation and Hue Vector Smoothing, IEICE 20th Workshop on Circuits and Systems, 2008.

R. Kimmel, Demosaicing: Image Reconstruction from Color CCD Samples, IEEE Trans. on Image Processing. Vol. 8, No. 9, 1999.

Boris Ajdin, Matthias B. Hullin, Christian Fuchs, Hans-Peter Seidel, Hendrik P. A. Lensch: Demosaicing by Smoothing along 1D Features. Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.

(ICC –international color consortium)

color management system

capture the properties of all devices

─ camera and lighting

─ monitor settings

─ output properties

common interchange space

sRGB standard as a definition of RGB

ICC Profiles

input device

(e.g. camera)

input profile

profile

connection

output device

(e.g. printer)

output profile

display

device

monitor)

monitor

profile

profile connection spaces

─ CIELAB (perceptual linear)

─ linear CIEXYZ color space

can be used to create an high dynamic range image in the profile connection space

allows for a color calibrated workflow

ICC Profiles and HDR Image Generation

input device

(e.g. camera)

input profile

profile

connection

output device

(e.g. printer)

output profile

Computational Optical Imaging - Optique Numerique --...

Documents

L’Enregistrement Optique : Comment ça marchesdg-master.com/lesestoff/2-GIS-DON-archivage-numerique...Jean-José Wanègue : « L’enregistremJean-José Wanègue : « Les disques

Ideal Optique Elbeuf - Optique de sport

JURA NUMERIQUE...2. La montée en débit pour apporter rapidement un service aux Jurassiens L’objectif est d’améliorer les débits en apportant la fibre optique au cœur de la

Cours d’electronique numerique - Freebertrand.granado.free.fr/Licence/ue201/coursimp.pdf · 2009-01-29 · Cours d’electronique numerique Cours d’electronique numerique Bertrand

système numerique

SCHEMA DIRECTEUR D’AMENAGEMENT NUMERIQUE DE … · sur la fibre optique jusqu’à l’habitant (FttH), déployée dans le cadre d’un réseau équilibré et ... 800 Km de fibre

Computational Optical Imaging - Optique Numerique -- Image ...giana.mmci.uni-saarland.de/website-template/...panoramas_3Dbasic… · Blend the two together to create a mosaic

Computational Optical Imaging - Optique Numerique ...giana.mmci.uni-saarland.de/website-template/... · Ivo Ihrke / Winter 2013 Advantages simple approach works quite reliable accuracy

Accessibilite Numerique

Optique Géométrique Optique Physique.pdf

Computational Optical Imaging - Optique Numeriquegiana.mmci.uni-saarland.de/website-template/... · Joseph Nicéphore Niépce 1765 - 1833 First photograph. Ivo Ihrke / Winter 2013

Lexique numerique

Jounee numerique

Audio Numerique

Cours Numerique

Électronique numerique

Startegie numerique

L’Identite Numerique

Portfolio Numerique

Computational Optical Imaging - Optique Numerique ...giana.mmci.uni-saarland.de/website-template/lectures/Computational... · Matchmoving in Cloverfield. Ivo Ihrke / Winter 2013 References