ADMINISTRIVIA SENSORayan/courses/cse559a/PDFs/lec... · 2019. 8. 29. · Install Anaconda, LaTeX, Git Submit your Public Key Submit Problem Set 0 2 ADMINISTRIVIA Oﬀice Hours Arushee

CSE 559A: Computer Vision

Fall 2020: T-R: 11:30-12:50pm @ Wrighton 300 / Zoom

Instructor: Ayan Chakrabarti ([email protected]).Course Staff: Adith Boloor, Patrick Williams

Sep 17, 2020

http://www.cse.wustl.edu/~ayan/courses/cse559a/

1

ADMINISTRIVIAADMINISTRIVIAAll information @ http://www.cse.wustl.edu/~ayan/courses/cse559a/

Read through course websiteSyllabusProblem Set and Resource SectionLate PoliciesCollaboration & Academic Honesty Policies

Join PiazzaReview Pre-req SlidesInstall Anaconda, LaTeX, GitSubmit your Public KeySubmit Problem Set 0Post on piazza if you have issues with your setup.

2

THE PINHOLE CAMERATHE PINHOLE CAMERA

3


4


5


6


7


8


9


10


11


12


13


14


15


16


17


18


19


20


21


22

SENSORSENSOR

: Light energy, per unit area per unit time, arriving at point at time

Here, are real numbers (in meters) denoting actual position on the sensor plane.

: Intensity measured by the sensor element at grid location

Here, , are integers, indexing pixel location.

Note the convention for “indexing”, while for because it is a function.

: is a sensitivity function for a pixel, assuming (0,0) is the center of the pixel.

is ideally 1 inside pixel (for example, within [-W/2,-H/2] to [W/2,H/2)), and 0 outside. But may haveattenuation at boundaries.

is the location (in meters) of the center of the sensor element

is the sensitivity function for the pixel at , “centered” at .

E(x, y, t) (x, y) t

x, y

I[ , ]nx ny ,nx ny

nx ny

[] () E

p(x, y)

p(⋅, ⋅)

,xnx yny

p(x − , y − )xnx yny [ , ]nx ny ( , )xnx xny

23

SENSORSENSOR: Light energy, per unit area per unit time, arriving at point at time

Here, are real numbers (in meters) denoting actual position on the sensor plane.

: Intensity measured by the sensor element at grid location

Here, , are integers, indexing pixel location.

Note the convention for “indexing”, while for because it is a function.

: is a sensitivity function for a pixel, assuming (0,0) is the center of the pixel.

is ideally 1 inside pixel (for example, within [-W/2,-H/2] to [W/2,H/2)), and 0 outside. But may haveattenuation at boundaries.

is the location (in meters) of the center of the sensor element

is the sensitivity function for the pixel at , “centered” at .

Defining as the “quantum efficiency” of the sensor: Ratio of Light Energy to Charge/Voltage

Rate at which charge/voltage increases in sensor element at time .

An image capture involves “exposing” the image for an interval (seconds).

So the total intensity is going to involve integrating the charge/voltage rate over that interval.

E(x, y, t) (x, y) t

x, y

I[ , ]nx ny ,nx ny

nx ny

[] () E

p(x, y)

p(⋅, ⋅)

,xnx yny

p(x − , y − )xnx yny [ , ]nx ny ( , )xnx xny

q

∫ E(x, y, t) p(x − , y − ) q dx dyxnx yny,nx ny t

T

24

SENSORSENSOR

are integers indexing pixels in image array.

is spatial location

is recorded pixel intensity.

is light “power” per unit area incident at location on the sensor plane at time

is the “center” spatial location of the pixel / sensor element at .

is spatial sensitivity of the sensor (might be lower near boundaries, etc.)

is quantum efficiency of the sensor (photons/energy to charge/voltage)

is the duration of the exposure interval.

CCD/CMOS sensors measure total energy or “count photons” that arrived during exposure.

I[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtnx ny ∫T

t=0

xnx yny

,nx ny

(x, y)

I[ , ]nx ny

E(x, y, t) (x, y) t

( , )xnx yny [ , ]nx ny

p(x, y)

q

T

25

SENSORSENSOR

I[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtnx ny ∫T

t=0

xnx yny

is the ideal unquantized pixel intensityI[ , ]nx ny

26

SENSORSENSOR

[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T

t=0

xnx yny

I ← I 0

27

SENSORSENSOR

SHOT NOISESHOT NOISE

Caused by uncertainty in photon arrival

Actual number of photons is a discrete random variable with Poisson distribution

is the “expected” number of photons. In our case,


t=0

xnx yny

I ← I 0

K

P(K = k) = λke−λ

k!

λ ∝ I 0

28

SENSORSENSOR

SHOT NOISESHOT NOISE

Property of Poisson distribution: Mean and Variance both equalO�en, shot noise is modeled with additive Gaussian noise with “signal dependent” variance:

where (Gaussian random noise with mean 0, variance 1).


t=0

xnx yny

I ← I 0

I ← + I 0 I 0‾‾√ ϵ1

ϵ ∼ (0, 1)

I ∼ ( , )I 0 I 0

29

SENSORSENSOR

AMPLIFICATION & ADDITIVE NOISEAMPLIFICATION & ADDITIVE NOISE

Signal amplified by gain before digitization. Based on ISO (higher for higher ISO).

Some signal-independent Gaussian noise added before and a�er amplification.

where and are parameters (lower for high quality sensors),and are noise variables, all independent.


t=0

xnx yny

I ← + I 0 I 0‾‾√ ϵ1

g g

I ← g × ( + + ) +I 0 I 0‾‾√ ϵ1 σ2aϵ2a σ2bϵ2b

σ2a σ2b

, ,ϵ1 ϵ2a ϵ2b (0, 1)

I ← + +gI 0

⏟Amplified Signal

g I 0‾‾√ ϵ1

Amplified Shot Noise

( + )g2σ2

2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

Amplified and un-amplified additive noise

30

SENSORSENSOR

DIGITIZATIONDIGITIZATION

Final step is rounding and clipping (by an analog to digital converter)


t=0

xnx yny

I ← g + g + I 0 I 0‾‾√ ϵ1 ( + )g2σ2

2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

I ← Round(g + g + )I 0 I 0‾‾√ ϵ1 ( + )g2σ2

2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

I = min( , Round(g + g + ))Imax I 0 I 0‾‾√ ϵ1 ( + )g2σ2

2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

31

SENSORSENSOR

ignoring sensor saturation, dark current, …

WHY STUDY THIS ?WHY STUDY THIS ?

To understand the degradation process of noise (if we want to denoise / recover from ).

To prevent degradation during capture, because we control exposure time and ISO / gain .

To understand the different trade-offs for loss of information from noise, rounding, and clipping.


t=0

xnx yny


2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

I 0 I

T g

32

SENSORSENSOR

ROUNDING VS CLIPPINGROUNDING VS CLIPPING

Ignoring noise, what is the optimal for a given ?

Keep low so that most values of are below .

But if is too low, a lot of the variation will get rounded to the same value.


t=0

xnx yny


2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

g [ , ]I 0 nx ny

g g [ , ]I 0 nx ny Imax

g

33

SENSORSENSOR

34

SENSORSENSOR

35

SENSORSENSOR

36

SENSORSENSOR

37

SENSORSENSOR

Note that here, our ‘ideal’ intensity is , everything else is noise.

LIGHT VS AMPLIFICATIONLIGHT VS AMPLIFICATION

Say we have chosen the optimal target values for the product . Is it better:

To have a higher and lower magnitude

To have a lower and higher magnitude

Depends, based on


t=0

xnx yny


2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

gI 0

gI 0

g I 0

g I 0

,σ2a σ2b

38

SENSORSENSOR

Note that here, our ‘ideal’ intensity is , everything else is noise.

LIGHT VS AMPLIFICATIONLIGHT VS AMPLIFICATION

Say we have chosen the optimal target values for the product . Is it better:

To have a higher and lower magnitude

To have a lower and higher magnitude

Depends, based on

Additional Reading (if interested):S. Hasinoff, F. Durand, W.T. Freeman, “Noise-Optimal Capture for High Dynamic Range Photography,” CVPR 2010.


t=0

xnx yny


2a σ2

2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2

gI 0

gI 0

g I 0

g I0

,σ2a σ2b

39

SENSORSENSOR

So how do we increase ?

Better sensors (higher )

Larger sensor elements: over a larger area.

But we’ve gone the other way: cameras stuff more ‘megapixels’ in smaller form factors.


t=0

xnx yny

I 0

q

p(⋅, ⋅) > 0

40

SENSORSENSOR

Increase exposure time ?

If scene is static and camera is stationary:

doesn’t change with

If scene is moving …


t=0

xnx yny

T

E(x, y, t) t ⇒ ∝ TI 0

41

SENSORSENSOR

Increase itself. How ?

Take pictures outdoors, or under brighter lights.Don’t use a pinhole camera !


t=0

xnx yny

E(x, y, t)

42

LENSESLENSES

43

LENSESLENSES

44

LENSESLENSES

45

LENSESLENSES

46

LENSESLENSES

47

LENSESLENSES

48

LENSESLENSES

49

LENSESLENSES

50

LENSESLENSES

51

LENSESLENSES

52

LENSESLENSES

53

LENSESLENSES

54

LENSESLENSES

55

TRADEOFFSTRADEOFFSPhotographers think about these tradeoffs every time they take a shot

Dynamic range and what part of the image should be well exposed (rounding and clipping)

Choosing between:ISO i.e. Gain & noiseExposure Time & motion blurF-stop i.e. aperture size & defocus blur

56

COLORCOLOR

We le� out at an important term in this equation: wavelength


t=0

xnx yny

57

COLORCOLOR

Light carries different amounts of power in different wavelengths

now refers to power per unit area per unit wavelength

In wavlength , incident at at time

Both spectral and spatial density function

: Quantum efficiency also a function of wavelength

CMOS/CCD sensors are sensitive (have high ) across most of the visible spectrum

Actually extend to longer than visible wavelengths (near infra red)Why cameras have NIR filter, to prevent NIR radiation from being ‘superimposed’ on the image

Q: But this measures ‘total’ power in all wavelengths. How do we measure color ?

Ans: By putting a color filter in front of each sensor element.

[ , ] = [∫ E(λ, x, y, t) p(x − , y − ) q(λ) dλ dx dy]dtI 0 nx ny ∫T

t=0

xnx yny

E(λ, x, y, t)

λ (x, y) t

q(λ)

q

58

COLORCOLOR

is the transmittance of a color filter for color channel

E.g, will transmit power in (be high for) wavelengths in the red part of the visible spectrum and attenuate power in (be low for) other wavelengths.Sometimes also called “color matching functions”

[ , , c] = [∫ E(λ, x, y, t) p(x − , y − ) (λ) q(λ) dλ dx dy]dtI 0 nx ny ∫T

t=0

xnx yny Πc

for c ∈ {R,G,B}

Πc c

ΠR

(actually XYZ not RGB)

59

COLORCOLOR

But we can only put one filter in front of each sensor element / pixel location.So color cameras “multiplex” color measurements: they measure a different color channel at each location.Usually in an alternating pattern called the Bayer pattern:

Note: a disadvantage is that color filters block light, so measured values are lower.

That’s why black and white / grayscale cameras are “faster” than color cameras.

[ , , c] = [∫ E(λ, x, y, t) p(x − , y − ) (λ) q(λ) dλ dx dy]dtI 0 nx ny ∫T

t=0

xnx yny Πc

for c ∈ {R,G,B}

I 0

60

FINAL STEPSFINAL STEPSFinal steps in camera processing pipelines (except for some DSLR cameras shooting in RAW):

Filter Colors to Standard RGB:

Cameras o�en use their own color filters .

Apply a linear transformation to map those measurements to standard RGB.White-balance: scale color channels to remove color cast from a non-neutral illuminant.Tone-mapping:

The simplest form is “gamma correction” (approximately raising each intensity to the power )

Done based on standard developed for what old display devices expectedFits the full set of measurable colors into the gamut that can be displayed / printedModern cameras o�en do more advanced processing (to make colors look vibrant)

Compression

And that’s how you get your PNG / JPEG images !

Optional Additional Reading: Szeliski Sec 2.3

Πc

(1/2.2)

61

Documents

ADMINISTRIVIA SENSORayan/courses/cse559a/PDFs/lec... · 2019. 8. 29. · Install Anaconda, LaTeX, Git Submit your Public Key Submit Problem Set 0 2 ADMINISTRIVIA Oﬀice Hours Arushee