Motion perception in displays SFU 2011.pptljilja/cnl/guests/Motion_perception_in_displays_SFU_2011.pdf · CRT spatially nonlinear MTF hard to assess, use ... Current Challenges for

Motion Perception in Displays

Motion Perception in Displays

Scott Daly

Dolby Laboratories

Scott Daly

Dolby Laboratories

Sim

on

Fra

ser

Un

iver

sity

Oct

. 2

011

��LCTV Basics

LCTV Basics

�Transm

ission m

odulation, Spatial , Color, etc.

��Basics of Spatiotemporal vision

Basics of Spatiotemporal vision

�Motion

�Eye m

ovements

�Eccentricity

��LCD Temporal Issues

LCD Temporal Issues

�Overdrive

�

Outline

�Dynamic Gamma

�Display Temporal Rendering Function

�Analysis of Temporal LCD approaches

��Perceptual Appearance: Motion Sharpness Effect

Perceptual Appearance: Motion Sharpness Effect

��Standardized Metrics

Standardized Metrics

��Conclusions/Summary

Conclusions/Summary

��What’s next:

What’s next:

�Other Temporal Artifacts

�What Does Motion Really Look Like?

LCTV Basics

LCTV Basics

Light Modulation via Liquid Crystals

�LCD is a “transm

issive” display

�Light is not created by the liquid crystals themselves

�A light source behind the panel shines through the display (CCFL, LED)

�Diffusion panel behind the LCD scatters and re-directs the light evenly

Glass Substrate

Transparent

Electrode

Alignment Layer

Backlight direction

Polarizing Filter

with Retardation Film

�Driving the Display

�2 polarizing transparent panels (One Vertical , One Horizontal)

�Liquid crystal solution sandwiched in betw

een

�Liquid crystals are rod-shaped m

olecules

-Bend light in response to an electric current

-Act like a shutter –allow light to pass through or block (or attenuate)

Alignment Layer

Liquid Crystals

Color Filter

Glass Substrate

RG

B

RG

BR

GB

Polarizing Filter w/ret

Pixels to resolution

45" LCD “full HDTV”

6.22 million pixels

�Physical resolution vs.

#Pixel Dim

ensions

�Full HD (1920 x 1080 progressive)

achieved in 2003

�Full HD now shown up to 108” for LCTV

0.5135mm

0.5135mm

1,920(H)×

RGB

×1,080(V) Progressive�4k x 2k pixel resolution shown by several

manufacturers (65”; 24 m

illion pixels)

�Usually, pixel physical resolution for

LCTV is near 45 ppi

Salient Characteristics of LCD: MTF & PSF

Width

Horizontal gap

Length Vertical gap

�LCD MTF does not vary with gray level or

spatial neighbors

�Rigid pixel via fixed aperture + steady

Backlight

�MTF is sinc function based on subpixel

dim

ensions

�color crosstalk correction sometimes needed Vertical gap

�CRT spatially nonlinear � ��

MTF hard to assess, use

�Spatial superadditivity in H direction :

�Spatial sub-additivity if power supply not

powerful enough

Salient Characteristics of LCD: MTF details

Sinc MTF for MI-600 (G plane)

MI-600 MTF and CSF

Width

Horizontal gap

��Comparison to visual system “MTF” = CSF

Comparison to visual system “MTF” = CSF

•Sinc is only a gradual LPF within HVS CSF “window”

•Viewing distance = 2000 pix

•~2H for HDTV, ~4H for VGA

0.1

0.2

0.3

0.4

0.1

0.2

0.3

0.4

0

0.2

0.4

0.6

0.81 H SF (cy/pix)

V SF (cy/pix)

MTF

Sinc MTF for MI-600 (G plane)

0.1

0.2

0.3

0.4

0.5

0.1

0.2

0.3

0.4

0.5

0

0.2

0.4

0.6

0.81

H SF (cy/pix)

V SF (cy/pix)

MTF

Fig

ure

13A

: S

inc

MT

F f

or

MI-

600 F

igure

13B

: M

I-600 a

nd C

SF

Length Vertical gap

Current Challenges for LC TV

�High Dynamic Range at consumer cost

�Wide Color Gamut at consumer cost

�Ultra high resolution 4k x 2k and up

�Achieving perfect motion fidelity :

1. Speeding up response tim

e for pixels

•How fast a pixel can change color without blurring

•Currently <= 2-4 m

illiseconds cites, but not for all gray level transitions

2. Hold-response blur (problem with Plasm

a also)

3. Judder (frame rate issue, problem with CRTs and Plasm

a also)

�Human visual perception plays a role in perform

ance

Some Basics in Spatiotemporal Vision

Some Basics in Spatiotemporal Vision

Properties of the Visual System

Properties generally dissected along


these dim

ensions:

these dim

ensions:

•Luminance Level

•Spatial Frequency

•Local Spatial Content

•Temporal Frequency


•Motion

•Global Color

•Eccentricity

•Depth

Properties of the Visual System



these dim

ensions:

these dim

ensions:

•Spatial Frequency

• •Temporal Frequency


•Motion

• •

Engineering vs. Physiological Models of the Visual System

��Engineering Models of

Engineering Models of

visual behavior aim

visual behavior aim

for mathematical

for mathematical

descriptions of key

descriptions of key

functionality

functionality

��Psychophysics and

Psychophysics and

black

black--box m

odeling

box m

odeling

have gotten the m

ost

have gotten the m

ost

mileage for practical




applications

applications

��While physiological

While physiological

plausibility is helpful,

plausibility is helpful,

simplification is

simplification is

desired

desired

��No need to m

odel

No need to m

odel

down to the

down to the

neurotransm

itter

neurotransm

itter

��How is more

How is more

important than where

important than where

Spatial Frequency

��Spatial behavior constant with visual

Spatial behavior constant with visual

angle (degrees)

angle (degrees)

��Spatial frequencies specified in

Spatial frequencies specified in

cycles/degree (cpd, cy/deg)


��Spatial frequency behavior

Spatial frequency behavior

described with

described with CSF

CSF(contrast

(contrast

sensitivity function)

sensitivity function)

•Sim

ilar to OTF of optics, MTF of

•Sim


electrical systems, but it is

nonlinear and adaptive

•Measured with psychophysics

��One of the m

ost useful, and widely

One of the m


used properties of visual system


Campbell and Robson ’66


Spatial Frequency

��Spatial behavior constant with visual

Spatial behavior constant with visual

angle (degrees)

angle (degrees)

��Spatial frequencies specified in

Spatial frequencies specified in



��Spatial frequency behavior

Spatial frequency behavior

described with

described with CSF

CSF

(contrast sensitivity function)

(contrast sensitivity function)

•Sim


•Sim


electrical systems, but it is

nonlinear and adaptive

•Measured with psychophysics

��One of the m


One of the m






Spatial Frequency Sensitivity


>100cd/m

2

0.0001 cd/m

2


Log Contrast Sensitivity

Log Spatial Frequency

Log Contrast Sensitivity

Spatial Frequencyin Application

��The m

ax spatial frequency that can be displayed digitally is the Nyquist frequency

The m

ax spatial frequency that can be displayed digitally is the Nyquist frequency

��It is ½ the sampling frequency (e.g., 500 pixels can display at most 250 cycles/pixel)

It is ½ the sampling frequency (e.g., 500 pixels can display at most 250 cycles/pixel)

��Common m

ax frequency seen by humans (I.e, CSF) is 30 cy/deg for medium brightness

Common m

ax frequency seen by humans (I.e, CSF) is 30 cy/deg for medium brightness

•Highest m

ax ever seen is 60 cy/deg (very high brightness, Carlson @ RCA)

��Examples of visual Nyquist frequencies and viewing distances for common displays:

Examples of visual Nyquist frequencies and viewing distances for common displays:

�NTSC (425 lines) at 6H

(2550 pixels):

22 cy/deg


(2550 pixels):

22 cy/deg


(1275 pixels):

11

�XGA (1024x768) at 3H

(2304 pixels): 20

�SXGA (1280x1024) at 1H

(1024 pixels):

9

�1366 x 720 HDTV at 3H

(2160 pixels):

19

�Full HDTV (1920x1080) at 6H

(6480 pixels):

57


(3240 pixels):

28


(2160 pixels): 19

2D Spatial Frequency

��2D frequencies im

portant for im

ages

2D frequencies im

portant for im

ages

��2D CSF is

2D CSF is not

notrotationally symmetric (isotropic)

rotationally symmetric (isotropic)

��Lack of sensitivity near 45 degrees, called the oblique effect

Lack of sensitivity near 45 degrees, called the oblique effect

200

250

30

35

40

V spatial frequency (cy/deg)

2D Spatial CSF

10

20

30

10

20

30

0

50

100

150

200

H SF cpd

V SF cpd

S

510

15

20

25

30

35

40

5

10

15

20

25

H spatial frequency (cy/deg)

V spatial frequency (cy/deg)

Temporal Frequency

��CSF for temporal frequencies also

CSF for temporal frequencies also

has been m

easured and m

odeled

has been m

easured and m

odeled

��To right is shown temporal CSF for

To right is shown temporal CSF for

different light adaptation levels for

different light adaptation levels for

luminance

luminance

•Top curve is best for mid-bright

display applications

display applications

��Opponent Color

Opponent Colorsignals temporal

signals temporal

CSF also has about 1/2 the

CSF also has about 1/2 the

bandwidth and sensitivity of the

bandwidth and sensitivity of the

luminance

luminance

��DeLange 52, Kelly 60s

DeLange 52, Kelly 60s--70s, W

atson

70s, W

atson

80s

80s

Spatiotemporal Frequency

��Psychophysical data m

easurement of

Psychophysical data m

easurement of

spatio

spatio--temporal CSF is common

temporal CSF is common

•Robson 66

•Van Nes, Koenderinck, Bouman 67

•Kelly 79

•Kelly and Burbeck 80

0.51

Standing W

ave

•Kelly and Burbeck 80

��Test signal is product of spatial and

Test signal is product of spatial and

temporal frequency m

odulation

temporal frequency m

odulation

•Standing W

ave

•Counterphase flicker

0

200

400

0

2

4

6-1.5-1

-0.50

0.5 spatial position

time

Spatiotemporal CSF

��Spatiotemporal CSF (measured with counterphase flicker)

Spatiotemporal CSF (measured with counterphase flicker)

��Window of visibility

Window of visibility

��Data shows max visible temporal frequency (CFF) near 50 cy/sec

Data shows max visible temporal frequency (CFF) near 50 cy/sec

•CFF = Critical Fusion Frequency = m

ax temporal frequency that can be seen

1.52

2.53

log temporal frequency (cy/sec)

Spatiotemporal CSF

2.53

log Sensitivity

Spatiotemporal CSF

•D.H Kelly 79

•Koenderinck &

van Doorn 79

(bim

odal)

-1-0.5

00.5

11.5

2-1

-0.50

0.51

1.5

log spatial frequency(cy/deg)


-1

0

1

2-1

0

1

20

0.51

1.52

log SF (cy/deg)

log TF (cy/sec)

log Sensitivity

(bim

odal)

•Burbeck &

Kelly 80

(excitatory-

inhibitory

separable

version)

��Thus 60 fps usually causes no visible flicker (foveal)

Thus 60 fps usually causes no visible flicker (foveal)

��Movie film at 24 fps causes visible flicker, so projectors shutter each

Movie film at 24 fps causes visible flicker, so projectors shutter each

frame 2 or 3 tim

es to increase fundamental temporal frequency

frame 2 or 3 tim

es to increase fundamental temporal frequency

•Before the 1920s , movies were called “the flickers’

Brightness and Light Adaptation effects on T-CSF

��Higher brightness

Higher brightness ��

Increase in peak sensitivity of temporal CSF

Increase in peak sensitivity of temporal CSF

��Higher brightness

Higher brightness ��

Increase in bandwidth of temporal CSF

Increase in bandwidth of temporal CSF

��CFF = Critical Fusion Frequency (CSF bandwidth cut

CFF = Critical Fusion Frequency (CSF bandwidth cut--off)

off)

Fer

ry-P

ort

er L

aw

Fer

ry-P

ort

er L

aw -

>

CFF and Eccentricity

��CFF= critical fusion frequency.

CFF= critical fusion frequency.

•Defined as frequency when 100% m

odulation signal looks identical to flat-field

•Viewer does not see any flicker

��For fovea and typical display light levels, CFF around 55 Hz

For fovea and typical display light levels, CFF around 55 Hz

��For periphery at same light levels, it can increase to over 80Hz

For periphery at same light levels, it can increase to over 80Hz

Motion

Motion and Retinal Velocity

��For objects in real

For objects in real

world, Velocity

world, Velocity

more important

more important

than flicker

than flicker

��Standing waves can

Standing waves can

be de

be de--composed

composed

into traveling waves

into traveling waves

��Smooth tracking eye

Smooth tracking eye

movements can

movements can

0100

200

300

400

500

-3-2-10123Standing W

ave = Two Opposing Travelling W

aves

spatial position

amplitude

SF=6/512 cy/pix TF=1/8 cy/sec V=(1/8)/(6/512)= 11 pix/sec

t=0 ; 0.5++ ; 4 secs

--> <--

movements can

movements can

reduce image

reduce image

velocity on the

velocity on the

retina

retina

��Spatiovelocity CSF

Spatiovelocity CSF

by W

atanabe ‘68

by W

atanabe ‘68

��Retinal Velocities &

Retinal Velocities &

stabilization

stabilization

��Retinal Velocity

Retinal Velocity

CSFs by Kelly from

CSFs by Kelly from

Motion & Vision

Motion & Vision

series 79

series 79

-1-0.5

00.5

11.5

20

0.51

1.52

2.53

log spatial frequency (cy/deg)

log visual sensitivity

CSFs for Different Retinal Velocities

0.001

0.012

0.15 deg/sec

3

11

32

64 128

200

Sampled Motion and the W

indow of Visibility

Watson, Ahumada, Farrell 86



indow of Visibility



��Rectangular support shown is window of visibility (idealized separable version)

Rectangular support shown is window of visibility (idealized separable version)

•Max spatial = 50 cy/deg (depending on conditions, well studied)

•Max temporal = 30 cy/sec (depending on conditions and visual eccentricity, well studied)

��Undersampled m

otion

Undersampled m

otion

��Replications due to sampling = temporal aliases

Replications due to sampling = temporal aliases

��Note: this would look awful

Note: this would look awful


indow of Visibility

��Camera constrained window of visibility (not HVS)

Camera constrained window of visibility (not HVS)

��Aliasing vs. Blur tradeoffs at im

age capture via Temporal LPF prefilter

Aliasing vs. Blur tradeoffs at im

age capture via Temporal LPF prefilter

via exposure aperture length

via exposure aperture length

via illumination duration

via illumination duration

Andrew Davidhazy @ RIT

Andrew Davidhazy @ RIT


indow of Visibility



��Example of sm

oothly perceived m

otion

Example of sm

oothly perceived m

otion

��Sampling rate increases spreads out replications

Sampling rate increases spreads out replications

��Preventing aliases in window of visibility results in smooth true m

otion

Preventing aliases in window of visibility results in smooth true m

otion

��Sampling rate depends on object speed and spatial content

Sampling rate depends on object speed and spatial content

•(I.e., bandwidth)


indow of Visibility

��Watson, Ahumada, Farrell 86


��Now that we have smooth m

otion by keeping aliases out of the window of

Now that we have smooth m

otion by keeping aliases out of the window of

visibility……

visibility……

��We still need to worry about motion blur due to capture aperture

We still need to worry about motion blur due to capture aperture

•Thus the use of shorter capture tim

e than the frame duration

Relations betw

een Temporal, Spatial, and Motion

)0,

,(

),

,(

ty

tx

lt

yx

ly

xυ

υ−

−=

)(

),

()

,,

(t

yy

xx

yx

ty

xf

ff

ff

Lf

ff

L+

+=

υυ

δ

�Its 3D Fourier spectrum is given by

�Translational motion can be defined as

),

,(

ty

xf

ff

L

0=

++

ty

yx

xf

ff

υυ

is n

on z

ero o

nly

on t

he

pla

ne

def

ined

by

�The m

otion of an object causes temporal component in the

spatiotemporal spectrum.

�The temporal component is proportional to spatial frequency and velocity

Relations betw

een Temporal, Spatial, and Motion and MTF

)(

).(

)(

)(

),

(t

xt

xx

xt

xs

fT

fT

ff

fL

ff

L+

=υ

δ

Sp

atio

-tem

pora

l sp

ectr

um

is

low

pas

s fi

lter

ed b

y t

he

ST

CS

F ,

as w

ell

as d

isp

lay M

TF

(co

mb

ined

ST

syst

em M

TF

: T

)

When

eye

accu

rate

ly t

rack

s th

e m

oti

on, th

e re

tinal

im

age

is

pure

ly s

pat

ial

)(

)(

)(

)(

xx

xx

xs

fT

fT

fL

fL

υ=

)(

)(

)(

xx

xx

df

Tf

Tf

Tυ

=

Sp

atia

l tr

ansf

er f

unct

ion d

ue

to d

isp

lay s

pat

io-t

emp

ora

l M

TF

Sp

atia

l M

TF

Tem

pora

l M

TF

Advanced Issues in Spatiotemporal Vision

Advanced Issues in Spatiotemporal Vision

Properties of Visual System: Motion:retinal velocity

��Retinal Velocities

Retinal Velocities

��No Eye Movements Occur

No Eye Movements Occur

��Image velocity = retinal velocity

Image velocity = retinal velocity


Spatiovelocity CSF

(stabilized retina)

(stabilized retina)

-1-0.5

00.5

11.5

20

0.51

1.52

2.53




0.001

0.012

0.15 deg/sec

3

11

32

64 128

200


-1

0

1

2-1

0

1

2-10123

log SF (cy/deg)

log V (deg/sec)


Spatiovelocity CSF

-10

12

3-1

-0.50

0.51

1.52

2.53


log velocity (deg/sec)

Spatiovelocity CSF

Properties of Visual System: Motion:Eye Movements

��Eye m

ovement’s tracking changes the window of visibility

Eye m

ovement’s tracking changes the window of visibility

��B. Girod 93, Perfect (and m

andatory) object tracking

B. Girod 93, Perfect (and m

andatory) object tracking

Properties of Visual System: Motion:Eye Movements

Types of eye m

ovements:

Types of eye m

ovements:

��Saccadic Eye m

ovements (jumps)

Saccadic Eye m

ovements (jumps)

•Usually > 160-300 deg/sec

•With larger display, larger saccades will still fit on

screen, giving m

ore of a feeling of being in real

world

��Smooth Pursuit Eye Movements

Smooth Pursuit Eye Movements

(tracking )

(tracking )

•80 deg/sec for 90 degree field of view (+-45 deg)

•30 deg/sec for 30 deg field of view

050

100

150

0

20

40

60

80

100

120

140

160

Target Velocity (deg/sec)

Eye Tracking Velocity (deg/sec)

Smooth Pursuit Eye Tracking

o - Observer KC

•Some retinal slippage (slope= 0.9)

��Drift Eye m

ovements (very small )

Drift Eye m

ovements (very small )

•responsible for the prevention of im

age fading due

to low S of spatial & temporal CSFs

•No expected consequences of large screen on these

•Approx 0.10 to 0.15 deg/sec

•Other sm

all eye m

ovements: Tremor,

Microsaccades

��Data from Meyer 85 : Smooth tracking data

Data from Meyer 85 : Smooth tracking data

•Red line is model we use for eye m

ovements

•-smooth pursuit + baseline drift as minim

um


10-1

100

101

102

10-1

100

101

102


Eye Tracking Velocity (deg/sec)

Smooth Pursuit Eye Tracking

o - Observer KC

Properties of Visual System: Motion:

��Problem: spatial CSFs vs. velocity are narrower than usual CSF

Problem: spatial CSFs vs. velocity are narrower than usual CSF

��Static CSF viewing does not result in stabilized image on retina

Static CSF viewing does not result in stabilized image on retina

��EEye drifts and sm

all pursuit m

ovements cause retinal velocities during CSF

ye drifts and sm

all pursuit m

ovements cause retinal velocities during CSF

examination

examination

103

CSF M

odels: Drift Range vs. Natural Static

103

CSF M

odels: Drift Range vs. Natural Static

10-1

100

101

102

100

101

102

Spatial Frequency (cy/deg)

Sensitivity

<--Modified Kelly Model w/ Drift Range

(0.1-2.0 deg/s)

<- Static CSF

10-1

100

101

102

100

101

102


Sensitivity

<--Envelope of Drift Range

(0.1-2.0 deg/s)

<- Static CSF

��This gives us more confidence in the m

odel for spatial attributes

This gives us more confidence in the m

odel for spatial attributes

Eye Movement Model Spatiovelocity CSF

��Use best case eye

Use best case eye

movements for

movements for

detection of moving

detection of moving

targets

targets

��Eye Movement

Eye Movement

Model

Model

��Shifts image

Shifts image

-1

0

1

2-1

0

1

2-10123

log SF (cy/deg)

log V (deg/sec)


Spatiovelocity CSF

-10

12

3-1

-0.50

0.51

1.52

2.53



Spatiovelocity CSF

��Shifts image

Shifts image

velocities to retinal

velocities to retinal

velocities that are

velocities that are

low

low

��Daly ’98 (SPIE HVEI)

Daly ’98 (SPIE HVEI)

-1

0

1

2-2

0

2-10123

log SF (cy/deg)

log V (deg/sec)

log visual sensitivityEye Movement Model Spatiovelocity CSF

-10

12

3-1

-0.50

0.51

1.52

2.53






Spatiovelocity CSF

using Eye m

ovement

using Eye m

ovement

model

model

��ω ωωωω ωωω=

= vvρ ρρρρ ρρρ

(cy/sec) = (deg/sec)(cy/deg)

(cy/sec) = (deg/sec)(cy/deg)

ωω=

= temporal frequency

temporal frequency

v =

v = velocity

velocity

ρρ=

= spatial frequency

spatial frequency

-1

0

1

2-2

0

2-10123

log SF (cy/deg)

log V (deg/sec)

log visual sensitivityEye Movement Model Spatiovelocity CSF

-10

12

3-1

-0.50

0.51

1.52

2.53




ρρ=

= spatial frequency

spatial frequency

��Rotation back into

Rotation back into

spatiotemporal CSF

spatiotemporal CSF

including effects of

including effects of

eye m

ovements

eye m

ovements

•Can be used to

assess smoothness

of motion

-1

0

1

2-1

0

1

20123

log SF (cy/deg)

log TF (cy/sec)


Eye Movement Model Spatiotemporal CSF

-1-0.5

00.5

11.5

2-1

-0.50

0.51

1.52

2.53

log spatial frequency(cy/deg)

log temporal frequency (cy/sec)Eye Movement Model Spatiotemporal CSF

Spatiotemporal visibility demos

Application of SV EMM m

odel : Analysis of Digital Video Form

ats

��Analysis of interlace, flicker and resolution issues

Analysis of interlace, flicker and resolution issues

��Use spatiotemporal CSF to analyze progressive and interlace parameters

Use spatiotemporal CSF to analyze progressive and interlace parameters

•720 lines progressive @

60 fps, 1080 lines progressive @

30 fps, 1080 lines

interlace @

60 fps

•all have sim

ilar uncompressed data rates

��Viewing distance = 3H

Viewing distance = 3H

2.5

Video Nyquist Boundaries + ST CSF

2.5Video Nyquist Boundaries + EMM ST CSF

-1-0.5

00.5

11.5

2-0.50

0.51

1.52

spatial frequency (cy/deg)

temporal frequency (cy/sec)

720P @

3H

1080I @ 3H

1080P @

3H

-1-0.5

00.5

11.5

2-0.50

0.51

1.52



720P @

3H

1080I @ 3H

1080P @

3H

Different Viewing Distances

2.5

Video Form

at Nyquist Boundaries (6H)

2.5

Video Form

at Nyquist Boundaries (9H)

��Analysis of interlace, flicker and resolution issues

Analysis of interlace, flicker and resolution issues

��Use spatiotemporal CSF to analyze progressive and interlace parameters

Use spatiotemporal CSF to analyze progressive and interlace parameters

•720 lines progressive @

60 fps, 1080 lines progressive @

30 fps, 1080 lines

interlace @

60 fps

•SD signal of 480P also considered (some DVDs)

��Increase Viewing distance to 6H and 9H

Increase Viewing distance to 6H and 9H --> Interlace advantage lost

> Interlace advantage lost

-1-0.5

00.5

11.5

2-0.50

0.51

1.52



720P @

6H

1080I @ 6H

480P @

6H -->

-1-0.5

00.5

11.5

2-0.50

0.51

1.52



720P @

9H

1080I @ 9H -->

480P @

9H ->

Speculative Video Form

at

0.51

1.52

2.5


360Hz 1080I @ 3H

Speculative Video Form

at

360 Hz 1080I @ 3H

360 Hz 1080I @ 3H

-1-0.5

00.5

11.5

2-0.50



��Auxiliary issues:

Auxiliary issues:

•Interlace is more difficult to compress

•2H becoming m

ore common w

ith large displays, so 1080 not enough

•Cost

•Trumbull’s Showscan (explored up to 100 Hz): some considered too realistic

and not cinematic

Closer Examination of Spatiovelocity CSF via

Closer Examination of Spatiovelocity CSF via

Eye Tracking

Eye Tracking

Verification of Eye Movement Model & SV CSF

��Laird, Pelz, Rosen, Montag and Daly (2006)

Laird, Pelz, Rosen, Montag and Daly (2006)

��Spatiovelocity Model based on Kelly’s experiments

Spatiovelocity Model based on Kelly’s experiments

•Using retinal stabilization to control velocities on the

retina

•No directed eye m

ovements

��However, in real im

age viewing applications,

However, in real im

age viewing applications,

•eyes will actually be in m

otion,

•And generally be directed as well

•And generally be directed as well

��The Spatiovelocity m

odel may not be valid when the

The Spatiovelocity m

odel may not be valid when the

eyes are actually in m

otion…

eyes are actually in m

otion…

•… if auxiliary signals from eye control circuitry to V5,

the m

otion area, affect …..??

��Build/optimize 2D spatio

Build/optimize 2D spatio--velocity CSF m

odel

velocity CSF m

odel

•Further refine Daly (Kelly+EMM) model

•Incorporate calculated retinal velocities

•Study effects of eye m

ovements on retinal velocity

sensitivity

��Equipment & Methodology:

Equipment & Methodology:

•Sony Trinitron MultiScan G420

CRT

•ASL Series 504 Remote

Eyetracker

•2IFC

Mean Lum. of screen

Mean Lum. of screen

60 cd/m

60 cd/m

22

Dist Obs. from Screen

Dist Obs. from Screen

84 cm

84 cm

Horiz. Deg span of

Horiz. Deg span of

23.95

23.95oo

Experimental Setup

Horiz. Deg span of

Horiz. Deg span of

screen

screen

23.95

23.95oo

Size of stim

ulus

Size of stim

ulus

2.46

2.46oox 2.46

x 2.46oo

��Stimuli:

Stimuli:

•Gabor

(contrast, frequency, velocity)

•Disembodied Edge

(contrast, velocity)

Eye tracking velocity calculations

Eye tracking velocity calculations

Conversion from position to degrees:

Conversion from degrees to velocity:

Tested spatiotemporal frequencies

Tested spatiotemporal frequencies

ων=

Spat Freq

Spat Freq

(Cyc/Deg)

(Cyc/Deg)

Temporal Freq (Hz)

Temporal Freq (Hz)

10

10

20

20

30

30

442.5

2.5

5.0

5.0

7.5

7.5

881.25

1.25

2.5

2.5

3.75

3.75

16

16

0.625

0.625

1.25

1.25

ρν=

deg

ree

cycl

esse

ccy

cles

sec

deg

=

Retinal velocities with and without directed eye m

ovements

��Experiment tests 4 cases:

Experiment tests 4 cases:

•mixtures of Gabor velocity, fixation points, and envelope:

Time

Trad

CSF

Time

ST

CSF

Gabor does

Gabor does

not move

not move

Sine m

oves, but

Sine m

oves, but

envelope & fixation

envelope & fixation

do not:

do not:

Retinal velocity

Retinal velocity

Time

Time

Time

SV CSF

No eye

tracking

Time

SV CSF

eye tracking

Fixation m

oves

Fixation m

oves

with Gabor:

with Gabor:

Retinal velocity

Retinal velocity

depends on eye

depends on eye

tracking

tracking

Gabor moves,

Gabor moves,

but fixation

but fixation

does not:

does not:

Retinal velocity if

Retinal velocity if

observer can ignore

observer can ignore

envelope?

envelope?

��Eye fixation is good (able to ignore m

oving object, if requested)

Eye fixation is good (able to ignore m

oving object, if requested)

��Moving sines (fixed envelope) = m

oving gabor (m

oving envelope)

Moving sines (fixed envelope) = m

oving gabor (m

oving envelope)

Retinal velocities without directed eye m

ovements

��Eye tracking is good, results similar to static

Eye tracking is good, results similar to static

��No signals related to eye m

otion affect neural processing (no intercedent)

No signals related to eye m

otion affect neural processing (no intercedent)

Retinal ‘stasis’ with and without directed eye m

ovements

��Data shifted horizontal for separability, since they superimpose closely

Data shifted horizontal for separability, since they superimpose closely

��Eye tracking removes the decrease in Sensitivity with increasing

Eye tracking removes the decrease in Sensitivity with increasing

temporal frequency , for all tested spatial frequencies

temporal frequency , for all tested spatial frequencies

��Maybe m

otion sharpening at 4 cpd?

Maybe m

otion sharpening at 4 cpd?

Retinal velocities with and without directed eye m

ovements

�The red dots correspond

to the points in the table

Sensitivity results on Spatiovelocity CSF m

odel

10Hz

20Hz

30Hz

4 cpd

2.5 deg/sec

5 deg/sec

7.5 deg/sec

8 cpd

1.25 deg/sec

2.5 deg/sec

3.75 deg/sec

16 cpd

0.625 deg/sec

1.25 deg/sec

�The velocities result from the

particular spatial and temporal

frequency combination.

Fine tuning parameters of SV m

odel

()

()

−

⋅⋅

⋅⋅

⋅=

max

12

12

10

4ex

p2

,ρπρ

πρρ

cc

vc

cc

kv

CSF

RR

3

2

3lo

g

⋅

+=

Rv

cs

sk

Wh

ere:

�S

V C

SF

in r

etin

al v

eloci

ties

, v

r, a

nd s

pat

ial

freq

uen

cy ρ

22

13

log

⋅+

=R

vc

ss

k

()

22

1m

ax+

=R

vc

pρ

�K

elly

model

modif

ied t

o f

it d

ata

•(K

elly

mo

del

on

ly a

t lo

w L

A l

evel

, an

d n

ois

ier

dis

pla

ys

of

the

pas

t)

�N

on-l

inea

r le

ast

squar

es r

outi

ne:

•

Sen

siti

vit

y v

alu

es f

rom

mo

del

fit

to

ex

per

imen

tal

resu

lts

Test of model on combined frequencies

Test of model on combined frequencies

��Experiment 5

Experiment 5

•Moving edge results

•Sensitivity to blurring of edge

•As a function of edge contrast

Test of the SV m

odel

Test of the SV m

odel

��Revised SV CSF m

odel based on new parameters (inset)

Revised SV CSF m

odel based on new parameters (inset)

��Prediction results of moving edges via m

odel :

Prediction results of moving edges via m

odel :

•Based on W

atson

& Ahumada JOV

2005

•Use 2D integral of

CSF x signal

spectrum to

spectrum to

model sensitivity

•Perfect eye

tracking assumed

•Channels not

needed since no

masking ??

•OK, but could be

better

(facilitation?)

�S

ensi

tiv

ity

det

erm

ined

by

ret

inal

vel

oci

ty

•N

ot

affe

cted

by e

ye

movem

ents

�S

ensi

tiv

ity

sim

ilar

fo

r 2

ty

pes

of

mo

tio

n

•M

ovin

g s

inuso

ids

wit

hin

Gab

or

Verification of Eye Movement Model & SV CSF -Summary

•G

abor

movin

g a

cross

fie

ld o

f vie

w

�O

pti

miz

ed 2

D s

pat

io-v

elo

city

CS

F m

od

el

•M

ore

app

lica

ble

to T

V i

mag

ery

•U

se o

f re

tinal

vel

oci

ty a

nd u

nst

abil

ized

sti

muli

LCD Temporal Basics

LCD Temporal Basics

�W

hy

do

es L

CD

mo

tio

n b

lur

hap

pen

?

�L

CD

Tem

po

ral

MT

F c

om

po

nen

ts�

Tem

po

ral-

resp

on

se b

lur

&

�H

old

-ty

pe

blu

r (

tem

po

ral

ren

der

ing

fu

nct

ion

)�

Ho

ld-t

yp

e b

lur

(te

mp

ora

l re

nd

erin

g f

un

ctio

n)

�L

CD

mo

tio

n b

lur

mo

del

ing

�L

CD

mo

tio

n b

lur

anal

ysi

s�

Slo

w-r

esp

on

se b

lur

vs.

ho

ld-t

yp

e b

lur

�A

nal

ysi

s o

f P

rop

ose

d s

olu

tio

ns

Slow-response blur : LCD Temporal Characteristics

�Inputvs.

Outputtemporalresponsesshown

�Overallspeedandasymmetryare

important

Level 0

Level 0

010

20

30

40

50

60

7080

90

100

120

140

160

�Slowerresponseshavemore

temporalLPF

andleadto

motionblur

�Asymmetric

responsesleadto

HSFflicker

�Overdrive

Level 0

Level 2

Level 4

Level 6

Level 8

Level 0

Level 2

Level 4

Level 6

Level 8

Output

Input

Out 0

Out 2

Out 4

Out 6

Out 8

In 0In 1

In 2In 3

In 4 In 5

In 6In 7

In 8

0

20

40

60

80

Response

Time in ms

Output

Input

Level 0

Level 2

Level 4

Level 6

Level 8

Level 0

Level 2

Level 4

Level 6

Level 8

0102030405060

Response Time

(ms)

Output

Input

Spatial consequences of Temporal LPF Spectrum after

temporal lowpass

f t

f x

f t

f x

Spectrum at the

retina with eye

Spectrum of

motion sequence

Temporal

low-pass

of the

f t

f x

020

40

60

80

100

120

0

0.2

0.4

0.6

0.81

Temporal Frequency

Temporal MTF

f t

f x

Original image

spectrum

Klo

mp

enh

ou

wer

20

04

temporal lowpass

of display

retina with eye

tracking

motion sequence

of the

display

spectrum

�T

he

moti

on o

f an

ob

ject

cau

ses

tem

pora

l co

mp

onen

t in

the

spat

ial/

tem

pora

l sp

ectr

um

.

�T

his

sp

ectr

um

is

low

-pas

s fi

lter

ed b

y t

he

dis

pla

y s

pat

ial/

tem

pora

l tr

ansf

er f

unct

ion.

�T

he

eye

trac

kin

g c

ause

s th

e re

tina

imag

e to

hav

e p

ure

sp

atia

l co

mp

onen

t of

spec

trum

wit

hout

any

tem

pora

l co

mp

onen

t.

�B

ut

the

tem

pora

l lo

w-p

ass

filt

erin

g i

n t

he

dis

pla

y r

educe

s th

e sp

atia

l b

andw

idth

of

the

reti

na

imag

e, w

hic

h c

ause

s th

e p

erce

pti

on o

f m

oti

on b

lur.

Overdrive

Overdrive

ImprovingLCD Temporal Characteristics with Overdrive

�Slowerresponsesleadto

motionblur

�OverdriveLUT–from

graylevelto

graylevel(intendedto

necessary

map)

Targ

et

Over

dri

ve

No O

ver

dri

ve

Sta

rt

t

Sta

rt

Targ

etOverdrive Value

�Okumura

01,Sekiya02

LCD Temporal Response and its Temporal MTF

0.5

0.6

0.7

0.8

0.91

LCD temporal response

32 m

s

8 m

s

2 m

s

OD

0.5

0.6

0.7

0.8

0.91

Temporal MTF

32 m

s

8 m

s

2 m

s

OD

�T

emp

ora

l over

dri

ve

can e

ffec

tive

imp

rove

the

tem

pora

l M

TF

�

thus

reduci

ng t

he

moti

on b

lur

�A

t p

eak o

f H

VS

tem

pora

l C

SF

(8H

z), over

dri

ve

can e

ven

exce

ed a

2m

s te

mp

ora

l re

sponse

05

10

15

20

25

30

35

40

0

0.1

0.2

0.3

0.4

t in m

s

LCD temporal response

0

10

20

30

40

50

60

0

0.1

0.2

0.3

0.4

Temporal Frequency (Hz)

Temporal MTF

Designing a temporal overdrive algorithm

Target value

Y1

Y2

LC

D

Pre

vio

us

Val

ue

Y2

Y3

Fra

me

bu

ffer

Tar

get

val

ues

dn

-3d

n-2

dn

-1d

n

LC

D m

od

el

Overdrive algorithm results

8

8

72

01020

30405060

8

104

010

2030405060

8 40

72

104

136

168

200

232

255

No Overdrive

With Overdrive

72

136

200

255

72

136

200

255

Output

Input

8

72

136

200

255

200

0

Output

Input

�Note that overdrive m

akes temporal responses more

symmetrical: this essentially eliminates the flickering

artifacts

Temporal responses w/ OD are generally in range of 3-5ms

Dynamic Gamma Method

for Overdrive Analysis

for Overdrive Analysis

Dynamic Gamma Approach

�(Static) Display Gamma:

0.4

0.6

0.81

1.2

1.4

Display Output

10-2

10-1

100

Display Output

050

100

150

200

250

300

0

0.2

0.4

Input Digital Counts

100

101

102

103

10-3

Input Digital Counts

γd

cy=

Feng et al 04 & 05

Definition of first order Dynamic Gamma

90

% p

oin

t

targ

et

Outp

ut

DG

val

ue

Zn

-1

Zn

-2

Zn

-2

Zn=

25

5

Z=

0

Dri

vin

g W

avef

orm

Res

po

nse

tim

e

t

Fra

me

per

iod

�T

he

LC

D i

np

ut/

ou

tpu

t re

lati

on

ship

ch

ang

es w

ith

tim

e w

hen

dis

pla

yin

g m

oti

on

�D

yn

amic

gam

ma

val

ue:

th

e o

utp

ut

val

ue

mea

sure

d a

t th

e en

d o

f t

he

firs

t fr

ame

�C

an u

se t

he

sam

e eq

uip

men

t as

res

po

nse

tim

e m

easu

rem

ent

�A

dv

anta

ges

ov

er u

se o

f re

spo

nse

tim

es

Fra

me

No

.

0

1

Zn=

0

150

200

250

300

Output Level

0

32

64

96

128

160

192

224

255

Representation: Table vs. Figure

0032

32

64

64

96

96

128

128

160

160

196

196

224

224

255

255

0000

22

22

39

39

61

61

82

82

111

111

140

140

155

155

220

220

……

…

25

5

Mea

sure

d v

alues

()

nn

nz

df

d,

1−=

dn

-1:

25

5

050

100

150

200

250

300

0

50

100

150

Driving Value

Output Level

25

25

55135

135

141

141

15

15

55175

175

186

186

203

203

214

214

236

236

255

255

Sta

rtin

g v

alu

e

Tar

get

val

ue

……

…

0

Mea

sure

d v

alues

Eac

h c

urv

e re

pre

sents

dif

fere

nt

pre

vio

us

val

ue

�F

irst

ord

er Dγ

mo

del

s th

e ed

ge

mo

tio

n

0

Derivation of overdrive lookup table

200

250

300

0

16

32

48

64

80

96

112

050

100

150

200

250

300

0

50

100

150

Target Level

Output Level

112

128

144

160

176

192

208

224

240

255

�To go from 32 (previous frame) to 64: needs OD value of 130

Application for comparing LCD systems

100

150

200

250

300

Output Level

0

32

64

96

128

160

192

224

255

100

150

200

250

300

Output Level

0

32

64

96

128

160

192

224

255

Slo

w L

CD

No

OD

Slo

w L

CD

wit

h O

D

Dynamic Gamma useful for comparing LCD systems

(overdrive + inherent temporal response)

Assessment of overdrive perform

ance with dynamic gamma:

050

100

150

200

250

300

0

50

Driving Value

050

100

150

200

250

300

0

50

Driving Value

050

100

150

200

250

300

-500

50

100

150

200

250

300

Target Level

Output Level

050

100

150

200

250

300

0

50

100

150

200

250

300

1st

ord

er �

Edge

2

nd

advan

tage �

real

vid

eo

Fas

t L

CD

wit

h O

DF

ast

LC

D w

ith

OD

(2

nd-o

rder

dyn

amic

γ)

Current Overdrive algorithm results

40.0

60.0

80.0

100.0

120.0

Response time ms

response time ms

40.0

60.0

80.0

100.0

120.0

Without Overdrive

With Overdrive

0**

64**

128**

192**

255**

0*

32*

64*

96*

128*

160*

192*

224*255*

0.0

20.0

Input

Response time ms

Output

input

Output

response time ms

0**

64**

128**

192**

255**

0*

32*

64*

96*

128*

160*

192*

224*255*

0.0

20.0

Current Overdrive algorithm results

Example of visual consequences:

Conventional driver

High Perform

ance Overdrive

Display Temporal Rendering Function

Display Temporal Rendering Function

Comparative Display Basics of Temporal Aperture

tR

eal

worl

d t

emp

ora

l w

avef

orm

tC

RT

tem

pora

l outp

ut

(im

puls

e-ty

pe)

LC

D t

emp

ora

l outp

ut

(hold

-ty

pe)

LCD Motion Blur

��LCD

LCD’’s slow response:

s slow response: slow

slow--resp

onse

blur

resp

onse

blur

•physical;

•can be captured by a fixed-position camera

tt

Idea

l L

CD

tem

pora

l

outp

ut

(0-r

esp

onse

)

Pra

ctic

al L

CD

tem

pora

l

outp

ut

(slo

w r

esp

onse

)

��LCD’s hold

LCD’s hold--type rendition + HVS’ sm

ooth pursuit & lowpass

type rendition + HVS’ sm

ooth pursuit & lowpass

filtering:

filtering: hold

hold--type blur

type blur

•perceptual;

•only happen when human eyes are tracking;

•can NOT be captured by a fixed-position camera

•Lindholm

96 , Parker 97, Kurita 98, Kurita 01

Role of eye tracking in LCD hold-type blur

��Eye integrates along

Eye integrates along

tracking path (10

tracking path (10--50ms, LA)

50ms, LA)

��For CRT display, integration

For CRT display, integration

of eye tracking path causes

of eye tracking path causes

no m

ixing of black and

no m

ixing of black and

white displayed elements

white displayed elements

along path

along path

��Result for CRT is sharp

Result for CRT is sharp

CRT: impulse

FRAME 0

FRAME 1

FRAME 2

PERCEIVED

t

��Result for CRT is sharp

Result for CRT is sharp

moving edge

moving edge

��For LCD display, eye track

For LCD display, eye track

path goes through regions of

path goes through regions of

white and black displayed

white and black displayed

elements, so that mixing of

elements, so that mixing of

signals occurs due to

signals occurs due to

temporal integration of the

temporal integration of the

eye

eye

��Result for LCD (hold) is a

Result for LCD (hold) is a

blurred m

oving edge

blurred m

oving edge

FRAME 0

FRAME 1

FRAME 2

SHARP EDGE

BLURRED EDGE

PERCEIVED

LCD: hold

EYE M

OVEMENT:

TRACKING EDGE

Eye tracking Demo

Eye tracking Demo

Image examples

t

Ret

ina

imag

e o

f a

mo

vin

g e

dg

e

on

ho

ld d

isp

lay w

ith

eye

trac

kin

g

�h

old

blu

r

Ret

ina

imag

e of

a m

ovin

g e

dge

on

imp

uls

e dis

pla

y w

ith e

ye

trac

kin

g

�N

om

oti

on b

lur

Quan

tita

tive

Sim

ula

tion

)(

sin

)(

Tf

cf

Tt

th

=

The

effe

ct o

f te

mp

ora

l hold

is

ver

y s

imil

ar t

o s

pat

ial

aper

ture

eff

ect

of

CC

D

senso

r. T

he

tem

pora

l M

TF

is

giv

en b

y a

sin

cfu

nct

ion.

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.91

Temporal MTF

32 m

s

8 m

s

2 m

s

OD

Motion Blur Due to Hold: Temporal MTF

Motion Blur Due to Hold: Temporal MTF

)(

sin

)(

Tf

cf

Tt

th

=

�F

or

a fa

ster

LC

D p

anel

, th

e te

mp

ora

l M

TF

is

lim

ited

by t

he

ho

ld e

ffec

t. T

her

e is

dim

inis

hin

g g

ain

in

fu

rth

er i

mp

rov

ing

LC

D t

emp

ora

l re

spo

nse

.

�F

or

a g

iven

tem

po

ral

sam

pli

ng

, th

e o

nly

way

to

red

uce

ho

ld b

lur

is t

o r

edu

ce

tem

po

ral

aper

ture

.

�P

lot

incl

ud

es L

CD

tem

po

ral

resp

on

se +

ho

ld a

per

ture

010

20

30

40

50

60

0

0.1

Temporal Frequency (Hz)

Analysis of Temporal LCD System Issues

Analysis of Temporal LCD System Issues

Pan et al 05

Pan et al 05

Sim

plified display-perception chain

(,

,)

(,

,)

Ix

yt

Ix

vt

yv

tt

=+

+

Inp

ut:

ass

um

ing

th

e d

yn

amic

dis

cret

e co

nte

nt

I d(x

,y,t

) i

s an

im

age

I c(x

,y,t

)

mo

vin

g a

t a

con

stan

t sp

eed

HV

S

S&

H

(ht(

t))

Sm

oo

th

pu

rsu

it

Lo

wp

ass

filt

er

LC

D/C

RT

I d(x

,y,t

)I s

(x,y

,t)

I m(x

,y,t

)I o

(x,y

,t)

(,

,)

(,

,)

dc

xy

Ix

yt

Ix

vt

yv

tt

=+

+

(,

,)

(,

,)*

()

sd

tI

xy

tI

xy

th

t=

(2)

Sm

oo

th p

urs

uit

eye

mo

vem

ent:

()

(,

,)

(,

,)*

(,

)*

()

om

xyt

Ix

yt

Ix

yt

xy

t=

ΛΛ

(,

,)

(,

,)

ms

xy

Ix

yt

Ix

vt

yv

tt

=−

−

(1)

Sa

mp

le-a

nd

-ho

ld :

(3)

Lo

wp

ass

fil

ter:

ht (t

) is

th

e te

mp

ora

l re

con

stru

ctio

n f

un

ctio

n o

f an

LC

D o

r C

RT

Λxy

(x,y

) &

Λt(

t) a

re s

pat

ial

& t

emp

ora

l im

pu

lse

resp

on

se f

un

ctio

ns

of

the

filt

er

com

pen

sate

s m

oti

on

to

mak

e th

e o

bje

ct s

till

on

ret

ina.

The general LCD m

otion blur model

'

(,

,)

('

,'

,'

")(

,)

(")

"(

')'

oc

xy

xy

tt

t

Ix

yt

Ix

vt

py

vt

qt

tt

pq

td

pd

qd

th

tdt

∞ =−∞

=−

−−

−−

−Λ

Λ∫∫∫∫

•T

he

spat

ial

and

tem

po

ral

low

pas

s im

pu

lse

fun

ctio

ns

(Λxy

(x,y

) an

d Λ

t(t)

) a

re

un

kn

ow

n

•A

ssu

min

g t

hat

HV

S h

as t

he

sam

e lo

wp

ass

imp

uls

e fu

nct

ion

s fo

r L

CD

an

d

CR

T, an

d u

sin

g i

mag

e p

erce

ived

on

CR

T a

s a

refe

ren

ce

•T

he

inp

ut-

ou

tput

rela

tio

nsh

ip o

f th

e ch

ain

:

CR

T, an

d u

sin

g i

mag

e p

erce

ived

on

CR

T a

s a

refe

ren

ce

'

(,

,)

(',

',')

(')

'oL

CD

CR

TL

CD

ox

yt

t

Ix

yt

Ix

vt

yv

tt

th

tdt

∞ =−∞

=−

−−

∫

The general LCD m

otion blur model

�T

he

LC

D t

emp

ora

l re

con

stru

ctio

n f

un

ctio

n h

t L

CD

(t)

affe

cts

spat

iall

y a

nd

tem

po

rall

y.

�T

he

reco

nst

ruct

ion f

un

ctio

n h

tLC

D(t

) ca

n b

e m

easu

red

dir

ectl

y o

r d

eriv

ed

fro

m t

he

tem

po

ral

wav

efo

rm.

'

(,

,)

(',

',')

(')

'oL

CD

CR

TL

CD

ox

yt

t

Ix

yt

Ix

vt

yv

tt

th

tdt

∞ =−∞

=−

−−

∫

fro

m t

he

tem

po

ral

wav

efo

rm.

�W

hen

htL

CD

(t)

is δ

-fu

nct

ion, th

en L

CD

an

d C

RT

hav

e th

e sa

me

resu

lt

(mo

tio

n b

lur

do

es n

ot

exis

t)

�G

ener

ally

, th

e m

od

el i

s n

ot

in t

he

form

of

con

vo

luti

on.

�T

he

mo

tio

n s

pee

d v

xan

d t

he

LC

D r

eco

nst

ruct

ion f

un

ctio

n j

oin

tly d

eter

min

e

mo

tio

n b

lur.

�F

aste

r th

e m

oti

on

is,

mo

re b

lurr

ed t

he

per

ceiv

ed i

mag

es a

re.

�W

ider

th

e re

con

stru

ctio

n f

un

ctio

n i

s, m

ore

blu

rred

th

e p

erce

ived

im

ages

are.

Th

e re

con

stru

ctio

n f

un

ctio

n i

s th

e k

ey

Blur width calculated by the m

odel

Blur width calculated by the m

odel

1. A

ssu

me

a v

irtu

al h

ori

zon

tall

y

mo

vin

g s

har

p e

dg

e p

erce

ived

on

CR

T

2. C

alcu

late

th

e p

erce

ived

ed

ge

0x

L+

D L

inte

nsi

ty

⊗ ⊗⊗⊗

3. C

alcu

late

th

e b

lur

wid

th o

f th

e

calc

ula

ted

per

ceiv

ed e

dg

e

()

()*

(/

)L

CD

CR

TL

CD

oo

tx

Ix

Ix

hx

v=

2. C

alcu

late

th

e p

erce

ived

ed

ge

usi

ng

th

e re

con

stru

ctio

n

fun

ctio

n o

f th

e L

CD

BW

0x

L+

D L

inte

nsi

ty

L+

10%

D

L+

90%

D

Poin

t 1

Poin

t 2

Th/2

1 0

t

Th

Traditional LCD (slow response blur + hold-type blur)

t

The

tem

pora

l w

avef

orm

(lin

ear

tran

siti

on b

etw

een

fram

es)

Th

1 0

t

2T

h

The

reco

nst

ruct

ion

funct

ion (

linea

r

tran

siti

on b

etw

een

fram

es)

t0

0.2T

0.4T

0.6T

0.8T

T1.2T

1.4T

1.6T

1.8T

2T

0

0.2

0.4

0.6

0.81

tT

h

2T

hT

he

tem

pora

l w

avef

orm

(sin

e tr

ansi

tion b

etw

een

fram

es)

The

reco

nst

ruct

ion

funct

ion (

linea

r

tran

siti

on b

etw

een

fram

es)

Blu

r w

idth

: 1.1

vT

x

-15

-10

-5

05

10

15

20

0

0.2

0.4

0.6

0.81

1.2

Position (pixel)

Intensity

Perceived Edge on CRT

Perceived Edge on LCD (the first-ordre hold model)

Perceived Edge on LCD (the sine hold model)

CR

TL

CD

(fi

rst-

ord

er)

LC

D

(sin

e)

intensity

t

(a)

Th=

T

1/T 0

t

(b)

The ideal LCD temporal response (hold-type blur only)

Blu

r w

idth

: 0.8

vT

-15

-10

-50

510

15

20

0

0.2

0.4

0.6

0.81

1.2

Position (pixel)

Intensity

x

intensity

Cu

rren

t L

CD

Idea

l L

CD

�H

old

-ty

pe

+sl

ow

res

po

nse

blu

r: 1

.1vT

�H

old

-ty

pe

blu

r: 0

.8vT

�so

slo

w r

esp

on

se b

lur:

1.1

vT-0

.8vT

=0

.3vT

Hold-type blur vs. slow-response blur

�so

slo

w r

esp

on

se b

lur:

1.1

vT-0

.8vT

=0

.3vT

�7

0%

vs.

30

%

�S

o, h

old

-ty

pe

blu

r is

th

e m

ajo

r fa

cto

r

1)Black Data Insertion (BDI)

•Hong 04, Kim

ura 05

2)Backlight Flashing and Scrolling (BF)

•Fisekovic 01, Sluyterm

an 05

•Adaptive backlight flashing (60 and 120 Hz), Feng 06

Four key proposed solutions

•Adaptive backlight flashing (60 and 120 Hz), Feng 06

3)Frame Rate Doubling (FRD)

•Sekiya 02, Kurita 05

4)Motion-Compensated Inversing Filtering (MCIF)

•Klompenhauer 01, 05

Reconstruction functions of the four proposals

BD

I tT

/2T

2T

1 0

tT

/2T

2T

1 0

t

t

FR

D

tt

t

T

1/T

0

t

2T

T-T

12T-T

1T

2

T

1 0

t

BF

MC

IF

tt

Frame Rate Conversion based on Motion Compensation

Frame Rate Conversion based on Motion Compensation

temporal

Input Im

age

(60 frame/sec)

Converted sequence

(120 frame/sec)

1/60 sec

Images created

by frame rate

conversion

1/120 sec

1/60 sec

1/120 sec

Input

60p

Pre-

processing

Motion vector

estim

ation

Interpolation

Up-converted

output 120p

Motion vector estimation

Frame interpolation

Input

60p

Pre-

processing

Motion vector

estim

ation

Interpolation

Up-converted

output 120p

Motion vector estimation

Frame interpolation

Detect motion vector

Estimate object's motion in-

between Frame #1 and #2

as direction and speed

From frame#1 and

#2, create new

picture that shift its

position by

1/120sec

Interpolated picture

1/60 sec

Ｆｒａ

ｍｅ＃

１Ｆ

ｒａｍ

ｅ＃

１Ｆ

ｒａｍ

ｅ＃

１Ｆ

ｒａｍ

ｅ＃

１Ｆ

ｒａｍ

ｅ＃

２Ｆ

ｒａｍ

ｅ＃

２Ｆ

ｒａｍ

ｅ＃

２Ｆ

ｒａｍ

ｅ＃

２

BDI

BDI

BF

BF

FRD

FRD

MCIF

MCIF

Requirement on LCD

Requirement on LCD

temporal response

temporal response

High

High

Medium

Medium

High

High

No

No

Requirement on

Requirement on

backlight temporal

backlight temporal

response

response

No

No

High

High

No

No

No

No

Other Requirement

Other Requirement

No

No

Sync betw

een

Sync betw

een

LCD and

LCD and

backlight

backlight

Accurate

Accurate

motion

motion

estim

ation

estim

ation

Motion

Motion

estim

ation

estim

ation

Comparison betw

een different approaches

backlight

backlight

estim

ation

estim

ation

The ghosting artifact


Likely

Likely

Likely

Likely

No

No

No

No

The luminance

The luminance

reduction artifact

reduction artifact

Yes

Yes

Yes

Yes

No

No

No

No

flickering artifact

flickering artifact

Yes

Yes

Yes

Yes

No

No

No

No

Reduction of motion

Reduction of motion

blur (smaller the

blur (smaller the

number is, the

number is, the

better)

better)

50%

50% (limited

(lim

ited

by LCD

by LCD

temporal

temporal

response)

response)

25% or less

25% or less

(lim

ited by

(lim

ited by

backlight

backlight

temporal

temporal

response)

response)

50%

50% (limited

(lim

ited

by LCD

by LCD

temporal

temporal

response)

response)

??

Perceptual Motion Sharpening

Perceptual Motion Sharpening

Motion Sharpening

Motion Sharpening

��Ramachandran ’74 (observations on blurred m

ovie frames)

Ramachandran ’74 (observations on blurred m

ovie frames)

•things tend to look blurred when they are m

oving fast-----but---

•blurred edges look sharper when they are m

oving than w

hen stationary

��Poor tracking

Poor tracking � ��

blurred retina image

blurred retina image

��Motion sharpening effect

Motion sharpening effect � ��

perceived m

otion is sharper than the still

perceived m


images, which suggests that the


-1-0.5

00.5

11.5

20

0.51

1.52

2.53




0.001

0.012

0.15 deg/sec

3

11

32

64 128

200

perceived m


perceived m




perception of sm

ooth pursuit is different

perception of sm

ooth pursuit is different

from still image.

from still image.

��Less understood , higher order effect

Less understood , higher order effect

•Sharpness constancy

•Deblurring

��If m

otion sharpening effect is involved,

If m

otion sharpening effect is involved,

previous analysis based on retinal im

age

previous analysis based on retinal im

age

blur is insufficient

blur is insufficient

Motion induced Blur and Sharpening

Motion induced Blur and Sharpening

��Westerinck 90

Westerinck 90

��Studied perceived sharpness of im

ages

Studied perceived sharpness of im

ages

�with varying degrees of blur

�As a function of translational motion speeds

Conclusion: Evaluation of Motion Blur Reduction

Conclusion: Evaluation of Motion Blur Reduction

��Motion blur characterization

Motion blur characterization

•Objective m

ethod: measured retina image using a

simulated tracking camera –assuming perfect tracking

•Subjective m

ethod: Compared the perceived blur with

blurred edge of a still image

��Motion blur perception

Motion blur perception (

))

(*

)(

)/

()

()

(/

σxb

vx

gaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

��Motion blur perception

Motion blur perception

•The subjective m

ethod agrees with the objective

derived m

otion blur �

Perception of motion blur is

similar to perception of still im

age blur

•Backlight flashing can significantly reduce the

perception of motion blur.

Other Key Studies

Other Key Studies

20

30

40

50

60

頻度（％） Frequency (%)

20

30

40

50

60


Distribution of Moving Object Velocity

Average；；；；16.5 deg/sec

No.1

～No.32

The region where blur caused

by camera can be clearly seen.

Analysis of ITU-R BT-1210-3 test material for HDTV sequence

0

10

20

0～

10

10～

20

20～

30

30～

40

40～

50

50～

60

視覚

速度

(deg/s)

Frequency (%)

Velo

cit

y i

n v

iew

ing

an

gle

(d

eg

/sec)

0

10

20

0～

10

10～

20

20～

30

30～

40

40～

50

50～

60

視覚

速度

(deg/s)

Frequency (%)

Velo

cit

y i

n v

iew

ing

an

gle

(d

eg

/sec)

[3H

]

In term

s of the fastest motion within every sequence,

70% of the sequence distributed below 20[deg/sec]

T.Fujine et.al.;”Real-Life In-Home Viewing Conditions for FPDs and Statistical C

haracteristics of Broadcast Video

Signal,” Digest AM-FPD’06

345

Quality Scale ��

50% aperture

double-rate

Acceptance

limit

Threshold of

perception

Observer Study of Picture Quality Improvement

Observer Study of Picture Quality Improvement

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

12

05

10

15

20

25

30

Velocity in Viewing Angle (deg/sec)

Quality Scale

100% aperture

(hold type)

50% aperture

T.Kurita;

”Moving Picture Quality Improvement for Hold-type AM-LCDs,”SID

’01, 35.1, pp.986-989 (2001)

()

)(

*)

()

/(

)(

)(

/σx

bv

xg

au

sre

ctv

xtr

ix

step

xb

lurE

dg

e∗

=��

Basic

Basic

Analysis of Methods to Overcome Hold

Analysis of Methods to Overcome Hold--Blur

Blur

Analysis of Methods to Overcome Hold

Analysis of Methods to Overcome Hold--Blur

Blur

BDI

BDI (black data insertion)

BF

BF (backlight flashing)

FRD

FRD (frame doubling)

(frame doubling)

Pan

, F

eng

, &

Dal

y:

"Qu

anti

tati

ve

An

alysi

s o

f L

CD

Mo

tio

n B

lur

and

Per

form

ance

of

Ex

isti

ng A

pp

roac

hes

“ I

CIP

20

05

ASV1.0

Response time : 15ms

Response time 12ms

ASV2.2

≪ ≪≪≪Hold type drive

≫ ≫≫≫

7ms + Pseudo impulse drive

ASV3.0

≪ ≪≪≪Pseudo impulse drive

≫ ≫≫≫

5ms double rate LCD

w/ MC double rate drive

≪ ≪≪≪Motion Compensated 120Hz≫ ≫≫≫

BDI

BDI

BF

BF

FRD

FRD

Requirement on LCD temporal response

Requirement on LCD temporal response

High

High

Median

Median

High

High

Requirement on backlight temporal

Requirement on backlight temporal

response

response

No

No

High

High

No

No

Other Requirement

Other Requirement

No

No

Sync betw

een LCD

Sync betw

een LCD

and backlight

and backlight

Accurate m

otion

Accurate m

otion

estim

ation

estim

ation



Likely

Likely

Likely

Likely

No

No

The luminance reduction artifact

The luminance reduction artifact

Yes

Yes

Yes

Yes

No

No

flickering artifact

flickering artifact

Yes

Yes

Yes

Yes

No

No

Reduction of motion blur (smaller the

Reduction of motion blur (smaller the

number is, the better)

number is, the better)

50% (limited by LCD

50% (limited by LCD

temporal response)

temporal response)

25% or less (limited

25% or less (limited

by backlight

by backlight

temporal response)

temporal response)

50% (limited by

50% (limited by

LCD temporal

LCD temporal

response)

response)



Motion Picture Response Tim

e (MPRT)

Motion Picture Response Tim

e (MPRT)

Steps to Measure MPRT

1.Move an edge cross screen. The edge is made of a transition from one

gray level to another level. Total of 30 transitions (6 levels in digital

counts or L* space)

2.Using the pursuit camera to m

easure the blur width

3.MPRT is average blur edge width (BEW) norm

alized by the m

oving speed

-also referred to as E-BET (extended blurred edge tim

e)

MPRT basics and issues

MPRT basics and issues

��Motion blur can be characterized by m

otion picture response tim

e (MPRT)

Motion blur can be characterized by m

otion picture response tim

e (MPRT)

metric, which is measured with a tracking camera that simulates the eye

metric, which is measured with a tracking camera that simulates the eye

tracking of a m

oving edge

tracking of a m

oving edge

��The system is expensive and tim

e consuming

The system is expensive and tim

e consuming

��Theoretically, motion blur is a pure temporal issue that can be uniquely

Theoretically, motion blur is a pure temporal issue that can be uniquely

determ

ined by the temporal response function (via LTI: linear systems

determ

ined by the temporal response function (via LTI: linear systems

theory)

theory)

•Nonlinearities (of both LCD and HVS) are only reasons for failure of LTI

•Small amplitude signals m

ay be within linear region approxim

ation of

•Small amplitude signals m

ay be within linear region approxim

ation of

both

•Still, MPRT does not consider HVS effects (too m

uch norm

alization)

��In Q&A with Someya (Mitsubishi) at IDW05, he thought that MPRT from

In Q&A with Someya (Mitsubishi) at IDW05, he thought that MPRT from

temporal measurement is only accurate for hold displays, but not for

temporal measurement is only accurate for hold displays, but not for

impulse displays such as displays using backlight flashing and black data

impulse displays such as displays using backlight flashing and black data

insertion

insertion

��At SID 06, Klompenhouwer described advanced m

otion blur measurement

At SID 06, Klompenhouwer described advanced m

otion blur measurement

schemes as “inventing a complex system to m

easure a sim

ple temporal

schemes as “inventing a complex system to m

easure a sim

ple temporal

response”

response”

Motion Blur Measurement with Sim

ulated Tracking Camera

Motion Blur Measurement with Sim

ulated Tracking Camera

∫−

=L

CD

edt

vtx

Ex

E)

()

(

The simulated retina image is the integration of a sequence of

temporal captured frames in the m

otion tracking trajectory

∑∫ =

∆∆

−=

−=

: i

CC

De

LC

De

ti

tiv

xE

xE

dt

vtx

Ex

E

1

),

()

(

)(

)(

Captured Frames in one Display Frame Period

Captured Frames in one Display Frame Period

Frame 1

Frame 2

Frame 3

Frame 4

Frame 5

Via high speed digital camera (900 fps)

Frame 6

Frame 7

Frame 8

Frame 9

Frame 10

Summary

Summary

��Basic Spatiotemporal Vision

Basic Spatiotemporal Vision

��Spatiotemporal Vision with Eye Movements

Spatiotemporal Vision with Eye Movements

��LCD Motion Issues

LCD Motion Issues

•Temporal Response

•Overdrive

Summary

Summary

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

•Overdrive

•Temporal Rendering Function

��Observer study of LCTV m

otion sharpness

Observer study of LCTV m

otion sharpness

matching

matching

What’s next :

What’s next :

Other Temporal Artifacts


��Motion Blur and Sharpness … as discussed

Motion Blur and Sharpness … as discussed

��Flicker … m

entioned

Flicker … m

entioned

•Asymmetrical temporal response

•Periphery & Brightness issues

��Judder

Judder

•Stepper-like m

otion, seen with slower steady m

otions


Other Temporal Artifacts (

))

(*

)(

)/

()

()

(/

σxb

vx

gaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

•Stepper-like m

otion, seen with slower steady m

otions

•CRT’s fast temporal response is not desired

��Multiple Edges

Multiple Edges

•Examples from Backlight Flashing + m

ismatched Overdrive

•Hollywood is happy with 24 fps ! (looks cinematic)

oDCI

oAliasing control via cameraman & editors

��Human eye is poor for seeing m

otion

Human eye is poor for seeing m

otion

What does real

What does real--world Movement really look like?

world Movement really look like?

What does real



��Multiples edges

Multiples edges can

can be

be

seen with some types of


tracking & saccades

tracking & saccades

combinations ??

combinations ??

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

Marcel Duchamp (1912) Nude descending a staircase

Marcel Duchamp (1912) Nude descending a staircase

What does real



��Multiples edges

Multiples edges can

can be

be



tracking & saccades

tracking & saccades

combinations ??

combinations ??

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

��Motion blur can be seen if attentive ??

Motion blur can be seen if attentive ??

What does real



()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=



What does real



Csuri (1984) @ OSU

Csuri (1984) @ OSU

��Human eye is poor for seeing m

otion

Human eye is poor for seeing m

otion

•Muybridge 1870s->

��Multiples edges can be seen with some types of

Multiples edges can be seen with some types of

tracking & saccades combinations ??

tracking & saccades combinations ??

What does real





��Experience and attention have large effects on

Experience and attention have large effects on

motion perception

motion perception

References:

References:

Spatiotemporal analysis of displaying perceived object motion:

Spatiotemporal analysis of displaying perceived object motion:

•Frequency domain analysis, (Watson 85, Girod 93, Klompenhouwer 04)

•Spatiovelocity analysis (W

atanabe 68 Kelly 79, Adelson & Bergen 85, Daly 98,

Laird 06)

•Tim

e domain analysis (Adelson & Bergen 85, Pan 05)

•Motion blur perception (Ramachandran 74, Parker ’81, Westerink 90, Bex 95,

Takeuchi 05, Laird 06)

Motion blur in LCD:

Motion blur in LCD:

Understanding Motion Blur & LCD TV

Motion blur in LCD:

Motion blur in LCD:

•Caused by the hold-type temporal rendering m

ethod of LCDs combined with

the smooth pursuit eye m

ovement of human visual system (HVS)

–(Lindholm

96, Parker 97, Kurita 98, 01, Klompenhouwer 05, Pan 05)

Motion blur reduction approaches:

Motion blur reduction approaches:

•Temporal overdrive (Okumura 01, Sekiya, 02)

•Temporal aperture reduction: black data insertion -BDI (Hong 04, Kim

ura

05), backlight flashing (Fisekovic 01, Sluyterm

an 05)

•Frame rate doubling –

FRD (Sekiya 02, Kurita 05)

•Motion compensated inverse filtering –MCIF ( Klompenhouwer 01)

Thank you for your interest and patience

Thank you for your interest and patience

Reference Capability of Conference Projector

Reference Capability of Conference Projector

8bitramp

Bit Depth, Contrast, and Spatial CSF

�Displaywith:10bits,Cmax=0.95,Lmax=1000cd/m

^2(CSFpeak=500;upper50%)

Active Matrix Drive

Active element

(Transistor)

X electrode

Y electrode

Structure

�Transistors are attached to

each subpixel for precise and

X1

X2

Pixel

Y1

Y2

Circuitry

each subpixel for precise and

faster sw

itching of its gray

value

�X and Y electrodes are

form

ed on the same substrate

as the TFT array.

�Switching signals are

applied to the Y electrode.

Video is applied to the X

electrode

LCD Tonescale

20

40

60

80

10

0

12

0Transparency and derivative

of the S-curve function

0

020

40

60

80

100

Vo

ltag

e ap

plied

Transparency and derivative

of the S-curve function

VT

-cu

rve

der

ivat

ive

of

the

S c

urv

e fu

nct

ion

06

41

28

19

22

55

Corr

esp

on

din

g c

od

e val

ue

Fig

.5 S

-cu

rve

of

the

LC

D a

nd

its

der

iva

tive

.

�Voltage Transmission curve (device level tonescale)

�Inherent LCD tonescale is S-shaped, like film

LCD System Tonescale

�Gamma-correction via LUT usually performed

�for quality desktop LCD monitor, for LCTV other goals

Salient Characteristics of LCD: Brightness

You can see the difference when playing images whose entire area is bright.

Larg

e A

rea

Brightn

ess

CRT &

Plasm

a T

VAQUOS L

CD T

V

�No brightness dependence on area

�Independent Backlight + Transm

issive Modulation

230 cd/m

2

58 cd/m

2

Peak b

rightn

ess

470 cd/m

2

470 cd/m

2

Assessment along Image Quality Dim

ensions

2002

2005

0

0.2

0.4

0.6

0.81

Response Time

(msec)

Contrast

[Dark]

Power

Consumption

(W) ※20”

Set Price

(yen/inch)

Life

(houｒ)

Under

7 10

20

30

10000

10000

over

100%

80

60

40

20

200

300

200

100000

10000

1000

100

Under 30

50

70

90

20000

15000

10000

5000

Under

1000

80

40

2800

600

400

80

200

0.00

0.20

0.40

0.60

0.80

1.00

Response Time

(msec)

Power

Consumption

(W) ※20”

Set Price

(yen/inch)

Color Gamut

(vs.NTSC)

Contrast

[Bright

(300lx)]

Life

(houｒ)

Contrast

[Dark]

100000

10000

1000

100

Under 30

50

70

90

20000

15000

10000

5000

Under

1000

Under

7 10

20

30

10000

2800

1200

600

10000

80

500

over

100%

80

60

40

20

200

300

200

80

40

Color Gamut

(vs.NTSC)

Contrast

[Bright (300lx)]

High Speed Response Technology

Wide Color B/L System / High Transmissivity Panel

Optical Optimized Panel (Higher Contrast)

Brightness

[All screen (cd/㎡

)

Size

(inch)

Brightness

[Peak]

(cd/㎡

)

Pixel Density

(PPI)

100%

300

400

500

600

300 400 500

60

600

40

20

over

200

160

120

80

Brightness

[All screen]

(cd/㎡

)

Size

(inch)

Brightness

[Peak]

(cd/㎡

)

Pixel Density

(PPI)

400

500

600

300 400 500

60

600

4020

over

200

160

120

LCD (ASV)

CRT

PDP

OLED (EL)

Aside: Spatial Superadditivity of CRT

�Superadditivity is a nonlinearity that makes it hard to determ

ine MTF

(as well as apply linear systems theory in image processing applications)

�These m

easurements on b/w

CRT without shadow m

ask, which

introduces phase complications as well.

�Data from Naim

an ’92 (SPIE HVEI)

Salient Characteristics of LCD: Low Reflection

�

Comparison of Reflection

When the room brightness is 300 LUX (normal brightness), liquid

crystal's low reflection rate greatly reduces mirror effects.

LCD TV

Plasma or CRT TV

LC TV Status 2006

0 000

0.2

0.20.2

0.20.4

0.40.4

0.40.6

0.60.6

0.60.8

0.80.8

0.81 111

Resp

onse

Tim

eR

esp

onse

Tim

eR

esp

onse

Tim

eR

esp

onse

Tim

e(R

oom

Tem

p.)

(Room

Tem

p.)

(Room

Tem

p.)

(Room

Tem

p.)

Resp

onse

Tim

eR

esp

onse

Tim

eR

esp

onse

Tim

eR

esp

onse

Tim

e(L

ow

Tem

p)

(Low

Tem

p)

(Low

Tem

p)

(Low

Tem

p)

Contr

ast

Contr

ast

Contr

ast

Contr

ast

(Brigh

t)(B

righ

t)(B

righ

t)(B

righ

t)

Contr

ast

Contr

ast

Contr

ast

Contr

ast

Brigh

tness

Brigh

tness

Brigh

tness

Brigh

tness

Pow

er

Cons.

Pow

er

Cons.

Pow

er

Cons.

Pow

er

Cons.

Vie

win

g A

ngl

eV

iew

ing

Angl

eV

iew

ing

Angl

eV

iew

ing

Angl

e

2004

2004

2004

2004

NOW

NOW

NOW

NOW

( usual “sales” caveat)

�Other Remaining Challenges

�High Dynamic Range, Wide Color Gamut, High-frame rate, Cost

0 000C

ontr

ast

Contr

ast

Contr

ast

Contr

ast

(Dar

k)(D

ark)

(Dar

k)(D

ark)

Colo

r G

amut

Colo

r G

amut

Colo

r G

amut

Colo

r G

amut

(EB

U)

(EB

U)

(EB

U)

(EB

U)

Colo

r D

epth

(10 b

it)

Colo

r D

epth

(10 b

it)

Colo

r D

epth

(10 b

it)

Colo

r D

epth

(10 b

it)

Bla

ck

Shift

B

lack

Shift

B

lack

Shift

B

lack

Shift

Brigh

tness

Brigh

tness

Brigh

tness

Brigh

tness

[Peak

][P

eak

][P

eak

][P

eak

]

Brigh

tness

Brigh

tness

Brigh

tness

Brigh

tness

[Avg

.][A

vg.]

[Avg

.][A

vg.]

2004

2004

2004

2004

ＣＲ

ＴＣ

ＲＴ

ＣＲ

ＴＣ

ＲＴ

Eccentricity and Periphery

Eccentricity : Position in visual field

Eccentricity : Position in visual field

•0 degrees eccentricity refers to w

here your eyes are pointed, corresponds to

fovea in retina

•90 degrees eccentricity is near edges of visual field (periphery)

Spatial Bandwidth of eye reduces in periphery

Spatial Bandwidth of eye reduces in periphery

Cones are densely packed in fovea : high spatial sampling

Cones are densely packed in fovea : high spatial sampling --> high bandwidth

> high bandwidth

They become less dense as eccentricity increases

They become less dense as eccentricity increases

100

101

100

101

102

103

CSFs for Eccentricity Range 0-25 deg


Contrast Sensitivity

Properties of Visual System: Eccentricity

��How eccentricity changes across image as viewing distance changes

How eccentricity changes across image as viewing distance changes

(left)

(left)

•Assuming viewer looking at center of im

age (pixel = 320)

��Eccentricity m

odel predictions of how visual spatial sensitivity varies across image

Eccentricity m

odel predictions of how visual spatial sensitivity varies across image

(right)

(right)

1H at pixel=320

2H ""

4H ""

30

Eccentricity vs. Location (640x480 image)

1H at center

2H ""

4H ""

1

Visual Sensitivity vs. Location

6H ""

0100

200

300

400

500

600

05

10

15

20

25

Image location (pixel)

Eccentricity (deg)

6H ""

2H at 160

0100

200

300

400

500

600

0

0.2

0.4

0.6

0.8

Image location

Sensitivity

Eye Movement Model –maxim

um smooth pursuit velocity

��Rotation into Spatiotemporal CSF with Eye Movement model

Rotation into Spatiotemporal CSF with Eye Movement model

��Max smooth pursuit velocity condition dependent, some unknowns

Max smooth pursuit velocity condition dependent, some unknowns

•Predictive vs. non-predictive m

otion

•Oscillation : eye m

ovements can actually lead

•Field of view

2

2.5


EMM (max 2 deg/sec) ST CSF

2

2.5



2

2.5

log temporal frequency (cy/sec)EMM (max 30 deg/sec, Westheimer) ST CSF

-1-0.5

00.5

11.5

2-0.50

0.51

1.5



-1-0.5

00.5

11.5

2-0.50

0.51

1.5



-1-0.5

00.5

11.5

2-0.50

0.51

1.5



Eye Movement Model –maxim

um smooth pursuit velocity

��Rotation into Spatiotemporal CSF with Eye Movement model

Rotation into Spatiotemporal CSF with Eye Movement model

��Max smooth pursuit velocity condition dependent

Max smooth pursuit velocity condition dependent

•Predictive vs. non-predictive m

otion

•Oscillation : eye m

ovements can actually lead

•Field of view

��If pursuit could be as high as 1200

If pursuit could be as high as 1200 deg

deg/sec , looks like traditional ST

/sec , looks like traditional ST

CSF, but with higher flicker fusion




-1-0.5

00.5

11.5

2-0.50

0.51

1.52

2.5




Measured eyetracks

Measured eyetracks

��Transfer relative eye positions to degrees:

Transfer relative eye positions to degrees:

Eye tracking data of a person tracking a m

oving Gabor

�velocity variations due to instrumentation or physiology ?

LCD Flickering

�Asymmetric

temporalresponsesleadto

HSFflicker(HighSpatialFrequency)

INPUT SIGNAL (slow))

DISPLAY RESPONSE

INPUT MEAN

MOVING IMAGE PATTERN SEEN AT SINGLE PIXEL

(Fast)

DISPLAY MEAN

TIM

E

LUMINANCE

DISPLAY MEAN

DISPLAY MEAN

(Faster)

SLA’s Hi-speed Image Measurement System

��The camera captures im

age sequences at at least 4

The camera captures im


times of the LC TV frame rate (4x60). {Dalsa, Uniq}


Co

mp

ute

r

��The camera captures im


The camera captures im




��One computer drives both the camera and LC TV to

One computer drives both the camera and LC TV to

synchronize the data capture and processing.

synchronize the data capture and processing.

��The algorithm & software developed by SLA m

akes

The algorithm & software developed by SLA m

akes

data capture and processing 100% automatic and

data capture and processing 100% automatic and

real

real--time.

time.

��The m

easurement is

The m

easurement is fast

fast(about 20 m

inutes for each

(about 20 m

inutes for each

temperature) and the hardware system is

temperature) and the hardware system is compact

compact..

Application for comparing types of Overdrive

50

100

150

200

250

300

20

30

40

50

60

70

80

90

100

Fir

st O

rder

Dyn

amic

Gam

ma

No

n-m

od

el B

ased

OD

Mo

del

Bas

ed O

D

050

100

150

200

250

300

00

10

20

30

40

50

60

70

80

90

100

0

10

050

100

150

200

250

300

-500

50

100

150

200

250

300

Target Level

Output Level

050

100

150

200

250

300

-500

50

100

150

200

250

300

Target Level

Output Level

Sec

on

d O

rder

Dyn

amic

Gam

ma

150

200

250

300

Temporal Response

Definition of second order dynamic gamma

Zn+

1

Zn

Fir

st o

rder

Dyn

amic

gam

ma

Sec

on

d o

rder

Dyn

amic

gam

ma

dn=

f(d

n-2

, z n

, z n

+1)

-6-4

-20

24

68

-500

50

100

150

Time in Frame

Temporal Response

Fra

me

No.

0

1

Zn-1

Z

n-2

Z

n-2

2

�S

econd o

rder

DG

model

s m

oti

on o

f m

ore

com

ple

x i

mag

e, t

hus

it i

s m

ore

rea

list

ic m

easu

re o

f m

oti

on b

lur

of

real

vid

eo

Examples: second order responses

150

200

250

300

Temporal Response

150

200

250

300

Temporal Response

-5-4

-3-2

-10

12

34

-500

50

100

Time in Frame

Temporal Response

-5-4

-3-2

-10

12

34

-500

50

100

150

Tim

e in Frame

Temporal Response

Analysis of Backlight Flashing

Analysis of Backlight Flashing

Further analysis for backlight flashing

Further analysis for backlight flashing

��With backlight flashing, the m

otion blur is greatly

With backlight flashing, the m

otion blur is greatly

reduced. The m

otion image looks clear and shows

reduced. The m

otion image looks clear and shows

more realism

more realism

��Object Evaluation

Object Evaluation

•The display output was captured w

ith a 240 Hz

camera

camera

•Retina image was derived from integration along

the m

otion trajectory (assuming perfect eye

tracking, a la Laird)

��Subjective Evaluation

Subjective Evaluation

•to quantify the amount of motion blur relative to

the perception of still im

age blur at various

backlight flashing w

idths

Der

ive

reti

na

imag

e fr

om

cap

ture

d f

ram

e b

y i

nte

gra

tion a

long

the

moti

on t

raje

ctory

Predicting Retinal Image

No

Ov

erd

riv

e, n

o f

lash

ing

No

ov

erd

rive,

fla

shin

g a

t 1

/8

du

ty c

ycl

e

No Overdrive

No Overdrive

Mu

ltip

le e

dg

es !

Using Overdrive

Using Overdrive

Ov

erd

rive,

No

fla

shin

gO

ver

dri

ve,

fla

shin

g a

t 1

/8

du

ty c

ycl

e

Motion Sharpening Study

Motion Sharpening Study

��Hammet & Bex ’95, tried to parse out:

Hammet & Bex ’95, tried to parse out:

��Sharpness constancy : don’t see blur until

Sharpness constancy : don’t see blur until

error signal of blur is visible

error signal of blur is visible

��Deblurring: neural processing generates

Deblurring: neural processing generates

high spatial frequencies

high spatial frequencies

��Blur matching experiment:

Blur matching experiment:

•1cy/deg Moving sine vs. blurrable square w

ave

•Unadapted vs. adapted to high SF

•Unadapted vs. adapted to high SF

•Adapted assumed to knockout or at least

reduce high SFs

•Decrease on plot means more Motion

Sharpening

•Sharpness constancy (top-down version) should

not be affected by high SF adaptation

•Data indicates some Deblurring processing

��Experiment flaw : sine waves don’t blur

Experiment flaw : sine waves don’t blur

with LPF

with LPF

Observer Study on Perceived Motion Blur

Observer Study on Perceived Motion Blur

with Backlight Flashing

with Backlight Flashing

Observer study for Motion blur matching with edge

Observer study for Motion blur matching with edge

��Subjective experiment

Subjective experiment

•Compare the perceived m

otion blur with still image blur

•Characterizing the effectiveness of backlight flashing in reducing

motion blur

•Can determ

ine if “motion sharpening effect” occurs

��Experiment conditions

Experiment conditions



•4 m

oving speeds: 27, 53, 80, 94 degrees per second

•4 flashing duty cycle: 1 (hold), ½

, ¼, and 1/8

•6 observers

��Feng HVEI 06 (SPIE Electronic Imaging Conference)

Feng HVEI 06 (SPIE Electronic Imaging Conference)

Observer study task

Observer study task

Per

fect

shar

p

edge

movin

gS

imu

late

d e

dg

e ≈

Mea

sure

d

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

�G

auss

ian e

dge

sim

ula

ted w

ith e

dge

wid

th p

aram

eter

σ

�O

bse

rver

adju

sts σ

to i

ncr

ease

the

blu

r w

idth

and/o

r ad

just

vto

reduce

the

blu

r w

idth

.

�M

ovin

g e

dge

com

par

ed a

gai

nst

sti

ll e

dge

edge

movin

gS

imu

late

d e

dg

e ≈

Mea

sure

d

reti

na

blu

r as

sum

ing

per

fect

trac

kin

g

y = 0.9286x - 0.9576

R2 = 0.9919

0.00

5.00

10.00

15.00

20.00

25.00

30.00

Subjective Matched Width

Results of Motion Blur Matching

Results of Motion Blur Matching

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

-5.00

0.00 0

510

15

20

25

30

Modeled Width

�Modeled width = sim

ulated retinal blur, assuming perfect eye tracking

�Overdrive with different combinations of speed and flashing duty cycles

(including no flashing)

�The regression line shows good correlation with a slope of 0.93, which is

very close to the unit slope. This indicates the perceived blur closely m

atches

the retina blur.

�Motion sharpening effects did not occur (too m

uch m

asking due to edge?)

��The LCD m

odel blur is m

odeled using Fourier

The LCD m

odel blur is m

odeled using Fourier

analysis: It is the combination of display

analysis: It is the combination of display

temporal low pass filtering and eye tracking

temporal low pass filtering and eye tracking

��To reduce m

otion blur

To reduce m

otion blur � ��

improve the display

improve the display

temporal MTF

temporal MTF

Summary of motion blur study

Summary of motion blur study

()

)(

*)

()

/(

)(

)(

/σx

bv

xgaus

rect

vx

tri

xst

epx

blu

rEdge

∗=

•Temporal overdrive to improve LCD temporal response

•Reduce the temporal aperture function to reduce

motion blur due to hold

��Implemented overdrive and backlight flashing

Implemented overdrive and backlight flashing

on a LCD with LED backlight.

on a LCD with LED backlight.

•The LED can be flashed at various duty cycle in sync

with LCD driving

Distribution of Text Scroll Velocity in TV Broadcasting

Average；；；；13.8 [deg/sec]

Aver

age；

13.8

Max

；35.9

From 71 programs of BS-digital broadcasting

20

30

40

50

60


“Telop”, which doesn't contain camera blur, mostly

distributes below 20[deg/sec]

Fujine et.al.;”Real-Life In-Home Viewing Conditions for FPDs and Statistical Characteristics of

Broadcast Video Signal,” Digest AM-FPD’06

0

10

20

0～

10

10～

20

20～

30

30～

40

40～

50

50～

60

視角

速度

(deg/sec)

頻度（％）

Vel

oci

tyin

vie

win

gan

gle

(deg

/sec

)[3

H]

Frequency (%)

MPRT m

easurement equipment

MPRT m

easurement equipment

�Photal –Otsuka electronics MPRT-2000

�Photal –Otsuka electronics MPRT-2000

-Equipping a pursuit CCD camera enables the evaluation close to that by eye perception

-A unique algorithm as a tim

e-based norm

alization perm

its the comparison among the

different types of displays

-Processes for measurements and analyses are computer-controlled including for moving

picture displays on the sample FPD

-Moving picture characteristic estim

ation for optional bitmap pictures

-Expansion of lineup of selections for many purposes・

The system w

ith a color CCD camera, the addition to the conventional system w

ith a

luminance filtered CCD camera ・

The pursuit color camera for a small display, the

addition to the conventional system for a m

edium and large display

Motion Blur and Temporal Response

Motion Blur and Temporal Response

Without Hold Effect (LCD is driven at infinite high frame rate),

the retina image is the temporal response function of a step

function Et(t)

)/

()

(v

xE

xE

t−

=With finite frame rate, motion blur is the

)(

)/

()

(

)(

)(

)(

)(

)(

0

0

/

vtx

dtd

t

vx

ttt

rect

vx

Ex

E

tE

th

dt

rect

th

xE

⊗−

==

⊗=∫

−

∞−

h(t

)

With finite frame rate, motion blur is the

convolution of the temporal response and the

temporal aperture function

x

Documents

Motion perception in displays SFU 2011.pptljilja/cnl/guests/Motion_perception_in_displays_SFU_2011.pdf · CRT spatially nonlinear MTF hard to assess, use ... Current Challenges for