Upload
ashoka-vanjare
View
240
Download
2
Embed Size (px)
Citation preview
Fourier Analysis of Video Signals& Frequency Response of the
HVS
(Chapter 2)
Outline
• Fourier transform over multidimensional space– Continuous space FT (CSFT)– Discrete space FT (DSFT)– Sampled space FT (SSFT)
• Frequency domain characterization of video signals– Spatial frequency– Temporal frequency– Temporal frequency caused by motion
• Frequency response of the HVS– Spatial frequency response– Temporal frequency response and flicker– Spatio-temporal response– Smooth pursuit eye movement
Fourier Transform
4
• Fourier: a periodic function can be represented by the sum of sines/cosines of different frequencies, multiplied by a different coefficient (Fourier series)
• Non-periodic functions can also be represented as the integral of sines/cosines multiplied by weighing function (Fourier transform)
Fourier Transform
5
Introduction to the Fourier Transform
• f(x): continuous function of a real variable x• Fourier transform of f(x):
dxuxjxfuFxf ]2exp[)()()(
• (u) is the frequency variable.• The integral shows that F(u) is composed of an
infinite sum of sine and cosine terms and…• Each value of u determines the frequency of its
corresponding sine-cosine pair.
6
Introduction to the Fourier Transform
• Given F(u), f(x) can be obtained by the inverse Fourier transform:
)()}({1 xfuF
duuxjuF ]2exp[)(
• The above two equations are the Fourier transform pair.
7
Introduction to the Fourier Transform
• Fourier transform pair for a function f(x,y) of two variables:
dxdyvyuxjyxfvuFyxf ])(2exp[),(),()},({
dudvvyuxjvuFyxfvuF ])(2exp[),(),()},({1
and
where u,v are the frequency variables.
8
Continuous Space Signals
• K-D Space Signals
• Convolution
• Example function– Delta function
kK Rxxx ],...,,[ ),( 21xx
yyyxxx dhhkR )()()(*)(
9
Continuous Space Signals
• Delta function properties
)fdxx)f2exp(
)x)xxx)
)xx)xx*x)
0T0
00
00
K
K
R
R
j
10
Continuous Space Fourier Transform (CSFT)
• Forward transform
• Inverse transform
• Any signal can be expressed as a linear combination of complex exponential functions
variablefrequency domain continuous - ],...,[fth wi
) 2exp()()(
2,1K
K
R
Tc
Rfff
djk
xxfxf
fxffx djkR
Tc ) 2exp()()(
11
Continuous Space Fourier Transform (CSFT)
• Most properties of 1D and 2D Fourier Transforms can be extended to K-D case.
• Convolution theorem
)(*)()()()()()(*)(ffxxffxx
cc
cc
HhHh
12
Continuous Space Systems
• General system over K-D continuous space (input and output)
• Linear and Space-Shift Invariant (LSI) System
• LSI systems can be completely described by its impulse response
kRT xxx )),(()(
)()()()(*)()(
)()(fffxxx
xx
ccc HhTh
)())(())()(()()(
00
22112211
xxxxxxxx
TT
13
Discrete Space Signals
• K-D Space Signals
• Convolution
• Example function– Delta function
KK Znnn ],...,,[),( 21nn
KZ
hhm
mmnnn )()()(*)(
14
Discrete Space Fourier Transform (DSFT)
• Forward transform
• Inverse transform
• Convolution theorem
)2/1,2/1(, :period lFundamenta
1 of periodwith dimension each in periodic is )(
)2exp()()(
kK
d
R
Td
fI
jK
f
f
nfnfn
KI
Td dj
f
fnffn )2exp()()(
)(*)()()()()()(*)(ffnnffnn
dd
dd
HhHh
Frequency domain characterization of video signals
• Video is 3D signal: x,y,t• Can apply CSFT and DSFT with K=3
• Spatial frequency
• Temporal frequency
• Temporal frequency caused by motion
tfyx ff ,
16
Spatial Frequency
• Spatial frequency measures how fast the image intensity changes in the image plane
• Spatial frequency can be completely characterized by the variation frequencies in two orthogonal directions (e.g horizontal and vertical)– fx: cycles/horizontal unit distance
– fy : cycles/vertical unit distance
• It can also be specified by magnitude and angle of change
)/arctan( ,22xyyxm fffff
17
Illustration of Spatial Frequency
18
Angular Frequency
height-pictureper cyclesin is f whereee)cycle/degr(f180
ff
/2for (degree)180(radian)/22(radian))2/arctan(2
sss
hd
dhdhdhdh
Problem with previous defined spatial frequency:Perceived speed of change depends on the viewing distance.
19
Angular Frequency
• Spatial and angular frequencies are related• Angular frequency increases as the viewing distance increases• Angular frequency can be defined along horizontal direction as well
• Angular frequency depends on the signal and the viewing condition
20
Temporal Frequency
• Temporal frequency measures temporal variation (cycles/s)
• In a video, the temporal frequency is spatial position (2D) dependent, as every point may change differently
• For a fixed (x,y) cycles/sec = Hz• Temporal frequency is caused by camera or object
motion (equivalent)• Maximum temporal frequency (for all points)
21
Temporal Frequency caused by Linear Motion
22
),(),,( 0 tvytvxtyx yx
Relation between Motion, Spatial and Temporal Frequency
)AssumptionIntensity (Constant is at timepattern image the),,( is
0at pattern image theAssume ).,( speed with movingobject an Consider
0 tyx
tvv yx
derive
23
)( :frequency temporaland spatial, motion,between Relation
0at zeronot is ),,( Hence,
)(),(),,( 0
yyxxt
yyxxttyx
yyxxtyxtyx
fvfvf
fvfvffff
fvfvffffff
The temporal frequency of the image of a moving object depends on motion as well as the spatial frequency of the object.
Example: A plane with vertical bar pattern, moving vertically, causes no temporal change; But moving horizontally, it causes fastest temporal change
If scene has multiple objects, then we can subdivide into small regions with uniform motion with constant speed
Relation between Motion, Spatial and Temporal Frequency
24
)( :frequency temporaland spatial, motion,between Relation
yyxxt fvfvf
Relation between Motion, Spatial and Temporal Frequency
The temporal frequency of the image of a moving object depends on motion as well as the spatial frequency of the object.
change) spatialgreatest of direction in the movesobject hen greatest w theisfrequency (temporal
),( toorthogonal is )v,( if 0 (b)
)brightness shomogeneou (e.g.
and of any valuesfor 0 then 0 if (a)
y yxxt
yxtyx
ffvf
vvfff
25
Illustration of the Relation
Frequency response of the HVS
• Temporal frequency response and flicker• Spatial frequency response• Spatio-temporal response• Smooth pursuit eye movement
27
Temporal Response
Critical flicker frequency: The lowest frame rate at which the eye does not perceive flicker.
Provides guideline for determining the frame rate when designing a video system.
Critical flicker frequency depends on the mean brightness of the display:
60 Hz is typically sufficient for watching TV.
Watching a movie needs lower frame rate than TV
28
Temporal Response
• Experiment with human observer• Screen with varied brightness (t)
ysensitivitcontrast - /1noticeablejust flicker -mmidentify tovary
frequencymean - ,brightnessmean - )2cos1()(
min
min
m
mfBftmBt
• Different curves for different B• reduce sensitivity due to “persistence of vision”
29
Temporal Response
• Different curves for different B• “troland” – units of light entering retina• critical frequency 20 – 80 Hz• Computer display 9000 trolands• movies very low ~ 1 troland• note: brightness at the retine, i.e. depends on viewing distance• Hence
• Movies 24fps• TV 50/60fps• SVGA 72fps
30
Spatial Response
• Experiment with human observer• Screen with varied brightness (x)• can be measured in x, y, etc
ysensitivitcontrast - /1noticeablejust changes spatial -m
midentify to vary each for brightnessmean -
)2cos1()(
min
min
m
mfB
fxmBt
• 3 curves – eye movement stabilization• “saccadic” eye movement changes contrast sensitivity• top curve – no eye stabilization
31
Spatiotemporal Response
• Previously: temporal frequency response at zero spatial frequency• Previously: spatial frequency response at zero temporal frequency• Now need to combine
page)next (graph y sensitivitcontrast - /1noticeablejust changes oralspatiotemp -m
midentify to vary and ofset afor brightnessmean -
))2cos()2cos(1(),,(
min
min
m
mffB
tfxfmBtyx
tx
tx
• Graph – next page• Note: experiment under normal saccadic eye movement• much higher sensitivity when tracking moving object compare to a flickering wave• Smooth pursuit eye movement vs. saccadic eye movement
32
Spatiotemporal Response
The reciprocal relation between spatial and temporal sensitivity was used in TV system design:
Interlaced scan provides tradeoff between spatial and temporal resolution.
33
Smooth Pursuit Eye Movement
• Smooth Pursuit: the eye tracks moving objects• Net effect: reduce the velocity of moving objects on the retinal plane, so that
the eye can perceive much higher raw temporal frequencies than indicated by the temporal frequency response.
yyxxtyyxxtt
yx
yyxxt
yyxxtyxtx
yx
yx
vvvvffvfvff
vv
fvfvf
fvfvffffff
ttvytvxtyxtyxtyxvv
~,~ if 0~
and )~~(~
:)~,~(at moving is eye when theretina at thefrequency temporalObserved
)( :onlymotion object by causedfrequency Temporal
)~~,,(),,(~FT takingand ),~,~(),,~
scoordinate retine andscreen - ),,(~ ,),,( and ),(movement eye Assume
y
(a) no smooth persuit
(b) 2 deg/s
(c) 10 deg/s
Outline
• End of Chapter 2