Upload
roland-page
View
217
Download
0
Embed Size (px)
Citation preview
Medical University of Lübeck
Institute for Signal Processing
Nonlinear visual coding from an intrinsic-geometry perspective
E. Barth* & A. B. Watson
NASA Ames Research Centerhttp://vision.arc.nasa.gov
Supported by DFG grant Ba 1176/4-1 to EB and NASA grant 199-06-12-39 to ABW
Medical University of Lübeck
Institute for Signal Processing
Intrinsic dimensionality in 2D
• i0D: constant in all directions
• i1D: constant in one direction
• i2D: no constant direction
Medical University of Lübeck
Institute for Signal Processing
i1D
• e.g. straight lines and edges, gratings
FTf (x, y) =g(ξ)
Medical University of Lübeck
Institute for Signal Processing
i2D
• e.g. corners, line ends, curved edges and lines
FTf (x, y) =g(ξ,ζ )
Medical University of Lübeck
Institute for Signal Processing
Intrinsic dimensionality in 3D
• i0D: constant in all (space-time) directions
• i1D: constant in 2 directions
• i2D: constant in one direction
• i3D: no constant direction
Medical University of Lübeck
Institute for Signal Processing
i1D
• e.g. drifting spatial grating
f (x, y, t) =g(ξ)FT
Medical University of Lübeck
Institute for Signal Processing
i2D
e.g. drifting corner, flashed grating
f (x, y, t) =g(ξ,ζ )FT
Medical University of Lübeck
Institute for Signal Processing
i3D
e.g. flow discontinuities, flashed corners
f (x, y, t) =g(ξ,ζ,τ)FT
Medical University of Lübeck
Institute for Signal Processing
Intrinsic dimensionality and motion
• FT of (rigid) motion signal is in a plane
motion ⇔ i2D
The visual input as a hypersurface
luminance
( x , y , t , f ( x , y , t ))
f ( x , y , t )
hypersurface
Visualization of surfaces is easier:(x, y, f (x, y))
Medical University of Lübeck
Institute for Signal Processing
Geometric view on intrinsic dimensionality
i1D++
mean curvature
i2D+
Riemann curvature tensor
i3D
Gaussian curvature
Hypersurface Geometry
H ≠0 R ≠0 K ≠0
Medical University of Lübeck
Institute for Signal Processing
Curvature and motion
(“plane” = “more than line but no volume”)
motion ⇔ R≠0 ∧¬ K ≠0
Medical University of Lübeck
Institute for Signal Processing
The Riemann tensor R
• most important property of (hyper)surfaces
• measures the curvature of the (hyper)surface
• has 6 independent components in 3D
• vanishes in 1D.
Medical University of Lübeck
Institute for Signal Processing
The Riemann tensor components
R1 =fxxfyy− fxy
2
1+ fx2 + fy
2 + ft2
R2 =fxxftt − fxt
2
1+ fx2 + fy
2 + ft2 R3 =
fyyftt − fyt2
1 + fx2 + fy
2 + ft2
R6 =fxyftt− fxtfyt
1+ fx2 + fy
2 + ft2 R4 =
fxxfyt − fxtfxy
1+ fx2 + fy
2 + ft2 R5 =
fxyfyt − fyyfxt
1 + fx2 + fy
2 + ft2
are nonlinear combinations of derivatives, i.e., of linear filters with various spatio-temporal orientations.
Medical University of Lübeck
Institute for Signal Processing
R components and speed v
Multiple representation of speed.
R3
R1
=v2 (cosθ)2
R2
R4
=−v (sinθ)R4
R1
=−v (sinθ)
R3
R5
=v (cosθ)
R2
R1
=v2 (sinθ)2
R5
R1
=v (cosθ)R6
R4
=v (cosθ)
R6
R5
=−v (sinθ)
f : ′ f (x−tvcosθ,y−tvsinθ)
Medical University of Lübeck
Institute for Signal Processing
R and direction of motion q
R6
R3
=− (tanθ )R2
R6
= (tanθ)
R2
R3
= (tanθ)2
Multiple, distributed representation of direction.
R4
R5
=− (tanθ )
f : ′ f (x−tvcosθ,y−tvsinθ)
Medical University of Lübeck
Institute for Signal Processing
Sectional curvatures
x
′ t ′ x t
yvx =
Ky ′ t −Ky ′ x
2 Kxy
(R5 ) : R3221 =fxyfyt − fyyfxt
1 + fx2 + fy
2 + ft2 =
12
( fyyf ′ t ′ t − fy ′ t2 )−( fyyf ′ x ′ x − fy ′ x
2 )
1 + fx2 + fy
2 + ft2
Medical University of Lübeck
Institute for Signal Processing
Direction tunings of R componentsverticalmotion
horizontalmotion
Kooi, 1993
QuickTime™ and aAnimation decompressorare needed to see this picture.
“abolished illusion”
Rodman & Albright, 1989
Analytical predictions based on R components
Typical Type II MT neuron, macaque monkey
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1-1
-0.5
0
0.5
1-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
Direction tuning
Orientation tuning
Orthogonal orientation and direction tunings
Recanzone, Wurtz, & Schwarz, 1997
Analytical predictions based on R components
Typical MT neuron, macaque monkey
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
-1
-0.5
0
0.5
1
Multiple motions
Medical University of Lübeck
Institute for Signal Processing
(Reference to 3D world of moving objects is not needed.)
Conclusion
Hypothesis that a basic (geometric) signal property (the intrinsic dimensionality) is encoded in early- and mid-level vision explains– orientation selectivity (derivatives, and R2, R3)– endstopping
(all R components are endstopped for translations)– velocity selectivity
– direction selectivity– some global-motion percepts (by integration)– some properties reported for MT neurons.