Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Higher-order Statistical Modeling based
Deep CNNs (Introduction) Peihua Li
Dalian University of Technology http://peihuali.org
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment
, ( )XX f x
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment
, ( )XX f x
2 21( ) ii
E X xN
= ∑
1( ) ii
E X xN
= ∑i.i.d. samples
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2 21( ) i
i
E X xN
= ∑2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment 1( ) i
i
E X xN
= ∑i.i.d. samples
, ( )XX f x
3rd-order moment
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2 21( ) i
i
E X xN
= ∑2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment 1( ) i
i
E X xN
= ∑i.i.d. samples
, ( )XX f x
4th-order moment
1st-order moment
1( )E XN ∈
= ∑x
x
p∈Rx
3rd-order moment
3 1( )E XN ∈
= ⊗ ⊗∑x
x x x
x
x
x3pR
2nd-order moment
2 1( )
1
TE XN
N
∈
∈
=
= ⊗
∑
∑x
x
xx
x x
x
x
2pR
What is Higher-order?─Statistical Moments Random vector
Images courtesy of “Kernel Pooling for Convolutional Neural Networks”
What is Higher-order?─Statistical Moments
( )Xf x
Probaiblity density is everything ( )Xf xRandom vector
What is Higher-order?─Statistical Moments
( )( ) ( )jX jxX XE e f x e dxω ωω
+∞
−∞Φ = = ∫ ( )Xf x
Characteristic Function=Probability Density
The characteristic function is defined as
1( ) ( )2
jxX Xf x e dxωω
π+∞ −
−∞= Φ∫
Fouriere Transform Pair
Random vector
What is Higher-order?─Statistical Moments
( )( ) ( )jX jxX XE e f x e dxω ωω
+∞
−∞Φ = = ∫ ( )Xf x
Characteristic Function=Probaiblity Density
The characteristic function is defined as
1( ) ( )2
jxX Xf x e dxωω
π+∞ −
−∞= Φ∫
Fouriere Transform Pair
Random vector
Moments matter ( )( ) jXX E e ωωΦ =
( )Xf x
Fouriere Transform Pair
0 0
22 2
( ) ( )! !
1 ( ) ( ) ( ) .2! !
k kk k
k k
kk k
j X jE E Xk k
j jjE X E X E Xk
ω ω
ω ω ω
∞ ∞
= =
= =
= + + + + +
∑ ∑
If we know characteristic function , we know everything. ( )X ωΦ
What is Higher-order?─Statistical Moments
( )( ) ( )jX jxX XE e f x e dxω ωω
+∞
−∞Φ = = ∫ ( )Xf x
Characteristic Function=Probability Density
The characteristic function is defined as
1( ) ( )2
jxX Xf x e dxωω
π+∞ −
−∞= Φ∫
Fouriere Transform Pair
Random vector If we know characteristic function , we know everything.
0
( ) ( )!
kk k
Xk
j E Xk
ω ω∞
=
Φ =∑( )Xf xFouriere Transform Probability density Characteristic function
1st-order moment 1( )E XN ∈
= ∑x
x
2 1 1( ) TE XN N∈ ∈
= = ⊗∑ ∑x x
xx x x
2nd-order moment
3rd-order moment 3 1( )E XN ∈
= ⊗ ⊗∑x
x x x
( )X ωΦMoments matter
What is Higher-order?─Signal Perspective
Convolution is a linear transformation
= +y b Wxx y
( )= + + ⊗ +y b Wx H x x
Multi-variable Taylor series:
1st-order term (Linear term) ( )f=y x
What is Higher-order?─Signal Perspective
Convolution is a linear transformation
[ ]( , ) (0,0) T u uf u v f u v
v v
= + + +
w H
Two variable Taylor series:
1 2w u w v+ 2 211 12 222h u h uv h v+ +
Linear term
What is Higher-order?─Signal Perspective
Convolution is a linear transformation
[ ]( , ) (0,0) T u uf u v f u v
v v
= + + +
w H
Two variable Taylor series:
1 2w u w v+ 2 211 12 222h u h uv h v+ +
Linear term
Higher order enhances non-linear modeling capability
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
Why Higher-order? Hand-crafted Features
Learned Features
0
( ) ( )!
kk k
Xk
j E Xk
ω ω∞
=
Φ =∑( )Xf xProbability density Characteristic function
Why Higher-order? Hand-crafted Features
Learned Features
0
( ) ( )!
kk k
Xk
j E Xk
ω ω∞
=
Φ =∑( )Xf xProbability density Characteristic function
Higher-order moments can better characterize real-word distributions
Global average pooling
Why Higher-order?
1 1 1
1
1
cov( , ) cov( , ) cov( , ) cov( , ) cov( , ) cov( , )
j
i i j
p p j
x x x x
x x x x
x x x x
channel correlation
width
height
p channel
X
jxix
2 1( ) TE XN ∈
= ∑x
xx
2nd-order moment
CNN
What does each channel indicate?
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
width
height
p channel
Why Higher-order?
Body | channel 452
Head | channel 123
Hind claw | channel 448
Legs | channel 99
Tail | channel 174
Front claw | channel 333
CNN
What does each channel indicate?
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
width
height
p channel
Why Higher-order?
Body | channel 452
Head | channel 123
Hind claw | channel 448
Legs | channel 99
Tail | channel 174
Front claw | channel 333
Physical Interpretation For object recogniton, 2nd-order moment capture dependency of different parts
Context of the object
Why Higher-order?
What does each channel indicate?
width
height
p channel
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
Bookcase | channel 97
Drawer | channel 360
Carpet | channel 260
Plant | channel 384
Decoration | channel 44
Floor | channel 459
CNN
Why Higher-order?
What does each channel indicate?
width
height
p channel
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
Bookcase | channel 97
Drawer | channel 360
Carpet | channel 260
Plant | channel 384
Decoration | channel 44
Floor | channel 459
Physical Interpretation For scene images, 2nd-order moment capture dependency of different objects
Context of the scene
Why Higher-order?
What does each channel indicate?
width
height
p channel
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
Bookcase | channel 97
Drawer | channel 360
Carpet | channel 260
Plant | channel 384
Decoration | channel 44
Floor | channel 459
3rd-order moment or direct distribution
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
Overview of Speaker Wangmeng Zuo received the Ph.D. degree in computer application technology from the Harbin Institute of Technology, Harbin, China, in 2007. He is currently a Professor in the School of Computer Science and Technology, Harbin Institute of Technology. His current research interests include image enhancement and restoration, object detection, visual tracking, and image classification. He has published over 70 papers in toptier academic journals and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal of Electronic Imaging, and the Guest Editor of Neurocomputing, Pattern Recognition, IEEE Transactions on Circuits and Systems for Video Technology, and IEEE Transactions on Neural Networks and Learning Systems.
Overview of Speaker Qilong Wang received the Ph.D. Degree in the School of Information and Communication Engineering, Dalian University of Technology in 2018. He is currently a lecturer in the College of Intelligence and Computing, Tianjin University. His research interests include visual classification and deep probability distribution modeling. He has published several papers in top conferences and referred journals including ICCV, CVPR, ECCV, NIPS, IJCAI, TPAMI, TIP and TCSVT.
Overview of Speaker Peihua Li is a professor of Dalian University of Technology. He received Ph.D degree from Harbin Institute of Technology in 2003, and then worked as a postdoctoral fellow at INRIA/IRISA, France. He achieved the honorary nomination of National Excellent Doctoral dissertation in China. He was supported by Program for New Century Excellent Talents in University of Chinese Ministry of Education. His team won 1st place in large-scale iNaturalist Challenge spanning 8000 species at FGVC5 CVPR2018, 2nd place in Alibaba Large-scale Image Search Challenge 2015. His research topics include deep learning and computer vision, focusing on image/video recognition, object detection and semantic segmentation. He has published papers in top journals such as IEEE TPAMI/TIP/TCSVT and top conferences including ICCV/CVPR/ECCV/NIPS. As a principal investigator, he receives funds from National Natural Sceince Foundation of China (NSFC), Chinese Ministry of Education and Huawei Technologies Co., Ltd.
http://peihuali.org/
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
Overview of Tutorial─Part 1
Higher-order:
Higher-order:
Overview of Tutorial─Part 2
Overview of Tutorial─Part 3
Overview of Tutorial─Part 4