Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
http://bit.ly/idl2018
Xavier [email protected]
Associate ProfessorUniversitat Politecnica de CatalunyaTechnical University of Catalonia
The Neural Network Zoo(& some research at UPC)
Day 4 Lecture 4
#DLUPC
2
Acknowledgements
Fjodor Van Veen, “The Neural Network Zoo”
The Asimov Institute (2016)
3
Acknowledgements
4
A Perceptron
J. Alammar, “A visual and interactive guide to the Basics of Neural Networks” (2016)F. Van Veen, “The Neural Network Zoo” (2016)
5
Neural Network = Multi Layer Perceptron
2 layers of perceptronsF. Van Veen, “The Neural Network Zoo” (2016)
6
Deep Neural Network (DNN)
F. Van Veen, “The Neural Network Zoo” (2016)
7Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks”
The hidden layers and the output depend from previous
states of the hidden layers
Recurrent Neural Network (RNN)
8
Recurrent Neural Network (RNN)
F. Van Veen, “The Neural Network Zoo” (2016)
9
Recurrent Neural Network (RNN)
F. Van Veen, “The Neural Network Zoo” (2016)
10
Recurrent Neural Network (RNN)
Víctor Campos, Brendan Jou, Jordi Torres, Xavier Giró-i-Nieto, Shih-Fu Chang, “Skip RNN: Learning to Skip State Updates in RNNs”. NIPS Workshop on Time Series 2018.
11
Recurrent Neural Network (RNN)
Deep Learning TV, “Recurrent Neural Networks - Ep. 9”
12
Convolutional Neural Network (CNN)
F. Van Veen, “The Neural Network Zoo” (2016)
13
Convolutional Neural Network (CNN)
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
LeNet-5
14
Convolutional Neural Network (CNN)
Orange
A Krizhevsky, I Sutskever, GE Hinton “Imagenet classification with deep convolutional neural networks” NIPS 2012.
AlexNet
15
Convolutional Neural Network (CNN)
Junting Pan, Kevin McGuinness, Elisa Sayrol, Noel O'Connor, and Xavier Giro-i-Nieto. "Shallow and Deep Convolutional Networks for Saliency Prediction." CVPR 2016.
JuntingNet
16
Convolutional Neural Network (CNN)
Marc Assens, Kevin McGuinness, Noel E. O’Connor, Xavier Giró-i-Nieto, “SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes”. ICCVW 2017. (winner Salient360 2017)
[ 600 x 300 ]
SaltiNet
17
Convolutional Neural Network (CNN)
Deep Learning TV, “Convolutional Neural Networks - Ep. 8”
18
Deconvolutional Neural Network (Deconv)
Junting Pan, SalGAN (2017)F. Van Veen, “The Neural Network Zoo” (2016)
19
Deep Residual Learning / Skip connections
F. Van Veen, “The Neural Network Zoo” (2016)
He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep Residual Learning for Image Recognition." CVPR 2016 [slides]
20Figure: Kilian Weinberger
Deep Residual Learning / Skip connections
21Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." CVPR 2015 & PAMI 2016.
Deep Residual Learning / Skip connections
22Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." MICCAI 2015
U-Net
Deep Residual Learning / Skip connections
23
CNN + RNN + Skip connections
Amaia Salvador, Míriam Bellver, Manel Baradad, Ferran Marqués, Jordi Torres, Xavier Giró-i-Nieto, “Recurrent Neural Networks for Semantic Instance Segmentation”. NIPS WiML 2017.
24
Dense connections
Dense Block of 5-layers with a growth rate of k=4
Connect every layer to every other layer of the same filter size.
Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017. [code]
DenseNet
25
Dense connections
Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017. [code]
DenseNet
26
Dense connections
Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017 [code]
27
Dense connections
Jégou, Simon, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. "The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation." CVPRW 2017 [code] [slides]
28
Autoencoder (AE)
F. Van Veen, “The Neural Network Zoo” (2016)
29
Variational Autoencoder (VAE)
F. Van Veen, “The Neural Network Zoo” (2016)
30F. Van Veen, “The Neural Network Zoo” (2016)
Restricted Boltzmann Machine (RBM)
31
Deep Belief Networks (DBN)
F. Van Veen, “The Neural Network Zoo” (2016)
32
Adversarial Networks
Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative adversarial nets." NIPS 2014Goodfellow, Ian. "NIPS 2016 Tutorial: Generative Adversarial Networks." arXiv preprint arXiv:1701.00160 (2016).
F. Van Veen, “The Neural Network Zoo” (2016)
33
Adversarial Networks
Junting Pan, Cristian Canton, Kevin McGuinness, Noel E. O’Connor, Jordi Torres, Elisa Sayrol and Xavier Giro-i-Nieto. “SalGAN: Visual Saliency Prediction with Generative Adversarial Networks.” CVPRW 2017.
Introduce an additional term to the less to improve performance.
34F. Van Veen, “The Neural Network Zoo” (2016)
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). [slides] [code]
Differentiable Neural Computers (DNC)
35
von Neumann architecture (1952) Neural Turing Machine (2014)
Differentiable Neural Computers (DNC)
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). [slides] [code]
End-to-end trainable
36
Differentiable Neural Computers (DNC)Add a trainable external memory to a neural network.
Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo et al. "Hybrid computing using a neural network with dynamic external memory." Nature 538, no. 7626 (2016): 471-476. [Post by DeepMind]
37
Differentiable Neural Computers (DNC)DNC can solve tasks reading information from a trained memory.
Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo et al. "Hybrid computing using a neural network with dynamic external memory." Nature 538, no. 7626 (2016): 471-476. [Post by DeepMind]
38
Differentiable Neural Computers (DNC)
[Siraj Raval on YouTube]
39
More architectures to come...
Yann LeCun, “A Path to AI”, Beneficial AI 2017.
von Neumann architecture (1952)
40
The Full Story
F. Van Veen, “The Neural Network Zoo” (2016)
41
The Prequel
Fjodor Van Veen, “Neural Network Zoo Prequel: Cells and Layers”(2017)
42
Questions ?