• Home

  • Documents

  • 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner...
39
Visual Computing – Data Analysis – Consulting [email protected] René Donner Deep Learning

20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

  • Upload
    ngophuc

  • View
    259

  • Download
    1

Embed Size (px)

Citation preview

Page 1: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

Visual Computing – Data Analysis – Consulting [email protected]

René Donner

Deep Learning

Page 2: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Overview

2

The (amazing) things Deep Learning can do!

How does it work?

!

How can you start with DL?!

!

!

!

Page 3: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Roughly …

3

Deep learning finds patterns in data corresponding to high-level, abstract concepts

!

!

!… just like we do, be we are better …

!

!

!… currently. In most tasks, not all, any more.

!

!

Page 4: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

What it can be used for

4

Image recognition

Text understanding, translation

Voice recognition

Playing video games

Driving cars

…

!

!

Page 5: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Image recognition

5

Google Research Blog

Page 6: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Image recognition

6

Google Image Search

Page 7: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Scene labeling

7

http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html

Page 8: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Text recognition

8

!

http://www.pyimagesearch.com/2014/09/22/!getting-started-deep-learning-python/

Large-Scale Deep Learning for Intelligent Computer Systems, !Jeff Dean, Google, BayLearn 2015

Page 9: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Text understanding

9

2013 Glove: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher and Christopher D. Manning

Page 10: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Word embeddings

10

Page 11: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Word embeddings

11

Page 12: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Information extraction / Reasoning

12

MetaMind

Page 13: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Information extraction / Reasoning

13

MetaMind

Page 14: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

It is not perfect

14

!

!

!

!

http://www.news.cornell.edu/stories/2015/03/images-fool-computer-vision-raise-security-concernsderStandard.at

Page 15: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Some well know research groups

15

Stanford / BaiduAndrew Ng

!

NYU / FacebookYann LeCun

!

UToronto / GoogleGeoffrey Hinton

Page 16: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

NVIDIA

16

brand new: M40(same as Geforce GTX Titan X)

Images: NVIDIA website

Page 17: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

How does it work?

Page 18: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Difference to classic ML

18http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html

Page 19: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning

19

http://theanalyticsstore.ie/deep-learning/

Page 20: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Visualization

20

1. Layer

higher Layers

Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012

Page 21: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning

21

How does it work?

http://theanalyticsstore.ie/deep-learning/ http://stats.stackexchange.com/questions/114385/!what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma

Page 22: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Optimization

22

Stochastic gradient descent

!

!

!

!

!

Automatic differentiation

blog.datumbox.com

Page 23: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Local minima

23

Less problematic than thought - saddle points

https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg

Page 24: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning

24

Low level features of color images

https://www.coursera.org/course/neuralnets

Page 25: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning

25http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf

Page 26: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

ImageNet topologies

26

ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky

“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015

Page 27: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

MNIST - Demo

27

Page 28: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

MNIST

28

http://deeplearning4j.org/rbm-mnist-tutorial.html

Page 29: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning - why does it work?

29

Can cope with huge amounts of data

Learns small invariances

Overcomplete, sparse, representations

Learn Embedding

Lots of data

Recent advance: it is actually computable!

Page 30: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning - pros

30

Not-domain specific

Supervised / Semi-supervised / Unsupervised

Classification / regression in last layer

Simple math

Hip

Page 31: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Deep learning - cons

31

Lots of meta-parameters

Needs a lot of data

Very compute intensive

Hip

Page 32: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

Getting started with DL

Page 33: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Frameworks

33

Many different DL toolboxes

Efficiency important (GPU)

Attention to numerical issues

Page 34: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Frameworks

34

Caffehttp://caffe.berkeleyvision.org/Plain text filesFastest CNN, GPU

!Keras

https://github.com/fchollet/kerasPython, on top of Theano

!TensorFlow

http://tensorflow.org/Python, by Google

!MXNet

https://github.com/dmlc/mxnetPython, R, Julia

Slid

e fro

m c

affe

tuto

rial

Page 35: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Tensorflow

35

General gradient descent library

Page 36: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Tutorials

36

Stanford tutorial:https://deeplearning.stanford.edu/wiki/index.php/UFLDL_TutorialMatlab code snippets

videolectures.nethttp://videolectures.net/deeplearning2015_montreal/

courserahttps://www.coursera.org/course/neuralnets

!

!

!

!

!

Page 37: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Practical hints

37

Bengio ArxivPractical Recommendations for Gradient-Based Training of Deep Architectures, http://arxiv.org/abs/1206.5533http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html!

Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challengehttp://benanne.github.io/2014/04/05/galaxy-zoo.html

!

Relevant conferences NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)CVPR, ICMLMany interesting papers on arxiv.org

Page 38: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

René Donner Deep Learning

Current research topics

38

Parallelization

What is deep learning, actually?

Alternative, faster, simpler methods

Multi-domain, transfer learning

Page 39: 20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

Visual Computing – Data Analysis – Consulting [email protected]

René Donner

Deep Learning


Deep Learning: An Introduction from the NLP Perspective Deep Learning - An...Deep Learning Approach 1 Deep Learning Approach 2 Disclaimer I am not (yet) an expert in Deep Learning

Deep Learning: An Introduction from the NLP Perspective Deep Learning - An...Deep Learning Approach 1 Deep Learning Approach 2 Disclaimer I am not (yet) an expert in Deep Learning

Documents
Introducing Deep Learning with MATLAB - Systematics · Introducing Deep Learning with MATLAB6 Inside a Deep Neural Network ... Deep learning is a subtype of machine learning. With

Introducing Deep Learning with MATLAB - Systematics · Introducing Deep Learning with MATLAB6 Inside a Deep Neural Network ... Deep learning is a subtype of machine learning. With

Documents
Deep materials informatics: Applications of deep learning in … · deep learning: † Deep learning requires big data: In many cases, the biggest limiting factor for applying deep

Deep materials informatics: Applications of deep learning in … · deep learning: † Deep learning requires big data: In many cases, the biggest limiting factor for applying deep

Documents
20151216 - Booosting - Wim Meilink EconStruct

20151216 - Booosting - Wim Meilink EconStruct

Technology
Demystifying Deep Learning - MathWorks · Demystifying Deep Learning. 2 What’s Deep Learning and why should I care? A practical approach to Deep Learning (for images) –Transfer

Demystifying Deep Learning - MathWorks · Demystifying Deep Learning. 2 What’s Deep Learning and why should I care? A practical approach to Deep Learning (for images) –Transfer

Documents
Lecture 1: Introduction to Deep Learning - uvadlc.github.io · Lecture 1: Introduction to Deep Learning Efstratios Gavves. UVA DEEP LEARNING COURSE UVA DEEP LEARNING COURSE –EFSTRATIOS

Lecture 1: Introduction to Deep Learning - uvadlc.github.io · Lecture 1: Introduction to Deep Learning Efstratios Gavves. UVA DEEP LEARNING COURSE UVA DEEP LEARNING COURSE –EFSTRATIOS

Documents
Deep Learning Lab Course 2017 (Deep Learning …ais.informatik.uni-freiburg.de/teaching/ws17/deep...Deep Learning Lab Course 2017 (Deep Learning Practical) Labs: (Computer Vision)

Deep Learning Lab Course 2017 (Deep Learning …ais.informatik.uni-freiburg.de/teaching/ws17/deep...Deep Learning Lab Course 2017 (Deep Learning Practical) Labs: (Computer Vision)

Documents
TygerBurger Kuilsrivier 20151216

TygerBurger Kuilsrivier 20151216

Documents
Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

Documents
TygerBurger Eersterivier/Blue Downs 20151216

TygerBurger Eersterivier/Blue Downs 20151216

Documents
Learning Deep Learning

Learning Deep Learning

Technology
Diario Resumen 20151216

Diario Resumen 20151216

News & Politics
Essf 20151216 essf full

Essf 20151216 essf full

Documents
Deep Learning - Oxford Statisticspalamara/teaching/SML19/HT19_lecture15.pdf · SB2b/SM4 - Deep Learning Deep Learning Infrastructure! Computational Infrastructure critical to deep

Deep Learning - Oxford Statisticspalamara/teaching/SML19/HT19_lecture15.pdf · SB2b/SM4 - Deep Learning Deep Learning Infrastructure! Computational Infrastructure critical to deep

Documents
Deploying Enterprise Deep Learning Masterclass Preview - Enterprise Deep Learning

Deploying Enterprise Deep Learning Masterclass Preview - Enterprise Deep Learning

Software
20151216 Succeed Letter - Lakewood Health System

20151216 Succeed Letter - Lakewood Health System

Documents
THE DEEP LEARNING REVOLUTION - Nvidia...THE DEEP LEARNING REVOLUTION Superhuman Breakthroughs in Modern Artificial Intelligence Powered by GPUs JOIN THE DEEP LEARNING ERA DEEP LEARNING

THE DEEP LEARNING REVOLUTION - Nvidia...THE DEEP LEARNING REVOLUTION Superhuman Breakthroughs in Modern Artificial Intelligence Powered by GPUs JOIN THE DEEP LEARNING ERA DEEP LEARNING

Documents
Executing Deep Learning Strategies Masterclass Preview - Enterprise Deep Learning

Executing Deep Learning Strategies Masterclass Preview - Enterprise Deep Learning

Software
20151216 cross foods

20151216 cross foods

Food
Lecture 1: Introduction to Deep Learning · uva deep learning course –efstratios gavves introduction to deep learning - 1

Lecture 1: Introduction to Deep Learning · uva deep learning course –efstratios gavves introduction to deep learning - 1

Documents
Introduction to Deep Learning - Joyce Hojoyceho.github.io/cs534_s17/slide/25-deep-learning.pdf · Deep Learning: Overview ... Review: Backpropogation https: ... • Deep Learning

Introduction to Deep Learning - Joyce Hojoyceho.github.io/cs534_s17/slide/25-deep-learning.pdf · Deep Learning: Overview ... Review: Backpropogation https: ... • Deep Learning

Documents
From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Documents
TygerBurger Brackenfell 20151216

TygerBurger Brackenfell 20151216

Documents
Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Documents
Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Documents
Zuger presse 20151216

Zuger presse 20151216

Documents
Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Documents
Deep Learning I - Korea Universitydsba.korea.ac.kr/wp/wp-content/seminar/Deep learning/Deep Learning... · Deep Learning I Deep learning is research on learning models with multilayer

Deep Learning I - Korea Universitydsba.korea.ac.kr/wp/wp-content/seminar/Deep learning/Deep Learning... · Deep Learning I Deep learning is research on learning models with multilayer

Documents
fast.ai - Learning Deep Learning

fast.ai - Learning Deep Learning

Data & Analytics
Lecture 3: Deep Learning Optimizations - GitHub Pagesuvadlc.github.io/lectures/nov2019/lecture3.pdf · Lecture 3: Deep Learning Optimizations Deep Learning @ UvA. UVA DEEP LEARNING

Lecture 3: Deep Learning Optimizations - GitHub Pagesuvadlc.github.io/lectures/nov2019/lecture3.pdf · Lecture 3: Deep Learning Optimizations Deep Learning @ UvA. UVA DEEP LEARNING

Documents