Deep Learning Tutorial (I)IntroductionGuan Wang

Contents

Overview

Industry Landscape

System architectures

Bleeding edges

Overview

Deep neural networks learn to do the following

Yearbook• prior 2012

• Machine learning was more about SVM, Graphical Models, non-parametric baysian, and simple hacks, e.g., decision tree, naïve bayesian, regressions, etc

•2012~2013

• Deep Neural Network, big data, mature distributed computing architectures

• Refreshing accuracy record in image recognition tasks

• Feed forward Neural Networks, CNN, RBM

•2014~2015

• Bridging CV, NLP and expanding to other domains

• New architectures and new ML tasks

• RNN, LSTM, RL

•2016~future

• Larger models on CV, NLP, keep expanding to other domains

• Larger systems, larger models on CNN, LSTM, etc

Industry Landscape

Hardware Optimized for Neural Networks

Google TPUNVIDIA

Deep Learning Service on cloud

Computer Vision (crowded market)Generic Algorithms & API providers•Special object recognition (face, etc)•General object recognition (image search, etc)•Moving object detection & recognition (pedestrian detection, etc)•Image understanding (visual QA, artify, etc)•Video understanding (video search, etc)Verticals•Satellite image analysis (understanding civil developments)•Home & office place security & surveillance•User interest analysis and high precision targeting (ads)•ADAS & Autonomous Driving•Robotics and drones•The list goes on and on

Natural Language Understanding (crowded market)

Generic Algorithms & API providers•Personal assistance (x.ai, api.ai, etc)•Chat bots (facebook ecosystem, viv.ai, etc)•Knowledge understanding (IBM watson, etc)

Verticals•Customer service•Travel management•Financial service•Smart homes•Connected cars•The list goes on and on

System Architecture

Every good machine learning algorithm deserves its own system architecture. -- one of my mentors

Distributed ArchitecturesGeneric solvers•Stochastic Gradient Descent•Coordinate Descent•MCMC•ADMM•…Design Choices•sync or async•CPU or GPU cluster•Online training or offline training•...Examples•Parameter Server•DistBelief•Tensorflow

Flexible solutions•CoreOS, etcd, Docker•Kubernetes•Pachyderm•Mesos•etcBig data ecosystem•Spark & tachyon•yarn•hdfs•etcDeep learning tools•Tensorflow•Torch-IPC•DistBelief•Petuum•GraphLab

Standalone ToolkitsCPU with GPU speedup •Torch•Caffe•Theano•Tensorflow

Good for research or small applications

Embedded Support•Tiny-CNN•Tensorflow-embedded

Good for phone apps, raspberry Pi, cars, drones, robotics

Bleeding edge directions

CNN

CNN on Text ClassificationR-CNN, fast R-CNNAttention Model

LSTMSeq2seq model•Translation•Dialogue system•Time series analysis•Text classification

Memory NetworksExternal memory on RNN•Translation•Dialogue system•QA

Reinforcement LearningDeep Q-LearningFunction approximation on•Policy function•Value function

Deep Generative ModelAdversarial networks

The End

Thank You!