Deploying Enterprise Deep Learning Masterclass Preview - Enterprise Deep Learning

Deploying Enterprise Deep Learning

Sam Putnam, Enterprise Deep Learning, LLC

July 27, 2017

Sam Putnam

@edeeplearning

July 27

Deep Learning Made This Video

https://www.extremetech.com/g00/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they?i10c.referrer=https%3A%2F%2Fwww.google.com%2F

Preview - Slide Available at deeplearningconf.com

Sam Putnam

@edeeplearning

And This Song!

Sam Putnam

@edeeplearning

July 27

And Predicted These Housing Prices!

https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb

http://www.mirror.co.uk/news/uk-news/experts-predict-house-prices-could-10724816

Run the Code ->This Afternoon

Sam Putnam

@edeeplearning

July 27

OK, so what is Deep Learning?

http://p.migdal.pl/imgs/2017-04-30-learning-deep-learning/deep_learning_meme_keras.png

Sam Putnam

@edeeplearning

July 27

Deep Learning is Neural Networks

http://p.migdal.pl/imgs/2017-04-30-learning-deep-learning/deep_learning_meme_keras.png

Sam Putnam

@edeeplearning

July 27

Well, Artificial Neural Networks. But they are based off of how the brain works.

https://www.extremetech.com/g00/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they?i10c.referrer=https%3A%2F%2Fwww.google.com%2F

Sam Putnam

@edeeplearning

July 27

This is a Neuron: Electrical Signals Are Sent By Axons and Received By Dendrites

http://www.newworldencyclopedia.org/entry/Dendrite

Sam Putnam

@edeeplearning

July 27

This is a Single Artificial Neuron With Two Inputs and Two Weights

Sam Putnam

@edeeplearning

July 27

The Artificial Neuron is Also Called a “Perceptron”. It “Perceives” Inputs.

Sam Putnam

@edeeplearning

July 27

b is the bias, and to activate the neuron:

Sam Putnam

@edeeplearning

Click Play, Visualize Neuron’s Performance

http://playground.tensorflow.org/

Sam Putnam

@edeeplearning

Trust Me, When you add more inputs, your decision boundary becomes a “plane”

When you’re talking about Deep Learning you’re talking about Artificial Neural Networks

of Neurons

@edeeplearning

In particular, Deep Learning = Deep Artificial Neural Networks of Neurons

@edeeplearning

This it it - Just more than one layer of neurons between the input and output!

@edeeplearninghttp://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/

You Tried The Line, Now Add A Hidden Layer To Capture More Complex Data Separations

This is No Line You Are Tweaking! This Is a More Complex Function

More Neurons + A Deeper Network = More Sophisticated Representations

@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground

Look at the last hidden layer - it is doing a pretty good job separating the data

This Architecture is called a Feedforward Neural Network

There’s a Lot Going on Here. Do You Really Want to Change Those Weights One at a Time?

Solution: Gradient Descent - Look How the Lowest Loss (0) Is Sought Out on the Right

What other Architectures Are There? Well, what about for Time Series Data

@edeeplearninghttps://deeplearning4j.org/lstm.html

Recurrent Neural Networks are Networks with Loops in Them

@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/

Information is passed from one step of the network to the next

Recurrent Neural Networks (RNNs) Contain A Single Layer (Yellow Box)

Recurrent Neural Networks (RNNs) cannot handle long term dependencies

LSTM Recurrent Neural Networks (RNNs) Contain Four Layers (Yellow Box)

Long Short Term Memory Networks (LSTMs) Can Handle Long Term Dependencies

@edeeplearning

Sigmoid Activation Function Scales Your Input Between 0 and 1

http://neuralnetworksanddeeplearning.com/chap4.html

Sigmoid Activation Function Scales Your Input Between 0 and 1

@edeeplearning

Tanh Activation Function Scales Your Input Between -1 and 1

@edeeplearning

RELU Activation Function Cuts off Negative Inputs

@edeeplearning

What About Image Data? For That, Convolutional Neural Networks (Right)

@edeeplearninghttp://cs231n.github.io/convolutional-networks/

Convolution Just Means Picking Out Representative Elements from a Bunch of

Pixels (F represents all the below)

@edeeplearning

This is What Convolution Looks Like, a Little Boring, But Look How It Reduces the Size of

the Data (3x3 on the Right!)

@edeeplearning

So Convolution Is a Mathematical Operation (Using a Weight Filter, Let’s Look!), but Even Simpler is

Max Pooling, just take the Biggest Pixel!

@edeeplearning

A Pre-Trained Image Recognition Neural Network Looks Like This (Input At Left) - Let’s

Zoom In

@edeeplearninghttps://arxiv.org/pdf/1409.4842.pdf

Sam Putnam

@edeeplearning

July 27

Deep Learning in Production Considerations

Help me find the source for this nice diagram? Lost it :(

Sam Putnam

@edeeplearning

July 27

Solution - Use What I Know and Am Allowed To Use, i.e. Kaggle, AWS, TF, Jupyter

Help me find the source for this nice diagram? Lost it :(

Sam Putnam

@edeeplearning

July 27

What I Did - A Real Estate Agent Told Me About His Problem & Pointed Me To The Data

https://www.fortunebuilders.com/how-to-become-a-real-estate-agent/

Sam Putnam

@edeeplearning

July 27

I Took a Holdout Set of the Data and Left the Rest for Train/Validate

https://www.fortunebuilders.com/how-to-become-a-real-estate-agent/

Sam Putnam

@edeeplearning

July 27

Started with Feature Selection and then Linear Regression in Good old Excel, WITH the subject matter

expert (the real estate agent). Did Pretty Well.

https://www.youtube.com/watch?v=SQkpLMLoqww

Sam Putnam

@edeeplearning

July 27

How Well? 18 percent Mean Error and 26 percent Median Error before Feature Selection using Weighted Importance, Did some thinking:

Sam Putnam

@edeeplearning

July 27

Feature Importance, for Feature Selection, Often Use After Deep Learning Step, to Gain Interpretability

Sam Putnam

@edeeplearning

July 27

https://www.kaggle.com/samdeeplearning/naive-subsample-5-10-city/output

GBM (Boosted Decision Tree) - with Hyperparameter Tweaking - Yields Strong Model for this Problem

Sam Putnam

@edeeplearning

July 27

https://www.kaggle.com/samdeeplearning/naive-subsample-0-25-xgb/notebook

Starts to Degrade Upon Subsampling only 1/4 of the Trees

Sam Putnam

@edeeplearning

July 27

Sam Putnam

@edeeplearning

July 27

Want To See If Neural Network Can Improve Performance

Sam Putnam

@edeeplearning

July 27

Building off of a preexisting housing prices regression model

Sam Putnam

@edeeplearning

July 27

Set Input Layer (n_hidden_1 is input layer here) to number of features (27)

Sam Putnam

@edeeplearning

July 27

Combining the features right off the bat by using a wide first hidden layer (200 nodes)

Sam Putnam

@edeeplearning

July 27

Set number of classes to 1 for regression problem

Sam Putnam

@edeeplearning

July 27

Multilayer Perceptron = Feedforward Perceptron We Looked At Before

Sam Putnam

@edeeplearning

July 27

RELU - As Mentioned in Executing Strategies Yesterday, not using negative features/weights

Sam Putnam

@edeeplearning

July 27

Training - Can tell performance needs tweaking, pretty good on houses 2-6

123456

Estimate on LeftPrediction on Left

Sam Putnam

@edeeplearning

July 27

Let’s Dive Into the TensorFlow Housing Code

https://www.kaggle.com/samdeeplearning/deep-neural-network-for-starters-r/edit

Sam Putnam

@edeeplearning

July 27

What Next? Go Deep Learning!

https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6

Sam Putnam

@edeeplearning

July 27

1) Identify an architecture for your problem

Sam Putnam

@edeeplearning

July 27

2) Identify a Framework, Be Willing to Try a New Framework

https://blog.algorithmia.com/deploying-deep-learning-cloud-services/

Sam Putnam

@edeeplearning

July 27

3) Build or tailor a model for your application, Don’t be Afraid to Reference Academic Code

Sam Putnam

@edeeplearning

July 27

4)Tweak the Parameters for your Model

Sam Putnam

@edeeplearning

July 27

5)Train on a (probably, Nvidia, maybe Intel) GPU if need be (for image data, you need it)

https://www.youtube.com/watch?v=2NrgPdGSXhE

Sam Putnam

@edeeplearning

July 27

6) Validate, test, check in the real world. Iterate!

https://medium.com/towards-data-science/train-test-split-and-cross-validation-in-python-80b61beca4b6

Sam Putnam

@edeeplearning

July 27

Weekend trial: Josh Gordon’s Machine Learning Recipes. If the code doesn’t work for you (I am 99 %

sure it will), let me know!

Sam Putnam

@edeeplearning

July 27

Like R? Try this notebook on my Kaggle

Sam Putnam

@edeeplearning

July 27

Submit your predictions, even! (disclaimer: middle of the pack result)

Sam Putnam

Thank you to Google and others who have published diagrams and photos. Slides are for today only.

@edeeplearning

Questions/Comments: Sam@EDeepLearning.com

Thank you

July 27

Deploying Enterprise Deep Learning Masterclass Preview - Enterprise Deep Learning

Software

Deep Learning/AI Lifecycle with Dell EMC and bitfusion · the enterprise with easy attachable elastic GPUs for deep learning development, as well as the creation of a cost effective

Lecture 1: Introduction to Deep Learning - uvadlc.github.io · Lecture 1: Introduction to Deep Learning Efstratios Gavves. UVA DEEP LEARNING COURSE UVA DEEP LEARNING COURSE –EFSTRATIOS

Deep Enterprise Gujarat India

Drishyam.ai: A Deep learning based Image recognition enterprise platform in the Home Improvement domain

An Introduction to Deep Learning · Deep Learning An Introduction to ... •Deep Learning •Feed Forward Network •Problems of Deep Learning •AutoEncoders •Restricted Boltzmann

Deep Learning and Recurrent Neural Networks in the Enterprise

Intro to Deep Learning - bpTT Bright Minds … · 17/08/2020 · Intro to Deep Learning Deep Sequence Modeling Deep Computer Vision Deep Generative Modeling Deep Learning Applications

Introduction to Deep Learning - Computer Graphics€¦ · Introduction to Deep Learning CS468 Spring 2017 Charles Qi. What is Deep Learning? Deep learning allows computational models

Improving Shared GPU Clusters for DL Training Workloadssigfast.or.kr/fiss/fiss19/Myeongjae_Jeon.pdf · Deep Learning at Enterprise •Deep learning (DL) is popular –Speech, Image,

deep freeze Enterprise

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

The Enterprise AI Platform Cross ... - star-cooperation.com · Artificial Intelligence Machine Learning Deep Learning Amazon uses Machine Learning to ship packages before they are

Enterprise Deep Learning with DL4J

Deep Learning and Artistic Style Transfer for Videos - Enterprise Deep Learning

Deep Learning - Oxford Statisticspalamara/teaching/SML19/HT19_lecture15.pdf · SB2b/SM4 - Deep Learning Deep Learning Infrastructure! Computational Infrastructure critical to deep

Deep Learning - South Dakota State UniversityDeep Learning Deep learning (also known as deep structured learning, hierarchical learning or deep machine learning) is a branch of machine

Tutorial Deep Freeze Enterprise

Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Introduction to Deep Learning - Joyce Hojoyceho.github.io/cs534_s17/slide/25-deep-learning.pdf · Deep Learning: Overview ... Review: Backpropogation https: ... • Deep Learning