Upload
danny-liu
View
165
Download
15
Embed Size (px)
Citation preview
PowerPoint
Machine Learning, Deep Learning and Data Analysis
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEOutline2Overview of ML, DL and Data Analysis What is Machine LearningTake a Look At Linear RegressionOther ML Algorithms at a GlanceWhat is Neural Network?What is Deep Learning?Deep Learning using TensorFlowData AnalysisCase 1, 2 and 3Multivariate Analysis
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
2
My BackgroundFAB/Mobile Phone/Device
System/Application DevelopmentEDA
Validation ToolData Center
Cloud IaaS / PaaS developmentMaster of Computer and Information Science (Cleveland State University, USA)Master of Industrial Engineering (NCTU, Taiwan) 0.5 year1 year3 yearsNetworking
Switch L2/L3 ProtocolsAnd SDN2 years6 yearsITRI
ML/DLBig Data Analysis
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
My Experience for Machine Learning4!!Hope giving you an experience and guidelineTake courses:Coursera: Machine Learning ( Got Certificate )Udemy: Data Science: Deep Learning in Python ( ongoing)Study on-line resources:YoutubeML/DL tutorials and so onhttps://morvanzhou.github.io/http://bangqu.com/gpu/bloghttp://www.jiqizhixin.com/insightsGet you hands dirtyPython programmingTensorFlow Deep Learning LibraryScikit-Learn LibraryNumby, Pandas, matplotlib,
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
From AI to Deep Learning5: 2017 Google https://www.youtube.com/watch?v=uZ-7DVzRCy8
https://blogs.nvidia.com.tw/2016/07/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/CPU/GPUBig DataAlgorithmsBreakthrough
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML, DL and Data Analysis6
Visually LinkingWhat we focus todayhttps://whatsthebigdata.com/2016/10/17/visually-linking-ai-machine-learning-deep-learning-big-data-and-data-science/
??Data Analysis
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Machine Learning
7
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMachine Learning definition8Arthur Samuel (1959). Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.Tom Mitchell (1998) Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMachine Learning definition9Suppose your email program watches which emails you do or do not mark as spam, and based on that learns how to better filter spam. What is the task T in this setting? Classifying emails as spam or not spam. (T)Watching you label emails as spam or not spam. (E)The number (or fraction) of emails correctly classified as spam/not spam. (P)None of the abovethis is not a machine learning problem
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Machine Learning ?Without writing any custom code specific to the problemFeed data to the generic algorithmIt builds its own logic
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTETwo styles of Machine LearningSupervised Learning
Unsupervised Learning
Use the logic to predict the sales price
figure out if there is a pattern or grouping or something
FeaturesLabel
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat are machine learning algorithms?Regression AlgorithmsLinear RegressionLogistic RegressionLASSODecision Tree AlgorithmsClassification and Regression Tree (CART)Iterative Dichotomiser 3 (ID3)C4.5 and C5.0 (different versions of a powerful approach)Bayesian AlgorithmsNaive BayesClustering Algorithms (unsupervised)k-MeansSupport Vector MachinesPrincipal Component AnalysisAnomaly DetectionRecommender SystemsArtificial Neural Network Algorithms
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELets Take a Look At Linear Regression13
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression14The Hypothesis Function
Cost Function
Gradient Descent for Multiple Variables
https://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEGradient Descent15How to choose learning
https://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEGradient Descent16Convergence of gradient descent with an appropriate learning rate
Cost Function https://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression17Training data with linear regression fit
https://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEOther ML Algorithms at a Glance18
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELogistic Regression19Training data with decision boundary
linear decision boundaryno linear decision boundaryhttps://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTESupport Vector Machines20The difference between the kernels in SVMLinearPolynomialGaussian (RBF)SigmoidSVM (Gaussian Kernel) Decision BoundaryChoose gamma ( auto )
Gaussian (RBF)Non-linear decision boundaryhttps://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEK-Means21The original 128x128 image with 24-bit color (three 8-bit )using K-means (K=16) to use find the 16 colors that best group (cluster) the pixels in the 3-dimensional RGB space.
K=3 and computing centroid means Iterativelyhttps://www.coursera.org/learn/machine-learning/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEPrincipal Component AnalysisAn example to deal with image dimension reduction and proximate recovery.
Faces Dataset
Recovered faces
Principal components
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Neural Network(We will review the previous concepts a little bit)23
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML -- write that program by ourselvesTo estimate the price of a houseIf we could just figure out the perfect weights to use that work for every house, our function could predict house prices!How to do that with ML?def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
price = 0 # a little pinch of this
price += num_of_bedrooms * .841231951398213
price += sqft * 1231.1231231
price += neighborhood * 2.3242341421
price += 201.23432095
return price
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML -- write that program by ourselvesStep 1 Initialize weights to 1.0Step 2See the difference and how far off the function is at guessing the correct price
Step 3Repeat Step 2 over and over with every single possible combination of weights.
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML -- What about that whole try every number bit in Step 3?
is what represents your current weights. J() means the cost for your current weights.Clever waysto quickly find good values for those weights without having to try very many.If we graph this cost equation for all possible values of our weights for number_of_bedroomsandsqftGradient Descent
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
26
Making SmarterGuesses27We ended up with this simple estimation function
def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
price = 0 # a little pinch of this
price += num_of_bedrooms * 0.123
price += sqft * 0.41
price += neighborhood * 0.57
return price
alinearrelationship with the input
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEIf there is more complicated situation?Different of weights for the different house sizes
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is a Neural NetworkNow we have four different price estimates. Lets combine those four price estimates into one final estimate.
neuronsThis is a neural network
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is a Neural NetworkHuman Brains
http://www.slideshare.net/tw_dsconf/ss-62245351
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is a Neural Network
Different connections lead to different structured network.http://www.slideshare.net/tw_dsconf/ss-62245351
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
31
Fully Connected Feedforward Network
http://www.slideshare.net/tw_dsconf/ss-62245351
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEFully Connected Feedforward Network
Deep Learninghttp://www.slideshare.net/tw_dsconf/ss-62245351
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEFully connected feedforward networkMatrix Operation
http://www.slideshare.net/tw_dsconf/ss-62245351
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEOutput LayerSoftmax layer as the output layer
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTENeural Network Playgroundhttp://playground.tensorflow.org/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Deep Learning?
37
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Deep Learning?38Deep Learning is Large Neural NetworksDeep Learning attracts lots of attention
http://static.googleusercontent.com/media/research.google.com/en//people/jeff/BayLearn2015.pdf
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhy Deep Learning?The more data, the more performance.
Game ChangerDL accuracy/performance is more than 99%
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
39
Deep Learning Models40Convolutional Neural NetworkInception-V3Recurrent Neural NetworkLSTMAuto-encoderReinforcement LearningQ-LearningPolicy GradientWide and Deep LearningRecommender system
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEDeep Learning is not so simpleBackpropagation an efficient way to compute Gradient DescentOverfittingChoosing Loss functionSquare Error, Cross Entropy, and so onMini-Batch Too deep ( many hidden layers )ReLU, MaxOut, Learning RatesMomentumAdam ( optimizer )Weight DecayDropout
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEBackpropagation42A common method of trainingartificial neural networksand used in conjunction with anoptimization methodsuch asgradient descent.
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEUnderfitting and OverfittingBias-Variance Tradeoff
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEConvolutional Neural Network (CNN)Why CNN is for image?The first layer of fully connected network would be very large
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEThe solution is Convolution
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEThe solution is Convolution
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEConvolutional Neural Network (CNN)
Adding Even MoreSteps
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
47
Convolutional Neural Network (CNN)
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEDeep Learning using TensorFlow
49
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression in TensorFlow50
X_data array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, Y_data array([ 0. , 0.29999667, 0.59997333, 0.89991 , 1.19978668, 1.49958339, 1.79928013, 2.09885695, 2.39829388, 2.69757098, 2.99666833, 3.29556602,
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression in TensorFlow51
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression in TensorFlow52
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMNIST53MNIST dataset55000 samples (50000 for training, 5000 for testing)For each sample, it has X, y parts. X are the image with 28*28 pixels in 8 bit gray scaleY is the label answer: 0, 1, 2, , 9
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMNIST54X, y can be represented as follows
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMNIST55
If you want to get the accuracy more than 99%, check it out:https://gotocon.com/dl/goto-london-2016/slides/MartinGorner_TensorflowAndDeepLearningWithoutAPhD.pdf92%
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEImage Recognition and Retraining Inception-v3 model is ready and made by Googleit took Google researchers two weeks to build on a desktop with eight NVidia Tesla K40s.It can recognize > 1000 categoriesRetrainingTo prepare the new images and categoriesDo training and testing
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEPlate Number RecognitionThere is an example using UKs Plate Number and Character to train TensorFlow CNN modelTake 3 days with GPU Card (GTX 750 TI)
http://matthewearl.github.io/2016/05/06/cnn-anpr/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTETechnical breakthrough for Deep-ANPR
http://matthewearl.github.io/2016/05/06/cnn-anpr/
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEAutoencoderEnccode the input data (MNIST data) and then decode it backIt is similar to PCA
autoencoderoriginal
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELSTM (RNN)It is a special kind of RNN, capable of learning long-term dependencies
LSTM Training with
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTERecurrent Neural Network61
RNN Modeltraining
outputPlay this map with Super Mario Makerhttps://medium.com/@ageitgey/machine-learning-is-fun-part-2-a26a10b68df3
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
61
GPU62Setup NVIDIA Quadro M6000 GPU CardInstall driver (NVIDIA-Linux-x86_64-375.26.run)CUDA Tookit 8.0 (cuda-repo-ubuntu1404-8-0-local_8.0.44-1_amd64.deb)CuDNN v5.1 (cudnn-8.0-linux-x64-v5.1.tgz)The M6000s performance could be affected by not enough power supplyOnly 1.3 ~ 1.7 Times faster than GTX 750 Ti
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE63
Quadro M6000GTX 750 Ti
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEAOI Defect Classification64
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEA deepCNN model for AOI65Image recognition model, Inception-v3To achieve reasonable performance on hard visual recognition tasks
Defect Classification
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE(Big) Data Analysis
66
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis67The steps to do data analysisData CollectionFrom CSV files, database, and so on.Data Preprocessing ( very very very important )Regularization, Normalization, Table Join, Feature ExtractionReduce the dimensions .Feature SelectionSelect the important featuresMachine Learning / Deep LearningTo train the modelTo do the Prediction and Classification by the trained modelApply or implement to systemBut, still needs:domain experts involved!!Studying related papers and researches
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEAnalysis Tools and Libraries68Open Sources(Python)Machine LearningSciKit-LearnNumPyMatplotlibPandasDeep LearningTensorFlowKerasHadoop & SparkCommercial Software ( rare to use)PolyAnalyst 6.5SAS
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis69In my experience with data analysis, I belong to a Rookie
http://www.slideshare.net/tw_dsconf/ss-71780267
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis Case 1
70
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEESL in Details71Trainingpython nn_esl1.py --do_training True --epoch 350Prediction:python nn_esl1.py --x1 10 --x2 25 --x3 0.2 --x4 20 --epoch 350('Result: ', array([[ 145.28657532, 193.63188171]], dtype=float32))Original Label Result Data: [145.69, 190.95]5%Feature SelectionThreshold = 0.25 array([0, 1]) D, PThreshold = 0.1 array([0, 1, 3])D, P, H
> 500 raw data
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE72
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE73ESL Web App Architecture 73Dell Server(Ubuntu 14.04) VirtualBoxVirtual Machine (Ubuntu 14.04)NVIDIA Quadro M6000 GPU Card (12GB)
ESL Web App
UsersTensorFlowTrained Model
TensorFlow
2Apply the trained ESL Model
3Query
Criteria: Scenario, Timing, Power, Electrical-SI/Pi, Thermal, Mechanical
1Train ESL Model Output:
2.5D TSVPoP2.5D Fan-OutInFO PoP
4Predict the module design
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis Case 2
74
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE75Bond Time, Power, and Force Pull
#9(y = x * w + b) w = np.array([[ 0.31186092], [ 0.26080179], [ 0.19781925]])b = np.array([[ 0.22946352]])
# Normalization $ teX = np.array([[150/np.max(data[:,0]), 95/np.max(data[:,1]), 500/np.max(data[:,2])]]) array([[ 0.83333333, 1.0, 0.71428571]])# Prediction $ teX.dot(w) + barray([[ 0.89144887]])# $ (teX.dot(w) + b) * 952array([[ 848.65932832]]) : 842
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE76
9Pull
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMultivariate Analysis
77
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMultivariate Analysis78
http://www.slideshare.net/tw_dsconf/ss-71780267
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEReferenceMachine Learning is Fun! MediumMachine Learning is Fun! Part 2 MediumMachine Learning is Fun! Part 3: Deep Learning and ... - MediumDeep Learning TutorialFIRST CONTACT WITH TENSORFLOWhttps://ireneli.eu/2016/02/03/deep-learning-05-talk-about-convolutional-neural-network%EF%BC%88cnn%EF%BC%89/http://www.slideshare.net/tw_dsconf/ss-71780267https://morvanzhou.github.io/tutorials/python-basic/https://media.readthedocs.org/pdf/python-for-multivariate-analysis/latest/python-for-multivariate-analysis.pdfhttp://blog.topspeedsnail.com/http://www.leiphone.com/news/201702/vJpJqREn7EyoAd09.htmlPython scikit-learnhttps://machine-learning-python.kspax.io/version >= 0.17
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEThank You80
,
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE