Setting up your ML application
Train/dev/testsetsdeeplearning.ai
Andrew Ng
Applied ML is a highly iterative process
Idea
Experiment Code
# layers
# hidden units
learning rates
activation functions
…
Andrew Ng
Train/dev/test sets
Andrew Ng
Mismatched train/test distribution
Training set:Cat pictures from webpages
Dev/test sets:Cat pictures from users using your app
Not having a test set might be okay. (Only dev set.)
Setting up your ML application
Bias/Variancedeeplearning.ai
Andrew Ng
Bias and Variance
high bias “just right” high variance
Andrew Ng
Bias and Variance
Cat classification
Train set error:
Dev set error:
Andrew Ng
High bias and high variance
!"
!#
Setting up your ML application
Basic “recipe” for machine learningdeeplearning.ai
Andrew Ng
Basic recipe for machine learning
Regularizing your neural network
Regularizationdeeplearning.ai
Andrew Ng
Logistic regression
min$,& '(), *)
Andrew Ng
Neural network
Regularizing your neural network
Why regularization reduces overfittingdeeplearning.ai
Andrew Ng
How does regularization prevent overfitting?
!"!#!$
%&
high bias “just right” high variance
Andrew Ng
How does regularization prevent overfitting?
Regularizing your neural network
Dropoutregularizationdeeplearning.ai
Andrew Ng
Dropout regularization
!"!#
$%
!&
!'
!"!#
$%
!&
!'
Andrew Ng
Implementing dropout (“Inverted dropout”)
Andrew Ng
Making predictions at test time
Regularizing your neural network
Understandingdropoutdeeplearning.ai
Andrew Ng
Why does drop-out work?
Intuition: Can’t rely on any one feature, so have to spread out weights.
!"!# $%
!&
Regularizing your neural network
Other regularizationmethodsdeeplearning.ai
Andrew Ng
Data augmentation
4
Andrew Ng
Early stopping
# iterations
Setting up your optimization problem
Normalizing inputsdeeplearning.ai
Andrew Ng
Normalizing training sets
!"
!# 5
3
!#
!"
!#
!"
Andrew Ng
Why normalize inputs?! ", $ = 1
'(*
+,-ℒ(01 + , 0(+))
"
$
"
$
"$
!Unnormalized:
"$
!Normalized:
Vanishing/explodinggradientsdeeplearning.ai
Setting up your optimization problem
Andrew Ng
Vanishing/exploding gradients!"!# =
$%
Andrew Ng
Single neuron example!"!#!$!%
&'( = *(,)
Numerical approximationof gradientsdeeplearning.ai
Setting up your optimization problem
Andrew Ng
Checking your derivative computation
!
Andrew Ng
Checking your derivative computation
!
Gradient Checkingdeeplearning.ai
Setting up your optimization problem
Andrew Ng
Gradient check for a neural network
Take ! " , $["], … ,! ( , $ ( and reshape into a big vector ).
Take +! " , +$["], … , +! ( , +$ ( and reshape into a big vector d).
Andrew Ng
Gradient checking (Grad check)
Gradient Checkingimplementation notesdeeplearning.ai
Setting up your optimization problem
Andrew Ng
Gradient checking implementation notes
- Don’t use in training – only to debug
- If algorithm fails grad check, look at components to try to identify bug.
- Remember regularization.
- Doesn’t work with dropout.
- Run at random initialization; perhaps again after some training.