CS 4803 / 7643: Deep Learning · “2-layer Neural Net”, or “1-hidden-layer Neural Net” “3-layer Neural Net”, or “2-hidden-layer Neural Net” Slide Credit: Fei-FeiLi,

CS 4803 / 7643: Deep Learning

Zsolt Kira Georgia Tech

Topics: – Neural Networks– Computing Gradients

– (Finish) manual differentiation

Administrivia• PS1/HW1 out

• Instructor Office hours: 10-11am Mondays CODA 11th floor, conference room C1106 Briarcliff– This week only: Thursday 11am (same room)

• Start thinking about project topics/teams

(C) Dhruv Batra & Zsolt Kira 2

Do the Readings!


Recap from last time


(Before) Linear score function:

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Neural networks: without the brain stuff

6


(Now) 2-layer Neural Networkor 3-layer Neural Network

Neural networks: without the brain stuff


● Not really how neurons work (they are more complex)● This is the same thing as● Started with binary output (0/1) depending on whether sum is larger than 0

(or threshold if no bias is used)● Still want to add non-linearity to increase representational power

7

Impulses carried toward cell body

Impulses carried away from cell body

This image by Felipe Peruchois licensed under CC-BY 3.0

dendrite

cell body

axon

presynaptic terminal


8

sigmoid activation function

Impulses carried toward cell body

Impulses carried away from cell body

This image by Felipe Peruchois licensed under CC-BY 3.0

dendrite

cell body

axon

presynaptic terminal


Sigmoid

tanh

ReLU

Leaky ReLU

Maxout

ELU


Activation functions

With ReLU it’s what we showed before

“Fully-connected” layers“2-layer Neural Net”, or“1-hidden-layer Neural Net”

“3-layer Neural Net”, or“2-hidden-layer Neural Net”


Neural networks: Architectures

Representational Power: • Any continuous functions with 1 hidden layer• Any function with 2 hidden layers• Doesn’t say how many nodes we’ll need!

11


(Now) 2-layer Neural Networkor 3-layer Neural Network

Equivalent to Math Viewpoint!


x hW1 sW2

3072 100 10

Strategy: Follow the slope

In 1-dimension, the derivative of a function:

In multiple dimensions, the gradient is the vector of (partial derivatives) along each dimension

The slope in any direction is the dot product of the direction with the gradientThe direction of steepest descent is the negative gradient


Gradient Descent Pseudocodefor i in {0,…,num_epochs}:

for x, y in data:𝑦 𝑆𝑀 𝑊𝑥𝐿 𝐶𝐸 𝑦, 𝑦

? ? ?

𝑊 ≔ 𝑊 𝛼

Some design decisions:• How many examples to use to calculate gradient per iteration? • What should alpha (learning rate) be?

• Should it be constant throughout?• How many epochs to run to?

Full sum expensive when N is large!

Approximate sum using a minibatch of examples32 / 64 / 128 common


Stochastic Gradient Descent (SGD)

How do we compute gradients?• Analytic or “Manual” Differentiation

• Symbolic Differentiation

• Numerical Differentiation

• Automatic Differentiation– Forward mode AD– Reverse mode AD

• aka “backprop”


gradient dW:

[-2.5,?,?,?,?,?,?,?,?,…]

(1.25322 - 1.25347)/0.0001= -2.5

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347


W + h (first dim):

[0.34 + 0.0001,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25322

Manually Computing Gradients

17

• Our function: linear with sigmoid activation

• Definition of sigmoid

• Sigmoid has a nice derivative!

• Start with our loss function• Calculate derivative:

2*inside*derivative of inside• Also get rid of y (constant!) leaving

a negative sign for rest

• Move negative out, include derivative of g (g’) and multiple by derivative of inside

• Define some terms

• Write with new terms and substitute derivative of sigmoid in

Plan for Today• Decomposing a function• Backpropagation algorithm (w/ example)• Math

– Function composition– Vectors– Tensors


How to Simplify? • Calculating gradients for large functions is

complicated

• Idea: Decompose the function and compute local gradients for each part!


Computational Graphs• Notation


Example


+

sin( )

x1 x2

*

x

W

hinge loss

R

+ Ls (scores)

*


Computational Graph

Logistic Regression as a Cascade

(C) Dhruv Batra and Zsolt Kira 23

Given a library of simple functions

Compose into a

complicate function

Slide Credit: Marc'Aurelio Ranzato, Yann LeCun

input image

loss

weights

Figure copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.

Convolutional network (AlexNet)


Figure reproduced with permission from a Twitter post by Andrej Karpathy.

input image

loss

Neural Turing Machine


How do we compute gradients?• Analytic or “Manual” Differentiation

• Symbolic Differentiation

• Numerical Differentiation

• Automatic Differentiation– Forward mode AD– Reverse mode AD

• aka “backprop”


Any DAG of differentiable modules is allowed!

Slide Credit: Marc'Aurelio Ranzato(C) Dhruv Batra & Zsolt Kira 27

Computational Graph

Directed Acyclic Graphs (DAGs)• Exactly what the name suggests

– Directed edges– No (directed) cycles– Underlying undirected cycles okay

(C) Dhruv Batra 28

Directed Acyclic Graphs (DAGs)• Concept

– Topological Ordering

(C) Dhruv Batra 29

Directed Acyclic Graphs (DAGs)

(C) Dhruv Batra 30

Key Computation: Forward-Prop

(C) Dhruv Batra & Zsolt Kira 31Slide Credit: Marc'Aurelio Ranzato, Yann LeCun

Key Computation: Back-Prop


How to Simplify? • Calculating gradients for large functions is

complicated

• Idea: Decompose the function and compute local gradients for each part!

• We will use the chain rule to calculate the gradients– We will receive an upstream gradient from layer after us– We will then use that to compute local gradients using chain

rule


Neural Network Training• Step 1: Compute Loss on mini-batch [F-Pass]






Neural Network Training• Step 1: Compute Loss on mini-batch [F-Pass]• Step 2: Compute gradients wrt parameters [B-Pass]






Neural Network Training• Step 1: Compute Loss on mini-batch [F-Pass]• Step 2: Compute gradients wrt parameters [B-Pass]• Step 3: Use gradient to update parameters


41


Backpropagation: a simple example

42



43

e.g. x = -2, y = 5, z = -4



44

e.g. x = -2, y = 5, z = -4

Want:



45

e.g. x = -2, y = 5, z = -4

Want:



46

e.g. x = -2, y = 5, z = -4

Want:



47

e.g. x = -2, y = 5, z = -4

Want:



48

e.g. x = -2, y = 5, z = -4

Want:



49

e.g. x = -2, y = 5, z = -4

Want:



50

e.g. x = -2, y = 5, z = -4

Want:



51

e.g. x = -2, y = 5, z = -4

Want:



52

e.g. x = -2, y = 5, z = -4

Want:



53

e.g. x = -2, y = 5, z = -4

Want:

Chain rule:

Upstream gradient

Localgradient



54

Chain rule:

e.g. x = -2, y = 5, z = -4

Want: Upstream gradient

Localgradient



55

e.g. x = -2, y = 5, z = -4

Want:

Chain rule:

Upstream gradient

Localgradient



56

Chain rule:

e.g. x = -2, y = 5, z = -4

Want: Upstream gradient

Localgradient



Patterns in backward flow




add gate: gradient distributor



add gate: gradient distributorQ: What is a max gate?



add gate: gradient distributormax gate: gradient router



add gate: gradient distributormax gate: gradient routerQ: What is a mul gate?



add gate: gradient distributormax gate: gradient routermul gate: gradient switcher



(C) Dhruv Batra and Zsolt Kira 64

Summary• We will have a composed non-linear function as our model

– Several portions will have parameters

• We will use (stochastic/mini-batch) gradient descent with a loss function to define our objective

• Rather than analytically derive gradients for complex function, we will modularize computation

– Back propagation = Gradient Descent + Chain Rule

• Now:– Work through mathematical view– Vectors, matrices, and tensors– Next time: Can the computer do this for us automatically?

• Read:– https://explained.ai/matrix-calculus/index.html– https://www.cc.gatech.edu/classes/AY2020/cs7643_fall/slides/L5_gradients

_notes.pdf

Matrix/Vector Derivatives Notation• Read:

– https://explained.ai/matrix-calculus/index.html– https://www.cc.gatech.edu/classes/AY2020/cs7643_fall/slide

s/L5_gradients_notes.pdf

• Matrix/Vector Derivatives Notation• Vector Derivative Example• Extension to Tensors• Chain Rule: Composite Functions

– Scalar Case– Vector Case– Jacobian view– Graphical view– Tensors

• Logistic Regression Derivatives


Documents

CS 4803 / 7643: Deep Learning · “2-layer Neural Net”, or “1-hidden-layer Neural Net” “3-layer Neural Net”, or “2-hidden-layer Neural Net” Slide Credit: Fei-FeiLi,