CS4501: Introduction to Computer VisionNeural Networks +Backpropagation

• Softmax Classifier• Generalization / Overfitting• Pytorch

Last Class

• Global Features• The perceptron model• Neural Networks – multilayer perceptron model (MLP)• Backpropagation

Today’s Class

Prediction

Supervised Machine Learning StepsTraining Labels

Training Images

Training

Training

Image Features

Image Features

Testing

Test Image

Learned model

Learned model

Slide credit: D. Hoiem

5

Supervised Learning –Softmax Classifier

!" = [!"%!"'!"(!")]Extract features

+, = -,%!"% + -,'!"' + -,(!"( + -,)!") + /,+0 = -0%!"% + -0'!"' + -0(!"( + -0)!") + /0+1 = -1%!"% + -1'!"' + -1(!"( + -1)!") + /1

2, = 345/(345+348 + 349)20 = 348/(345+348 + 349)21 = 349/(345+348 + 349)

Run features through classifier

;<" = [2,2021]

Get predictions

6

Last Class:(mini-batch) Stochastic Gradient Descent (SGD)

!(#, %) =(−log./,01230(#, %)/∈5

6 = 0.01

for e = 0, num_epochs do

end

Initialize w and b randomly

:!(#, %)/:# :!(#, %)/:%Compute: and

Update w:

Update b:

# = # − 6:!(#, %)/:#% = % − 6:!(#, %)/:%

Print: !(#, %) // Useful to see if this is becoming smaller or not.

end

for b = 0, num_batches do B is a smallset of trainingexamples.

Generalization• Generalization refers to the ability to correctly classify never before

seen examples• Can be controlled by turning “knobs” that affect the complexity of

the model

Training set (labels known) Test set (labels unknown)

Overfitting

!"## $ is high !"## $ is low !"## $ is zero!

OverfittingUnderfittingHigh Bias High Variance

% is linear % is cubic% is a polynomial of

degree 9

Pytorch: Project Assignment 4

• http://vicenteordonez.com/vision/

Image Features• In your Project 4: Nearest Neighbors + Softmax Classifier features are:

Image: 3x32x32Feature: 3072-dim vector

Image Features: Color

Photo by: marielito

slide by Tamara L. Berg

Image Features: Color

Image Features: Color

However, these are all images of people but thecolors in each image are very different.

Image Features: HoG

Scikit-image implementation

Paper by Navneet Dalal & Bill Triggs presented at CVPR 2005 for detecting people.

Image Features: HoG

Paper by Navneet Dalal & Bill Triggs presented at CVPR 2005 for detecting people.Figure from Zhuolin Jiang, Zhe Lin, Larry S. Davis, ICCV 2009 for human action recognition.

+ Block Normalization

Image Features: GIST

The “gist” of a scene: Oliva & Torralba, 2001

Image Features: GIST

Oriented edge response at multiple scales (5 spatial scales, 6 edge orientations) Hays and Efros,

SIGGRAPH 2007

Image Features: GIST

Aggregated edge responses over 4x4 windowsHays and Efros, SIGGRAPH 2007

Image Features: Bag of (Visual) Words Representation

slide by Fei-fei Li

slide by Fei-fei Li

Summary: Image Features

• Many other features proposed• LBP: Local Binary Patterns: Useful for recognizing faces.• Dense SIFT: SIFT features computed on a grid similar to the HOG features.• etc.

• Largely replaced by Neural networks• Still useful to study for inspiration in designing neural networks that

compute features.

Perceptron ModelFrank Rosenblatt (1957) - Cornell University

More: https://en.wikipedia.org/wiki/Perceptron

! " = $1, if* +,",-

,./+ 1 > 0

0, otherwise

";

"<

"=

">

*

+;+<+=+>

Activationfunction

Perceptron ModelFrank Rosenblatt (1957) - Cornell University

More: https://en.wikipedia.org/wiki/Perceptron

! " = $1, if* +,",-

,./+ 1 > 0

0, otherwise

";

"<

"=

">

*

+;+<+=+>

!?

Perceptron ModelFrank Rosenblatt (1957) - Cornell University

More: https://en.wikipedia.org/wiki/Perceptron

! " = $1, if* +,",-

,./+ 1 > 0

0, otherwise

";

"<

"=

">

*

+;+<+=+>

Activationfunction

Activation Functions

ReLU(x) = max(0, x)Tanh(x)

Sigmoid(x)Step(x)

Pytorch - Perceptron

Two-layer Multi-layer Perceptron (MLP)

!"!#!$!%

&

'"'#'$'%

&

&

&

&

()"

”hidden" layer

("

Loss / Criterion

Forward pass

!"!#!$!%

&

'"'#'$'%

&

&

&

&

()" ("

*+ =& -"+.'+/

+01+ 3" !+ = 567896:(*+)

=" =& -#+!+/

+01+ 3#

(" = 567896:(=+)

>9?? = >((", ()")

Backward pass

!"!#!$!%

&

'"'#'$'%

&

&

&

&

()" ("

*+*,-

= **,-

/012304(,-)*+*!7

898:;

= 88:;

/012304(<-) 898=);

898=);

= 88=);

+((", ()")

*+*'7

= ( **'7& ?"-@'-

A

-BC+ E")

*+*,-

*+*?"-@

= *'7*?"-@

*+*'7

*+*!7

= ( **!7& ?#-!-

A

-BC+ E#)

*+*<"

*+*?#-

= *!7*?#-

*+*!7

GradInputs

GradParams

Pytorch – Two-layer MLP + Regression

Pytorch – Two-layer MLP + LogSoftmax

# of Hidden Units

Pytorch – Two-layer MLP + LogSoftmax

LogSoftmax + Negative Likelihood Loss

Bookmark Opportunity!

Questions?

34