CS 1699: Intro to Computer Vision Introductionkovashka/cs1699_fa15/... · Quadratic optimization problem: Minimize Subject to y i (w·x i +b) ≥ 1 wT w 2 1 negative ( 1) : 1 positive

CS 1699: Intro to Computer Vision

Bias-Variance Trade-off +Other Models and Problems

Prof. Adriana KovashkaUniversity of Pittsburgh

November 3, 2015

Outline

• Support Vector Machines (review + other uses)

• Bias-variance trade-off

• Scene recognition: Spatial pyramid matching

• Other classifiers– Decision trees

– Hidden Markov models

• Other problems– Clustering: agglomerative clustering

– Dimensionality reduction

Lines in R2

0 bxw

c

aw

y

xx

0 bcyax

Let

w

00, yx

D

w

xw ||

22

00 b

ca

bcyaxD

distance from

point to line

Kristen Grauman

Support vector machines

• Want line that maximizes the margin.

1:1)(negative

1:1)( positive

by

by

iii

iii

wxx

wxx

Support vectors

For support, vectors, 1 bi wx

Distance between point

and line: ||||

||

w

wx bi

www

211

M

ww

xw 1

bΤ

For support vectors:

C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998

Margin

http://www.umiacs.umd.edu/~joseph/support-vector-machines4.pdf

Finding the maximum margin line

1. Maximize margin 2/||w||

2. Correctly classify all training data points:

Quadratic optimization problem:

Minimize

Subject to yi(w·xi+b) ≥ 1

wwT

2

1

1:1)(negative

1:1)( positive

by

by

iii

iii

wxx

wxx

One constraint for each

training point.

Note sign trick.



Finding the maximum margin line

• Solution:

b = yi – w·xi (for any support vector)

• Classification function:

• Notice that it relies on an inner product between the test

point x and the support vectors xi

• (Solving the optimization problem also involves

computing the inner products xi · xj between all pairs of

training points)

i iii y xw

by

xf

ii

xx

xw

i isign

b)(sign )(

If f(x) < 0, classify as negative, otherwise classify as positive.



• Datasets that are linearly separable work out great:

• But what if the dataset is just too hard?

• We can map it to a higher-dimensional space:

0 x

0 x

0 x

x2

Nonlinear SVMs

Andrew Moore

Examples of kernel functions

Linear:

Gaussian RBF:

Histogram intersection:

)2

exp()(2

2

ji

ji

xx,xxK

k

jiji kxkxxxK ))(),(min(),(

j

T

iji xxxxK ),(

Andrew Moore

Allowing misclassifications

Maximize margin Minimize misclassification

Slack variable

The w that minimizes…

Misclassification cost

# data samples

What about multi-class SVMs?

• Unfortunately, there is no “definitive” multi-class SVM formulation

• In practice, we have to obtain a multi-class SVM by combining multiple two-class SVMs

• One vs. others– Training: learn an SVM for each class vs. the others– Testing: apply each SVM to the test example, and assign it to the

class of the SVM that returns the highest decision value

• One vs. one– Training: learn an SVM for each pair of classes– Testing: each learned SVM “votes” for a class to assign to the

test example

Svetlana Lazebnik

Evaluating Classifiers

• Accuracy

– # correctly classified / # all test examples

• Precision/recall

– Precision = # predicted true pos / # predicted pos

– Recall = # predicted true pos / # true pos

• F-measure

= 2PR / (P + R)

• Want evaluation metric to be in some range, e.g. [0 1]

– 0 = worst possible classifier, 1 = best possible classifier

Precision / Recall / F-measure

• Precision = 2 / 5 = 0.4

• Recall = 2 / 4 = 0.5

• F-measure = 2*0.4*0.5 / 0.4+0.5 = 0.44

True positives

(images that contain people)

True negatives

(images that do not contain people)

Predicted positives

(images predicted to contain people)

Predicted negatives

(images predicted not to contain people)

Accuracy: 5 / 10 = 0.5

Support Vector Regression

Regression

Regression is like classification except the labels are real valued

Example applications:

‣ Stock value prediction

‣ Income prediction

‣ CPU power consumption

Subhransu Maji

Regularized Error Function for Regression

N

n

nn wty1

22

2}{

N

n

nn wtxyEC1

2

2))((

In linear regression, we minimize the error function:

Use the Є-insensitive error function:

An example of Є-insensitive error function:

Adapted from Huan Liu

|)(|

0

yxfE

E |)(| yxffor

otherwise

true value predicted value

Slack Variables for Regression

nnn yty

For a target point to lie

inside the tube:

Introduce slack variables

to allow points to lie

outside the tube:

nnn

nnn

xyt

xyt

)(

)(

Adapted from Huan Liu

0

0

n

n

Subject to:

N

n

nn wC1

2

2

1)(

Minimize:

Next time:Support Vector Ranking

Outline








Generalization

• How well does a learned model generalize from

the data it was trained on to a new test set?

Training set (labels known) Test set (labels

unknown)

Slide credit: L. Lazebnik

Generalization• Components of generalization error

– Bias: how much the average model over all training sets differs

from the true model

• Error due to inaccurate assumptions/simplifications made by

the model

– Variance: how much models estimated from different training

sets differ from each other

• Underfitting: model is too “simple” to represent all the

relevant class characteristics

– High bias and low variance

– High training error and high test error

• Overfitting: model is too “complex” and fits irrelevant

characteristics (noise) in the data

– Low bias and high variance

– Low training error and high test error


Bias-Variance Trade-off

• Models with too few parameters are inaccurate because of a large bias (not enough flexibility).

• Models with too many parameters are inaccurate because of a large variance (too much sensitivity to the sample).

Slide credit: D. Hoiem

Fitting a model

Figures from Bishop

Is this a good fit?

With more training data

Figures from Bishop

Bias-variance tradeoff

Training error

Test error

Underfitting Overfitting

Complexity Low Bias

High Variance

High Bias

Low Variance

Err

or


Bias-variance tradeoff

Many training examples

Few training examples

Complexity Low Bias

High Variance

High Bias

Low Variance

Test E

rror


Choosing the trade-off

• Need validation set

• Validation set is separate from the test set

Training error

Test error

Complexity Low Bias

High Variance

High Bias

Low Variance

Err

or


Effect of Training Size

Testing

Training

Generalization Error

Number of Training Examples

Err

or

Fixed prediction model

Adapted from D. Hoiem

How to reduce variance?

• Choose a simpler classifier

• Use fewer features

• Get more training data

• Regularize the parameters


Regularization

Figures from Bishop

Characteristics of vision learning problems

• Lots of continuous features

– Spatial pyramid may have ~15,000 features

• Imbalanced classes

– Often limited positive examples, practically infinite negative examples

• Difficult prediction tasks

• Recently, massive training sets became available

– If we have a massive training set, we want classifiers with low bias (high variance is ok) and reasonably efficient training


Remember…

• No free lunch: machine learning algorithms are tools

• Three kinds of error

– Inherent: unavoidable

– Bias: due to over-simplifications

– Variance: due to inability to perfectly estimate parameters from limited data

• Try simple classifiers first

• Better to have smart features and simple classifiers than simple features and smart classifiers

• Use increasingly powerful classifiers with more training data (bias-variance tradeoff)


Outline








Beyond Bags of Features: Spatial Pyramid Matching

for Recognizing Natural Scene Categories

CVPR 2006

Svetlana Lazebnik ([email protected])

Beckman Institute, University of Illinois at Urbana-Champaign

Cordelia Schmid ([email protected])

INRIA Rhône-Alpes, France

Jean Ponce ([email protected])

Ecole Normale Supérieure, France

http://www-cvr.ai.uiuc.edu/ponce_grp

http://www-cvr.ai.uiuc.edu/ponce_grp

Bags of words


1. Extract local features

2. Learn “visual vocabulary” using clustering

3. Quantize local features using visual vocabulary

4. Represent images by frequencies of “visual words”

Bag-of-features steps


…

Local feature extraction

Slide credit: Josef Sivic

Learning the visual vocabulary

…



Clustering

…



Clustering

…


Visual vocabulary

Image categorization with bag of words

Training1. Extract bag-of-words representation

2. Train classifier on labeled examples using histogram values as features

Testing1. Extract keypoints/descriptors

2. Quantize into visual words using the clusters computed at training time

3. Compute visual word histogram

4. Compute label using classifier


What about spatial layout?

All of these images have the same color histogramSlide credit: D. Hoiem

Spatial pyramid

Compute histogram in each spatial bin


Spatial pyramid

[Lazebnik et al. CVPR 2006]Slide credit: D. Hoiem

http://www.di.ens.fr/sierra/pdfs/cvpr06b.pdf

Level 2

Level 1

Level 0

Feature histograms:

Level 3

Total weight (value of pyramid match kernel):

Pyramid matchingIndyk & Thaper (2003), Grauman & Darrell (2005)

Matching using pyramid and histogram intersection for some particular visual word:

Original images

Adapted from L. Lazebnik

xi xj

K( xi , xj )

Scene category dataset

Fei-Fei & Perona: 65.2%

Multi-class classification results (100 training images per class)

Fei-Fei & Perona (2005), Oliva & Torralba (2001)

http://www-cvr.ai.uiuc.edu/ponce_grp/data


http://www-cvr.ai.uiuc.edu/ponce_grp/data

Scene category confusions

Difficult indoor images

kitchen living room bedroomSlide credit: L. Lazebnik

Caltech101 dataset

Multi-class classification results (30 training images per class)

Fei-Fei et al. (2004)

http://www.vision.caltech.edu/Image_Datasets/Caltech101/Caltech101.html


http://www.vision.caltech.edu/Image_Datasets/Caltech101/Caltech101.html

Outline








Decision tree classifier

Example problem: decide whether to wait for a table at a restaurant, based on the following attributes:1. Alternate: is there an alternative restaurant nearby?

2. Bar: is there a comfortable bar area to wait in?

3. Fri/Sat: is today Friday or Saturday?

4. Hungry: are we hungry?

5. Patrons: number of people in the restaurant (None, Some, Full)

6. Price: price range ($, $$, $$$)

7. Raining: is it raining outside?

8. Reservation: have we made a reservation?

9. Type: kind of restaurant (French, Italian, Thai, Burger)

10. WaitEstimate: estimated waiting time (0-10, 10-30, 30-60, >60)






Sequence Labeling Problem

• Unlike most computer vision problems, many

NLP problems can viewed as sequence labeling.

• Each token in a sequence is assigned a label.

• Labels of tokens are dependent on the labels of

other tokens in the sequence, particularly their

neighbors (not i.i.d).

foo bar blam zonk zonk bar blam

Adapted from Ray Mooney

Markov Model / Markov Chain

• A finite state machine with probabilistic

state transitions.

• Makes Markov assumption that next state

only depends on the current state and

independent of previous history.

Ray Mooney

Sample Markov Model for POS

0.95

0.05

0.9

0.05stop

0.5

0.1

0.8

0.1

0.1

0.25

0.25

start0.1

0.5

0.4

Det Noun

PropNoun

Verb

Ray Mooney

Sample Markov Model for POS

0.95

0.05

0.9

0.05stop

0.5

0.1

0.8

0.1

0.1

0.25

0.25

start0.1

0.5

0.4

Det Noun

PropNoun

Verb

P(PropNoun Verb Det Noun) = 0.4*0.8*0.25*0.95*0.1=0.0076

Adapted from Ray Mooney “Mary ate the cake.”

Hidden Markov Model

• Probabilistic generative model for sequences.

• Assume an underlying set of hidden (unobserved) states in which the model can be (e.g. parts of speech).

• Assume probabilistic transitions between states over time (e.g. transition from POS to another POS as sequence is generated).

• Assume a probabilistic generation of tokens from states (e.g. words generated for each POS).

Ray Mooney

Sample HMM for POS

PropNoun

JohnMaryAlice

Jerry

Tom

Noun

catdog

carpen

bedapple

Det

a thethe

the

that

athea

Verb

bit

ate sawplayed

hit

0.95

0.05

0.9

gave0.05

stop

0.5

0.1

0.8

0.1

0.1

0.25

0.25

start0.1

0.5

0.4

Ray Mooney

Outline









Clustering Strategies

• K-means– Iteratively re-assign points to the nearest cluster

center

• Mean-shift clustering– Estimate modes

• Graph cuts– Split the nodes in a graph based on assigned links with

similarity weights

• Agglomerative clustering– Start with each point as its own cluster and iteratively

merge the closest clusters

Agglomerative clustering






How to define cluster similarity?- Average distance between points, maximum

distance, minimum distance

How many clusters?- Clustering creates a dendrogram (a tree)

- Threshold based on max number of clusters or based on distance between merges

dis

tan

ce

Adapted from J. Hays

Why do we cluster?

• Summarizing data– Look at large amounts of data– Represent a large continuous vector with the cluster number

• Counting– Histograms of texture, color, SIFT vectors

• Segmentation– Separate the image into different regions

• Prediction– Images in the same cluster may have the same labels

Slide credit: J. Hays, D. Hoiem


Dimensionality Reduction

Figure from Genevieve Patterson, IJCV 2014

Documents

CS 1699: Intro to Computer Vision Introductionkovashka/cs1699_fa15/... · Quadratic optimization problem: Minimize Subject to y i (w·x i +b) ≥ 1 wT w 2 1 negative ( 1) : 1 positive