Embed Size (px)
Citation preview
CS 1699: Intro to Computer Vision
Intro to Machine Learning & Visual Recognition – Part II
Prof. Adriana KovashkaUniversity of Pittsburgh
October 27, 2015
Homework
• HW3: push back to Nov. 3
• HW4: push back to Nov. 24
• HW5: make half-length, still 10% of overall grade, out Nov. 24, due Dec. 10
• Take out a small piece of paper and vote yes/no on this proposal
Other announcements
• Piazza
– Can get participation credit by asking/answering
• Feedback from surveys
Plan for today
• Visual recognition problems
• Recognition pipeline
– Features and data
– Challenges
• Overview of some methods for classification
• Challenges and trade-offs
Some translations
• Feature vector = descriptor = representation
• Recognition often involves classification
• Classes = categories (hence classification = categorization)
• Training = learning a model (e.g. classifier), happens at training time from training data
• Classification = prediction, happens at test time
Classification
• Given a feature representation for images,
learn a model for distinguishing features from
different classes
Zebra
Non-zebra
Decision
boundary
Slide credit: L. Lazebnik
Image categorization
• Cat vs Dog
Slide credit: D. Hoiem
Image categorization
• Object recognition
Caltech 101 Average Object ImagesSlide credit: D. Hoiem
Image categorization
• Place recognition
Places Database [Zhou et al. NIPS 2014]Slide credit: D. Hoiem
Region categorization
• Material recognition
[Bell et al. CVPR 2015]Slide credit: D. Hoiem
Recognition: A machine
learning approach
The machine learning
framework
• Apply a prediction function to a feature representation of
the image to get the desired output:
f( ) = “apple”
f( ) = “tomato”
f( ) = “cow”Slide credit: L. Lazebnik
The machine learning
framework
y = f(x)
• Training: given a training set of labeled examples {(x1,y1),
…, (xN,yN)}, estimate the prediction function f by minimizing
the prediction error on the training set
• Testing: apply f to a never before seen test example x and
output the predicted value y = f(x)
output prediction
function
image
feature
Slide credit: L. Lazebnik
Prediction
Steps
Training
LabelsTraining
Images
Training
Training
Image
Features
Image
Features
Testing
Test Image
Learned
model
Learned
model
Slide credit: D. Hoiem and L. Lazebnik
BOARD
Popular global image features
• Raw pixels (and simple
functions of raw pixels)
• Histograms, bags of features
• GIST descriptors [Oliva and
Torralba, 2001]
• Histograms of oriented gradients
(HOG) [Dalal and Triggs, 2005]
Slide credit: L. Lazebnik
• Color
• Texture (filter banks or HOG over regions)L*a*b* color space HSV color space
What kind of things do we compute histograms of?
Slide credit: D. Hoiem
What kind of things do we compute histograms of?• Histograms of descriptors
• “Bag of visual words”
SIFT – [Lowe IJCV 2004]
Slide credit: D. Hoiem
Prediction
Steps
Training
LabelsTraining
Images
Training
Training
Image
Features
Image
Features
Testing
Test Image
Learned
model
Learned
model
Slide credit: D. Hoiem and L. Lazebnik
Recognition training data• Images in the training set must be annotated with the
“correct answer” that the model is expected to produce
“Motorbike”
Slide credit: L. Lazebnik
Challenges: robustness
Illumination Object pose Clutter
ViewpointIntra-class
appearanceOcclusions
Slide credit: K. Grauman
Challenges: importance of context
Slide credit: Fei-Fei, Fergus & Torralba
Painter identification
• How would you learn to identify the author of a painting?
Goya Kirchner Klimt Marc Monet Van Gogh
Plan for today
• Visual recognition problems
• Recognition pipeline
– Features and data
– Challenges
• Overview of some methods for classification
• Challenges and trade-offs
One way to think about it…
• Training labels dictate that two examples are the same or different, in some sense
• Features and distances define visual similarity
• Goal of training is to learn feature weights so that visual similarity predicts label similarity– Linear classifier: confidence in positive label is a weighted
sum of features – What are the weights?
• We want the simplest function that is confidently correct
Adapted from D. Hoiem
BOARD
Supervised classification
• Given a collection of labeled examples, come up with a function that will predict the labels of new examples.
• How good is some function that we come up with to do the classification?
• Depends on– Mistakes made
– Cost associated with the mistakes
“four”
“nine”
?Training examples Novel input
Slide credit: K. Grauman
Supervised classification
• Given a collection of labeled examples, come up with a function that will predict the labels of new examples.
• Consider the two-class (binary) decision problem– L(4→9): Loss of classifying a 4 as a 9
– L(9→4): Loss of classifying a 9 as a 4
• Risk of a classifier s is expected loss:
• We want to choose a classifier so as to minimize this total risk.
49 using|49Pr94 using|94Pr)( LsLssR
Slide credit: K. Grauman
Supervised classification
Feature value x
Optimal classifier will minimize total risk.
At decision boundary, either choice of label yields same expected loss.
If we choose class “four” at boundary, expected loss is:
If we choose class “nine” at boundary, expected loss is:
So, best decision boundary is at point x where
4)(4) | 4 is (class4)(9 )|9 is class( LPLP xx
9)(4 )|4 is class( LP x
9)(4) |4 is P(class4)(9 )|9 is class( LLP xx
Slide credit: K. Grauman
Supervised classification
To classify a new point, choose class with lowest expected loss; i.e., choose “four” if
)94()|4()49()|9( LPLP xx
Feature value x
Optimal classifier will minimize total risk.
At decision boundary, either choice of label yields same expected loss.
Loss for choosing “four” Loss for choosing “nine”
Slide credit: K. Grauman
Disclaimers
• We will often assume the same loss for all possible types of misclassifications
• We won’t always build probability distributions – often we’ll just find a decision boundary (using discriminative methods)
• What’s the simplest classifier you can think of?
Nearest neighbor classifier
f(x) = label of the training example nearest to x
• All we need is a distance function for our inputs
• No training required!
Test
exampleTraining
examples
from class 1
Training
examples
from class 2
Slide credit: L. Lazebnik
K-Nearest Neighbors classification
k = 5
Slide credit: D. Lowe
• For a new point, find the k closest points from training data
• Labels of the k points “vote” to classify
If query lands here, the 5
NN consist of 3 negatives
and 2 positives, so we
classify it as negative.
Black = negative
Red = positive
1-nearest neighbor
x x
xx
x
x
x
x
o
oo
o
o
o
o
x2
x1
+
+
Slide credit: D. Hoiem
3-nearest neighbor
x x
xx
x
x
x
x
o
oo
o
o
o
o
x2
x1
+
+
Slide credit: D. Hoiem
5-nearest neighbor
x x
xx
x
x
x
x
o
oo
o
o
o
o
x2
x1
+
+
Slide credit: D. Hoiem
What are the tradeoffs of having a too large k? Too small k?
A nearest neighbor recognition example:
im2gps: Estimating Geographic Information from a Single Image.
James Hays and Alexei Efros.CVPR 2008.
http://graphics.cs.cmu.edu/projects/im2gps/
Where in the World?
Slides: James Hays
Where in the World?
Slides: James Hays
Where in the World?
Slides: James Hays
How much can an image tell about its geographic location?
Slide credit: James Hays
Nearest Neighbors according to gist + bag of SIFT + color histogram + a few others
Slide credit: James Hays
6+ million geotagged photosby 109,788 photographers
Slides: James Hays
A scene is a single surface that can be
represented by global (statistical) descriptors
Spatial Envelope Theory of Scene RepresentationOliva & Torralba (2001)
Slide Credit: Aude Olivia
Scene Matches
[Hays and Efros. im2gps: Estimating Geographic Information from a Single Image. CVPR 2008.] Slides: James Hays
Slides: James Hays
Scene Matches
[Hays and Efros. im2gps: Estimating Geographic Information from a Single Image. CVPR 2008.] Slides: James Hays
[Hays and Efros. im2gps: Estimating Geographic Information from a Single Image. CVPR 2008.] Slides: James Hays
Scene Matches
[Hays and Efros. im2gps: Estimating Geographic Information from a Single Image. CVPR 2008.] Slides: James Hays
[Hays and Efros. im2gps: Estimating Geographic Information from a Single Image. CVPR 2008.] Slides: James Hays
The Importance of Data
[Hays and Efros. im2gps: Estimating Geographic Information from a Single Image. CVPR 2008.]Slides: James Hays
Nearest neighbor classifier
f(x) = label of the training example nearest to x
• All we need is a distance function for our inputs
• No training required!
Test
exampleTraining
examples
from class 1
Training
examples
from class 2
Slide credit: L. Lazebnik
Evaluating Classifiers
• Accuracy
– # correctly classified / # all test examples
• Precision/recall
– Precision = # retrieved positives / # retrieved
– Recall = # retrieved positives / # positives
• F-measure
= 2PR / (P + R)
Discriminative classifiers
Learn a simple function of the input features that correctly predicts the true labels on the training set
Training Goals
1. Accurate classification of training data
2. Correct classifications are confident
3. Classification function is simple
𝑦 = 𝑓 𝑥
Slide credit: D. Hoiem
Linear classifier
• Find a linear function to separate the classes
f(x) = sgn(w1x1 + w2x2 + … + wDxD) = sgn(w x)
Slide credit: L. Lazebnik
What about this line?
NN vs. linear classifiers• NN pros:
+ Simple to implement
+ Decision boundaries not necessarily linear
+ Works for any number of classes
+ Nonparametric method
• NN cons:
– Need good distance function
– Slow at test time (large search problem to find neighbors)
– Storage of data
• Linear pros:
+ Low-dimensional parametric representation
+ Very fast at test time
• Linear cons:
– Works for two classes
– How to train the linear function?
– What if data is not linearly separable?
Adapted from L. Lazebnik
