Embed Size (px)
Citation preview
Bag-of-Words based Image Classification
Joost van de Weijer
What is in the image ?
image classification: answers the question what is in the image.
Is there a person ?
Is there a suit-case ?
Is there car ?
Inspiration
The VOC Pascal challenge: a competition on image classification.Participants have to classify 20 classes in over 10.000 images.
Inspiration
http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2010/results/index.html
The Event Data Set
• 7 event classes: basketball, polo, rowing, castells, marathon, sailing, skiing.• each class has 50 images, devided in 30 training and 20 test images.
Project I
goal: build an image classification system which can successfully classify sport images.
title: Bag-of-Words based Image Classification.
competition: do so better than the other groups.
Why is this difficult ?
Variations in viewpoint and zoom.
Variations in pose.
Why is this difficult ?
Inter-class variation.
lighting changes.
Why is this difficult ?
Back-ground variation.
Maybe the background could help ?
similar backgrounds- different classes.
from images to frequency histogram
•Compute visual words:• detect local regions from a set of images.• describe every local region by a descriptor
• texture• color
• cluster all descriptors into visual words
Given a new image:• detect local regions from a set of image.
assign to visual word N
• assign every region to its nearest visual word.• compute visual word-image histogram
Bag of Visual Words representation
Feature Detection
Bag-of-Words
representation
normalize patches
No spatial relations.
pi(w|Miro)
pi(w|Dali)
Bag of Visual Words representation
Image Representation
2. Extractionshapetexturecolor
Image
1. Feature detection
shape words
3.Le
arn
voca
bula
ry
Shape Voc
image classificationimage retrieval
4. BOW
5. SVM/ distance measures
The Framework
Image Representation
Image
1. Feature detection
1.random
2. Extractionshapetexturecolor
2. RGB
shape words
3.Le
arn
voca
bula
ry
Shape Voc
3. random
4. BOW
4. nearest neighbor
image classificationimage retrieval
5. SVM/ distance measures
5. linear SVM
Existing Implementation:
50 % classification score
Existing Implementation:
properties of BOW implementation:
• you can improve any of the subroutines and analyze the changes based on the classification results.
• several team members can work on feature detection while others work on feature description.
• the final classification results allow us to compare the results between the groups.
Project I: Bag Bag-of-Words based Image Classification
goal : build an image classification system which can successfully classify sport images.
teaching objectives
you will learn:• to represent images robust to changes of cameras, object orientation, and illuminant color.• what photometric invariance theory is and how to apply it to a real-world problem.• understand and use the SIFT descriptor.• how to discretize image features (colors, shapes, and textures).• what the strong and weak points of BOW representations for images are. • how to evaluate retrieval and classification results.
Practical information:
Group Size:The project has to be made in groups of 3 students. Each group should decide on the following roles:
• responsible competition.• responsible presentation.• responsible report
Practical Information:All practical information can be found in the student guide (http://cat.cvc.uab.es/~joost/master.html )
If it is hard to work as a group you can partition the tasks:
• feature detection• feature description• vocabulary construction• learning/evaluation
All group-members should understand all steps in the final program !
Practical information:
Important Dates:22 jan - 19 Feb. : The project will last 1 month. 22 jan. : Start project.29 Jan. : Extra assignment will be handed out. Submission of first results in AP.5 Feb. : Discussion meeting + submission second results in AP.11 Feb. : Publication of final test set.12 Feb. : Discussion meeting with groups separately.15 Feb. : Final submission of classification results in AP for all classes.19 Feb. : Presentation of the project.22 Feb. : Final submission date for report.
Supervision:There will be project meetings on Tuesdays afternoon to discus progress.
For any questions during the three weeks of the project email ([email protected]) or come to office O/119 in the CVC.
Use “PROJECT I” as subject of your emails, which makes it easier to manage.
Practical information:
NotesThe final note will be based on:
• participation (15%)• presentation (25%)• report (50%)• competition (10%)
Bugs:For sure there will be several bugs in the code. If you find one, mail me, and I will notify the other groups. Thanks !
Practical information:
Competition:
Dates:29 Jan. : Submission of first results in AP (before 15:00).5 Feb. : Submission second results in AP (before 15:00)
labeled train set labeled test set
19/22 Feb. : Your report/final presentation is based on the labeled test set !
Practical information:
labeled train set no labels for test set !
Competition:
Dates:
11 Feb. : Publication of final test set.15 Feb. : Final submission results in AP for all calsses.
Practical information:
Final Report The final report has to be submitted on 22th of February. The report should contain the following chapters. • Introduction ( max 1 page )• Feature Detection (max 2 pages).• Feature Description (max 3 pages).• Visual Vocabulary and BOW representation (max 2 pages)• Classification (max 2 pages)• Object Detection (optional: max 2 pages)• Results (max 2 pages).• Conclusions (max 1 page)
What to do next ?
• make groups of and assign :•responsible competition
(send an email to me today or tomorrow )
• install the programs and play with the code. ( http://cat.cvc.uab.es/~joost/master.html )
• This week you should already start working on a feature detector.
What to do next ?
Good Luck !
