Introduction Reduced Problem Complete Problem ConclusionOutline
2
Slide 3
Background Gesture recognition is a strong upcoming field in
computer vision Gesture recognition can be seen as a way for
computers to begin to understand human body language 3
Slide 4
Motivation Existing Gesture recognition demand a long
configuration and training Different Gestures are been solved using
different approaches 4
Slide 5
Goals Learn and understand existing Gesture recognition
algorithms. Compare different approaches Design Gesture recognition
algorithm which reduces training time 5
Slide 6
Data The Data is compose from several set each contains: o
Gesture vocabulary (learning set) which contain only one sample per
gesture. o Test set which contain one or more gestures. Each of the
sets has different vocabulary features such as large/small gesture
hand/legs movement etc. 6
Slide 7
Data Train 7 Base gesture
Slide 8
Data Test 8 Multiple base gestures Large movements
Slide 9
Data - Test 9 Multiple base gestures Small movements
Slide 10
Challenges One shot learning - only one learning sample (unlike
the common approach of multi class classification) Tests videos
segmentation Same gesture can have different number of frames Each
set has different features (small/big gestures) 10
Slide 11
Introduction Reduced Problem Complete Problem ConclusionOutline
11
Slide 12
Reduced Problem Assume that each of the test movies has only
one gesture Goal: finding features space and distance function
which have good separation of the features space 12
Slide 13
Problem Approach Classic machine learning problem Select
Feature One Shot Match using similarity function 13
Slide 14
Features Motion Energy subtracting consecutive frames Space
Quantization 14
Slide 15
Features Harris Corner Detector Find interest point in the
difference image based on corner detection Space Time Interest
Points Extend Harris to the time domain 15
Slide 16
Features Harris Corner Detector Find interest point in the
image based on corner detection 16
Slide 17
Features Space Time Interest Points 17
Slide 18
Features 18 STIPHarris
Slide 19
Features Head Relative Interest Points 19
Slide 20
Features 20 Interest pointsHead Histogram
Slide 21
Distance Functions Good features space is defined not only by
the features but also by the distance (similarity) function
Different features need different distance functions 21
Slide 22
Principal Motion Using PCA Using principal component analysis
(PCA), to find the main motion vectors. For test set - project
feature onto each of train principals and evaluate similarity
22
Slide 23
Earth Moving Distance Given two sets of distribution, EMD will
measure the minimum cost to shift dirt from one distribution to the
other. 23
Slide 24
Perturbed Variations 24 Given two sets of distribution and
predefined value of permitted variations optimally perturbs the
distribution to best fit each other. Transportation problem under
permitted variations constrain
Slide 25
Perturbed Variations 25
Slide 26
Levenshtein Distance Measure the difference between two
sequences. Consider lengths and classification. 26
Slide 27
Results 27
Slide 28
Results 28 Top 10 Top 20
Slide 29
Results 29
Slide 30
Introduction Reduced Problem Complete Problem ConclusionOutline
30
Slide 31
Complete Problem Separate problems Basic Segmentation
(equal/movement) Whole problem solving approach Moving Window
Dynamic Time Warping (DTW) 31
Slide 32
Problem Approach Three different method to solve the problem:
Basic Segmentation (equal/movement) Moving Window Dynamic Time
Warping (DTW) 32
Slide 33
Moving Window Move a window along the test video. Assume each
window frames has only one gesture Preform basic analysis as did
before to and build the distance matrix 33
Slide 34
Moving Window 34
Slide 35
Moving Window After Sorting the Distance matrix we extract
labels and cuts 35 Several other operation (such as smoothing) are
done before extracting the final result
Slide 36
Dynamic Time Warping Create a state machine from train data:
Module standing position Form standing position can move to start
of base gestures Assume we can move forward, or stay in the same
sate. For a given gesture find the best path along the sate machine
36
Slide 37
Dynamic Time Warping 37
Slide 38
Results 38
Slide 39
Results 39
Slide 40
Results 40
Slide 41
Results 41 Top 10 Top 20
Slide 42
Introduction Reduced Problem Complete Problem ConclusionOutline
42
Slide 43
Conclusions Each approach receive better results in different
feature and similarity function Different algorithms has different
strengths (segmentation\recognition) Segmentation require standing
position model. 43
Slide 44
Conclusions 44 Pre-processing unsupervised algorithms help
better representing the data. There is still allot left to do on
the field
Slide 45
Future Work Try different models for the standing position to
improve segmentation results Try combing DTW for segmentation and
PCA for recognition. Use different unsupervised algorithms to
better represent the data. 45
Slide 46
References Ivan Laptev, "On Space-Time Interest Points, 2005
Hugo Jair Escalantea and Isabelle Guyonb, "Principal motion:
PCA-based reconstruction of motion histograms M.Harel, S.Manor,
"The Perturbed Variation, NIPS 2012 Elizaveta Levina, Peter Bickel
Department of Statistics, The EarthMovers Distance is the Mallows
Distance: Some Insights from Statistics. Ofir Pele,Michael Werman,
Fast and Robust Earth Movers Distances.2008 46