Pose EstimationOHAD MOSAFI

MotivationDetecting people in images is a key problem for video indexing, browsing and retrieval. the main difficulties are the large appearance variations caused by action, clothing, illumination, viewpoint and scale.

Finding people by samplingPeople can be quite accurately modeled as assemblies of cylinders and these assemblies are constrained by the kinematics of human joints.People are presented as collections of nine body segments one for the torso and two for each limb.

Finding people by samplingSymmetries- pairs of edge elements that are approximately symmetric about some symmetry axis and whose tangents are approximately parallel to that axis.Segments- extended group of symmetries which approximately share the same axis (expectation- maximization algorithm that assumes a fixed number of segments).Using learned likelihood model to form assemblies by sampling.Finally the set of assemblies is replaced with a smaller set of representatives, which are used to count people in the image.

ThumbtackLet = P(up), 1- = P(down)How to determine

Empirical estimate: 8 up, 2 down ->

Maximum Likelihood = P(up), 1- = P(down)Observe:Likelihood of the observation sequence depends on

Maximum LikelihoodMaximum likelihood finds

Extrema at

Maximum LikelihoodMore generally, consider a binary-valued random variable with , assume we observe ones, and zeroes.•Likelihood: •Derivative: Hence we have for the extrema:

Log-likelihood The function log: is a monotonically increasing function of xHence for any (positive valued) function f:

In practice often more convenient to optimize the log likelihood rather than the likelihood itself.

Log-likelihood Example:

Log-likelihood <- -> Likelihood Reconsider thumbtack: 8 up, 2 down.

Log-likelihood <- -> Likelihood

𝜆 𝑥1+ (1− 𝜆 )𝑥2

Finding Segments Using EMA symmetry fits a segment best when the midpoint of the symmetry lies on the segment’s symmetry axis, the endpoints lie half a segment width away from the axis, and the symmetry is perpendicular to the axis This yields the conditional likelihood for a symmetry given a segment as a four-dimensional Gaussian, and an EM algorithm can now fit a fixed number of segments to the symmetries.

Mixture of Gaussians

Mixture of Gaussians

Expectation Maximization

Expectation Maximization

EM Derivation

EM Derivation(cont.)

Jensen’s inequality

EM Derivation(cont.)

EM for Mixture of Gaussians

EM for Mixture of Gaussians

EM for Mixture of Gaussians

Representing Likelihood for people

The likelihood for a nine segment assembly is computed from a set of 41 geometric features.(Angles and distances between segments, length ratios, etc).

Features are independent with a relatively small error.

The main errors will be due to interactions between kinematic constraints on the hips and shoulders, and viewing pose.

Marginal LikelihoodGiven a set of independent identically distributed data points where according to some distribution parameterized by whee itself is a random variable described by the distribution , the marginal likelihood in general asks what's the probability is, where has been marginalized out

Building Assemblies Incrementally by Resampling

Fixed Permutation of labels {T,LUA,…}.Generate a sequence of multisets of samples.

fig. 2 The samples in are drown from marginal likelihood,.

Building Assemblies Incrementally by Resampling (cont)We generate the set of samples from using importance sampling.oa general technique for estimating properties of a

particular distribution, while only having samples generated from a different distribution from the distribution of interest

Form the set of sub-assemblies for all groups and all choices of .

Directing the Sampler We define equivalent assemblies to be those that label the same segment as a torso. (This is a good choice, because different people in an image will tend to have their torsos in different places) The highest likelihood assembly is found by a simple greedy algorithm.If there are segments in the image, we never have more than n sub-assemblies of each type, thus the algorithm run in time.

Examplesuppose that all of the segments in an assembly, except the lower left arm, are fixed.We are to choose the lower left arm that maximizes the likelihood of the resulting assembly.In our model, The lower left arm can be find by considering all the pairs of lower left arm (which can be any segment) and the upper left arm(which is fixed), and choosing the one with the highest marginal likelihood

Example(cont.)Now let us supposed that we have fixed a torso and, possible, some limbs, and we want to add the left arm that would maximize the likelihood of the result.First, we will find the highest-likelihood left arm for each choice of the upper arm. Since no feature involves the left arm and any other limb, we can choose the best left arm by considering all the pairs of torso(fixed) and a left arm, and choosing the one with the largest marginal likelihood.

Counting peopleThis sampling algorithm allows to count people in images. To estimate the number of people. We begin by selecting a small set of representative assemblies in the image and then use them for countingWe break the set of all assemblies in the image into (not necessarily disjoint) blocks — sets of assemblies such that any two assemblies from the same block have overlapping torsos. Then, the representative is chosen from each block as the assembly with the highest likelihood, over all assemblies available from the block.

ResultsTo learn the likelihood model , they used a set of 193 training images.(standing against a uniform background, all the views were frontal and all limbs were visible, although the configurations varied).The training set was expanded by adding the mirror image of each assembly, thus resulting in 386 configurations.The test set included:o145 control images with no peopleo228 – 1 people, 72- 2 people, 65 – 3 people. (COREL database)

Pictorial Structure of People People are represented using a 2D articulated appearance model composed of 15 part-aligned image rectangles surrounding the projections of body parts.

Each body part is a rectangle parameterized in image coordinates by its center , , its length or size and orientation .

Pictorial Structure of People The posterior likelihood of there being a body with parts at image location is the product of the data likelihood for the 15 parts and the prior likelihoods for the 14 joints:

- function measuring the degree of deformation of the model when part is placed at location and part is located at location .

Detecting Body Parts The ultimate goal is to detect people and label them with detailed part locations, in applications where the person may be in any pose and partly occluded.

Detecting and labelling body parts is a central problem in all component-based approaches. Clearly the image must be scanned at all relevant locations and scales, but there is also a question of how to handle different part orientations, especially for small, mobile, highly articulated parts such as arms and hands.

Feature Sets numerous feature sets have been suggested, including image pixel values, wavelet coefficients and Gaussian derivatives.

a feature set consisting of the Gaussian filtered image and its first and second derivatives is used.

The feature vector for an image rectangle at location-scale orientation contains the absolute values of the responses of the six Gaussian:

in the rectangle’s 14X24 windows. Thus, there are 14X24X6=2016 features per body.

TrainingUsing the 2016-dimensional feature vectors for all body parts

we trained two linear classifiers for each part:oSupport Vector MachineoRelevance Vector Machine.

training on the smallest sets of examples that give reasonable results — in our case, about 100.

SVM- Linear Separators Binary classification can be viewed as the task of separating classes in feature space:

wTx + b = 0

wTx + b < 0wTx + b > 0

f(x) = sign(wTx + b)

Linear SeparatorsWhich of the linear separators is optimal ?

SVM- Classification Margin

Distance from example xi to the separator is

Examples closest to the hyperplane are support vectors.

Margin ρ of the separator is the distance between support vectors.

w

xw br i

T

r

ρ

Maximum Margin ClassificationMaximizing the margin is good according to intuition

Implies that only support vectors matter; other training examples are ignorable.

Linear SVMs MathematicallyLet training set be separated by a hyperplane with margin . Then for each training example

Linear SVMs Mathematically For every support vector the inequality is an equality. After rescaling w and b by p/2 in the equality, we obtain that distance between each and the hyperplane is

Then the margin can be expressed as :

Linear SVMs Mathematically we can formulate the quadratic optimization problem

Which can be reformulated as:

Find w and b such that

is maximized

and for all (xi, yi), i=1..n : yi(wTxi + b) ≥ 1w

2

Find w and b such that

Φ(w) = ||w||2=wTw is minimized

and for all (xi, yi), i=1..n : yi (wTxi + b) ≥ 1

Solving the Optimization Problem

Need to optimize a quadratic function subject to linear constraints.Quadratic optimization problems are a well-known class of mathematical programming problems for which several (non-trivial) algorithms exist.

Find w and b such thatΦ(w) =wTw is minimized and for all (xi, yi), i=1..n : yi (wTxi + b) ≥ 1

Solving the Optimization ProblemThe solution involves constructing a dual problem where a Lagrange multiplier αi is associated with every inequality constraint in the primal (original) problem:

Find α1…αn such thatQ(α) =Σαi - ½ΣΣαiαjyiyjxi

Txj is maximized and (1) Σαiyi = 0(2) αi ≥ 0 for all αi

Quadratic ProgramA special type of mathematical optimization problemIt is the problem of optimizing (minimizing or maximizing) a quadratic function of several variables subject to linear constraints on these variablesThe quadratic programming problem can be formulated as:

Assume x belongs to space. Both x and c are column vectors with n elements (n×1 matrices), and Q is a symmetric n×n matrix

Minimize (with respect to x)

SVM-The Optimization Problem Solution

Given a solution α1…αn to the dual problem, solution to the primal is:

Each non-zero αi indicates that corresponding xi is a support vector. Then the classifying function is (note that we don’t need w explicitly):

Notice that it relies on an inner product between the test point x and the support vectors xi – we will return to this later. Also keep in mind that solving the optimization problem involved computing the inner products xi

Txj between all training points.

w =Σαiyixi b = yk - Σαiyixi Txk for any αk > 0

f(x) = ΣαiyixiTx + b

Not linearly separable cases

Soft Margin Classification What if the training set is not linearly separable?Slack variables ξi can be added to allow misclassification of difficult or noisy examples, resulting margin called soft.

Soft Margin Classification MathematicallyThe old formulation:

Find w and b such thatΦ(w) =wTw is minimized and for all (xi ,yi), i=1..n : yi (wTxi + b) ≥ 1

Soft Margin Classification MathematicallyModified formulation incorporates slack variables:

Parameter C can be viewed as a way to control overfitting it “trades off” the relative importance of maximizing the margin

and fitting the training data.

Find w and b such thatΦ(w) =wTw + CΣξi is minimized and for all (xi ,yi), i=1..n : yi (wTxi + b) ≥ 1 – ξi, , ξi ≥ 0

Soft Margin Classification – Solution

Dual problem is identical to separable case (would not be identical if the 2-norm penalty for slack variables CΣξi

2 was used in primal objective, we would need additional Lagrange multipliers for slack variables):

Find α1…αN such thatQ(α) =Σαi - ½ΣΣαiαjyiyjxi

Txj is maximized and (1) Σαiyi = 0(2) 0 ≤ αi ≤ C for all αi

Non-linear SVMs

Non-linear SVMs: Feature spaces General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable:

The “Kernel Trick”The linear classifier relies on inner product between vectors K(xi,xj)=xi

Txj

If every datapoint is mapped into high-dimensional space via some transformation Φ: x → φ(x), the inner product becomes:

K(xi,xj)= φ(xi) Tφ(xj)

A kernel function is a function that is equivalent to an inner product in some feature spaceThus, a kernel function implicitly maps data to a high-dimensional space (without the need to compute each φ(x) explicitly)

The “Kernel Trick” Example2-dimensional vectors x=[x1 x2]; let K(xi,xj)=(1 + xi

Txj)2,

Need to show that K(xi,xj)= φ(xi) Tφ(xj):

K(xi,xj)=

(1 + xiTxj)2

,=

( 1+ xi1xj1 + xi2xj2

)2=

[1 xi12 √2 xi1xi2 xi2

2 √2xi1 √2xi2]T [1 xj12 √2 xj1xj2 xj2

2 √2xj1 √2xj2] =

φ(xi) Tφ(xj)

, where φ(x) = [1 x12 √2 x1x2 x2

2 √2x1 √2x2]

Examples of Kernel FunctionsLinear: K(xi,xj)= xi

Txj

oMapping Φ: x → φ(x), where φ(x) is x itself

Polynomial of power p: K(xi,xj)= (1+ xiTxj)p

Gaussian (radial-basis function): K(xi,xj) =oMapping Φ: x → φ(x), where φ(x) is infinite-dimensionaloDecreases with distance and ranges between zero (in the limit) and

one (when x = x'), it has a ready interpretation as a similarity measure

Non-linear SVMs Mathematically Dual problem formulation:

The solution is:

Find α1…αn such thatQ(α) =Σαi - ½ΣΣαiαjyiyjK(xi, xj) is maximized and (1) Σαiyi = 0(2) αi ≥ 0 for all αi

f(x) = ΣαiyiK(xi, x)+ b

RVM -Relevance Vector Machines

Bayesian alternative to support vector machine. Limitations of the SVM:

◦two classes◦ large number of kernels.◦decisions at outputs instead of probabilities

Parsing Algorithm Given N candidate body part locations detected by each body part classifier , we are looking for a ‘parse’ of the scene into one or more ‘body trees’.

Given a detecntion score for all candidates we search for the best candidate as a function of their direct parents in the body tree.

Learning the Body Tree The articulation model is a linear combination of the differences between two joint locations.

Using positive and negative examples from the training set, they used a linear SVM classifier to learn a set of weights such that the score is positive for all positive examples, and negative for all negative examples.

Results- Detection of body partsThe RVM classifiers perform only slightly worse than their SVM counterparts, with mean false detection rates of 80.1% and 78.5% respectively.

The worst results are obtained for the torso and head models, The torso is probably the hardest body part to detect as it is almost entirely shapeless. It is probably best detected indirectly from geometric clues. In contrast, the head is known to contain highly discriminant features, but the training images contain a wide range of poses and significantly more training data

Results- Detection of Body Trees 100 training examples.We obtained correct detections rates of 72 % using RVM scores and 83 % using SVM scores.a majority of the body parts were correctly positioned in only 36 % of the test images for RVM and 55 % for SVM.

Second experiment: 200 training examples.Detention rate 76% for SVM and 88% for RVM.

ReferencesFinding People by Sampling Ioffe & Forsyth, ICCV 1999Probabilistic Methods for Finding People, Ioffe & Forsyth 2001Pictorial Structure Models for Object Recognition Felzenszwalb & Huttenlocher, 2000Learning to Parse Pictures of People Ronfard, Schmid & Triggs, ECCV 2002