AAM based Face Tracking with Temporal Matching and Face

SegmentationDalong Du

Outline

• Author Introduction• AAM Introduction• Abstract• Method and Theory• Experiment

Author Introduction

• Mingcai Zhou– Institute of Automation Chinese Academy of

Sciences

• Lin Liang– Microsoft Research Asia–

Author Introduction

• Jian Sun– Microsoft Research Asia• joined in July, 2003.

– Educational background • BS degree, MS degree and Ph.D degree from Xian

Jiaotong University in 1997, 2000 and 2003– Current research interests • Interactive compute vision (user interface + vision)• Internet compute vision (large image collection + vision)• stereo matching and computational photography

Author Introduction

• Yangsheng Wang– Director of Digital Interactive

Media Lab, Institute of Automation Chinese Academy of Sciences– Educational background• BS degree, MS degree and Ph.D degree from Huazhong

University of Science and Technology

AAM Introduction

• Shape Model• Appearance (Texture) Model• AAM Model Search

AAM—Shape Model• Face Q consists of N landmark points– – The geometry information of Q decouples into two parts:

• A shape S– Shape is the geometric information

invariant to a particular class of transformations

– e.g. Or other linear or nonlinear methods

• A transformation– θ– e.g. similarity s, R, t Or

Affine or others.– Similarity

»

x u b b

θ

x = (x1,y1, … , xn, yn)T

Same shape Different shape

AAM—Shape Model

• Shape Model Building– Given a set of shapes– Align shapes into common frame• Procrustes analysis

– Estimate shape distribution p(x)• Use PCA

The aligned shapes

AAM—Shape Model

• Shape Model Building, continued– Given aligned shapes, { }– Apply PCA• Compute mean and eigenvectors of covar.

– P – First t eigenvectors of covar. matrix– b – Shape model parameters

ix

Pbxx

AAM—Texture Model

• Building Texture Models– For each example, extract texture vector

– Normalise vectors (as for eigenfaces)– Build eigen-model

Texture, g

Warp tomeanshape

ggbPgg

1b12 12 2b22

22

AAM—Texture Model

• Warp method ),( :points Control ii yx )','( :points Warped ii yx

a b

c

x 'a

'b

'c

'x

cbax ''' cbax ' 1

10 and 10

if triangle theinside is

βα

x

AAM—Texture Model

• Warp method, continued

a b

c

x

)( ab

)( ac cba

cba

acabax

)1(

)()(

cbax

1

1111yyy

xxx

cba

cba

y

x

AAM—Model Search• Find the optimal shape parameters and appearance

parameters to minimize the difference between the warped-back appearance and synthesized appearance

( , )W x p

p

( ( ))I W p

( ( , ))I W x p

map every pixel x in the model coordinate to its corresponding image point( , )W x p

0s

Computed by the inverseCompositional parameter Update technique

Abstract

• Problems– Generalization problem– images with cluttered background

• How to do?– A temporal matching constraint in AAM fitting• Enforce an inter-frame local appearance constraint

between frames

– Introduce color-based face segmentation as a soft constraint

Method and Theory• Extend basic AAM to Multi-band AAM– The texture(appearance) is a concatenation of three

texture band values• The intensity (b)• X-direction gradient strength (c)• Y-direction gradient strength (d)

Method and Theory

• Temporal Matching Constraint– Select feature points with salient local appearances at

previous frame– Optimize the shape parameters to match the local

appearances at current frame

Method and Theory

• Temporal Matching Constraint, continued

– : a set of feature points• Selected by a corner detector and some semantic points

– : the face appearance of frame t-1– : the local patch corresponding to the j-th feature

point– : the average intensity of j-th patches of

frame t-1 and t respectively

t

1tA

jR

Normalize the illuminations of two patches

Method and Theory

• Temporal Matching Constraint, continued– Add a new term to the AAM cost function

• Empirically,

Can be efficiently minimized based on inverse compositional algorithm

Method and Theory

• Temporal Matching Constraint, continued– Be resistant to global illumination changes• Match local patches

– Do not suffer from the mismatched points• Feature matching is continuously refined by updating

the shape parameters during AAM fitting

Method and Theory

• Initialize shape– Good initial parameters -> good AAM fitting– Method • Selected feature points at frame t-1• Matched feature points at frame t• Remaining feature points after main direction filter

Method and Theory

• Initialize shape, continued– M matched points– Estimate the initial shape parameters

• represents the consistency of feature

points I’s direction• is the estimated position of the point I given

the shape parameters p• are the vertex coordinate of the triangle• are the triangle coordinate

1

M

i iz

0p

Gauss-Newton algorithm

Method and Theory

• Face Segmentation Constrained AAM– Problem: AAM tends to fit the face outline to the

background edges

– Method: segment the face region using an adaptive color model and constrain AAM fitting

Method and Theory

• Formalization

– Where are the locations of the selected outline points in the model coordinate

{ }kx

Wc = 0.01

Experiments

• RI: robust initialization• TO: temporal matching constraint • FS: face segmentation

Experiments

Thank you