Face Recognition and Retrieval in VideoBasic concept of Face Recog. & retrieval And their basic methods.

C.S.E.Kwon Min Hyuk

True? False?Q1 : in recently, face recognition

researches focus on video-based rather than still image-based (O / X)

Q2 : There is three approaches; (O/ X)◦1. key-frame based◦2. Temporal model based◦3. image set based

Face variation and expression make face recognition difficult. (O/X)

AnswersAll of statement is true.

Intro Q1 : Why do we need Face recogni-

tion system?◦ Increasing request to search specific peo-

ple related video contents

◦ Can be applied at security, human-com-puter inter action etc.

IntroQ2: What is recent trend of approach

to Face recog.?

◦ Traditionally , focused on Still image-based appr.

◦ Recently , focused on Video-based appr.

◦We can extract more information from video than that of still image.

General steps for Face recognitionWhere is face located in video

frame?◦ We should look for which part of the frame

is face. Face detecting and Tracking.

Recognizing face◦There is some basic approach for

Face Recog. Key fame-based approach Temporal Model-based approach Image set-based approach

Face detectionUsing statistical geometric model

◦ From the frame Extract appearance fea-tures such as edge, intensity, color(histogram)

To evolve the face detector by using machine learning tech.◦Adaboost◦Neural Network◦Support Vector Machine

Face trackingFace detection’s limit.

◦It detect only frontal or near frontal view.

Tracking face is needed to handle large head motions.

Face tracking◦Difficulties◦Method to solve

Face trackingDifficulty 1 : There is

◦Face appearance variation◦3D motion◦Background change

Method to solve: face online boosting◦Using tracked images in previous

frames.◦Applying current result to tracking

seq. for next frame. (real-time updat-ing feedback)

example in next slide

Example for online boosting al-gorithm.

So-called “adaptive tracker.”

Face tracking(con’d)Difficulty 2: The adaptive tracker

can adapt to non-targets.

Method to solve : add basal appear-ance of target ◦ Teaching the tracker about some basal ap-

pearance of the target. ◦ Basal appearance : Image set of various

target condition (face expression, pose etc)

After face detecting and trackingWe can determine the part of

frame where face is located.

Now, we can get to face recogni-tion.

Face RecognitionBasic steps for Face Recog.

◦1. get weak evidence in individual frame.

◦2. collect that evidence over time.◦3. lead(determine) reliable result.

Three approaches◦1. key-frame approach◦2. temporal model-based approach◦3. image set-based approach

Key-frame based ap-proachTreat each video as a collection

of images. Basic steps of the approach.

◦1. input data(still images, video)◦2. from data, extract images of the

target. Extracted images are called key-frames

or examplars.

◦3. matching them with all or subset of other video sequence(where the target is).

Key-frame based ap-proach(con’d)How can we get some ‘good’ key-

frame from input data?◦By image-based recognition◦In each frame, probe the nose and

eyes’ triangular structure.

If it is in the frame, then face recognition is performed. And key-frame is extracted.

Key-frame based ap-proach(con’d)

◦Applying K-Means clustering Cluster means a group whose elements

have some common property. This algorithm is grouping some data ob-

servations into one of cluster which has nearest mean.

Key-frame based ap-proach(con’d)Other algorithms

◦ Isomap algorithm◦ Combination of majority and probabilistic

voting.◦ And so on. (I’ll skip the details.)

Finally, all or subset of video sequence will be compared(matched) with extracted ‘good’ key-frame to determine recognition.

Temporal Model Based approachTo handle face dynamics

◦Ex: face expression(non-rigid)or head movement(rigid)

Using temporal sequence(continuous coherent)◦Ex> Using whole sequence of chang-

ing face dynamics as a image set.

Temporal Model Based ap-proach(con’d)Basic methods

◦Matching the face Trajectory. Trajectory means the moving face’s

path(orbit) through in surfaces. Two(model and object) trajectory dis-

tance accumulates recognition evidence over time.

Temporal Model Based ap-proach(con’d)Other method

◦Trained statistical face model Using density estimation.

◦Probabilistic approach Using time-series state space variables

◦Hidden Markov Model Fusing pose and person-discriminant fea-

tures.

I’ll skip all of details.

Image set based approachThis approach uses

◦both image collected over consecutive time

(similar with temporal image set)

◦And independent still image set (similar with key-frame)

Combination of both temporal and key-frame based approaches.

Two major approaches.◦Statistical modal-based◦Mutual subspace-based

Image set based ap-proach(con’d)Image set classification

◦Non-parametric sample based Compare representative images of each

image sets

◦Parametric model-based In terms of probabilistic, compare two

distributionsof each image set.

Image set based ap-proach(con’d)Statistical Model-based

◦To determine recognition, consider similarity of two manifolds manifold is large group (more than clus-

ter)which contains several cluster.

Drawback ◦Need to solve the difficult parameter

estimation problem.

Image set based ap-proach(con’d)Mutual Subspace-Based

Model(MSM)◦To determine Similarity between im-

age sets measure by the smallest principal angles

between subspaces.

CMSM is expansion of MSM◦Assume more constraints.

Face RetrievalIt is difficult to recognize face in the

uncontrolled condition like face dy-namics , light intensity, hair styles

◦There is two applications1. Person Retrieval 2. Cast listing

Person RetrievalFace recognition tech. are ap-

plied.Basic method

◦Using head model (with multiple tex-ture map)

◦Step1. rendering(extract or gener-ate) face images

◦Step2. identifying target face.◦Step3. updating the texture map of

the model.

Cast listing (cast : actors or charac-ters in film)

Automatic cast listing is interest-ing problem.

Based on face recognition◦Because face is repeatable cue in

the filmUsing image clustering method

For accuracy, Treat clothing ap-pearance additional cues for clustering.

Challenges and Future di-rectionDatabases.

◦Constructed in lab. enviro. Not a real world.

◦Limited face appearance of variation.Low-quality Video data

◦Exist lots of noise hard to filter out.Computational Cost

◦Face recognition requires quite high power devices.

ConclusionFace recog. can be applied in var-

ious area.Face detecting and Tracking.Three general Methods

◦Key-frame based◦Temporal model based◦Image set based

Person Retrieval and Cast listingChallenges to evolve Face recog.

system.