“Bayesian Identity Clustering”

Simon J.D. PrinceDepartment of Computer Science University College London.

James ElderCentre for Vision ResearchYork University.

http://pvl.cs.ucl.ac.uks.prince@cs.ucl.ac.uk

How many different people are present and which face corresponds to which person?

Could be 80 pics of the same person, or 1 pic each of 80 different people.

Difficult Task: Compound face recognition + choice of model size

The problem

The problem

Faces don’t necessarily all have the same pose

Face Clustering

Application 1: Photo content labelling

System analyzes a set of photos and provides one picture of each different individual present, which you label.

Labels are then propagated to your entire set of photos.

Face Clustering

Application 2: Security synopses

Here are a collection of face images captured by a pan/tilt security camera.

A crime is committed in the area.

How many people were present in the last few hours, and when did each enter and leave?

Face Clustering

Application 3: Web search

Here are images from Google image search given the query “john smith”.

Identify which pictures belong with which and present only one of each person as options rather than 100’s of pages with repeated images.

Latent variables for identity

(i) Face images depend on several interacting factors: these include the person’s identity (signal) and the pose, illumination (nuisance variables).

(ii) Image generation is noisy: even in matching conditions, images of the same person differ. This remaining variation comprises unmodeled factors and sensor noise.

(iii) Identity can never exactly be known: since generation is noisy, there will always be uncertainty on any estimate of identity, regardless of how we form this estimate.

(iv) Recognition tasks do not require identity estimates: in face recognition, we can ask whether two faces have the same identity, regardless of what this identity is.

..

IDENTITY DIM 1

IDE

NTI

TY D

IM 2

OBSERVED DIM 1

h3(explains x3)

h2(explains x2 and xp )

h1(explains x1)

Identity space explains data

in feature space.

Deterministic transformation +

additive noise

Each point represents abstract identity of person.

OBS

ERVE

D D

IM 3

OB

SE

RV

ED

DIM

2 .x3

x1. x2

•xp

.

GOAL: To build a generative model of the whole face manifold.

We hypothesize an underlying representation of identity (h) and generation process f(.) to create image (a generative model)

Add within-individual noise process e to explain why two images of same person are not identical

Latent Identity Variables

Latent Identity Variables (LIVs)

..

IDENTITY DIM 1

IDE

NTI

TY D

IM 2

OBSERVED DIM 1

h3(explains x3)

h2(explains x2 and xp )

h1(explains x1)

Identity space explains data

in feature space.

Deterministic transformation +

additive noise

Each point represents abstract identity of person.

OBS

ERVE

D D

IM 3

OB

SE

RV

ED

DIM

2 .x3

x1. x2

•xp

.

Hypothesize existence of underlying representation of identity:

If two identity variables take the same value then they represent same person

If two identity variables take different values, they represent different people

Latent Identity Variables (LIVs)

..

IDENTITY DIM 1

IDE

NTI

TY D

IM 2

OBSERVED DIM 1

h3(explains x3)

h2(explains x2 and xp )

h1(explains x1)

.

Deterministic transformation +

additive noise

OBS

ERVE

D D

IM 3

OB

SE

RV

ED

DIM

2 ..

x3

x1. x2

•xp

.•e

x1 = f(h1) + e

Hypothesize existence of underlying representation of identity:

If two points in identity space are the same then they represent same person

If two points in identity space differ, they represent different people

Latent Identity Variables (LIVs)

..

IDENTITY DIM 1

IDE

NTI

TY D

IM 2

OBSERVED DIM 1

h3(explains x3)

h2(explains x2 and xp )

h1(explains x1)

.

Deterministic transformation +

additive noise

OBS

ERVE

D D

IM 3

OB

SE

RV

ED

DIM

2 ..

x3

x1. x2

•xp

. •

x2 = f(h2) + ei

e

e

xp = f(h2) + ej

Hypothesize existence of underlying representation of identity:

If two points in identity space are the same then they represent same person

If two points in identity space differ, they represent different people

..

IDENTITY DIM 1

IDE

NTI

TY D

IM 2

OBSERVED DIM 1

h3(explains x3)

h2(explains x2 and xp )

h1(explains x1)

Identity space explains data

in feature space.

Deterministic transformation +

additive noise

Each point represents abstract identity of person.

OBS

ERVE

D D

IM 3

OB

SE

RV

ED

DIM

2 .x3

x1. x2

•xp

.

So a LIV model partitions of the data into signal and noise. In recognition we:

Compare the probability of data under different assignments between the data (left figure) and identity (right figure)

Acknowledge that values of identity variables are fundamentally uncertain so consider all possibilities (i.e. marginalize over them) – never estimate identity!

Latent Identity Variables (LIVs)

Probabilistic Linear Discriminant Analysis (Prince & Elder 2007)

13

Observed data from j th image of i th indivual

Overall mean

Weighted sum of basis functions F forbetween individual variation (identity)

Weighted sum of basis functions G forwithin-individual variation

noise

Probabilistic LDA

Or:

MEAN, m F(:,1) F(:,2)

NOISE, S

= + h1 + h2 +

IMAGE, xj

+ h3

F(:,3)

G(:,1) G(:,2) G(:,3)

++ w1j + w2j + w3j

Within individual and between individual variation

15

m+2F(:,1) m+2F(:,2) m+2(F:,3)

m+2G(:,1) m+2(G:,2) m+2G(:,3)

Between-individual variation

Within-individual variation

Model Comparison

16

x1 x2 x3

h2h1

x1 x2 x3

h2h1

h1 h2

x1 x2 x3

h3

1

x1 x2 x3

2

h1

4 5

h2

x1 x2 x3

3

h1

Frame clustering as model comparison

Data Extraction

XM2VTS database

Trained with 8 images each of 195 individuals

Test with remaining 100 individuals

Find facial feature points

Extract normalized vector of pooled gradients around each feature

Build a PLDA model of each

IMAGES: 2 3 4 5 6 7 8MODELS: 2 5 15 52 203 877 4140

For small N, we can compare all hypotheses

NO. OF IMAGES

% C

OR

REC

T

PLDA Model

Chance

PROBLEM 1: For larger N, the number of models is too large (10115 for 100 images) and we must resort to approximate methods.

Agglomerative approximation used – start with N different groups and merge the best two repeatedly until the likelihood stops increasing

PROBLEM 2:

We almost never get the answer exactly right when N=100. % Correct is not a good metric. How should we measure clustering performance

Model Explosion

Cluster Alignment

19

Clustering Metrics

• Rand Metric

• Variation of Information

• Precision and Recall

20

Which clusters are the best?

21Surely M1=M2<M3=M4?

Which clusters are the best?

22Propose using no of splits and merges required to match

Quantitative Results

23

XM2VTS Frontal Data

Always 4 images per person.

5,10,15,20,25 people

100 repeats for each datapoint with different people and different images

Qualitative Results

Split-Merge cost here = 0.97 (average was 0.98)

Cross pose case

What if there is more than one pose?

Tied PLDA

26

Observed data from j th image of i th Individual in k th pose

Overall mean for pose k

Weighted sum of basis functions F forbetween individual variation (identity) for this pose

Weighted sum of basis functions G forwithin-individual variation for this pose

noise

Tied PLDA

27

Between-individual variation(pose 1)

Within-individual variation(pose 1)

Between-individual variation(pose 2)

Within-individual variation(pose 2)

Quantitative Results

28

XM2VTS Frontal Data

Always 4 images per person.

5,10,15,20,25 people

100 repeats for each datapoint with different people and different images

Clustering Results: Mixed Pose

Split merge cost = 0.85 (average 0.865)

Qualitative Results

Conclusions

• Clustering of faces is a difficult problem– Multiple face recognition– Choice of model size

• Our method marginalizes over number of identities – can compare models of different size

• Extends to the cross-pose case

30

Learn more about these techniques via http://pvl.cs.ucl.ac.uk