Automated recognition of facial expressi ns and identity 2003 UCIT Progress Report

TAUCHI – Tampere Unit for Computer-Human Interaction

Automated recognition of facial expressi ns and identity

2003 UCIT Progress Report

Ioulia Guizatdinova

Research Group for Emotions, Sociality, and Computing University of Tampere

10.01.2004


Contents

• Research problems

• Aims and Tasks

• Facial landmark extraction : Methods

• Facial landmark extraction : Results

• Future Steps


Research Problems

“Automated recognition of facial expressions and identity”

• Face identification

• Recognition of facial expressions


Face classes

1

…

N

Person A

Person Z

…… … …

Face database

Input face

Face identification

Unrecognized face

still image/video signal

Recognition system

• Classification of input face to one of existing face classes stored in database

• Rejection of input face as unrecognized/unknown face

classification

rejection

Research Problems


• Facial expressions affect face recognition because a variability of facial landmarks in their appearance is high

• Humans are good in recognizing facial identity regardless of changes in facial expressions

• Computer-aided systems of face recognition are dramatically compromised by changes in facial expressions

Recognition of facial expressions

Research Problems


Aim of research

• The primary aim of this research is to do theoretical and experimental investigation on the possibilities to automatically recognize facial identity independent of changes in facial expressions

• For that purpose two 2D recognition systems will be developed

- Recognition system of facial expressions

- Expression-invariant system of facial identity recognition


Tasks

• Extraction and classification of facial landmarks, namely, regions of eyes/eye-brows, nose, and mouth from still images

• Detection and recognition of facial expressions - how facial muscle activations can change appearance of a face during emotional reactions?

• Expression-invariant recognition of facial identity


• Four regions of interest have been selected as most informative for further recognition steps

– right eye-brow / eye– left eye-brow / eye– nose– mouth

Methods of landmark extraction

• Uses knowledge on geometrical structure of human faces

• Based on geometrical features of facial landmarks, such as position of eyes/eye-brows, nose, and mouth

Feature-b ased method



Template-b ase method

• Represents a face as a feature map/template of original facial image

• Local oriented edges are used to construct a feature map of the facial image

• Orientation of edges has been determined with step of 22.5 and encoded as 0,1,…..15

01

2

15

3

4

5

6

789

14

13

12

11

10

22.5

°

Oriented edges extracted in left eye region



Database

• Tests were performed using Pictures of Facial Affect [1]

• 110 images with 7 basic facial displays: happiness, surprise, fear, anger, disgust and neutral expression

• Images were first normalized to three pre-set sizes 100X150, 200X300 and 300X400 in order to test the effect of image size to the operation of the algorithms

• In sum 110 x 3 = 330 images were used for algorithm testing

[1] Ekman, P., Friesen, W. V., & Hager, J.C. (2002) Facial Action Coding System (FACS). Published by A Human Face, Salt Lake City, UTAH: USA


Input face (RGB)Normalization (grey-level scale)

Different resolution levels

Resolution level 0Resolution level 2

Transformation

Pre-processing algorithms

• RGB – grey-level transformation

• Multiresolution image representation was performed using a recursive Gauss transformation

Facial landmark extraction


Final feature map

• Final feature map has been constructed on base of local oriented edges extracted in each point of grey-level image at each resolution level with exception of points which had the contrast values less than threshold

• Extraction of local edges has been performed by calculation of difference between two oriented Gaussians with shifted kernels, which allows determining both orientation and contrast of local edge

Map of detected points of interest

Points of interest have been grouped - if the distance between points of interest was less than the threshold the points were grouped - otherwise ignored



matching

0

100

200

300

400

500

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

2 3 4 5 6 10 11 12 13 14

Edge orientationN

umbe

r of

poi

nts

of in

tere

st

Right Eye

Left Eye

Nose

Mouth

Orientation portraits of the facial landmarks• Detected regions of interest have been

compared with orientation portraits of facial landmarks I have constructed earlier

• Regions which did not correspond to the portraits have been ignored

• Pre-knowledge about facial structure have been used



Finally, facial landmarks have been detected!

Examples of feature maps of high-contrast oriented edges detected from the expressive images


neutral disgust fear


Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes

70

80

90

100

L eye R eye Nose Mouth

Per

form

ance

(%)

100X150 200X300 300X450

Results


Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes

70

80

90

100

L eye R eye Nose Mouth

Per

form

ance

(%)

100X150 200X300 300X450

Results


Right Eye Left Eye

Nose Mouth

Performance of the feature detection system for three image sizes.N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust

(a)

(c) (d)

(b)

Results


Right Eye Left Eye

Nose Mouth

Performance of the feature detection system for three image sizes.N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust

(a)

(c) (d)

(b)

Results


• Algorithms are slow – about few seconds

• Errors in groupping points of interest (red rectangles a, b, c)

• Some landmarks are undetectable (d)

Results

Problems

(a) (b) (c) (d)


• To improve detection of nose and mouth regions two alternatives are proposed

- The first one is selection of different thresholds for detection and groupping of points of interest for different resolution levels

- The second alternative requires more careful processing of detected regions and searching different landmark parts such as eye and mouth corners and nostrils.

Results

Recommendations


Future steps

• Full article about automated expression-invariant detection of facial landmarks; short article about how emotions affect cognitive functioning and how this knowledge might be implicated for HCI

• To improve landmark detection; implement prototype of 2D recognition system of facial expressions (iExpRec)

• To implement and test iExpRec

• To implement of expression-invariant 2D facial identity recognition system (iFaceRec).


Thank you f r your attention!

Documents

Automated recognition of facial expressi ns and identity 2003 UCIT Progress Report