Static Facial Expression in Tough Conditions: Data, Evaluation Protocol and Benchmark

http://cs.anu.edu.au/fewhttp://cs.anu.edu.au/few

Abhinav Dhall

Roland Goecke

Simon Lucey

Tom Gedeon

1

Content

� Why?

� Database creation

� Comparison

� Protocols� Protocols

2

Motivation

� Facial Expressions are the facial changes in response to a persons Internal

Emotional States, Intentions, or Social Communications. Automatic Facial

Expressions Analysis is the science of making computer understand “empathy”

– the art of understanding others emotions.

� Real world application• Affective Computing

• Human Computer Interaction• Human Computer Interaction

• Intelligent Environments

• Emotion and Paralinguistic Communication

• Lie Detection

• Medical Conditions

� Depression, Autism, Pain Analysis

3

Real world images, Source Flickr CC license

Motivation

� Facial Expressions are the facial changes in response to a persons Internal

Emotional States, Intentions, or Social Communications. Automatic Facial

Expressions Analysis is the science of making computer understand “empathy”

– the art of understanding others emotions.

� Real world application• Affective Computing

• Human Computer Interaction• Human Computer Interaction

• Intelligent Environments

• Emotion and Paralinguistic Communication

• Lie Detection

• Medical Conditions

� Depression, Autism, Pain Analysis

� Can be categorized into• Video based analysis: Movies, YouTube!

• Image based analysis: Flickr, Facebook

� Geometric/Appearance

4

Real world images, Source Flickr CC license

Motivation

� Computer vision is a data driven research!

� A lot of interesting datasets available:

• Belfast [1]

• CMU PIE [2]

• FEEDTUM [3]

• JAFFE [4]

• MultiPIE [5]

• MMI [6]

• CK+ [7]

• Semaine [8] ……

5

Images from JAFFE and MultiPIE

Motivation

� Computer vision is a data driven research!

� A lot of interesting datasets available:

• Belfast [1]

• CMU PIE [2]

• FEEDTUM [3]

• JAFFE [4]

• MultiPIE [5]

• MMI [6]

• CK+ [7]

• Semaine [8] ……

But majority of them are recorded in “lab” control led environment!

6

Images from JAFFE and MultiPIE

Motivation - 2

� Labeled Faces In The Wild (Huang et al.)

7

Motivation - 2

� Labeled Faces In The Wild (Huang et al.)

� Hollywood action database (Laptev et al.)

8

Motivation - 2

� Labeled Faces In The Wild (Huang et al.)

� Hollywood action database (Laptev et al.)

� There is a void for similar databases in the facial expression research community.

9

Motivation - 2

� Labeled Faces In The Wild (Huang et al.)

� Hollywood action database (Laptev et al.)

� There is a void for similar databases in the facial expression research community.

� We present two databases extracted from movies:

• Acted Facial Expressions In The Wild (AFEW) - Temporal

• Static Facial Expressions In The Wild (SFEW) - Static

� Use Subtitle for Deaf and Hearing impaired (SDH) subtitles

10

Database Creation

11

56631 subtitles -> 7025 expressive subtitles -> 957 clips

AFEW creation steps

Database Statistics

� Expression - Theme expression of the scene

� StartTime - This denotes the start timestamp of the clip in the movie DVD

� Length - Duration of the clip in milliseconds

� Person - This contains various attributes describing the actor / character in the scene

• Pose - This denotes the head pose based on the labeler’s observation

• AgeOfCharacter – Age of character in the movie

12

• AgeOfCharacter – Age of character in the movie

• NameOfActor - Real name of the actor

• AgeOfActor – Age of the actor extracted the information from www.imdb.com

• ExpressionOfPerson – Expression category of the character

• Gender - Gender of the actor

� Number of Clips – 957

� Total number of samples – 1259

� Expression category – Angry, Disgust, Fear, Happy, Neutral, Sad and Surprise

� Length – 300-5400 ms

SFEW

� Static Facial Expressions In The Wild

� Frames extracted from AFEW

� 95 Subjects

� Expression: Angry, Disgust, Fear, Happy, Neutral, Sad and Surprise

� PPI protocol part of BEFIT challenge

13

Samples from SFEW

Comparison with other databases

� Comparing

• SFEW with JAFFE and MultiPIE databases

• AFEW with CK+ database

DB Const.

Method

Environ. Age

Range

Illum. Occlus. Search. Subject

Details

Multiple

Subject

14

Method Range Details Subject

AFEW Assisted Real 1-70 CTN Yes Yes Yes Yes

CK+ Manual Lab 18-50 C No No No No

SFEW Manual Real 1-70 CTN Yes No No No

JAFFE Manual Lab ? C No No No No

M-PIE Manual Lab 27.9

(avg)

C No No No No

Database comparison. Here CTN – Close To Natural, C - Controlled

Comparison with other databases

• Face detect

• Descriptors

• Local Binary Pattern (LBP) [8]

• Pyramid of Histogram Of Gradient (PHOG) [9]

• Local Phase Quantization (LPQ) [8]

• LBP-Three Orthogonal Planes (LBP-TOP) [8]

• Support Vector Machine (SVM)

15

Classification accuracy comparison for LBP, PHOG, LPQ and LBP-TOP for CK+ and AFEW

• Support Vector Machine (SVM)

Comparison with other databases

16

Classification accuracy comparison for PHOG and LPQ SFEW, MultiPIE and JAFFE

Protocols

Protocol SFEW AFEW

Training & Testing sets have:

Strictly Person Specific (SPS) same single subject same single subject

Partial Person Independent

(PPI)

seen & unseen subjects seen & unseen subjects

(PPI)

Strictly Person Independent

(SPI)

unseen subjects (SFEW

challenge @ BEFIT challenge)

unseen subjects

Cross Database Set Test sets only Test sets only

17

Different protocol for SFEW and AFEW

The results should be reported as correct classification accuracy, precision and recall .

Conclusion

� We have proposed two new “in the wild” datasets

� A semi-automatic approach based on subtitle parsing based

recommender is used for database collection

� The database has been annotated with dense attributes about the

subjects

18

subjects

� The only labeled multi people expression database

� Multiple protocols for experiments based on subject dependency

� The datasets and related information will be updated at

http://cs.anu.edu.au/few

References

1) E. Douglas-Cowie, R. Cowie, and M. Schröder. A new emotion database: Considerations, sources and scope.

In Speech Comm., 2000.

2) Terence Sim, Simon Baker, and Maan Bsat. The CMU Pose, Illumination, and Expression Database. IEEE

Transactions on Pattern Analysis and Machine Intelligence, 25(12):1615–1618, December 2003.

3) Frank Wallhoff. Facial expressions and emotion database, 2006. http://www.mmk.ei.tum.de/

waf/fgnet/feedtum.html.

4) M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In

Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition and Workshops,

FG’98, 1998

5) Ralph Gross, Iain Matthews, Jeffrey F. Cohn, Takeo Kanade, and Simon Baker. Multi-PIE. In Proceedings of 5) Ralph Gross, Iain Matthews, Jeffrey F. Cohn, Takeo Kanade, and Simon Baker. Multi-PIE. In Proceedings of

the Eighth IEEE International Conference on Automatic Face and Gesture Recognition, FG’2008, pages 1–8,

2008.

6) Maja Pantic, Michel Franc¸ois Valstar, Ron Rademaker, and Ludo Maat. Web-based database for facial

expression analysis. In Proceedings of the IEEE International Conference on Multimedia and Expo, ICME’05,

pages 317–321, 2005.

7) G. McKeown, M. F. Valstar, R. Cowie, and M. Pantic, “The semaine corpus of emotionally coloured character

interactions,” In Proceedings of the IEEE International Conference on Multimedia and Expo, ICME’10.

8) D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen, “Local binary patterns and its application to facial

image analysis: A survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: applications and

Reviews, pp. 1 –17, 2011.

9) A. Bosch, A. Zisserman, and X. Munoz, “Representing Shape with a Spatial Pyramid Kernel,” in Proceedings of

the ACM International Conference on Image and Video Retrieval, ser. CIVR ’07, 2007, pp. 401–408.

19

Thank You

20