Upload
ryan-thornton
View
223
Download
4
Tags:
Embed Size (px)
Citation preview
A FACEREADER-DRIVEN 3D EXPRESSIVE
AVATAR
Crystal Butler | Amsterdam 2013
2
Key FramesAction Units
QuantitativeBasis
Action unit combinations forthe basic emotions of disgust (hindering)and happiness (facilitating) were chosen
from the Facial Action Coding Systemby Ekman, Friesen and Hagar (2002)
and varied based on happinessresponse intensity of the source data.
Action Units
Examining inflection points onthe averaged response curve in
conjunction with visual inspectionof the video stimulus revealed key
transition points for facial expressions.
Key Frames
Happiness response curves gathered byFaceReader analysis from a prior studywere transformed to create intensity
dynamics for animated avatars. The avatars were designed to hinder or
facilitate happiness responses of participants.
Quantitative Basis
Facial Mimicry Experiments
THE ANIMATION DEVELOPMENT PROCESS
2
3
Data from Source Experiment
FACIAL EXPRESSIONS OF HAPPINESS, CONTROL CONDITION
Below is an intensity graph of averaged participant happiness responses over the course of viewing an amusing commercial. From the
original data set, only 12 participants who had consistently good quality facial fitting results in FaceReader were used. Of a potential
peak intensity of 1, the maximum average happiness score was .2433.
3
4
Data from Source, Normalized
(Hi -Hmin)/(Hmax/Hmin)
In order to develop more discriminable facial expressions for the avatar animation, the original data was normalized to span the full
range of possible FaceReader emotion scores from 0-1.
4
5
Video Fear Face
HAPPINESS DIPS DURING FEAR DISPLAY
5
QuickTime™ and aH.264 decompressor
are needed to see this picture.
6
Inflection points are frames at which the slope of the curve, dy/dx calculated using happiness measurements on either side of
the frame, changes sign, indicating that a local maximum or minimum has been reached and a change in trend of expressive
intensity has occured. Areas of high standard deviation were determined by calculating the SD at a point using its value plus the
five frames to either side. Frames with SDs in the top 30% form areas where rapid changes in happiness intensities are
occuring relative to the average.
vs
Determining Key Frames
High SD AreasInflection Points
6
7
Key Frames: The Eyes Have It
7
VISUAL INSPECTION OF INFLECTION POINTS ON THE VIDEO WINS
8
Hindering Expressions: Disgust
The FACS Investigator’s Guide lists six Action Unit (AU) combinations typical of disgust. Three involve action unit 9, which wrinkles the nose and is thought to be an innate physiological
response to noxious odors. Action Unit 10 is indicated in the other three combinations; it is also found in expressions of anger. Because of these differences, combinations with AU 9
were reserved for video scenes focused on food (Doritos) or the goat. Action Unit 10 combinations were applied to scenes in which the focus was on the human actor. In order to create
expressions strong enough to be recognizable, an intensity floor of .3 was applied to the average happiness measurements and then the AU intensities were normalized to range from
0-1. Combinations with a greater number of AUs were considered to be more intense, and were further broken down into slight, moderate, and strong expressions. Thresholds for AU
intensities were determined by dividing each group by nine.
Applied to happiness intensities from 0-.33.Slight: .11 = .377 normalizedModerate: .22 = .454 normalizedStrong: .33 = .531 normalized
AU 9 AU 10
9+17
8
Applied to happiness intensities from .34-.66.Slight: .44 = .608 normalizedModerate: .55 = .685 normalizedStrong: .66 = .762 normalized
9
9+16+25+26
Applied to happiness intensities from .67-.99.Slight: .77 = .839 normalizedModerate: .88 = .916 normalizedStrong: .99 = .993 normalized
10
Applied to happiness intensities from 0-.33.Slight: .11 = .377 normalizedModerate: .22 = .454 normalizedStrong: .33 = .531 normalized
Applied to happiness intensities from .34-.66.Slight: .44 = .608 normalizedModerate: .55 = .685 normalizedStrong: .66 = .762 normalized
Applied to happiness intensities from .67-.99.Slight: .77 = .839 normalizedModerate: .88 = .916 normalizedStrong: .99 = .993 normalized
10+17
10+16+25+26
9
About us
Key frames are illustrated with clips from the Doritos Goat 4 Sale video and renderings of the avatar’s expression as it would appear at that moment. The video frame number is given, along with the point Happiness Average (HA) and corresponding Action Unit designations per key frame.
Key Frames
Indicated within red arrow boxes
Based on intensity dynamics of the happiness graphSubjective use of additional AUs or modified intensities to create a more natural flow
Transitions
Storyboarding: Disgust
9
10
Hindering Video
DO EXPRESSIONS OF DISGUST REDUCE FEELINGS OF HAPPINESS?
10
11
Lip Corner Pull Duchenne Smile Open-lipped Smile Open-mouthed Smile
Facilitating Expressions: Happy
11
The FACS Investigator’s Guide lists six only two AU combinations typical of happiness, both closed-mouth smiles. Two more were created by adding AU 25 and AU 25+26 for open
smiles with greater intensity. In order to create expressions strong enough to be recognizable, an intensity floor of .3 was applied to the happiness measurements and then the AU
intensities were normalized to range from 0-1. Combinations with a greater number of AUs were considered to be more intense, and were further broken down into slight, moderate,
and strong expressions. Thresholds for AU intensities were determined by dividing the four combinations into three subgroups, for a total of twelve intensity ranges.
12 D
Applied to happiness intensities from 0-.2475.
Slight: .3583 normalizedModerate: .4167 normalizedStrong: .475 normalized
6+12
Applied to happiness intensities from .2476-.495.
Slight: .5333 normalizedModerate: .5917 normalizedStrong: .6497 normalized
6+12+25
Applied to happiness intensities from .496-7425.
Slight: .708 normalizedModerate: .7664 normalizedStrong: .8247 normalized
6+12+25+26
Applied to happiness intensities from .7426-.99.
Slight: .8831 normalizedModerate: .9414 normalizedStrong: .9999 normalized
12
About us
Key frames are illustrated with clips from the Doritos Goat 4 Sale video and renderings of the avatar’s expression as it would appear at that moment. The video frame number is given, along with the point Happiness Average (HA) and corresponding Action Unit designations per key frame.
Key Frames
Indicated within red arrow boxes
Based on intensity dynamics of the happiness graphSubjective use of additional AUs or modified intensities to create a more natural flow
Transitions
Storyboarding: Happy
12
13
Facilitating Video
DO EXPRESSIONS OF HAPPINESS MAGNIFY THAT EMOTION?
13
WHY
AVATARS?
15
This group from Northeastern University, USA, is working on medical applications
including:
Relational Agents Group
Some Current Research
AVATARS AND AGENTS
Affective Social Computing Lab Rachael Jack Institute for Creative Technologies
15
Christine Lisetti, Florida International University, develops FACS-based virtual
counselors to provide mental healthcare:“On-Demand VIrtual Counselor (ODVIC)
In this project, we design and implement the prototype of On-Demand VIrtual Counselor (ODVIC) intelligent virtual characters who can provide people access to effective behavior change interventions and help them find and cultivate motivation to change unhealthy lifestyles (e.g. excessive alcohol use, overeating).
An empathic Embodied Conversational Agent (ECA) delivers the intervention. The health dialog is directed by a computational model of Motivational Interviewing, a novel effective face-to-face patient-centered counseling style which respects an individual’s pace toward behavior change. “
Dr. Jack’s work at the University of Glasgow uses avatars generated by a FACS-based facial grammar to create expressions and
test recognition across cultures:
This University of Southern California group investigates a variety of uses for virtual
humans, focusing on education and training:
16
QuickTime™ and aH.264 decompressor
are needed to see this picture.
About us
Mirroring AU output for reliability checks or user feedback
Anonymize participant videos to alleviate privacy concerns
Create and identify a wide range of subtle expressions beyond the set of basics
Potential Applications
For FaceReader
16
Provide an instructional avatar within the interface to provide procedural guidelines and offer helpCreate customized avatars that allow for the study of reactions based on race, gender, and ageStudy the effect of the presence of an ‘other’ in various scenarios
Integrate as a tool for recognizing and coding Action Units (as in video at right)
Use FaceReader as an engine rather than an end product for driving affective, human-agent interactions
The Future?
Technology and Features
Avatar Creation Process
Goal: Use FaceReader’s Action Unit, Head and Eye
Poses, and Mask Data to drive a 3D Avatar
that Mirrors User Expressions in Real
time
19
faceshift for Modeling
AUTOMATED MODELING USING THE KINECT
19
20
Maya for Motion and TextureMANUAL ADJUSTMENTS AND ADDITIONS OF BLENDSHAPES
20
21
About us
Read the Action Unit values from the live FaceReader feed via a comma-delimited text file
Update the blendshapes in Maya using those AUs
Get the head angle values from the FaceReader text file and apply them to the neck joint
MEL Commands
Scripting in Maya
21
Get the face mask texture from the live FaceReader feed and apply it to the UV map on the model Grab a point color from the current mask texture and apply it to the shading node for the head
Set an animation key frame so the capture can be played back
Check for a radio button change indicating that the current facial texture should be fixed
Hair, eye movement, blending at the edge of the facial mask, and SPEED!
What’s Missing?
22
QuickTime™ and aH.264 decompressor
are needed to see this picture.
The BetaPRIOR TO APPLYING TEXTURES OR ADJUSTING BLENDSHAPES (AND TO SHOW OFF VOICE ACTIVATION!)
22
23
The Avatar NowFACEREADER VS FACESHIFT SMACKDOWN
23
24
Subtle Expressions
SOME EXAMPLES FROM FACEREADER AU VIDEO CAPTURES
24
25
About us
Surprise is the most well-recognized emotion
Disgust is the most poorly recognized emotion
Anger and disgust are frequently confounded
General Observations
FaceReader AU Recognition
25
A PILOT STUDY OF FACEREADER RESPONSES TO TYPICAL AU COMBINATIONS FOUND IN BASIC EXPRESSIONS OF EMOTION
Data is based on a series of 37 videos made to display the Action Units that comprise the prototypes and major variants of the basic emotions according to the FACS Investigator’s Guide. Each video is 30 seconds, with 3 seconds of neutral expression at the beginning and end and a full range of AU intensities that peak around 15 seconds. A full analysis of each video with emotion and AU recognition is available.
Fear is often mistaken for surprise
Due to the absence of AU 11, AUs 9 and 10 are mistakenly coded in sadness
Discrimination between AUs 9 and 10 is poor, as is differentiation between AUs 23 and 24
WHO WANTS TO
TRY IT?