Affective Computing: Machines with Emotional Intelligence

Hyung-il Ahn

MIT Media Laboratory

• Expressing emotions

• Recognizing emotions

• Handling another’s emotions

• Regulating emotions \

• Utilizing emotions /

(Salovey and Mayer 90, Goleman 95)

Skills of Emotional Intelligence:

if “have emotion”

We have pioneered new technologies to recognize human affective information:

Sensors, pattern recognition and common sense reasoning to infer emotion from physiology, voice, face, posture & movement, mouse pressure

Mind-Read: Recognizing complex cognitive-affective states from joint face and head movements

Future “teacher for every learner”

Can we teach a chair to recognize behaviors indicative of interest and boredom? (Mota and Picard)

Sit upright Lean Forward Slump Back Side Lean

What can the sensor chair contribute toward inferring the student’s state: Bored vs. interested?

Results (on children not in training data, Mota and Picard, 2003):

9-state Posture Recognition: 89-97% accurateHigh Interest, Low interest, Taking a Break: 69-83% accurate

Detecting, tracking, and recognizing facial expressions from video (IBM BlueEyes camerawith MIT algorithms)

Complex Mental States(subset)

Concentrating

Disagreeing

Interested

Thinking

Unsure

AbsorbedConcentratingVigilant

DisapprovingDiscouragingDisinclined

AskingCuriousImpressedInterested Brooding

ChoosingThinkingThoughtful

BaffledConfusedUndecidedUnsure

Affective-Cognitive Mental StatesBaron-Cohen et al. AUTISM RESEARCH CENTRE, CAMBRIDGE

Agreeing

AssertiveCommittedPersuadedSure

Knows when a person/customer is:• Concentrating, and does not interrupt unless very

important• Thinking, and can pause to let you think• Unsure, and can offer to explain differently• (Not) interested in what it says• (Dis)agreeing, and can adjust response respectfully

Technology that understands and responds to human experience like a caring, respectful person would, for example:

Technology with people sense will perceive cognitive-affective states, e.g., before interrupting

hmm … Roz looks busy. Its probably not a good time to bring this up

Analysis of nonverbal cues

Inference and reasoning about mental states

Modify one’s actionsPersuade others

Feature point tracking

Head pose estimation

Facial feature extraction

Head & facial action unit recognition

Head & facial display recognition

Mental state inference

Hmm … Let me think about this

Experimental Evaluation ConclusionsInferring Cognitive-Affective State from Facial+Head movements (el Kaliouby, 2005)

Other examples:

Agree

Disagree

75% sit in front of computers (static)

Back pain/injury = #2 cause of missed work

Physical movement helps prevent/reduce back pain

Goals :- Fostering healthy posture- Building social rapport- Improved task performance

(Affect-Congruent behavior)

Robotic Computer (RoCo) :World’s first physically animated computer

Animated Desktop Monitor: RoCo = Robotic Computer

QuickTime™ and aCinepak decompressor

are needed to see this picture.

RoCo Behavior

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

• NOT when: you’re concentrating, interested, in the middle of an engaging task, or otherwise attentive/focused on the monitor’s content.

• Might make a micro-movement when you’re looking away or blinking in the middle of a task.

• Might make a larger movement to attract a new user, bow to welcome, or when user shifts tasks and hasn’t shifted posture (etc.)

When should RoCo move? (Future work & not topic of this paper, but important to mention)

RoCo’s postures congruous to the user affect

N=(17)

“Stoop to Conquer” : Posture and affect interact to influence computer users’ comfort and persistence in problem solving tasks

People tend to be more persistent and feel more comfortable when RoCo’s posture is congruous to their affective state

“Stoop to Conquer”: Posture congruent with emotion improves persistence (# tracing attempts, two different experiments)

RoCo’s Posture:

Human State:

Slumped Neutral Upright

Success (“you scored 8/10”) N=30

8.2 8.3 12.0

Failure (“you scored 3/10”) N=19

9.6 7.4 6.9

A multi-modal affective-cognitive measures for product evaluation with computational models of predicting customer decisions

Predicting customer product preferences by combining information about emotion and cognition

We are creating new computational models to measure human affective experience and to predict human decision-making & preference

Background findings to inform new research:

The brain uses both emotion (affect) and cognition in decision making

-> model should combine both affect and cognition

A person in an experiment is likely to cognitively bias their self-report of what they like.

-> method should not rely on only self-report

When a person is cognitively loaded they are more likely to use emotion in decision-making.

-> method should slightly load person cognitively

Background findings to inform new method:

Multiple measures of affect provide most robust assessment:

-> method can measure affective physiology (face, skin conductance) as well as behavior and self-report

Sweeter beverages are preferred on the first sip; long-term accumulation of something mildly bad is required before it is “bad enough to notice”

-> method should require lots of sips of every beverage

More complete understanding of consumer desire

Skin Conductance

ANTICIPITORY FEELING

Arousal

Multi-Dimensional

Response Physical

NUMBER OF SIPSAmount Consumed

Facial Expression

AFFECTIVE LIKING

Emotions

Self ReportCOGNITIVE LIKING

Purchase intent Liking

Expectation

Videos of Testing• Here is a sneak preview of my project. Make sure to look

for consumers emotions that may not be captured in self reported questions.

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Test Products

• Stronger Performer – – Pepsi Vanilla

• Performed in top 25%, green region, in Directions HUT

• Weaker Performer – – Pepsi Summer mix

• Performed in lower 40%, lower yellow region, in Directions HUT

Products chosen with clear performance differences

• Two techniques performed simultaneously – Facial Imaging and Head Positioning

Tracking face muscle movements to interpret emotions

– Galvanic Skin Response (GSR)Measures Arousal, used as an intensity measure for emotions

Affective Computing

Facial HeadExpression Position

• Concentrating

• Thinking

• Confused

• Interested

• Agreeing• Disagreeing

Affective-Cognitive Mental States

GSR Shows Intensity

+ = Interpretation

Method: Choice Technique

• Choice technique - respondent selected one of two vending machines

– Process is repeated 30 times– Eventually respondents realized each machine favors a

different product and will select the vending machine hoping to receive their favored product

– 70/30 probability of either product coming out of either machine

Two cups on each side of the computer: Pepsi Vanilla and Pepsi Summer Mix

Use of straws avoided blocking facial reaction

Method - General Set-Up

Machine 2Machine 1

135 246 135 246

Experimental Set Up

Machine Selection Sip on Resulted Beverage Answer Questions

Method - Step 1

• Each vending machine directed you to sip a beverageRANDOMLY CHOOSE A VENDING MACHINE

Method- Step 2

RESPONDENTS SIP RESULTED BEVERAGE

Method – Step 3• Answer Questionnaire used in standard CLT

– Overall Liking (beverage and machine)– Purchase Intent, Comparison to Expectation

• Reselect a machine • 30 machine selections were made

Method – Step 4

Data collection timeline

Start

SelectOutcome

Evaluate Start (Next trial)

Choice 2 70% Mix30% Vanilla

Choice 1 70% Vanilla30% Mix

Measuring

ANTICIPITORY FEELING(hope/dread)

Skin conductance

vanillaor mix

Sip

How muchdo you like

the sip?

Measuring AFFECTIVE LIKING

(initial reaction)

Facial expressionSkin conductance

Measuring

COGNITIVE LIKING

(post reaction)

Self-report

Question

Data collected throughout experiment

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Videos of Testing

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Videos of Testing

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Videos of Testing

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Videos of Testing

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Analysis• Our hypothesis is that joining quantitative and qualitative methodologies will help provide understanding of consumers’ real product evaluations

Discussion

Skin Conductance

ANTICIPITORY FEELING

Arousal

Multi -Dimensional

Response

PhysicalNUMBER OF SIPSAmount Consumed

Facial Expression

AFFECTIVE LIKING

Emotions

Self ReportCOGNITIVE LIKING

Purchase intent Liking

Expectation