ICat as a Companion Robot
Siska FitrianieDragos DatcuAlin ChituL.J.M Rothkrantz
{s.fitrianie, d.datcu, a.g.chitu, l.j.m.rothkrantz}@ewi.tudelft.nlhttp://mmi.tudelft.nl
Man-Machine-Interactionhttp://mmi.tudelft.nl
2
Human AIBO Interactionhttp://mmi.tudelft.nl/~aibo
Man-Machine-Interactionhttp://mmi.tudelft.nl
3
Requirements and Challenges
• Requirements:– Develop a system architecture for a companion cat– Design and develop multimodal system to portray
emotions in iCat as a result of interaction with humans.– Design of a personality model adapted to iCat.
• Challenges:– Development of a complete set of an HCI system: input sensing, input
processor, input fusion, dialog management, output fission.– Use of iCat’s specific software that is new and still under development– Little is still known or standardized in the cognitive model for humans.
Man-Machine-Interactionhttp://mmi.tudelft.nl
4
Requirements in other words …
9:00 AM
11:00 AM 14:00 AM 16:00 AM
iCat! What’s today whether?
iCat!, Let’s play tic tac
toe!
iCat .. Read me a story, please ..
8:00 AM
Get a life! I am still sleeping!
What was that?? Hey .. where is everybody??
I feel so lonely!!! I wish I have a companion …
Are you out of your mind? I am sleeping!!!
Today is Monday, 27 March 2006 .. It will be
bright sunshine but a bit cold …
Hurray!! Finally I have a friend!!
OK!!I do the
first move!
OK, ok! I’ll do it. Don’t
cry ..
Man-Machine-Interactionhttp://mmi.tudelft.nl
5
Multimodal Human Computer Interaction
SpeechRecognition
Facial ExpressionRecognition
LanguageGenerator
EmotionAnalysis
Fusion Input &Interpretation
Dialog ActionManager
Fission +Emotion Generator
AnimationGenerator
Man-Machine-Interactionhttp://mmi.tudelft.nl
6
Fusion Input(Feature Level)
• Audio-Video Fusion for Automatic Speech Recognition
Speech in waveform
Mel-FrequencyCepstral Coefficients
(MFCCs)and
Lip Geometry Estimation(LGEs)
5-State Left-Right
HMMs
Recognized text
39 mfcc +
11 lge
Why don’t you come home alone uncle Oliver?
Face detection/ tracking
Mouth detection/ tracking
Man-Machine-Interactionhttp://mmi.tudelft.nl
7
Emotion Analysis:Emotion Extraction from Text (1)
VerbNet
AffectiveLexicon
Database
XML Ontology
text
UTTERANCE
CLAUSE
AGENT PREDICATE THEME ADVERBIAL
you treated me
wrong
EMOTION-OBJECT
CONJUNCTION
CONTRAST
But,
CLAUSE +
EmotionWords
Extraction
EmotionExtraction
fromSentence
Knowledgebase
Mapping toEmotion Type &Activation level
annotated text
41 emotiontypes & 3
activation levels
If (agent==pronoun)&&(theme==“myself”) and (emotion-object==adverbial)Then intention=“emotionally-directed”
If (intention==”emotionally-directed&& (pleasantness==”unpleasant”)Then emotion-type is “anger”
...
EmotionThermometers
<UTTERANCE> …. <CLAUSE></emotion emotion-type=neutral> <CONJUNCTION><CONTRAST>but</CONTRAST> </CONJUNCTION> <AGENT>you</AGENT> <PREDICATE>have treated</PREDICATE> <THEME>me</THEME> <ADVERBIAL> </emotion emotion-type=”anger” degree=“average”> <EMOTION-OBJECT pleasantness=”neg” degree=”average”> wrong </EMOTION-OBJECT> </ADVERBIAL> </CLAUSE> …</UTTERANCE>
Parsing treegeneration
Sentence types:(a) emotionally active toward an object(b) emotionally directed by an object(c) emotions that provoked by an object(d) emotions that experienced towards an object(e) appraisal toward an object
Man-Machine-Interactionhttp://mmi.tudelft.nl
8
Emotion Analysis:Emotion Extraction from Text (2)
Active
Passive
PleasantUnpleasant
Angry, apparent, attentive, comic, cordial,courteous, covetous, curious, darling, dear,eager, evident, fierce, fine, full, funny, furious,good, grabby, grasping, great, hearty, hot,jovial, merry, mordacious, nice, palpable, plain,respected, terrific, troubled, unquestionable,warm, well
1
Affable, anxious, appreciated,bright, correct, cute, itchy, right,superb, sweet, treasured,ungratified, untroubled
2
Agreeable, clear,dead, hopeful,indignant, kind,obvious, petrified
3
Comfortablefamiliar, positive,prosperous
4Amazed, astonished, astounded,calm, considered, cozy, dazed,dramatic, dumbfounded, exhausted,fatigued, fit, flabbergasted, serene,soft speechless, stupefied
5
Accepted, approved,bad, concentrated,desperate, elated,fearful, isolated, joyful,joyous, mean, terrible,tired, uncertain
6
Affected, annoyed, aroused, close, confused,contemptible, crazy, choleric, defeated,deplorable, depressed, disabled, disappointed,disgusted, dispirited, disturbed, down, excited,frustrated, gentle, gloomy, horrible, humiliated,ill, impetuous , improper, intent, irascible,irritated, lamentable, obsessed, obstinate, sad,shamed, sick, stimulated, stirred, unhappy,unreasonable, upset, worried
7
Tasty, tender8 flabbergasted
amazedastonished
tired
passivecalm
serenefrustated
defeated
disappointed
unhappy
disgustedworried
disturbed
dispiritedill
gloomy
fearful
sad
aroused
exited
abhorrentdesperate
eager
annoyedirritated
heartyangry
fiercefurious
indignantcurious
active
elatedjoyful
anxious
covetousgrabby
2D space of Affective Lexicon database based on relative distance of words meaning from WordNet and Multidimensional Scaling
Man-Machine-Interactionhttp://mmi.tudelft.nl
9
Emotion Analysis:Emotion Extraction from Facial Information
Viola & Jones features
Facial Characteristic Points
Face detection- Viola & Jones features- SVM classifier
Facial Characteristic Point (FCP) Detection- Viola & Jones features- SVM classifier
Facial expression recognition- FCP based model- SVM classifier
Face detection - Viola & Jones features - SVM classifier
Facial expression recognition - Viola & Jones features - SVM classifier
Man-Machine-Interactionhttp://mmi.tudelft.nl
10
Emotion Analysis:Emotion Extraction from Speech Prosody
Filtering
Segmentation - Multi-frame based analysis
Feature extraction
Emotion recognition - Boosting techniques, - SVM classifier
ROC graph for classifier selection
Man-Machine-Interactionhttp://mmi.tudelft.nl
11
Dialog Action Manager
Dynamic Context
Static Context
WorldModel
Intention/Belief
recognition
Linguistic/Semantic
Knowledge
DiscourseKnowledge
TaskKnowledge
User Model
PlanReasoning
DiscourseInterpretation
SemanticParsing
ResponsePlanning
FusedInput data
Response
ResponseContent
Assembly
DialogControl
InformationControl
Information
Task StateUser Mental
State
UserEmotion
State
DiscourseState
Man-Machine-Interactionhttp://mmi.tudelft.nl
12
Fission Output
World Model
OUTPUT
Fission
-USER ADAPTATION-CONTEXT/SITUATION ADAPTATION-SCHEDULING TIMING-SYNCHRONIZING MODALITIES
LanguageGeneration
EmotionGenerator
SpeechFacial
expression-MODALITY
DAM
Dialog action
-INFORMATION EXTRACTION (*-DIALOG PROVIDER
AnimationGenerator
User Model
Man-Machine-Interactionhttp://mmi.tudelft.nl
13
Fission Output:Language Generator
• Template-based (includes dialog act and emotion as parameters)
<aiml>
<dialog-act name="DIALOG-ACT"><topic name="TOPIC">
<user-group name="USER-GROUP"><appraisal name="APPRAISSAL">
<info name="INFO-TYPE">
<category><that>TOPIC-HISTORY</that><pattern>INPUT</pattern>
<template><random>
<li><sentence_lattice> ... </sentence_lattice></li>
... //other sentences
</random></template>
</category>
... //other categories
</info>
...//other categories with less available info
</appraisal>
... // other categories with different appraisal type
</user-group>
... //other categories for different user-group
</topic>
... //different topic
</dialog-act>
... // different dialog-act
</aiml>
Man-Machine-Interactionhttp://mmi.tudelft.nl
14
Fission Output:Emotion Generator (nPME model) - 1
Personality
Standards,Principles, Goals
Moods,Emotions
Behaviour(+ new need)
Content-basedInformatione.g. location
Interpretation of the world (external)Need(Internal)
Man-Machine-Interactionhttp://mmi.tudelft.nl
15
Fission Output:Emotion Generator (nPME model) - 2
Maslow Pyramid of Needs
OCEAN Model of Personality: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism
Mood: Level of valence and arousal.
Goals, Preferences, Standards (Rule-based)
41 Emotion Typesand
3 ActivationLevels
Man-Machine-Interactionhttp://mmi.tudelft.nl
16
The Point
• Symmetrical I/O modalities of user-system interaction
• Modular and general, as a starting point of any CHIM
• Possible applications:supervising, entertainment, tutor or help desk robot
• Current project development milestone:Input Sensing and Emotion Analysis