View
230
Download
9
Category
Preview:
Citation preview
Complementing Embodied Conversational Agents by Implementing ComplimentsMeasuring the Effect of Compliments in Embodied Conversational Agents
Group 23 Marieke Agterbos | Tim van Bremen | Carel Jansen | Lisa Oud | Justin Post
Research Question
To what extent do compliments have an impact on the
user experience of embodied conversational agents?
Questions
1. Are the ECA’s compliments recognized as compliments? (Part
I)
2. Experience with vs. without compliments (Part II)
A. Perceived liking of the agent
B. Perceived level of comfort
C. Perceived information quality
Background theory
Making machines more human (Picard, 2000)
Minimize gap between real life contact and digital contact
Lifelike ECA (Louwerse et al., 2009)
Is sensitive to moods and sentiments
Politeness and variation in language (Behnam & A. Niloufar, 2011)
Add value and believability
Function of Compliments (Manes en Wolfson, 1981)
Goodwill and solidarity
Three types of personal compliments (Jucker, 2009)
Implicit, explicit or indirect
Behavioral realism might be more important than photorealism (Groom, 2009)
Generating positive response
More productive conversation
Pre-test
Are the ECA’s compliments recognized as compliments? Participants read a transcript of an ECA conversation
Questionnaire on different parts of the conversation:
1. Bot: ‘Hello, how are you?’2. User: ‘Great, and you?’3. Bot: ‘Good, so am I.’4. User: ‘Good.’5. Bot: ‘What was the name of the lead guitar player of the band called “A band of Gypsies?”’6. User: ‘Jimi Hendrix.’7. Bot: ‘That is correct.’8. Bot: ‘Someone told me you are very bright.’
Pre-test results
‘Yes, this is a significant difference (p<0.01)’
84 participants (53, 31)
Pre-test results
Explicit Implicit Indirect
Non-contextual
‘You are such a nice person!’
‘People like you make this world a better
place.’
‘Someone told me you are very bright.’
‘I wish everyone was as nice as you.’
‘It seems like I’m having a conversation with an intellectual.’
‘My colleague told me you have a great
personality.’
Contextual
‘You are smarter than most respondents!’
‘You seem to know your instruments!’
‘According to our system, you score better than most
people!’
‘You answered really fast!’
‘It seems like you know your politics!’
‘According to our database, you know a
lot about music!’
Experiment Design
Measured variables Liking
Comfort
Information Quality
Contextual Non-contextual
Based on input data Based on ‘nice-to-hear’statements
Related to conversational content Unrelated to conversational content
‘You seem to know your instruments!’
‘People like you make this world a better place.’
Experiment
53 Participants
Conversation with bot Quiz setting
Questionnaire Perceived liking
Perceived level of comfort
Information quality
“How many strings does a guitar have?”“Who is the prime minister of the Netherlands?”
Results
Effects
Results H1. Contextual compliments have a positive effect on the user experience of an ECA.
H1-a. perceived liking of the agent.
H1-b. perceived level of comfort.
H1-c. perceived quality of information.
H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.
H2-a. perceived liking of the agent.
H2-b. perceived level of comfort.
H2-c. perceived quality of information.
H3. Contextual compliments have a more positive effect than non-contextual compliments.
H3-a. perceived liking of the agent.
H3-b. perceived quality of information.
H3-c. perceived level of comfort.
Results H1. Contextual compliments have a positive effect on non-contextual
compliments on the user experience of an ECA.
H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.
H3. Contextual compliments have a more positive effect than non-contextual compliments.
H4. Compliments have a positive effect on the user experience of an ECA.
H4-a. perceived liking of the agent.
H4-b. perceived quality of information.
H4-c. perceived level of comfort.
Conclusion
Compliments have a positive effect on the user experience of an ECA.
However, not on likingness.
The type of compliment matters
Non-contextual > contextual
Discussion
Pros:
Pretest, kind of compliments accounted for.
Pretest in line with experiment outcome.
All experiments under supervision of a test leader.
Improvements:
Sample too small
Language, not native language of all subjects
Variable mount of compliments given
Compliments when correctly answered
Definition of comfort, persuasiveness, likeliness might differ individually
Questions?
Measuring - Liking
‘I would like to be friends with her’
‘If I was upset, she would be able to cheer me
up.’
‘She is a bad listener’
Rated on Likert scale 1-7
Measuring - Level of comfort
During the conversation, I felt:
Comfortable
Involved
Open
Relaxed
Warm
Rated on Likert scale 1-7
Measuring - Information quality
The extent to which the information was:
Correct
Helpful
Persuasive
Insightful
Relevant
Rated on Likert scale 1-7
Context
Controlled environment
Participants from same population
Recommended