INTERACTING WITH AIMODULE 3Session 3, November 10, 2020
Amela Karahasanović, SINTEF and UiO
Module 3Living and working with AI
Objectives
Understanding of challenges related to use of AI infused systems in everyday life and at work
How to evaluate them?
When and how to use them?
What do we know about living and working with them?
2
Module 3Overview
Evaluation of interaction with AI [27th of October]
Human - AI partnership [3rd of November]
Lessons learned from studies of human – AI interaction [10th of November]
3
EXAMPLES
Human – human interaction -> Human - AI interactionHow to create patterns (building blocks)
for human interactions that can be used for human-robot interactions?
Interaction design patterns Interaction blocks - a visual prototyping
environment Evaluating usability of the environment
Sauppe, A., Mutlu, B., Design Patterns for Exploring and Prototyping Human-Robot Interactions, In Proc. of CHI 2014, 1439-1448, ACM
6
Human – human interaction
A lot of studies in linguistic and psychology on behaviour, but design patterns that can be translated in human-robot interactions were missing
16 participants from UoWM (19-62) assigned to 5 different interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation – discuss their educational experiences and
goals Collaboration – sorting the grocery bags Instructions – assembling some pipeline constructions Interview – a job interview, 14 questions Story telling – retelling a video, 7 minutes video
Human – human interaction
Analysis – coding (one researcher, 10% also by the other researcher to confirm reliability)
Codes for states – important events in interaction (question, answer) and the transition between the states to discover patterns
Identified 7 patterns
Design patterns for exploring and prototyping human-robot interaction
9
• Introductory monologue "Hi Welcome…"
• Questions and answer pair Storyteller: "Do you know who Marvin the Martian is?", Listener: "Oh yeah,…"
• Generic comment and personal comment "Interesting" "I had a similar experience once…"
• Monologue and generic commentStoryteller: "…and then, all of the sudden, thousands of these aliens appear at the Earth." Listener:"That's a lot"
Design patterns for exploring and prototyping human-robot interaction
10
• Instructions and actionInstructor: "Now connect the long pipe with the one shaped like an S" Student: <locates the pipe and connect it>
• Finished commentParticipant: "I think that's it, so we should be done"
• Wait (while the other participant is talking)
11
Implemented and evaluated an authoring environment based on these patterns with interaction designers and developers
13
Your turn Where studies of human-human interaction systems can
be useful? For which kind of systems? Which methods can be used for data collection?
1-2 minutes note for yourself
14
EXAMPLES
Evaluating chatbots
Evaluating and Informing the Design of Chatbots, Jain M. et al. 2018, DIS 2018
Evaluation of 8 chatbots with 16 first-time chatbot users over multiple sessions on the Facebook Messenger platform
Three days interactions with chatbots
Face-to-face semi-structured interview with the participants to elicit their understanding of the chatbots, perceived benefits/limitations, interesting conversations/experiences and areas of improvements
Quantitative data analysis: total interaction time, message count, interactive elements (composition of the chatbots' and participants' messages)
16
Chatbot evaluation
17
Chatbots: a trave agent, entertainment, a shopper, news, weather forecast, chit-chat, social chat, a quiz game
Quantitative data• similar amount of time with each chat-bot, the total
number of messages exchanged significantly different (chit-chat > shopper)
• game and chit-chat bots had more chatbot messages
Qualitative data
Three researchers developed together iteratively a coding schema
4 high-level themes:functionalityconversational intelligence
personality interface
18
Design implications
Chatbot designers
Clarify capabilities at the start and on-demand
Evaluate application-interface match
adding links to external webpages is not recommended; clear added value compare to search engine
Enable dialog efficiency through context resolution
Remain context from earlier conversations, intelligent questions
Consistent personality with small-talk and humor
Design for dialog failure – admitting failure
19
Design implications
Chatbot platform UX designer
Combine text-based interface with buttons and media
participants did not like when the chatbot, for example, opened a news article in a new browser window
Enable efficient input from users
Auto-suggestion buttons
Provide persistent view on chatbot capabilities and context
In the beginning, but also later, keeping the chatbot and the user in the same state of mind
20
Your turn What have you found in you chatbot evaluations? What
would you recommend to chatbot designers?
1-2 minutes note for yourself
21
EXAMPLES
ATM exampleEvaluation of DAC with ATCOs
Karahasanovic et al., 2019, Supporting Air Traffic Controllers During Sector Configuration Changes in Dynamic Airspace Configuration, SID 2019
23
Data collection
Observations
Log files (simulator, UI)
Screen captures
Video recording of screens
Interviews (audio records)
Audio records of the communication between the ATCOs and pilots
Questionnaires
24
Results overview
The overall workload - acceptable and the same or slightly reduced for DAC scenario
Few errors - mostly due to the simplifications of the protype's UI
Overall feedback positive
"Good", "almost perfect", "it enabled us to easily maintain the picture of the traffic"
Solutions for notifying and visualizing both horizontal and vertical changes were useful
Some improvements proposed
25
Results overview cont.
Situation awareness was very good DAC concept potentially useful for supervisors There is a need for a better alignment of the
change with the ATCOs workload
26
MED MIN MAX STDEV
All sessions 8 5 10 1,43
Ref. scenario 8 5 10 1,31
Sol. Scenario 8 5 9 1,56
EXAMPLES
RoboCare
• A system that assists elderly at home• Evaluation
• Video based trials, 8 short movies with different domestic scenarios
• User initiative and system initiative• 100 participants (59 – 90 years)• Questionnaire (background, evaluation)
Cesta et al. Monitoring elderly people with the robocare domestic environment: interaction synthesis and user evaluations, Computational Intelligence, Vol 27, 2011
Findings
High usefulness and acceptability
Useful for personal and environmental safety, reminding to take medications and finding objects
Not useful for providing suggestions
Appreciated help with cognitive problems (reminding them to do something, finding things)
Your turn What have you read/seen (news, articles, videos) about
validations of robots in care of elderly? You can check it quickly now.
4-5 minutes note for yourself
30
EXAMPLES
Can robots manifest a personalityAIBO (robot dog) 48 participants based on the personality scores Randomly assigned to the introvert or extrovert robot(4 conditions) Verbal cues Asked to interact with AIBO (verbally and tactile commands) AIBO respond (extrovert turn faster and wider) Questionnaire on personality, intelligence and attractiveness of AIBO Participant enjoyed the interaction more with a compatible robotLee, K. M., Peng, W., Jin, S., & Yan, C. (2006). Can robots manifest personality? An empirical test of personality recognition, social responses, and social presence in human-robot interaction. Journal of Communication, 56(4), 754-772.
EXAMPLES
Trust in automation An experiment to investigate how the experience of automated
driving change trust in automation and attitudes towards automation 72 participants different ages 19 – 79 (a half under 30) Questionnaire before and after a driving experience in the simulator Gaze recording Highly automated driving on a three line highway, 120 km/h Three take over scenarios, warned 7 seconds in advance, different
traffic densities Older participants more positive Driving increased trust
Trust in Automation – Before and After the Experience of Take-over Scenarios in a Highly Automated Vehicle, Gold et al., AHFE 2015
Module 3
35
The End