From Interaction to Understanding
Mark Billinghurst
The HIT Lab NZ, University of Canterbury
May 21st 2015
Lesson’s Learned ! Enjoy the journey ! Travel with others ! Have a well equipped vehicle ! The end always seems nearer than it really is ! Don’t be disappointed in the outcome ! Most Important: Have a Clear Vision
A Memex is a device in which an individual stores all his books … and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.
Vannevar Bush
The Ultimate Display The ultimate display would .. be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs .. would be confining, and a bullet .. would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.
Ivan E. Sutherland
Sutherland, I.E. The ultimate display. Information Processing 1965, Proc. IFIP Congress 65, 506-508.
Realizing the Vision ! 3D computer graphics ! Physical input devices ! Intuitive interaction
Sutherland, I. E. (1968, December). A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I (pp. 757-764). ACM.
Richard Bolt ! Founder MIT Media Lab ! The Human Interface: Where People
and Computers Meet ! Van Nostrand Reinhold, 1984.
Interacting with machines via speech, gesture, and gaze will not only be possible, but in fact will become the way most people--most of the time, and for most purposes--will deal with them.
Richard Bolt
“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”
Mark Weiser July 23, 1952 - April 27, 1999
Tangible User Interfaces (Ishii 97) ! Create digital shadows
for physical objects ! Foreground
! graspable UI
! Background ! ambient interfaces
Augmented Reality ! Defining Characteristics
! Combines Real and Virtual Images - Both can be seen at the same time
! Interactive in real-time - The virtual content can be interacted with
! Registered in 3D - Virtual objects appear fixed in space
Microsoft HoloLens
! Stereo see-through display ! Self contained ! Room scale tracking ! Gesture/speech interaction
Vision
“Using technology to create shared emotional experiences between users
and so create a deeper sense of empathy and understanding”
Empathic Computing 1. Computing systems that can understand your feelings and emotions 2. Computing systems that help you better understand the feelings of others
Empathy Computing Requirements
! Basic Requirements ! Making the technology transparent
! Empathy Definition ! Seeing with the eyes of another ! Hearing with the ears of another ! Feeling with the heart of another
Using AR for Empathy
! Augmented Reality can: ! Remove technology barriers ! Enhance communication ! Change perspective ! Share experiences ! Enhance interaction in real world
Wearable Computer
▪ Wearable computer – Ego-Vision view ▪ camera + processing + display + connectivity
Current Collaboration on Wearables
! First person remote conferencing/hangouts ! Limitations
! Single POV, no spatial cues, no annotations, etc
Sharing Space: Social Panoramas
! Capture and share social spaces in real time ! Enable remote people to feel like they’re with you
Technology
! Google Glass ! Image capture, viewpoint sharing
! Remote device (desktop, tablet) ! Immersive viewing, live annotation
Capture Emotions Data Capture
Feature Detection
Emotion Recognition
Emotion Representation
Empathic User Interface
Hardware
User Interface
Sensor Systems/Software
! Different systems developed/evaluated ! SSI framework (emoVoice, etc) ! e-Health, bitalino, arduino bio-sensors ! Wearscript wearable eye-tracker
!
!
Wearable Interface
! Google Glass + e-Health + Spydroid + SSI ! Measure GSR, pulse oxygen, ECG, pitch ! Share video and audio remotely ! Represent emotions back to Glass user (4 states)
! !
Desktop Interface
! Live video, real time emotion data ! See what sender sees, emotion representation
!
Emotion Representation
! How can we show what you’re feeling? ! Tested - Raw data, Visual tinting, Emotion labeling
!
! !Excited Happy
Early Results ! Colour overlay + video stream preferred
! Easier to understand
! Disagreement over best colour coding ! Different emotional response with diff. stimuli
! Scary movie best
! Remote users felt connected to local user ! Understanding of emotions
Analysis
Table 2. Correlation between different streaming scenarios obtained from the feedback of the local and remote users
Experiment 3 (Creating a shared experience while watching an audio Visual Watching an audio visual)
Happy Sad Neutral Excited Case1
(Only video stream) 0.77 0.84 0.85 0.85 Case 2 (Case 1 + Color Overlay) 0.92 0.91 0.85 0.98
Case 3 (Case 2 + Graphical Analysis) 0.86 0.75 0.85 0.94
Capturing Space: Real World Capture
! Hands free AR ! Portable scene capture (color + depth)
! Projector/Kinect combo, Remote controlled pan/tilt
! Remote expert annotation interface
AR + Smart Sensors + Social Networks
! Track population at city scale (mobile networks) ! Match population data to external sensor data
! medical, environmental, etc
! Mine data to improve social services
Research Challenges ! How to convey emotion? ! How to measure empathy? ! Interface/interaction models? ! How to communicate emotion? ! Scaling up to city/country scale?
Take Home Messages ! Find people to travel with ! Equip the vehicle ! Enjoy the journey ! Don’t be disappointed in the outcome Most Important: Find your vision