Embed Size (px)
Citation preview
DESIGNING OUTSTANDING AUGMENTED REALITY
EXPERIENCES
Mark Billinghurst [email protected]
May 17th 2016
Shenzen, China
• adsf
• jg
The Digital Divide
• Screens are windows into digital spaces • Separation between digital and physical world
Augmented Reality
Vuforia Smart Terrain
Augmented Reality Technology that: 1. Combines Real and Virtual Images
• Both can be seen at the same time
2. Interactive in real-time • The virtual content can be interacted with
3. Registered in 3D • Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
• Web based AR • Flash, HTML 5 • Marketing, education
• Outdoor Mobile AR • GPS, compass tracking • Viewing Points of Interest in real world
• Handheld AR • Vision based tracking • Marketing, gaming
• Location Based Experiences • HMD, fixed screens • Museums, point of sale, advertising
Typical AR Experiences
AR Business Today
• Around $600 Million USD in 2014 (>$2B 2016) • 70-80+% Games and Marketing
Market Projections
cf. 2014 computer game market = $84 Billion USD
DESIGNING AR EXPERIENCES
What Makes a Good AR Experience? • Compelling
• Engaging, ‘Magic’ moment
• Intuitive, ease of use • Uses existing skills
• Anchored in physical world • Seamless combination of real and digital
Example: colAR (Quiver)
• Turn colouring book pages into AR scenes • Markerless tracking, use your own colours..
• Try it yourself: http://www.colARapp.com/
• Interface Components • Physical components • Display elements
• Visual/audio • Interaction metaphors
Physical Elements
Display Elements Interaction
Metaphor Input Output
AR Interface Elements
AR Interface Design • Combines physical + virtual object design • Physical
• Tangible controllers and objects
• Virtual • Virtual graphics and audio
Physical Design • Industrial Design
• Type of Objects • Purposely built – affordances
• “Found” – repurposed
• Existing – already at use in marketplace
How are these used?
”… the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...]
Affordances provide strong clues to the operations of things.”
(Norman, The Psychology of Everyday Things 1988, p.9)
Affordance
Physical vs. Virtual Affordances • Physical Affordances
• Look and feel of real objects • Shape, texture, colour, weight, etc • Industrial Design
• Virtual Affordance • Look of virtual objects • Copy real objects • Interface Design
Design of AR Enhanced Objects
• Make affordances obvious
• Object affordances visible
• Give feedback
• Provide constraints
• Use natural mapping
• Use good cognitive model
Example: Haunted Book/AR Book
• Camera hidden in lamp object • AR content seamlessly integrated into real book • Natural page turning/manipulation interaction
Interface Design Path 1/ Demonstration: Working Prototype
2/ Copying: Adoption of Interaction Techniques from other interface metaphors
3/ Creation: Development of new interface metaphors appropriate to the medium
4/ Theory: Development of formal theoretical models for predicting and modeling user actions
Desktop WIMP
Virtual Reality
Augmented Reality
Example: VR Interfaces
• Copying: Virtual Windows/keyboards • Creation: World in Miniature
AR Interaction Metaphors
• AR Lens/Window • simple (conceptually!), unobtrusive
• 3D User Interfaces (VR) • expressive, creative, require attention
• Tangible Interfaces (TUI) • Embedded into conventional environments
• Tangible AR • Combines TUI input + AR display
AR Lens
• Information is registered to real-world context • Hand held AR displays
• Interaction • 2D/3D virtual viewpoint control • Limited input/interactivity
• Applications • Context-aware information
Tourism, gaming
3D AR Interfaces • Virtual objects displayed in 3D physical space and manipulated • HMDs and 6DOF head-tracking
• 6DOF hand trackers for input
• Interaction • Viewpoint control
• Traditional 3D interaction: • manipulation, selection, etc.
• VR techniques
Tangible User Interfaces (Ishii 97)
• Create digital shadows for physical objects
• Foreground • graspable UI
• Background • ambient interfaces
Tangible Interface: Augmented Groove
• Collaborative Instrument • Physically Based Interaction • Map actions to Midi output
• Translation, rotation • Tilt, shake
Lessons from Tangible Interfaces • Advantages
• Physical objects make us smart • Objects aid collaboration • Objects increase understanding
• Disadvantages • Difficult to change object properties • Limited display capabilities • Separation between object and display
Back to the Real World • AR overcomes limitation of TUIs
• enhance display possibilities • merge task/display space • provide public and private views
• TUI + AR = Tangible AR • Apply TUI methods to AR interface design • TUI for input, AR for output
Tangible AR Design Principles • Tangible AR Interfaces use TUI principles
• Physical controllers for moving virtual content • Support for spatial 3D interaction techniques • Time and space multiplexed interaction • Support for multi-handed interaction • Match object affordances to task requirements • Support parallel activity with multiple objects • Allow collaboration between multiple users
Tangible AR: Tiles • Tiles semantics
• data tiles • operation tiles
• Operation on tiles • proximity • spatial arrangements • space-multiplexed
Poupyrev, I., Tan, D., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001). Tiles: A mixed reality authoring interface. In INTERACT 2001 Conference on Human Computer Interaction (pp. 334-341).
Example: LevelHead
Case Study: LevelHead
• Physical Components • Real blocks
• Display Elements • Virtual person and rooms
• Interaction Metaphor • Blocks are rooms
Design for Technology Limitations • Understand the platforms used and design for limitations
• Hardware, software platforms
• Eg Handheld AR game with visual tracking • Use large screen icons • Consider screen reflectivity • Support one-hand interaction • Consider the natural viewing angle • Do not tire users out physically • Do not encourage fast actions • Keep at least one tracking surface in view
Art of Defense Game
Seamless Design
• Design to reduce seams in the user experience • Eg: AR tracking failure, change in interaction mode
• Paparazzi Game • Change between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media, and Humanities, p.19-28, October 26-29, 2011
Consider the Whole User
Consider Your User
• Mobile Phone AR User • Probably Mobile • One hand interaction • Short application use • Need to be able to multitask • Use in outdoor or indoor environment • Want to enhance interaction with real world
Social Acceptance
• People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses • 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)
TAT Augmented ID
Google Glass Anyone?
Designing AR Experiences • Create a compelling experience • Intuitive and ease of use • Anchor in the real world • Design affordances for physical + virtual elements • Create the appropriate Interaction Metaphor • Design for technology limitations • Consider the whole user (Social, cultural, ..)
RESEARCH DIRECTIONS
Looking to the Future
What’s Next?
Research Enables New Experiences • Gesture/multimodal Input
• Natural user interaction
• Collaborative experiences • Capturing rich communication cues
• Experience capture • Sharing surroundings
• Empathic Computing • Creating understanding
• Etc..
Gesture Based Interaction
• Use free hand gestures to interact • Depth camera, scene capture
• Multimodal input • Combining speech and gesture
HIT Lab NZ Microsoft Hololens
Meta SpaceGlasses
Social Panoramas
• Google Glass • Capture live image panorama (compass + camera)
• Remote device (tablet) • Immersive viewing, live annotation
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 303-304). IEEE.
Empathy Glasses
• Combine together eye-tracking, display, face expression • Implicit cues – eye gaze, face expression
+ +
Pupil Labs Epson BT-200 AffectiveWear
Remote Collboration
• Eye gaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
Holoportation
• Augmented Reality + 3D capture + high bandwidth • http://research.microsoft.com/en-us/projects/holoportation/
CONCLUSION
Conclusions • AR enables connection back to real world
• Great AR experiences need good design • Physical + virtual components, interaction metaphor • Design for technology limitations • Need to consider whole user
• Many directions for future research • Natural interaction, collaboration, experience capture
