View
1.207
Download
2
Category
Preview:
DESCRIPTION
Talk on Designing Augmented Reality Experiences given by Mark Billinghurst at the AWE 2013 conference on June 5th 2013
Citation preview
Designing Augmented Reality Experiences
Mark Billinghurst
mark.billinghurst@hitlabnz.org
The HIT Lab NZ, University of Canterbury
June 5th 2013
How Would You Design This?
Put nice AR Picture here – and video
Or This?
DARE 101 1. Know the Technology 2. Design for User Experience
All aspects of user experience
3. Follow good Interaction Design principles Discover, Design, Evaluate
4. Consider all the Design Elements Physical, Virtual and Metaphorical
5. Know Future Research Directions
Know the Technology
What is Augmented Reality? Defining Characteristics (Azuma 97)
• Combines Real and Virtual Images – Both can be seen at the same time
• Interactive in real-time – The virtual content can be interacted with
• Registered in 3D – Virtual objects appear fixed in space
Azuma, R., A Survey of Augmented Reality, Presence, Vol. 6, No. 4, August 1997, pp. 355-385.
AR From Science Fiction to Fact
1977 – Star Wars
2008 – CNN
AR Part of MR Continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real Environment
Augmented Reality (AR)
Augmented Virtuality (AV)
Virtual Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
Core Technologies Combining Real and Virtual Images
• Display technologies Interactive in Real-Time
• Input and interactive technologies Registered in 3D
• Viewpoint tracking technologies Display
Processing
Input Tracking
Display Technologies Types (Bimber/Raskar 2003)
Head attached • Head mounted display/projector
Body attached • Handheld display/projector
Spatial • Spatially aligned projector/monitor
HMD Optical vs. Video see-through Optical: Direct view of real world -> safer, simpler Video: Video overlay -> more image registration options
Display Taxonomy
AR Input Technologies Tangible objects
• Tracked items Touch (HHD)
• Glove, touch Gesture
• Glove, free-hand Speech/Multimodal Device motion
• HHD + sensors
Tracking Technologies Active
• Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location
Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision
• Marker based, Natural feature tracking, model based
Hybrid Tracking • Combined sensors (eg Vision + Inertial)
Design for User Experience
“The product is no longer the basis of value. The
experience is.”
Venkat Ramaswamy The Future of Competition.
Interaction Design
experiences
services
products
components
Valu
e
Gilmore + Pine: Experience Economy
Function
Emotion
experiences
applications
tools
components
Designing AR Experiences
Tracking, Display, Input
Authoring
Interaction
Usability
The Value of Good User Experience
Kenya: 20c
My house: 50c
Starbucks: $3.50
Good Experience Design Reactrix
Top down projection Camera based input Reactive Graphics No instructions No training
Would You Wear This?
User Experience is All About You Designing good user
experience involves many aspects
Consider all the needs of the user Especially context of
use
Web Based AR Flash, HTML 5 based AR Marketing, education
Outdoor Mobile AR GPS, compass tracking Viewing Points of Interest in real world
Handheld AR Vision based tracking Marketing, gaming
Location Based Experiences HMD, fixed screens Museums, point of sale, advertising
Typical AR Experiences
What Makes a Good AR Experience? Compelling
Engaging, ‘Magic’ moment
Intuitive, ease of use Uses existing skills
Anchored in physical world Seamless combination of real and digital
Demo: colAR
Turn colouring books pages into AR scenes Markerless tracking, use your own colours..
Try it yourself: http://www.colARapp.com/
Follow Good Interaction Design Principles
Interaction Design
Answering three questions: What do you do? - How do you affect the world? What do you feel? – What do you sense of the world? What do you know? – What do you learn?
The Design of User Experience with Technology
“Designing interactive products to support people in their everyday and working lives”
Preece, J., (2002). Interaction Design
Interaction Design Process
Interaction Design
AR UI Design Consider your user Follow good HCI principles Adapt HCI guidelines for AR Design to device constraints Using Design Patterns to Inform Design Design for you interface metaphor Design for evaluation
Consider Your User Consider context of user
Physical, social, emotional, cognitive, etc
Mobile Phone AR User Probably Mobile One hand interaction Short application use Need to be able to multitask Use in outdoor or indoor environment Want to enhance interaction with real world
AR vs. Non AR Design
Design Guidelines Design for 3D graphics + Interaction Consider elements of physical world Support implicit interaction
Characteristics Non-AR Interfaces AR Interfaces
Object Graphics Mainly 2D Mainly 3D
Object Types Mainly virtual objects Both virtual and physical objects
Object behaviors Mainly passive objects Both passive and active objects
Communication Mainly simple Mainly complex
HCI methods Mainly explicit Both explicit and implicit
Maps vs. Junaio
Google Maps 2D, mouse driven, text/image heavy, exocentric
Junaio 3D, location driven, simple graphics, egocentric
Design to Device Constraints Understand the platform and design for limitations
Hardware, software platforms
Eg Handheld AR game with visual tracking Use large screen icons Consider screen reflectivity Support one-hand interaction Consider the natural viewing angle Do not tire users out physically Do not encourage fast actions Keep at least one tracking surface in view 32
Art of Defense Game
Design Patterns “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice.”
– Christopher Alexander et al.
Use Design Patterns to Address Reoccurring Problems
C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.
Handheld AR Patterns Title Meaning Embodied Skills Device Metaphors Using metaphor to suggest available player
actions Body A&S Naïve physics
Control Mapping Intuitive mapping between physical and digital objects
Body A&S Naïve physics
Seamful Design Making sense of and integrating the technological seams through game design
Body A&S
World Consistency Whether the laws and rules in physical world hold in digital world
Naïve physics Environmental A&S
Landmarks Reinforcing the connection between digital-physical space through landmarks
Environmental A&S
Personal Presence The way that a player is represented in the game decides how much they feel like living in the digital game world
Environmental A&S Naïve physics
Living Creatures Game characters that are responsive to physical, social events that mimic behaviours of living beings
Social A&S Body A&S
Body constraints Movement of one’s body position constrains another player’s action
Body A&S Social A&S
Hidden information The information that can be hidden and revealed can foster emergent social play
Social A&S Body A&S
Example: Seamless Design
Design to reduce seams in the user experience Eg: AR tracking failure, change in interaction mode
Paparazzi Game Change between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media, and Humanities, p.19-28, October 26-29, 2011
Example: Living Creatures
Virtual creatures respond to real world events eg. Player motion, wind, light, etc Creates illusion creatures are alive in the real world
Sony EyePet Responds to player blowing on creature
36
Rapid Hardware Prototyping
Speed development time by using quick hardware mockups Handheld connected to PC, LCD screen, USB phone keypad,
Camera
Can use PC tools for rapid application development Flash, Visual Basic, etc
Build Your Own Google Glass
Rapid Prototype Glass-Like HMD Myvu HMD + headphone + iOS Device + basic glue skills
$300 + less than 3 hours construction http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/
Why Evaluate AR Applications? To test and compare interfaces, new technologies,
interaction techniques To validate the efficiency and efficient the AR
interface and system Test Usability (learnability, efficiency, satisfaction,...) Get user feedback, Better understand your users Refine interface design Better understand your end users ...
HIT Lab NZ Usability Survey A Survey of Evaluation Techniques Used in
Augmented Reality Studies Andreas Dünser, Raphaël Grasset, Mark Billinghurst
reviewed publications from 1993 to 2007 Extracted 6071 papers which mentioned “Augmented
Reality” Searched to find 165 AR papers with User Studies
Types of Experiments and topics Sensation, Perception & Cognition
How is virtual content perceived ? What perceptual cues are most important ? How to visualize augmented/overlay information on real environment? Visual search/attention/salience issues of human performance
Interaction How can users interact with virtual content ? Which interaction techniques are most efficient in certain context ?
Collaboration & Social issues How is collaboration in AR interface different ? Which collaborative cues can be conveyed best ? Privacy and security issues of AR interface
Gabbard Model for AR Design
1. user task analysis 2. expert guidelines-based evaluation 3. formative user-centered evaluation 4. summative comparative evaluations
Gabbard, J.L.; Swan, J.E.; , "Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design,” Visualization and Computer Graphics, IEEE Transactions on, vol.14, no.3, pp.513-525, May-June 2008
Gabbard Model in Context
Consider All Design Elements
Interface Components Physical components Display elements
- Visual/audio Interaction metaphors
Physical Elements
Virtual Elements Interaction
Metaphor Input Output
AR Design Elements
AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design
Design of Objects Objects
Purposely built – affordances “Found” – repurposed Existing – already at use in marketplace
Affordance The quality of an object allowing an action-
relationship with an actor An attribute of an object that allows people to
know how to use it - e.g. a door handle affords pulling
Affordance Led Design Make affordances perceivable
Provide visual, haptic, tactile, auditory cues
Affordance Led Usability Give feedback Provide constraints Use natural mapping Use good cognitive model
Example: AR Chemistry
Tangible AR chemistry education (Fjeld) Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction
approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich, Switzerland.
Input Devices
Form informs function and use
AR Interaction Metaphors Information Browsing
View AR content
3D AR Interfaces 3D UI interaction techniques
Augmented Surfaces Tangible UI techniques
Tangible AR Tangible UI input + AR output
1. Information Browsing Information is registered to
real-world context Hand held AR displays
Interaction Manipulation of a window
into information space Applications
Context-aware information displays
Rekimoto, et al. 1997
2. 3D AR Interfaces Virtual objects displayed in 3D
physical space and manipulated HMDs and 6DOF head-tracking 6DOF hand trackers for input
Interaction Viewpoint control Traditional 3D user interface
interaction: manipulation, selection, etc.
Kiyokawa, et al. 2000
3. Augmented Surfaces Basic principles
Virtual objects are projected on a surface Physical objects are used as controls for
virtual objects Support for collaboration
Rekimoto, et al. 1998 Front projection Marker-based tracking Multiple projection surfaces
Lessons from Tangible Interfaces
Physical objects make us smart Norman’s “Things that Make Us Smart” encode affordances, constraints
Objects aid collaboration establish shared meaning
Objects increase understanding serve as cognitive artifacts
TUI Limitations Difficult to change object properties
Can’t tell state of digital data Limited display capabilities
projection screen = 2D dependent on physical display surface
Separation between object and display Augmented Surfaces
4. Tangible AR Metaphor AR overcomes limitation of TUIs
enhance display possibilities merge task/display space provide public and private views
TUI + AR = Tangible AR Apply TUI methods to AR interface design
Tangible AR Demo
Use of natural physical object manipulations to control virtual objects
VOMAR Demo Catalog book:
- Turn over the page Paddle operation:
- Push, shake, incline, hit, scoop
Object Based Interaction: MagicCup Intuitive Virtual Object Manipulation
on a Table-Top Workspace
Time multiplexed Multiple Markers
- Robust Tracking Tangible User Interface
- Intuitive Manipulation Stereo Display
- Good Presence
Tangible AR Design Principles Tangible AR Interfaces use TUI principles
Physical controllers for moving virtual content Support for spatial 3D interaction techniques Time and space multiplexed interaction Support for multi-handed interaction Match object affordances to task requirements Support parallel activity with multiple objects Allow collaboration between multiple users
Example 1: AR Lens Physical Components
Lens handle - Virtual lens attached to real object
Display Elements Lens view
- Reveal layers in dataset
Interaction Metaphor Physically holding lens
Example 2: LevelHead
Physical Components Real blocks
Display Elements Virtual person and rooms
Interaction Metaphor Blocks are rooms
Know Future Research Directions
The Vision of AR
To Make the Vision Real.. Hardware/software requirements
Contact lens displays Free space hand/body tracking Speech/gesture recognition Etc..
Most importantly Usability/User Experience
Natural Interaction Automatically detecting real environment
Environmental awareness Physically based interaction
Gesture Input Free-hand interaction
Multimodal Input Speech and gesture interaction Implicit rather than Explicit interaction
AR MicroMachines AR experience with environment awareness
and physically-based interaction Based on MS Kinect RGB-D sensor
Augmented environment supports occlusion, shadows physically-based interaction between real and
virtual objects
Physics Simulation
Create virtual mesh over real world
Update at 10 fps – can move real objects
Use by physics engine for collision detection (virtual/real)
Use by OpenScenegraph for occlusion and shadows
Rendering
Occlusion Shadows
Gesture Input Architecture 5. Gesture
• Static Gestures • Dynamic Gestures • Context based Gestures
4. Modeling
• Hand recognition/modeling • Rigid-body modeling
3. Classification/Tracking
2. Segmentation
1. Hardware Interface
Results
Free Hand Multimodal Input Use free hand to interact with AR content Recognize simple gestures No marker tracking
Point Move Pick/Drop
Multimodal Architecture
Multimodal Fusion
Hand Occlusion
Conclusion
Conclusion
There is need for better designed AR experiences Through
use of Interaction Design principles understanding of the technology use of rapid prototyping tools rigorous user evaluation
There a number of important areas for future research Natural interaction, Multimodal interfaces, Intelligent agents, …
More Information • Mark Billinghurst
– mark.billinghurst@hitlabnz.org
• Websites – www.hitlabnz.org
80
Resources
Recommended