SVR2011 Keynote

Research Directions in Augmented Reality

Mark Billinghurst

The HIT Lab NZ

University of Canterbury

Augmented Reality Definition

� Defining Characteristics [Azuma 97]� Combines Real and Virtual Images

- Both can be seen at the same time

� Interactive in real-timeInteractive in real-time- The virtual content can be interacted with

� Registered in 3D- Virtual objects appear fixed in space

� Put AR pictures here

Augmented Reality Examples

Virtual Reality

� 1989…

Virtual Reality

� Immersive VR

� Head mounted display, gloves

� Separation from the real world

AR vs VR

� Virtual Reality: Replaces Reality

� Scene Generation: requires realistic images

� Display Device: fully immersive, wide FOV

� Tracking and Sensing: low accuracy is okay� Tracking and Sensing: low accuracy is okay

� Augmented Reality: Enhances Reality

� Scene Generation: minimal rendering okay

� Display Device: non-immersive, small FOV

� Tracking and Sensing: high accuracy needed

Milgram’s Reality-Virtuality continuum

Mixed Reality

Real Augmented Augmented Virtual

Reality - Virtuality (RV) Continuum

Environment Reality (AR) Virtuality (AV) Environment

AR History

AR Beginnings� 1960’s: Sutherland / Sproull’s

first HMD system was see-through

1960 - 80’s: US Air Force SuperCockpit (T. Furness)

� Early 1990’s: Boeing coined the term “AR.” Wire harness assembly application begun (T. Caudell, D. Mizell).

� Early to mid 1990’s: UNC ultrasound visualization projectultrasound visualization project

� Early 1990’s: Boeing coined the term “AR.” Wire harness assembly application begun (T. Caudell, D. Mizell).

� 1994 - : UNC Research

� Motion stabilized display, Hybrid tracking, Ultrasound visualization

A Brief History of AR

� 1996: MIT Wearable Computing efforts� 1998: Dedicated conferences begin� Late 90’s: Collaboration, outdoor, interaction� Late 90’s: Augmented sports broadcasts� 1998 - 2001: Mixed Reality Systems Lab

History Summary

� 1960’s – 80’s: Early Experimentation

� 1980’s – 90’s: Basic Research

� Tracking, displays

� 1995 – 2005: Tools/Applications� 1995 – 2005: Tools/Applications

� Interaction, usability, theory

� 2005 - : Commercial Applications

� Games, Medical, Industry

Medical AR Trials� Sauer et al. 2000 at Siemens

Corporate Research, NJ

� Stereo video see through

F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,MICCAI 2002

AR Reaches Mainstream

� MIT Technology Review � March 2007� list of the 10 most exciting

technologies� Economist� Economist

� Dec 6th 2007� Reality, only better

Virtual Reality, Augmented Reality

� Esquire Magazine

� Dec 2009 issue

� 12 pages AR content

Trend One: Browser Based AR

� Adobe Flash + camera + 3D graphics

� High impact

� High marketing value

� Large potential install base� Large potential install base

� 1.6 Billion web users

� Ease of development

� Lots of developers, mature tools

� Low cost of entry

� Browser, web camera

1983 – Star Wars

1999: AR Face to Face Collaboration

1998: SGI O2 2008: Nokia N95

CPU: 300 MhzHDD; 9GBRAM: 512 mbCamera: VGA 30fpsGraphics: 500K poly/sec

CPU: 332 MhzHDD; 8GBRAM: 128 mbCamera: VGA 30 fpsGraphics: 2m poly/sec

Trend Two: Mobile Phone AR

� Mobile Phones

� camera, sensors

� processor

� display� display

� AR on Mobile Phones

� Simple graphics

� Optimized computer vision

� Collaborative Interaction

Collaborative AR

� AR Tennis� Shared AR content

� Two user game

� Audio + haptic feedback

� Bluetooth networking

Location Aware Phones

Nokia NavigatorMotorola Droid

2009 - Outdoor Information Overlay

� Mobile phone based

� Tag real world locations � GPS + Compass input

� Overlay graphics data on live video� Overlay graphics data on live video

� Applications� Travel guide, Advertising, etc

� Wikitude, Layar, Junaio, etc..� Android based, Public API released

Layar (www.layar.com)

� Location based data

� GPS + compass location

� Map + camera view

� AR Layers on real world� AR Layers on real world

� Customized data

� Audio, 3D, 2D content

� Easy authoring

� Android, iPhone

Android AR Platform

� Architectural Application

� Loads 3D models

� a OBJ/MTL format

� Positions content in space� Positions content in space

� GPS, compass

� Intuitive user interface

� toolkit to modify the model

� Connects to back end model database

Mobile Outdoor AR

Client/Server

Web Interface

Add models

Androidapplication

Web application java and php server

Database serverPostgres

$784 million USD in 2014

Summary

� Augmented Reality has a long history going back to the 1960’s

� Interest in AR has exploded over the last two years and is being commercialized quicklyyears and is being commercialized quickly

� AR is growing in a number of areas

� Mobile AR

� Web based AR

� Advertising experiences

Looking to the Future

What’s Next?

Sony CSL © 2004

“The product is no longer the basis of value. The

experience is.”experience is.”

Venkat Ramaswamy

The Future of Competition.

PS3 - Eye of Judgement

� Computer Vision Tracking

� Card based battle game

� Collaborative AR

� October 24th 2007� October 24th 2007

experiences

applications

Building Compelling AR Experiences

Interaction

Usability

tools

components

Sony CSL © 2004

Tracking, Display

Authoring

AR Components

experiences

applications

Building Compelling AR Experiences

tools

components

Sony CSL © 2004

Tracking, Display

Low Level AR Libraries

� ARToolKit Enhancements

� Occlusion Handling

� SSTT

� Simple Spatial Template Tracking� Simple Spatial Template Tracking

� Opira

� Robust Natural Feature Tracking

Markerless Tracking

AR Tools

experiences

applications

Building Compelling AR Experiences

tools

components

Sony CSL © 2004

Tracking, Display

Authoring

AR Authoring

� Software Libraries

� OSGART, Studierstube, MXRToolKit

� Plugin to existing software

� DART (Macromedia Director)� DART (Macromedia Director)

� Stand Alone

� AMIRE, etc

� Next Generation

� iaTAR (Tangible AR)

mARx Plug-in

� 3D Studio Max Plug-in

� Can model and view AR content at the same time

BuildAR

� http://www.buildar.co.nz/

� Stand alone application

� Visual interface for AR model viewing application

� Enables non-programmers to build AR scenes

AR Applications

experiences

applications

Building Compelling AR Experiences

Interaction

tools

components

Sony CSL © 2004

Tracking, Display

Authoring

� Interface Components

� Physical components

� Display elements

AR Design Principles

- Visual/audio

� Interaction metaphors

Physical Elements

Display ElementsInteraction

MetaphorInput Output

Tangible User Interfaces (Ishii 97)

� Create digital shadows for physical objects

� Foreground� Foreground

� graspable UI

� Background

� ambient interfaces

Tangible AR Metaphor

� AR overcomes limitation of TUIs

� enhance display possibilities

� merge task/display space

provide public and private views � provide public and private views

� TUI + AR = Tangible AR

� Apply TUI methods to AR interface design

Tangible AR Design Principles

� Tangible AR Interfaces use TUI principles

� Physical controllers for moving virtual content

� Support for spatial 3D interaction techniques

� Support for multi-handed interaction� Support for multi-handed interaction

� Match object affordances to task requirements

� Support parallel activity with multiple objects

� Allow collaboration between multiple users

Case Study: 3D AR Lens

Goal: Develop a lens based AR interface

� MagicLenses� Developed at Xerox PARC in 1993

� View a region of the workspace differently to the rest� View a region of the workspace differently to the rest

� Overlap MagicLenses to create composite effects

3D MagicLenses

MagicLenses extended to 3D (Veiga et. al. 96)

� Volumetric and flat lenses

AR Lens Design Principles

� Physical Components� Lens handle

- Virtual lens attached to real object

� Display Elements� Display Elements� Lens view

- Reveal layers in dataset

� Interaction Metaphor� Physically holding lens

3D AR Lenses: Model Viewer

� Displays models made up of multiple parts

� Each part can be shown or hidden through the lens

� Allows the user to peer inside the model

� Maintains focus + context� Maintains focus + context

AR Lens Demo

HandHeld ARWearable AR

Output:Display Input &

Output

HMD vs Handheld AR Interface

Input

Handheld Interface Metaphors

� Tangible AR Lens Viewing

� Look through screen into AR scene

� Interact with screen to interact with AR content

- Eg Invisible TrainEg Invisible Train

� Tangible AR Lens Manipulation

� Select AR object and attach to device

� Use the motion of the device as input

- Eg AR Lego

Next Interaction Techniques

� Natural Gestures

� Depth sensing

� Natural body input

� Multimodal� Multimodal

� Speech + gesture

AR Experiences

experiences

applications

Building Compelling AR Experiences

Interaction

Usability

tools

components

Sony CSL © 2004

Tracking, Display

Authoring

Survey of AR Papers� Edward Swan (2005)

� Surveyed major conference/journals (1992-2004)- Presence, ISMAR, ISWC, IEEE VR

� Summary� 1104 total papers� 1104 total papers

� 266 AR papers

� 38 AR HCI papers (Interaction)

� 21 AR user studies

� Only 21 from 266 AR papers have formal user study (<8% of all AR papers)

Types of Experiments

� Perception� How is virtual content perceived ?

� What perceptual cues are most important ?

� InteractionInteraction� How can users interact with virtual content ?

� Which interaction techniques are most efficient ?

� Collaboration� How is collaboration in AR interface different ?

� Which collaborative cues can be conveyed best ?

AR Browser Interface

� Layar (www.layar.com)

� show POI on real world

� Typical Interface Elements

� Live camera view� Live camera view

� Radar view

� Virtual graphics of POI

� 2D map view

� Information area

Navigation

� How useful is AR view for navigation

� ego- vs. exo-centric

� Experiment� Experiment

� AR only

� Map only

� AR + map

Experiment Design

� Conditions

� AR: Using only an AR view

� 2D-map: Using only a top down 2D map view

� AR+2D-map: Using both an AR and 2D map view� AR+2D-map: Using both an AR and 2D map view

� Measures

� Time to complete, Distance travelled

� User preference, subjective measures

Paths Walked

� three different paths walked around campus

� between buildings/under trees

Performance Measures

Map

AR+Map

Map

AR+Map

� No difference between conditions

Average Time Taken (sec)

0 200 400 600 800 1000 1200 1400

AR

Average Distance Travelled (m)

0 200 400 600 800 1000

AR

Path Trails

User Feedback

� AR + Map easy to identify points of interest

� AR only hard to know where things werewhere things were

� Liked being able to switch between modes

� AR+Map preferred best

Typical User Comments

� “With the AR mode, I didn’t know where any of the buildings were, a couple of times I went round in a circle because I didn’t know where things were.”

� “I found the map interface the best one to use � “I found the map interface the best one to use because you are actually able to see the physical objects around you"

� “I used the map at the beginning to understand where the buildings were and the AR between each point”

Navigation Conclusion

� AR alone provides no improvement

� Lack of depth cues

� Difficult to create spatial awareness

� AR + Map preferred interface� AR + Map preferred interface

� Map for creating mental mode

� AR for near navigation

Conclusions

“We’re living in the experience economy and the customer is the star of the show. If I’m going to spend thousands of dollars on something. I want the whole on something. I want the whole experience to be a fairy-tale”

Milton Pedraza,

The Luxury Institute Illustrative

experiences

applications

Building Compelling AR Experiences

Interaction

Usability

tools

components

Sony CSL © 2004

Tracking, Display

Authoring

Conclusions

� AR is on the verge of commercialization

� There are interesting research opportunities in

� Developing AR Component Technology

� Build Easy to Use Tools� Build Easy to Use Tools

� Identify Application Domains

� Develop Compelling AR Experiences

More Information

• Mark Billinghurst

– mark.billinghurst@hitlabnz.org

• Websites– http://www.hitlabnz.org/ – http://www.hitlabnz.org/

– http://artoolkit.sourceforge.net/

– http://www.osgart.org/

– http://www.hitlabnz.org/wiki/buildAR/