COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

July 19th 2013

Lecture 2: AR Technology

Key Points from Lecture 1

Augmented Reality Definition   Defining Characteristics [Azuma 97]

 Combines Real and Virtual Images -  Both can be seen at the same time

  Interactive in real-time -  Virtual content can be interacted with

  Registered in 3D -  Virtual objects appear fixed in space

What is not Augmented Reality?

  Location-based services   Barcode detection (QR-codes)   Augmenting still images   Special effects in movies   …   … but they can be combined with AR!

Milgram’s Reality-Virtuality Continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

Metaverse

AR History Summary   1960’s – 80’s: Early Experimentation   1980’s – 90’s: Basic Research

  Tracking, displays

  1995 – 2005: Tools/Applications   Interaction, usability, theory

  2005 - : Commercial Applications  Games, Medical, Industry

Applications

  Medicine   Manufacturing   Information overlay   Architecture   Museum   Marketing   Gaming

AR Experience Design

“The product is no longer the basis of value. The

experience is.”

Venkat Ramaswamy The Future of Competition.

experiences

services

products

components

Valu

e

Sony CSL © 2004

Gilmore + Pine: Experience Economy

Function

Emotion

The Value of Good User Experience

20c

50c

$3.50

Good Experience Design   Reactrix

  Top down projection   Camera based input   Reactive Graphics   No instructions   No training

Apple: The Value of Good Design

  Good Experience Design Dominates Markets

iPod Sales 2002-2007

Using the N-gage

SideTalking   http://www.sidetalkin.com

Interaction Design

“Designing interactive products to support people in their everyday and working lives” Preece, J., (2002). Interaction Design

  Design of User Experience with Technology   Higher in the value chain than product design

  Interaction Design involves answering three questions:   What do you do? - How do you affect the world?   What do you feel? – What do you sense of the world?   What do you know? – What do you learn?

Interaction Design is All About You

  Users should be involved throughout the Design Process

  Consider all the needs of the user

Interaction Design Process

Gabbard Model for AR Design

1. user task analysis 2. expert guidelines-based evaluation 3. formative user-centered evaluation 4. summative comparative evaluations

Gabbard, J.L.; Swan, J.E.; , "Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design,” Visualization and Computer Graphics, IEEE Transactions on, vol.14, no.3, pp.513-525, May-June 2008

Gabbard Model in Context

experiences

applications

tools

components

Building Compelling AR Experiences

Tracking, Display

Authoring

Interaction

Usability

Summary   In order to build AR applications you need to

focus on the user experience   Great user experience is based on

  Low level AR component technology   Authoring tools   Application/Interaction design  User experience texting

AR Technology

experiences

applications

tools

components

Sony CSL © 2004

Building Compelling AR Experiences

Display, Tracking

Core Technologies  Combining Real and Virtual Images

•  Display technologies  Interactive in Real-Time

•  Input and interactive technologies  Registered in 3D

•  Viewpoint tracking technologies Display

Processing

Input Tracking

AR Displays

AR Displays

e.g. window reflections

Virtual Images seen off windows

e.g. Reach-In

Projection CRT Display using beamsplitter

Not Head-Mounted

e.g. Shared Space Magic Book

Liquid Crystal Displays LCDs

Head-Mounted Display (HMD)

Primarily Indoor Environments

e.g. WLVA and IVRD

Cathode Ray Tube (CRT) or Virtual Retinal Display (VRD)

Many Military Applications & Assistive Technologies

Head-Mounted Display (HMD)

e.g. Head-Up Display (HUD)

Projection Display Navigational Aids in Cars

Military Airborne Applications

Not Head Mounted (e.g. vehicle mounted)

Primarily Outdoor (Daylight) Environments

AR Visual Displays

Display Technologies

 Types (Bimber/Raskar 2003)  Head attached

•  Head mounted display/projector  Body attached

•  Handheld display/projector  Spatial

•  Spatially aligned projector/monitor

Display Taxonomy

Head Mounted Displays

Head Mounted Displays (HMD) -  Display and Optics mounted on Head -  May or may not fully occlude real world -  Provide full-color images -  Considerations

•  Cumbersome to wear •  Brightness •  Low power consumption •  Resolution limited •  Cost is high?

Key Properties of HMD   Field of View

  Human eye 95 degrees horizontal, 60/70 degrees vertical

  Resolution   > 320x240 pixel

  Refresh Rate   Focus

  Fixed/manual

  Power   Size

Types of Head Mounted Displays

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

The image cannot be displayed. Your computer

Occluded See-thru

Multiplexed

Immersive VR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

Non see-thru!Image source

& optics

virtual object

Virtual World

See-thru AR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

see-thru!combiner

Virtual Image superimposed!over real world object

real world

Image source

Optical see-through head-mounted display

Virtual images from monitors

Real World

Optical Combiners

Optical See-Through HMD

Epson Moverio Display

  Stereo see-through display   960 x 540 pixels, 23 degree Field of View, 60 Hz   $700 USD   Android Powered

View Through Optical See-Through HMD

DigiLens

  www.digilens.com

  Compact HOE   Solid state optics   Switchable Bragg Grating   Stacked SBG   Fast switching   Ultra compact

The Virtual Retinal Display

  Image scanned onto retina   Commercialized through Microvision

  Nomad System - www.mvis.com

Strengths of optical see-through AR   Simpler (cheaper)   Direct view of real world

  Full resolution, no time delay (for real world)   Safety   Lower distortion

  No eye displacement   but COASTAR video see-through avoids this

Video AR Architecture

Head!Tracker

Host !Processor

Graphics!renderer

Digital!Mixer Frame!

Buffer

head position/orientation

to network Display!Driver

Non see-thru!Image source

& optics

Head-mounted camera aligned to

display optics

Video!Processor

Video image of real world

Virtual image inset into video of real world

Video see-through HMD Video cameras

Monitors

Graphics

Combiner

Video

Video See-Through HMD

Video see-through HMD

MR Laboratory’s COASTAR HMD (Co-Optical Axis See-Through Augmented Reality) Parallax-free video see-through HMD

TriVisio   www.trivisio.com   Stereo video input

  PAL resolution cameras

  2 x SVGA displays   30 degree FOV   User adjustable convergence

  $6,000 USD

View Through a Video See-Through HMD

Vuzix Display

  www.vuzix.com   Wrap 920   $350 USD   Twin 640 x 480 LCD displays   31 degree diagonal field of view   Weighs less than three ounces

Strengths of Video See-Through AR   True occlusion

  Kiyokawa optical display that supports occlusion

  Digitized image of real world   Flexibility in composition  Matchable time delays  More registration, calibration strategies

  Wide FOV is easier to support

Optical vs. Video AR Summary   Both have proponents   Video is more popular today?

  Likely because lack of available optical products

  Depends on application?  Manufacturing: optical is cheaper  Medical: video for calibration strategies

Eye multiplexed AR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

Virtual Image inset into!real world scene

real world

Opaque!Image source

Virtual Image ‘inset’ into real

Virtual Vision Personal Eyewear

Virtual image inset into real world

Google Glass

View Through Google Glass

What's Inside Google Glass?

Spatial/Projected AR

Spatial Augmented Reality

  Project onto irregular surfaces   Geometric Registration   Projector blending, High dynamic range

  Book: Bimber, Rasker “Spatial Augmented Reality”

Projector-based AR

Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo

Projector

Real objects with retroreflective covering

User (possibly head-tracked)

Example of projector-based AR

Ramesh Raskar, UNC, MERL

Example of projector-based AR

Ramesh Raskar, UNC Chapel Hill

The I/O Bulb

  Projector + Camera   John Underkoffler, Hiroshi Ishii  MIT Media Lab

Head Mounted Projector

  Head Mounted Projector   Jannick Rolland (UCF)

  Retro-reflective Material   Potentially portable

Head Mounted Projector

  NVIS P-50 HMPD   1280x1024/eye   Stereoscopic   50 degree FOV   www.nvis.com

HMD vs. HMPD

Head Mounted Display Head Mounted Projected Display

Pico Projectors

  Microvision - www.mvis.com   3M, Samsung, Philips, etc

MIT Sixth Sense

  Body worn camera and projector   http://www.pranavmistry.com/projects/sixthsense/

Other AR Displays

Video Monitor AR

Video cameras Monitor

Graphics Combiner

Video

Stereo glasses

Examples

Virtual Showcase

  Mirrors on a projection table   Head tracked stereo   Up to 4 users   Merges graphic and real objects   Exhibit/museum applications

  Fraunhofer Institute (2001)   Bimber, Frohlich

Augmented Paleontology

Bimber et. al. IEEE Computer Sept. 2002

Alternate Displays

LCD Panel Laptop PDA

Handheld Displays   Mobile Phones

 Camera  Display   Input

Other Types of AR Display   Audio

  spatial sound   ambient audio

  Tactile   physical sensation

  Haptic   virtual touch

Haptic Input

  AR Haptic Workbench  CSIRO 2003 – Adcock et. al.

Phantom

  Sensable Technologies (www.sensable.com)   6 DOF Force Feedback Device

AR Haptic Interface

  Phantom, ARToolKit, Magellan