AUGMENTED

REALITY A/Prof Manolya Kavakli &

VISOR team

Majed Alrowaily

Shaghayegh Asgari

Guillaume Lucas

Farzana Alibay

Hessam Jahani

Department of Computing

Macquarie University

Sydney, Australia

Sydney Science Festival 2016 13th August 2016

Associated Research Groups

VISOR (Virtual and Interactive Simulations of Reality)

CEPET (Centre for Elite Performance, Expertise, and Training)

Understanding expertise is important for the design of training programs.

Principles and mechanisms proposed to underlie expertise can be used to evaluate the theories about basic cognitive processes and capacities, and thus explain human performance more generally (Loft et al, 2009).

CAVE @ ENSAM, Chalon Sur Saone, France

Simulation Hub & VR Lab 2015

Simulators

VISOR @ VR Lab

8 Postdocs

12 PhD students graduated

8 current PhD students

8 to start next term/year

7 Honours students

25 MIT students

11 MEng interns

6 Programmers

85 in total

VISOR: Virtual and Interactive Simulations of Reality

HCI

Virtual Reality

Augmented Reality

Graphics Programming

Simulation Design

Motion Tracking

Training & Expertise

Scene Complexity

Simulator Sickness

Novices and Experts

Performance

Cognitive Processing

Cognitive Load

Human Information Processing

Joint performance of Human Computer/ Machine Interfaces

Augmented Reality

A type of virtual reality in which

additional information,

otherwise imperceptible to the human system

is made perceptible and

registered with the display of the physical world.

Reality-Virtuality Continuum

Milgram (Milgram and Kishino 1994; Milgram, Takemura et al. 1994) describes a taxonomy that identifies how augmented reality and virtual reality work are related.

The real world and a totally virtual environment are at the two ends of this continuum with the middle region called Mixed Reality.

Augmented reality lies near the real world end of the line with the predominate perception being the real world augmented by computer generated data.

Augmented virtuality is a term created by Milgram to identify systems which are mostly synthetic with some real world imagery added such as texture mapping video onto virtual objects. This is a distinction that will fade as the technology improves and the virtual elements in the scene become less distinguishable from the real ones.

Augmented Reality

a live, copy, view of

a physical, real-world environment whose

elements are augmented by computer-generated

sensory input such as sound, video, graphics or

GPS data

Augmented Reality

Imagine a VR head mounted display,

but the display doesn't block out the regular view, it's just superimposed on it. Imagine walking around a building and "seeing" inside

the walls to the wiring, plumbing, and structure.

Or, seeing the tumor inside a patient's head as you hack away at it.

Focuses on enriching the natural environment

Extra information is displayed on a HMD Currently mainly used in the military with few civilian

applications

In the Touring Machine scenario,

the MARS unit acts as a campus information system, assisting a user in finding places and allowing her to query information about items of interest, like buildings, statues, etc.

MARS: Mobile Augmented Reality Systems

Mobile market

Rapid changes since 2007 in

Single purpose handheld devices:

Apple iPhone and

Android systems (Samsung’s Galaxy)

Tablet devices

(integrating smartphones and netbooks):

Apple iPad

Android systems (Samsung’s Galaxy)

The Layar Reality Browser An augmented reality app

(mobile phone application) that

shows you what is around you by displaying real time digital information (layers) on top of reality.

This digital information is called a ‘layer’.

Layers can provide services, such as finding ATMs, houses for sale and restaurants including reviews.

Layers can also provide an experience with interactivity, 3D objects and sounds for games and engaging guided tours. Many layers provide both.

AR platform? Essentially you’re using your phone like a camera

so you can see the real world as you pan around,

but the application utilizes your GPS location and G-Sensor

to determine exactly what your phone/camera is pointing towards and

provides you more information about it directly on your phone!

But Layar technology isn’t just an application…

its a platform.

The Layar Reality Browser first launched in June 2009 in The Netherlands,

announced that it’s second generation Reality Browser is now available globally on Android devices.

Many new content layers are available in the Reality Browser which vary from

Wikipedia, Twitter and Brightkite to local services like Yelp, Trulia, store locator’s, nearby bus stops, mobile coupons, Mazda dealers and tourist, nature and cultural guides.

http://www.youtube.com/watch?src_vid=b64_16K2e08&v=HW9gU_4AUCA&feature=iv&annotation_id=annotation_121798

http://www.youtube.com/watch?v=b64_16K2e08

http://layar.pbworks.com/w/page/7783224/Creating%20the%203D%20objects

AR games

In 2010:

To further grow the community of developers, Layar has released an additional 500 new API keys.

The supporting wiki (http://layar.pbworks.com),

with developer instructions and documentation, are all now available.

Augmented Reality Applications

Imagine that:

Signs could broadcast their message in several languages, being automatically picked up and displayed on the users phone in the appropriate language.

A map with GPS and/or building information could be projected on to the phone –> No more getting lost.

Virtual tour guides could be downloaded for any country and location.

Multimodal support for visual-impaired, using brail…

The possibilities are endless…

Using the GPS location, accelerometer and gyroscope of the smart tablet, and Google glasses, we can generate a mobile AR system.

The AR system (I-DeSIGN) will facilitate design communication by

using 3D architectural objects such as walls and windows

to push and pull to shape and create a virtual built environment,

whilst the architect has the benefit of having a superimposed image of the virtual world in physical reality.

iDesign A Multimodal Augmented Reality System for Spatial Design

Building Information Modeling

Simulation Hub

3D visualisation in Augmented Reality

Benefits of Augmented Reality

offers a combination of physical and virtual objects.

aids learning due to the combination of its relevance to the learner’s physical world and

the customizability of the simulation to offer a virtual world.

Therefore, the learning experience is superior to other media such as computer games and virtual reality because

does not separate the users from the reality but, instead, realistically transforms their reality.

This transformation is represented by a temporary scaffold between

what is real and virtual, in other words, known and unknown.

controls the accessibility, transparency and visibility of this scaffold as learning progresses.

AR frameworks & Scaffolding discuss learning from mainly three perspectives:

physical,

cognitive, and

contextual.

On the physical dimension, they argue that physical manipulation affords natural interactions and

encourages the creation of embodied representations to support learning.

On the cognitive dimension, they discuss how spatiotemporal alignment of information through AR systems can aid

student’s symbolic understanding by scaffolding the progression of learning, and result in improved understanding of abstract concepts.

On the contextual dimension, they argue that AR creates possibilities for interactive learning,

ultimately facilitating personal association and personally meaningful experiences.

VR & AR http://www.youtube.com/watch?v=bBjvqnKQsTI&list=P

LDF1BBECCE066EE5E

https://www.youtube.com/watch?v=V34gCw4fyLs

A standard VR system

A standard virtual reality system seeks to completely immerse the user in a computer generated environment.

This environment is maintained by the system in a frame of reference registered with the computer graphics system that creates the rendering of the virtual world. For this immersion to be effective, the egocentered frame of reference

maintained by the user's body and brain must be registered with the virtual world reference.

This requires that motions or changes made by the user will result in the appropriate changes in the perceived virtual world. Because the user is looking at a virtual world there is no natural connection

between these two reference frames and

a connection must be created.

(Azuma 1993).

A standard AR system An augmented reality system could be considered the ultimate

immersive system. The user can not become more immersed in the real world.

The task is to register the virtual frame of reference with what the user is seeing.

This registration is more critical in an augmented reality system because we are more sensitive to visual misalignments than to the type of vision-kinesthetic errors that might result in a standard virtual reality system.

The combination of real and virtual images into a single image presents technical challenges for designers of augmented reality systems. How to do this merging of the two images is a basic decision the designer must make.

Milgram and Kishino 1994; Milgram, Takemura et al. 1994 categorize augmented reality systems using Extent of Presence Metaphor that directly relates to the display that is used.

the multiple reference frames that must be related in an augmented reality system.

Display based AR At one end of the spectrum is monitor based viewing of the

augmented scene. This has sometimes been referred to as "Window on the World" (Feiner, MacIntyre et al. 1993) or Fish Tank virtual reality (Ware, Arthur et al. 1993).

The user has little feeling of being immersed in the environment created by the display. This technology is the simplest available. It is the technology that several other systems in the literature uses (Drascic, Grodski et al. 1993; Ahlers, Breen et al. 1994).

HMD based AR

To increase the sense of presence Head-mounted displays (HMD) have been widely used in virtual reality systems.

Augmented reality researchers have been working with two types of HMD: video see-through and

optical see-through.

The "see-through" designation comes from the need for the user to be able to see the real world view that is immediately in front of him even when wearing the HMD.

Video see-through AR The standard HMD used in virtual reality work gives the user complete

visual isolation from the surrounding environment.

Since the display is visually isolating the system must use video cameras that are aligned with the display to obtain the view of the real world.

This can be seen to actually be the same architecture as the monitor based display described above except that now the user has a heightened sense of immersion in the display.

Optical see-through AR The optical see-through HMD (Manhart, Malcolm et al. 1993) eliminates

the video channel that is looking at the real scene.

Instead, the merging of real world and virtual augmentation is done optically in front of the user. This technology is similar to heads up displays (HUD) that commonly appear in

military airplane cockpits and recently some experimental automobiles.

In this case, the optical merging of the two images is done on the head mounted display, rather than the cockpit window or auto windshield, prompting the nickname of HUD on a head.

See for calibration: http://stctutorial.icg.tugraz.at/

Main Advantage of Augmented Feedback

the ability to overlay virtual information on reality.

AR systems are successful because

they display information relevant to the user

at the appropriate time and location.

Most AR systems leverage this spatiotemporal contiguity by overlaying virtual information relevant to physical objects and spaces.

Thus, AR technology can bridge the gap between

physical manipulatives and their symbolic representation

by morphing the physical object into its representation.

Advantages of AR Feedback Physical

spatial and temporal alignment,

mapping between abstract and physical manipulatives,

Cognitive

situated cognition,

instructional scaffolding,

Contextual

engagement through personalisation,

motivation through emotional connection.

Disdvantages of AR Feedback

Cognitive load

Reality blur

blurring between borders of reality and virtual,

Dependency on multi-sensory information and stimulation

which refers to addiction to feedback and

Accessibility of AR technology.

Advantages and Disadvantages of AR Displays

discussed in greater detail by Azuma (Azuma 1995).

There are some performance issues: With both of the displays that use a video camera to view the real world there is a

forced delay of up to one frame time to perform the video merging operation.

At standard frame rates that will be potentially a 33.33 millisecond delay in the view seen by the user.

Since everything the user sees is under system control compensation for this delay could be made by correctly timing the other paths in the system.

Or, alternatively, if other paths are slower then the video of the real scene could be delayed.

With an optical see-through display the view of the real world is instantaneous so it is not possible to compensate for system delays in other areas.

On the other hand, with monitor based and video see-through displays a video camera is viewing the real scene. An advantage of this is that the image generated by the camera is available to the system to provide tracking information.

The optical see-through display does not have this additional information. The only position information available with that display is what can be provided by position sensors on the head mounted display itself.

How to develop AR apps

Wikitude forecast that AR/VR will hit $120 billion in revenue by 2020, with AR taking the lion’s share at around $90 billion and VR at $30 billion.

AR SDK for Google Glass, Epson Moverio, Vuzix M100 and ODG R-7. The Wikitude SDK is available for both Android and iOS devices.

Wikitude SDK + Cordova

Wikitude SDK offers a plugin for Unity3D so

you can integrate Wikitude’s computer vision engine into a game or application fully based on Unity3D.

Wikitude SDK works with up to 1000 images that can be recognized offline.

Apache Cordova is an open-source mobile development framework.

It allows you to use standard web technologies - HTML5, CSS3, and JavaScript.

Vuforia SDK Vuforia is an AR Software Development Kit (SDK) for mobile devices.

It uses Computer Vision technology to recognize and track planar images (Image Targets) and simple 3D objects, such as boxes, in real-time. This image registration capability enables developers to position and orient virtual

objects, such as 3D models and other media, in relation to real world images when these are viewed through the camera of a mobile device.

The virtual object then tracks the position and orientation of the image in real-time so that the viewer’s perspective on the object corresponds with their perspective on the Image Target, so that it appears that the virtual object is a part of the real world scene.

The Vuforia SDK supports a variety of 2D and 3D target types including ‘markerless’ Image Targets, 3D Multi-Target configurations, and a form of addressable Fiduciary Marker known as a Frame Marker.

Additional features of the SDK include localized Occlusion Detection using ‘Virtual Buttons’, runtime image target selection, and the ability to create and reconfigure target sets programmatically at runtime.

provides Application Programming Interfaces (API) in C++, Java, Objective-C, and the .Net languages through an extension to the Unity game engine.

supports both native development for iOS and Android

Using Unity Game Engine

One can introduce a plan as an image target to Unity.

Then, download the 3D form of buildings from google earth.

Afterwards, with one of compatible converter we need to convert 3D models to .FBX file.

As long as you have .FBX file, in Unity,

you can match it with the plan to show the building when the camera detects the plan in environment.

How to develop AR apps

**Setup the development environment:

1- Installing Cordova Cordova command-line runs on Node.js (https://nodejs.org) and

is available on NPM. Follow platform specific guides to install additional platform dependencies.

Open a command prompt or Terminal, and type npm install -g cordova.

2- Installing Android SDK http://developer.android.com/sdk/installing/index.html?pkg=tools

3- Downloading Wikitude sdk and Get your trail license http://www.wikitude.com/products/extensions/cordova-plugin-augmented-

reality/

http://www.wikitude.com/developer/licenses

**Setup Cordova application:

1- Create a project cordova create MyApp

2- Add a platform cd MyApp

cordova platform add android

3- Add Wikitude SDK Plugin

cordova plugin add $path-of-sdk

4- Add your wikitude license key at WikitudePlugin.js this._sdkKey = "Your-License-key";

5- Create WTC file for your target Image using Wikitude Target manager web application https://cas.wikitude.com/cas/login?service=https%3A%2F%2Ftargetmanag

er.wikitude.com%2F

6- Create Your AR world "See example application"

7- Load your AR world "See example application"

For more information see wikitude documentation http://www.wikitude.com/developer/documentation/phonegap

http://developer.android.com/tools/device.html

manolya.kavakli@mq.edu.au

http://www.comp.mq.edu.au/~manolya/

We are all looking for an answer but in fact what drives us is the question. Future isn’t written. It is designed.

Using Psycho-physiological feedback in AR

The HCI technologies that we study today will provide the interfaces of tomorrow.