Augmented Realityfor Consumer Devices (and shortcuts for Windows Phone)

Jared Bienzjbienz@microsoft.comjaredbienz.wordpress.com@jbienz

Windows Phone Microsoft Corporation.

What will be covered

What is Augmented Reality? How AR is being used today Different approaches to AR (platform agnostic) Tools for building AR on Windows Phone Getting “crafty”

Windows Phone Microsoft Corporation.

What is Augmented Reality?

“A live direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data.”

- Wikipedia

Windows Phone Microsoft Corporation.

What is Augmented Reality?

“A view of the world through a digital “lens”, which allows the scene to be changed or appear differently than the real-world environment.”

- Jared Bienz

Windows Phone Microsoft Corporation.

AR Approaches

Windows Phone Microsoft Corporation.

Vision-Based (Games)

“These Magic Cards Are The ‘Holy Shit’ Moment of Nintendo’s New Toy”- Gizmodo

(March 2011)

Windows Phone Microsoft Corporation.

Vision-Based (Apps)

Kogan concept app

Windows Phone Microsoft Corporation.

Sensor-Based (Games)

Live Butterflies Viewer / Game

Windows Phone Microsoft Corporation.

Sensor-Based (Apps)

PhotoSynth for iPhone (yes, we know)

Windows Phone Microsoft Corporation.

Vision vs. SensorVision+ No sensors required (only access to camera feed)+ Existing libraries for tracking tags (code is fairly simple)- Tied to specially printed tags (in range, good angle)- CPU intensive

Sensor+ Requires no printed tags+ Not nearly as CPU intensive- Code is more complex (due to dealing with sensors)- Most current examples are tied to a fixed location- Requires sensors

Windows Phone Microsoft Corporation.

Steps for Building Vision-Based AR1. Capture video from the camera2. Using a vision library to locate tag in scene

(position) and determine its scale and skew3. Working backward, calculate the real-world

camera location and direction4. Transform virtual scene (possibly using a virtual

camera)5. Draw virtual scene on top of camera feed6. Repeat

Windows Phone Microsoft Corporation.

Vision-Based AR on Windows Phone

SLAR Toolkit - slartoolkit.codeplex.com

00:37

00:37

Windows Phone Microsoft Corporation.

Sensor-Based AR

𝑥

𝑦

𝑧

XNA

Virtual World

𝑥𝑦

𝑧

Camera and Sensors Augmented World+ =

Windows Phone Microsoft Corporation.

Steps for Building Sensor-Based AR1. Position items in “virtual world” (WP use XNA)2. Determine how device is rotated using Motion APIs3. Rotate virtual world to match device rotation4. Capture video from camera5. Draw virtual world on top of camera feed

a) (WP) Use XNA to render in XNA objects – or – b) (WP) Convert XNA space to Silverlight transforms

6. Repeat

Windows Phone Microsoft Corporation.

What Sensors are Required? Accelerometer

Camera

Compass

Gyro (best)

New in WP 7.5

Windows Phone Microsoft Corporation.

Connecting the camera to UI (WP)// Create PhotoCameracamera = new PhotoCamera(CameraType.Primary);

// Create a VideoBrush and connect it to the cameracameraBrush = new VideoBrush();cameraBrush.SetSource(camera);

// Fill rectangle with video from the camerasomeRectangle.Fill = cameraBrush;

Windows Phone Microsoft Corporation.

SensorsAccelerometer+ Very fast+ 3 Axis (X, Y, Z)- Not a good reference point

in motion

Compass+ Simple to use- Jitters- 1 Axis only (heading)

Gyro+ Fast+ 3 Axis (X, Y, Z)+ Always a reference

point, even in motion- Drifts- Not always available

Windows Phone Microsoft Corporation.

Motion “Sensor” (WP)

Motion Sensor

[cos γ .cos α −cos β . sin α+sin β . sin γ .cosα sin β . sin α+cos β .sin γ .cos

α

cos γ . sin α cos β .cos α+sin β . sin γ . sinα − sin β .cos α+cos β . sin γ .sin α

− sin γ sin β .cosγcos β .sin γ

]All the sensors + a bunch of math!

Use this sensor whenever available

Windows Phone Microsoft Corporation.

Using Motion “sensor” (WP)// Create Motion “sensor”if (Motion.IsSupported){

motion = new Motion();motion.TimeBetweenUpdates = TimeSpan.FromMilliseconds(20);motion.CurrentValueChanged += motion_CurrentValueChanged;motion.Start();

}

// Handle movementvoid motion_CurrentValueChanged(object sender, SensorReadingEventArgs<MotionReading> e){ …}

Windows Phone Microsoft Corporation.

Hardware Availability (WP)Accelerometer & Compass are required for motion. Highest quality if Gyro is available.

All 7.0 devices have Accelerometer Some 7.0 devices have Compass (Motion

enabled) All 7.5 devices will have Compass (Motion

enabled) Many 7.5 devices will have Gyro (Best motion)

Windows Phone Microsoft Corporation.

Debugging and Testing Only Accelerometer supported in the emulator

today Test for Motion.IsSupported and fail gracefully! AR should always be tested on a physical device

Windows Phone Microsoft Corporation.

Sensor-Based AR on Windows Phone

Step-by-step video - bit.ly/WPARBasic

Windows Phone Microsoft Corporation.

Geo Augmented Reality

Real World ObjectLocation + =

𝑥𝑦

𝑧

𝐴𝑙𝑡𝑖𝑡𝑢𝑑𝑒

𝐿𝑜𝑛𝑔𝑖𝑡𝑢𝑑𝑒

𝐿𝑎𝑡𝑖𝑡𝑢𝑑𝑒

Nort

h

𝑥

𝑦

𝑧

XNA

TexMex

TexMex

Your Location andDirection (sensors) +

Automatically Generated

Virtual World

AugmentedWorld

Alt,

Windows Phone Microsoft Corporation.

Steps for Building Geo-Based AR1. Get location of objects in real world (web service, etc.)2. Get location of the user (Location APIs)3. Calculate actual distance between user and objects4. Generate “virtual world” where user is at center5. Capture the direction the user is looking (Motion APIs)6. Rotate virtual world to match device rotation7. Capture video from camera8. Draw virtual items on top of camera feed9. Repeat

Windows Phone Microsoft Corporation.

Introducing: GART

Geo AR Toolkit - gart.codeplex.com

00:18

Windows Phone Microsoft Corporation.

Steps for using GART1. New Windows Phone project2. Add a reference to GART.dll3. Add ARDisplay control to page (use Expression

Blend)4. Add child views (map, heading, camera, 3D)5. Create an ARItem to represent a real-world thing6. Add ARItem to ARDisplay.ARItems collection

Windows Phone Microsoft Corporation.

GART Manages SensorsGART automatically manages the camera, motion and location APIs for you!

Call ARDisplay.StartServices in Page.OnNavigatedTo Call ARDisplay.StopServices in Page.OnNavigatedFrom

If there is a problem starting or stopping any sensor it’s bubbled through the ARDisplay.ServiceError event.

Windows Phone Microsoft Corporation.

GART Demo

Windows Phone Microsoft Corporation.

Getting Crafty

Windows Phone Microsoft Corporation.

Targus TG-42TT $14

Cable Ties: 30 cm $3

Cable Ties: 10 cm $3

Friction Mount $16

Spray Grip $3

Mainstays Fabric Bin $6

Total:$45

Windows Phone Microsoft Corporation.

Thank YouPlease let me help you build Windows Phone apps

gart.codeplex.com jbienz@microsoft.com jaredbienz.wordpress.com @jbienz on Twitter

Windows Phone Microsoft Corporation.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

© 2011 Microsoft Corporation.

All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.