Lab 5: Vuforia + Unity

CS410/510: Virtual RealityEhsan Aryafar earyafar@pdx.eduSam Shippey sshippey@pdx.edu

1

Review from last week● Vuforia: What/why● Go to https://developer.vuforia.com/● Sign up for an account● Upload an image under “target manager”● When choosing an image, select for:

○ Strong lines○ Good lighting○ <2Mb size

2

2

Review from last week● Key AR concepts

○ Marker-based AR vs Markerless AR○ A really rough view of how object detection works○ Know what SLAM means and why it’s important

3

3

Review from last week● Key AR concepts

○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,

within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from

there by adding things to the scene.○ A really rough view of how object detection works○ Know what SLAM means and why it’s important

4

4

Review from last week● Key AR concepts

○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,

within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from

there by adding things to the scene.○ A really rough view of how object detection works

■ Classification, bounding boxes, etc. Limited on phones, but works better than you would think.

○ Know what SLAM means and why it’s important

5

5

Review from last week● Key AR concepts

○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,

within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from

there by adding things to the scene.○ Know what SLAM means and why it’s important

■ Simultaneous Localization And Mapping. Depth maps, orientation, etc.

○ A really rough view of how object detection works

6

6

Review from last week● Key AR concepts

○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,

within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from

there by adding things to the scene.○ Know what SLAM means and why it’s important

■ Simultaneous Localization And Mapping. Depth maps, orientation, etc.

○ A really rough view of how object detection works■ Classification, bounding boxes, etc. Limited on phones, but works

better than you would think.7

7

● Limits to environmental understanding

Addendum to last week

8

8

● Limits to environmental understanding○ Object recognition and SLAM are hard

Addendum to last week

9

9

● Limits to environmental understanding○ Object recognition and SLAM are hard○ Phones have limited hardware and poor thermals

Addendum to last week

10

10

● Limits to environmental understanding○ Object recognition and SLAM are hard○ Phones have limited hardware and poor thermals○ Functionality for cutting edge software isn’t always particularly

user-friendly.

Addendum to last week

11

11

● Although neural approaches to object detection, SLAM, and other AR related tasks such as human segmentation are still hard, they’re now doable, even on phones.

● Even over the web.

Addendum to last week

12

12

Addendum to last week● Some examples from Tensorflow’s javascript API:

○ https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection

○ https://github.com/tensorflow/tfjs-models/tree/master/posenet

13Facemesh Posenet

13

Onto Unity!● Open the Unityhub and we’ll get to work building a super

basic AR application.

14

14

Editor settings● If you plan to build and

deploy this, make sure your platform’s build support is checked in the editor you installed.

● To check, go to Installs in the UnityHub.

15

15

What to do if this doesn’t build● This might or might not build for you● Why this happens

○ As previously discussed, iOS doesn’t necessarily play nice with Windows or Linux to the point where cross compilation is impossible.

○ Sometimes Android is annoying.○ Oftentimes Unity is annoying.

● What to do○ Rejoice! Vuforia can simulate some of this.○ If you really, really want to deploy quickly, we have other

technologies.16

16

Getting started● Create project

17

17

Getting started● Setup editor to work correctly● We want to be able to see:

18

18

Getting started● Setup editor to work correctly● We want to be able to see:

○ Package manager: Window->Package manager

The package manager lets us add packages, which are distinct from Assets in that they somehow extend not just your project’s functionality, but Unity’s as well, both the editor and the engine.

19

19

Getting started● Setup editor to work

correctly● We want to be able to see:

○ Package manager○ Game windows

These should be visible by default, but it’s easier if both the preview and the scene view are both turned on. 20

20

Adding Vuforia support● In the package manager,

find “Vuforia Engine AR”

21

21

Adding Vuforia support● In the package manager,

find “Vuforia Engine AR” ● Click “Install”

22

22

Adding Vuforia support● In the package manager,

find “Vuforia Engine AR” ● Click “Install”● Wait

23

23

Vuforia Camera● AR Cameras are different!● Even with marker-based AR, we still have to do 6dof

tracking.● Vuforia implements its own camera for this reason exactly.● Remember: It matters remarkably little where this camera

is. All positions are computed relative to the marker rather than the camera.

24

24

Vuforia Camera● Add by going to

“GameObjects”● -> Vuforia Engine● -> AR Camera● Delete the main camera.● Position somewhere you’ll

remember it.● Should look basically the

same as the default scene. 25

25

Adding the Vuforia marker● Time to add the markers we made last week through

Vuforia’s developer portal.

26

26

Adding the Vuforia marker● Time to add the markers we made last week through

Vuforia’s developer portal.● Go to the Vuforia developer portal

27

27

Adding the Vuforia marker● Time to add the markers we made last week through

Vuforia’s developer portal.● Go to the Vuforia developer portal● Target manager

28

28

Adding the Vuforia marker● Time to add the markers we made last week through

Vuforia’s developer portal.● Go to the Vuforia developer portal● Target manager● Find the target you created in lab 4

29

29

Adding the Vuforia marker● Time to add the markers we made last week through

Vuforia’s developer portal.● Go to the Vuforia developer portal● Target manager● Find the target you created in lab 4● Download it as a .unitypackage.

30

30

Adding the Vuforia Marker● Back in the Unity editor, go to● GameObject● -> Vuforia Engine● -> Image

31

31

Adding the Vuforia Marker● You should now have an image in the scene!

32

32

Adding the Vuforia Marker● You should now have an image in the scene!● Note: It may be extremely tiny because of the scale. Make

sure it’s there by zooming in.

33

33

Adding the Vuforia Marker● You should now have an image in the scene!● Note: It may be extremely tiny because of the scale. Make

sure it’s there by zooming in.● Time to add our image. Go to● Assets● -> Right click● -> Import package…● -> Custom package…● Find the .unitypackage you downloaded earlier

34

34

Adding the Vuforia Marker● Now, we want to configure the image we

have to match our database.● Click the image on the scene (Or

click+drag to select it)

35

35

Adding the Vuforia Marker● Now, we want to configure the image we

have to match our database.● Click the image on the scene (Or

click+drag to select it)● Examine the object in the inspector.

36

36

Adding the Vuforia Marker● Now, we want to configure the image we

have to match our database.● Click the image on the scene (Or

click+drag to select it)● Examine the object in the inspector.● Change the “database” and “image

target” attributes to point to what you want them to.

37

37

The actual app part● You now have a kind of working app!● Add some assets, as children of the image target.

38

38

The actual app part● You now have a kind of working app!● Add some assets, as children of the image target.● I’m uncreative, so I’ve put a cube here.

39

39

The actual app part● You can now work as if this was a more or less normal

app in Unity.● This part is the part you’re familiar with

40

40

Deployment, further reading● Deployment is intentionally left out of the close

instructions as otherwise it would exclude most of the iOS users.

● As such, as part of the homework, Android users are encouraged (but not required) to work out how to deploy to Android.

● This isn’t required because the sheer volume of software needed to do it makes it borderline unfair.

41

41

Deployment, further reading● Guide for mobile deployment

○ https://learn.unity.com/tutorial/building-for-mobile#5c7f8528edbc2a002053b4a1

● Tips for mobile deployment○ Make sure the phone is connected in the right way according to

the software. My phone is set, by default, to act something like a USB stick. This doesn’t go so great for sending apps to it. Make sure USB Debugging is enabled under developer settings, etc.

○ If your phone gets a bit hot testing, that’s unfortunately kind of normal.

42

42

Deployment, further reading● Deployment is intentionally left out of the close

instructions as otherwise it would exclude most of the iOS users.

● The homework○ Add some more things to your app! Assets, physics, etc. Knock

yourself out.○ Package it up and send it to us.

43

43

Deployment, further reading

44

● Guide for mobile deployment○ https://learn.unity.com/tutorial/building-for-

mobile#5c7f8528edbc2a002053b4a1● Tips for mobile Unity usage

○ Unity has support for mobile input, you can read about that here: https://learn.unity.com/tutorial/touch-input-for-mobile-scripting

○ You might want to minimize the number and complexity of elements on the screen, particularly in regard to physics.

44

Deployment, further reading● Deployment is intentionally left out of the close

instructions as otherwise it would exclude most of the iOS users.

● The homework○ Add some more things to your app! Assets, physics, etc. Knock

yourself out.○ Package it up and send it to us.○ How to do that: Find where it’s installed (Right click Assets->Open

in Containing Folder…), go up a level, package the folder your project is in in a zip, tarball, 500mb git repository, etc.

45

45

Deployment, further reading● Guide for mobile deployment

○ https://learn.unity.com/tutorial/building-for-mobile#5c7f8528edbc2a002053b4a1

● Tips for mobile deployment○ Make sure the phone is connected in the right way according to

the software. My phone is set, by default, to act something like a USB stick. This doesn’t go so great for sending apps to it. Make sure USB Debugging is enabled under developer settings, etc.

46

46

Further reading● Vuforia is easy to use, but certainly not the only AR

platform!● If you’d like to experiment more with Marker based AR,

consider using Three.JS + AR.JS, AWS Sumerian, or any one of around a dozen constantly shifting proprietary solutions.

47

47

Further reading● Vuforia is easy to use, but certainly not the only AR

platform!● If you’d like to experiment more with Marker based AR,

consider using Three.JS + AR.JS, A-Frame, AWS Sumerian, or any one of around a dozen constantly shifting proprietary solutions.

● If you’d like to experiment with markerless AR, try the same set of things, plus Unity’s AR Foundation.

48

48

Let’s experiment● I’ll sit around here to answer more questions if you want to

continue adding things while we’re here.● Feel free to leave and bug me on Slack later, or stick

around and try to debug things with this OR with your project now.

49

49