Open XR – And the Future of AR/VR/MR Gaming

Brent Insko, Virtual Reality Center of Excellence

October, 2017

Where is VR today?

2

3

Gartner Hype Cycle 2017

© 2017 Gartner, Inc.

What is holding VR back?

4

What is holding VR back?

• Content• Comfort• Cost

5

6

7

Platforms

8

Platforms VR Platforms

9

3 PCVR Verticals

Oculus Store Microsoft Store

Mixed Reality App

Mixed RealityRuntime

Oculus App

OculusRuntime

Mixed RealityHMDs

OculusRift

Steam Store

SteamVRApp

SteamVRRuntime

HTC Vive(+ HDK & others)

10

11

12

13

What is a VR Runtime?

14

15

High Level VR Software Stack

VR RuntimeServer Compositor

DirectX (+ other system APIs)

Windows

VR Application Browser

VRHardware

OpenXRAPIs

WebVRGame Engine

Application interface

Driver(HW) interface

• Takes Windows surfaces from multiple applications

• Composites onto a single desktop in 2.5D

• Handles transparency/ translucency

Windows Compositor… Desktop Window Manager (DWM)

VR Compositor

T1 T2 T3

• Composition• Time Warp• Barrel Distortion• Chromatic Aberration

Correction

VR Compositor

18

• Head and controller positions

• Controller state• Input events

• Game images• Haptics events

Information Flow

19

Why does Intel care?

20

VR Vision – The PC PlatformThe PC is doing more than ever to stimulate the user and drive their senses

Physically driven audioAudio effects that are true to life, reflecting off

objects and surface materials, accounting for distance and orientation.

Intense graphicsHigher resolution, faster frame rates, and depth capable displays, fully immerse players in the experience.

Hands-free inputDepth cameras incorporated into HMDs and

connected to the PC track arm and hand movement allow controller-less input

Multiplayer supportSimulation accounts for multiple players in the

same room – including both players VR experience being driven off of one high-performance PC.

Wireless freedomHigh-speed, low-latency wireless driving of all peripherals.

Haptic feedbackVest and controllers mimic in-game interactions to further stimulate the player and make interactions between teammates more physical.

Vest reacts with jolt from enemyweapon fire.

Vest and controllers relay low health condition.

Physically accept a healing item handed to you by a teammate. Object heft is conveyed to you.

1

2

3

We believe VR will evolve to allow multi-modal input and feedback. Here we show a variety of the technologies that the PC will be providing feedback to the user.

CPU WorkloadsParticle effects and physics, Artificial

intelligence, Destruction and Object Persistence, Initialization & level setup, Spatial binaural 3D

audio, frame rate, mega-tasking workloads

VR Vision – The Sensor PlatformFully immersing users in VR will include full awareness of their state and movement

Facial expression trackingMask sensors map your reactions to your in

game character, enhancing non-verbal communication and heightening involvement.

Object scanningDepth cameras in HMD allow scaling real

world objects to bring in game. Swing a favorite toy sword or bring a child’s art class

project into social spaces.

Eye trackingPlayers eyes are reflected in their in game character, expressing emotion and intent. Gaze tracking can unlock UI triggers, and drive rendering performance focus.

Biometric sensorsHeart rate, pupil dilation and galvanic skin response can be used to convey emotional responses such as fear or excitement.

Responses can be analyzed to tailor game difficulty, give cues to other players, or to limit fatigue.

At the same time, the PC will be capturing a wide array of conscious and unconscious input from the player, and feeding this back to the simulation for processing

Intuitive ControllersControllers evolve to intuitively

communicate greater degrees of intent, with analog triggers, contact sensing, and greater

degrees of affordance

Physics-driven presence Motion tracking of HMDs and controllers, combined with 3D cameras, and CPU-driven physics are used to track and confer full body articulation .

23

3rd Person Camera AI (Yoann Pignole Gamasutra 2015)

GFX

Source(PC) Sink(HMD)

Panel(Output)

Sensors(Input)

RenderTimeWarpBarrelDistortionChromaCorrection

Display(Scalar)

PoseInputImagingData

Tx Rx

USBDisplay

Displaylink Wireless VR

Intel’s partnership with ISVs on VR content

Intel Projects

2020 Virtual Reality Vision

Adventure together with friends, compete with rivals, or bring the world along for the show

Open Innovation & Best of class performance allow VR to thrive and lead on PCUsage: Truly Immersive multiplayer gaming allows players to travel to other worlds, whether next to each other or across the globe

Focus areas: Gaming, eSports, Social

Key capability areas (hypotheses)

• Mind-tripping quality: Photo-real, deep behavioral simulation, responsive enough to exceed sensory reqs. “Believed I was there”

• Wireless HMDs and peripherals and/or wearable form factors for unhindered and unlimited movement

• Multi sensory stimulation: physiological-transparent quality display & graphics, physically modeled 3D Audio, (haptic stimulation), (vestibular dampening),

• Multi-model input, sensing, and responsive simulation: eye-tracking, expression tracking, tactile feedback, biometric feedback (galvanic response, heart rate, etc), voice & outside audio cuing, environment capture

• Social & competitive: Your expressions & movements driving your presence in virtual interactions with others

New York Los Angeles London

WW Spectators

The Future of AR/VR/MR Gaming & OpenXR• Engaging more senses, better experiences

à more immersion

à more presence

• More innovation

25

The Future of AR/VR/MR Gaming & OpenXR• Engaging more senses, better experiences

à more immersion

à more presence

• More innovation

• Dependent on:

à more content more places

à more hardware more places

à easier cross-platform development & testing

26

Placeholder Footer Copy / BU Logo or Name Goes Here

28

à more content more placesà more hardware more placesà easier cross-platform development & testing