Stereos Copy

7/21/2019 Stereos Copy

http://slidepdf.com/reader/full/stereos-copy-56e42374c74cf 1/99

G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

GDC 2011NVIDIA SPONSORED

Samuel Gateau, NVIDIADominic Filion, Blizzard

Stereoscopic 3D Demystified:From Theory to Implementation in Sta



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

Outline

Fundamentals of Stereoscopic 3D

Stereo projection

Depth Perception & how to manage it

Stereoscopy in a game engine

Stereo rendering engine modifications & common pitf

3D vision driver

Starcraft 2



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

3D StereoscopicFundamentals



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

3D Stereoscopic Fundamentals

Stereo Projection



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

TWO EYES, ONE SCREEN, TWO IMAGES

Changes to the rendering pipe



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

In Mono

Eye space

ZY

X

Near plane

Scene is viewed from one eye

and projected with a perspectiveprojection along eye direction on

Near plane in ViewportMono Frustum

Scene

Viewport



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

In Stereo

Eye space

ZY

X

Scene

Near plane

S



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

In Stereo:

Two eyes

Eye space

ZY

X

Left and Right eyesShifting the mono eye along

the X axis

Scene

Near plane

S



G D C

2 0 1 1

N V I D I A

S P O N S O

R E D

S E S S I O N S

In Stereo:

Two eyes

Eye space

ZY

X


the X axisEye directions are parallels

Scene

Near plane

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

Virtual Screen

In Stereo: Two Eyes,

One Screen

Eye space

ZY

X

One “virtual” screen



Scene

Near plane

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

Scene

In Stereo: Two Eyes,

One Screen

Virtual Screen

Eye space

ZY

X

Left Frustum Right Frustum


Where the left and right

frustums converge



Near plane

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

In Stereo: Two Eyes, One Screen,

Two Images

Virtual Screen

Eye space

ZY

X

Two images2 images a

the near pla

Scene

Left

Image

Right

Image




Where the left and right

frustums converge

Near plane

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

In Stereo: Two Eyes, One Screen,

Two Images

Virtual Screen

Eye space

ZY

X

Scene

Left

Image

Right

Image

Left Image Right Image

Real Screen

Near plane

Two images2 images a

the near pla

Presented

each eyes

real screen

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

Stereoscopic Rendering

Render geometry twice

From left and right eyes

Into left and right images

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

DEFINING STEREO PROJECTION

Basic definitions so we all speak English

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

Stereo Projection

Human vision is really like 2 eyes looking at a parallel direction

Left Eye

Right Eye

Z

Y

X

Eye space

S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N

Stereo Projection

Stereo projection matrix is a horizontally offset version of regular mon

matrix

Offset Left / Right eyes along X axis

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Screen

N S



G D C

2 0 1 1

N V I D I A

S P O N S O R E D

S E S S I O N

Stereo Projection

Projection Direction is parallel to mono direction (NOT toed in)

Left and Right frustums converge at virtual screen

Left Eye

Right Eye

Mono Eye

Virtual

Screen

Z

Y

X

Eye space

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N

Parallel, NOT Toed in!

Historically, live camera mounted in parallel stereo would waste a lot

Waste view field is wasted film area

Left Eye

Right Eye

Mono EyeZ

Y

X

Eye space

Virtual

Screen

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N

Parallel, NOT Toed in!

Hence the Toed-in camera solution

But Toed in frustum introduces deformation which is really painful Image Planes are not parallel to the screen plane

This can be corrected in post production but not perfect

Left Eye

Right Eye

Mono EyeZ

Y

X

Eye space

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N

Interaxial

Distance between the 2 virtual eyes in eye space

The mono, left & right eyes directions are all parallels

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye Interaxial

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N

Separation

The normalized version of interaxial by the virtual screen width

More details in a few slides….

Screen wZ

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye Interaxial

VirtualScreen

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N

Convergence

Virtual Screen„s depth in eye space (“Screen Depth”)

Plane where Left and Right Frustums intersect

Convergence

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Virtual

Screen

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N

Parallax

Signed Distance on the virtual screen between the projected

positions of one vertex in left and right image

Parallax is function of the depth of the vertex

Z

Y

X

Eye space

Left Eye

Right Eye

Mono Eye

Virtual

ScreenParalla

Vertex depth

Convergence I n t e r a x i a l

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O N


Depth Perception

N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O

DEPTH PERCEPTION

Where the magic happens

N S

Vi l R l S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O Virtual vs. Real Screen

V

Virtual Space

ZY

X

The virtual screen is

perceived AS the real screenLeft Image Right Image

Real Screen

O N S

P ll i D th



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O Parallax is Depth

V

Virtual Space

ZY

X

Left Image Right Image

Real Screen

O N S

P ll i D th



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O Parallax is Depth

V

Virtual Space

ZY

X

Parallax creates the depth

perception for the userlooking at the real screen

presenting left and right

images

Left ImageRight Image

Real Screen

O N S

I / O t f th S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O In / Out of the Screen

Z

Y

X

Eye space

Left Eye

Right Eye

Mono Eye

ScreenOut of the Screen In the Screen

Convergence

Vertex Depth Parallax Vertex A

Further than Convergence Positive In the S



Equal Convergence Zero At the



Equal Convergence Zero At the S

Closer than Convergence Negative Out of th

O N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O

COMPUTING PARALLAX & PROJECTION M

Equations !!!

O N S

Computing Parallax



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O Computing Parallax

Thank you Thales

Z

Y

X

Eye space

Left Eye

Right Eye

Mono Eye


Parallax

S c r e e n

Depth

O N S

Computing Parallax



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O

S c r e e n

Computing ParallaxIn image space (not pixels but in range [0,1])

Z

Y

X

Eye space

Left Eye

Right Eye

Mono Eye

Depth


Parallax

O N S

Computing Parallax



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O

S c r e e n

Computing ParallaxAnd clip space for free

Z

Y

X

Eye space

Left Eye

Right Eye

Mono Eye

Depth


Parallax

O N S

Parallax in normalized image space



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O Parallax in normalized image space

P a r a l l a x i n n o r m a l i z e d i m a g e s p a c e

Depth

Separation

C o n v e r g e n c e

Parallax diverges quickly to negative infinity

for object closer to the eye

Parallax is 0 at screen depth

Maximum Parallax separation dista

O N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O


Managing the depth

O N S



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I O

REAL EYE SEPARATION

Take care of your audience

O N S

Real Eye Separation



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I Real Eye Separation

Interocular (distance between the eyes) is on

average 2.5” 6.5 cm

Equivalent to the visible parallax on screen for

objects at infinity

Depending on the screen width, we define a

normalized “Real Eye Separation”

Different for each screen model

A reference maximum value for

the Separation used in the stereo

projection for a comfortable experience

Real

Scree

Interocular

Para

O N S

Real Eye Separation is infinity



G D C

2 0 1 1

N V I D I A

S P O N S

O R E D

S E S S I Real Eye Separation is infinity

The maximum Parallax at infinity is

Separation Real Eye Separation should be used as

the very maximum Separation value

Rea

I O N S

Separation must be Comfortable



G D C

2 0 1

1

N V I D I A

S P O N S

O R E D

S E S S I Separation must be Comfortable

Never make the viewer look diverge

People don‟t have the same eyes

For Animation movie, separation must

be very conservative because of the

variety of the screen formats

IMAX vs Home theatre

For Interactive application, let the useradjust Separation

When the screen is close to the user (PC

scenario) most of the users cannot handle

more than 50% of the Real Eye Separation

Rea

I O N S

Real Eye Separation is the Maximum



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S Real Eye Separation is the Maximum

ParallaxReal ScReal Screen

I O N S

Safe Parallax Range



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S Safe Parallax Range

P a r a l l a x


Real Eye

-Real Eye

S I O N S



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S

PARALLAX BUDGET

Convergence and Separation working together

S I O N S

Parallax Budget



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S

Nearest

pixel

Farthest

pixel

Parallax

budget P a r a l l a x


Parallax BudgetHow much parallax variation is used in the fr

S I O N S

In Screen : Farthest Pixel



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S In Screen : Farthest Pixel

At 100 * Convergence, Parallax is 99% of the Separation

For pixels further than 100 * Convergence,

Elements looks flat on the far distance with no depth differentiation

Between 10 to 100 * Convergence, Parallax vary of only 9

Objects in that range have a subtle depth differentiation

P a

r a l l a x

Separation


S I O N S

Out of the Screen : Nearest pixel



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S Out of the Screen : Nearest pixel

At Convergence / 2, Parallax is equal to -Separation, out

Parallax is very large (> Separation) and can cause eye strains

P a r a l l a x


S I O N S

Convergence sets the scene in the



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S Convergence sets the scene in the

Defines the window into the virtual space

Defines the style of stereo effect achieved (in / out of the screen

P a r a l l a x

C o n v e r g e n c e 1

Far pixelNear pixel

C o n v e r g e n c e 2

Parallax

budget 1

Parallax

budget 2

S I O N S

Separation scales the parallax budg



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S p p g

Scales the depth perception of the frame

P a r a l l a x


Parallax

budget 1

Far pixelNear pixel

Parallax

budget 2

S I O N S

Adjust Convergence



G D C

2 0 1

1

N V I D I A

S P O N S O R E D

S E S S j g

Convergence is a Camera parameter driven by the lo

frame Artistic / Gameplay decision

Should adjust for each camera shot / mode

Make sure the scene elements are in the range

[ Convergence / 2, 100 * Convergence ]

Adjust it to use the Parallax Budget narratively Art direction on the stereo

Dynamic Convergence is a bad idea

Except for specific transition cases

S I O N S



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S S

Stereoscopy in a game e

S I O N S



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S S

RENDERING IN STEREO

Let‟s do it

S I O N S

Stereoscopic Rendering



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S p g

Render geometry twice Do stereo drawcalls

Duplica

Using co

resou

From left and right eyes Apply stereo projection Modify pro

Into left and right images Use stereo surfaces Duplicate r

S S I O N S

How to implement stereo projectio



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S

p p j

Start from the mono transformation stack

Inject the side, separation and convergence to get a stereo tra

stack

RasterizationVertex Shader

Worldspace

Eyespace

Clipspace

Normalizedspace

ViewTransform

ProjectionTransform

PerspectiveDivide

ViewportTransform

Stereo Projection Matrix


EyeSpace

Stereo Clipspace

StereoNormalized

space

StereoProjection

Transform

PerspectiveDivide

ViewportTransform

…


Clipspace

Stereo Clipspace

Stereo

Normalized

space

StereoSeparation

PerspectiveDivide

ViewportTransform

Eyespace

ProjectionTransform

…

Stereo shift on clip position

S S I O N S




G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S Right handed column major matrix ( OpenGL style )

S S I O N S




G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S Left handed row major matrix ( D3D9 style )

S S I O N S

Stereo shift on clip position



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S

Just before rasterization in the vertex shader, offset the clip po

parallax amount


Clip

space

Stereo Clip

space

StereoNormalized

space

Stereo

Separation

Perspective

Divide

Viewport

Transform

Eye

space

Projection

Transform…

S S I O N

S

Stereo rendering surfaces



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S

Screen

Left Image

Right Image

View dependent render targets

must be duplicated Back buffer

Depth Stencil buffer

Intermediate full screen render

targets used to process finalimage

High dynamic range, Blur, Bloom

Screen Space Ambient Occlusion

Left Image

S S I O N

S

Mono rendering surfaces



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S

View independent render targets DON‟T need

duplicated Shadow map

Spot light maps projected in the scene

Screen

S S I O N

S

How to do the stereo drawcalls ?



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S

Simply draw the geometries twice, in left and right versions of

Can be executed per scene pass

Draw left frame completely

Then Draw right frame completely

Need to modify the rendering loop

Or for each individual objects

Bind Left Render target, Setup state for left projection, Draw geo Bind Right render target, Setup state for right projection, Draw G

Probably less intrusive and more efficient in an engine ( 3D vision

Not everything in the scene needs to be drawn

Just depends on the render target type

S S I O N

S

When to do what?



G D C

2 0 1

1

N V I D I A

S P O N

S O R E D

S E S

Use CaseRender Target

TypeStereo Projection S

Shadow maps Mono NoUse Shadow projection

Main frame

Any Forward rendering passStereo Yes

Reflection maps Stereo

Yes

Generate a stereo

reflection projection

Post processing effect

(Drawing a full screen quad)Stereo

No

No Projection needed at all

Deferred shading lighting

pass

(Drawing a full screen quad)

Stereo

G-buffers

Yes

Be careful of the

Unprojection

Should be stereo

S S I O N

S



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E S

STEREO CULLING & FLOATING WINDOW

S S I O N

S

3D Objects Culling



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

When culling is done against the mono frustum…

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Screen

E S S I O N

S

3D Objects Culling



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

… Some in screen regions are missing in the right and left frustum

They should be visible

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Screen

E S S I O N

S

3D Objects Culling



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

… And we don‟t want to see out of the screen objects only in one

It disturbs the stereo perception

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Screen

E S S I O N

S

3D Objects Culling



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

Here is the frustum we want to use for culling

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Screen

E S S I O N

S

3D Objects CullingComputing Stereo Frustum origin offset



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

Z

Y

X

Eye spaceLeft Eye

Right Eye

Mono Eye

Screen

p g g

Z = Convergence / ( 1 + 1 / Separation )

Z

Interaxial

Convergence

Screen W

E S S I O N

S

3D Objects Culling



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

Culling this area is not always a good idea

Blacking out pixels in this area is better

Through a shader

Equivalent to the “Floating window” used

in movies

Left Eye

Right Eye

Mono Eye

Screen

E S S I O N

S



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

STEREO TRANSFORM STACK TRICKS

E S S I O N

S

Fetching Stereo Render Target



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E

When fetching from a stereo render target use the good textur

Render target is addressed in STEREO IMAGE SPACE

Use the pixel position provided in the pixel shader

Or use a texture coordinate computed in the vertex shader corre

Pixel Shader

Stereo ImageSpace

POSITION.xy

Fetch Texelat

POSITION.xy

Do something

with it

Stereo Render

Target

Pixel Shader

…

Mono ImageSpace

uv

Fetch Texelat

uv

Do somethingwith it

Stereo Render

Target

E S S I O N

S

Unprojection in pixel shader When doing deferred shading technique Pixel shader fetch the de



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S E When doing deferred shading technique, Pixel shader fetch the de

(beware of the texcoord used, cf previous slide)

And evaluate a 3D clip position from the Depth fetched and XY vi

Make sure to use a Stereo Unprojection Inverse transformation tospace

Otherwise you will be in a Stereo Eye Space !

Pixel Shader

Stereo Image

Space

POSITION.xy

Image

space

Normalized

space

Clip

space

Fetch Depth

at

POSITION.xy

Viewport

Inverse

Transform

Perspective

Multiply

Evaluate

Image Space

Position

Stereo

Depth Buffer

Pixel Shader

Stereo Image

Space

POSITION.xy

Image

space

Normalized

space

Clip

space

Fetch Depth

at

POSITION.xy

Viewport

Inverse

Transform

Perspective

Multiply

Evaluate

Image Space

Position

StereoDepth Buffer

E S S I O N

S



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S

NVIDIA 3D VISION DRIVER

Finally something useful

E S S I O N

S

How Games are Converted to 3D autom



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

SPC Game & DirectX

All DirectX games have

lighting, depth, models that are

already created in a 3D world

As the data is processed by the

GPU, our driver dynamically

renders each frame twice.

You can then

Since conve

time, there is

depth adjust

NVIDIA GPU & 3D vision driver 3D Disp

S E S S I O N

S

Programming for 3D vision driver



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S

NVIDIA 3D Vision driver automatically renders twpositions for any DirectX game or application and

the stereo images Nothing to do !

More control on 3D vision driver behavior with N

Activation, separation & convergence control

And Heuristics defined in game profile

That‟s when we work together to fine tune the 3D visbehavior for your game

Game developer writes their game as usual, but all effects will work in 3D as much as possible

F ll i b t ti

S E S S I O N

S

3D vision driver basic Behavior



G D C

2 0 1 1

N V I D I A

S P O N

S O R E D

S

What How When

Create

Stereo surface

Duplicate textures on

first usage

• If texture is rend

• And not squa

• If destination of

stereo surface

Do

Stereo Drawcall

Duplicate drawcall

Swap stereo resource

sides in between

drawcalls

• If render target

Apply

Stereo Projection

Shift SV_Position.x at

the end of the Vertex

shader

• If drawcall is ste

• And If heuristic

if SV_Position.w

* Very basic behavior, much more cases are available through Heuristics

S E S S I O N

S

Why use 3D Vision driver ?



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S

Much more easy than doing it yourself

Requires to fix only a few remaining issues with our h Much better performances right out of the box

Stereo driver works within the low level NVIDIA drive

avoiding a lot of the runtime cost

Marketing and ecosystem support from NVIDIA

S E S S I O N

S

3D Vision driver – Developer Resou



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S

NVIDIA 3D Vision Developer Website

http://developer.nvidia.com/object/3d_stereo_de

Developers Conference Presentations

Best practices guide

GDC, NVISION & Siggraph Presentations

New samples coming… NvAPI

http://developer.nvidia.com/object/nvapi.html

S E S S I O N

S

http://developer.nvidia.com/object/3d_stereo_dev.html



http://developer.nvidia.com/object/3d_stereo_dev.html



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S

Starcraft II in Stereo

S E S S I O N

S



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S

ItSucks.

S E S S I O N

S

Steroscopic support in StarCraft II



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S

StarCraft II was initially shipped without any code or design ge

stereoscopic support

Official support for stereoscopic stereo was added in 1.2.0 patc

Process to get all the kinks worked out was on the order of 3 w

issues, designing UI for it, etc.

Although fairly straightforward to deal with, stereoscopic is sti

in games and a lot of the information has to be gathered from

undocumented sources

That‟s why we had Nvidia fly over their rocket scientists to help us out

Here is our experience so the process can be much easier for yo

S E S S I O N

S

What worked out of the box



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S

3D objects

Billboards

Skyboxes

All the 3D objects must have a coherent depth relati

scene

Don‟t fake your 3D

Lighting effects are visible in 3D so should be compu Highlight and specular are probably best looking evaluated

origin

Reflection and refraction should be evaluated with stereo

S E S S I O N

S

What needed fixing#1- 2D UI



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

S E S S I O N S

2D UI



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

With no stereo projection (or depth zero)

Head Up Display interface

Menus

At the correct depth when interacting with the 3D scene

Labels or billboards in the scene

The w component coming out of vertex shader must not be 1

S E S S I O N S

2D to 3D conversionshader function



G D C

2 0

1 1

N V I D I A

S P O N S O R E D

float4 2Dto3DclipPosition(

in float2 posClip : POSITION, // Input position in clip

uniform float depth // Depth where to draw the

) : POSITION // Output the position in

{

return float4(

posClip.xy * depth, // Simply scale the posClip by

// to compensate for the divis

// performed before rasterizat

0, // Z is not relevant if the depth buffer is

// If needed Z = ( depth * f – nf )/(f – n)

// ( For DirectX )

depth ); // W is the Z in eye space

}

S E S S I O N S

What needed fixing#2 - Selection, Pointing in 3D



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

, g Selection UI or cursors interacting with the 3D scene don‟t wor

mono

Mouse cursor at the pointed object‟s depthCan not use the HW cursor

Crosshair

Need to modify the projection to take into account depth of po

Draw the UI as a 2D element at the depth of the scene where poi

Compute the depth from the graphics engine (can also be compubuffer)

S E S S I O N S

What needed fixing#3 - Issues with render targets



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

g Render targets may or may not be sampled different

eye depending on context

Driver currently relies on heuristics based on render

to determine what to do

Special registry keys give hints to heuristics for prop

per application

Heuristics are in flux, so currently must work closelyto determine the proper registry setup

S E S S I O N S

Render targets - bloom



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

D

S E S S I O N S

Render targets



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

Bloom render target

This is purely a 2D post-process with no depth

Hence each eye must sample the same point – bloom rende

not be doubled to be different for each eye

The default heuristic assumes render targets smaller

swap chain are stereo – incorrect for this case

Fixed by NVIDIA in the application profile

D

S E S S I O N S

Render targets - Water



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

D

S E S S I O N S

Render targets



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

Water reflection render target

Reflection angle varies per eye

In this case, render target must be doubled so that reflect

each eye

Again, fixed by the NVIDIA voodoo registry key

D

S E S S I O N S

Render targets



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

Terrain blending render target

Our terrain textures are pre-composited from several laye

rendering the layers to off-screen render targets at run-tim

Another purely 2D operation that has no relationship with

scene viewpoint

Heuristics assume square render targets are stereo

We got stuck – registry voodoo state could not be changed

breaking previous fixes What to do?

Use a 2048x2047 render target when using stereo

D

S E S S I O

N S

Render targets



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

Future direction

Definitely the most counter-intuitive part of fixing up issue

For now, you can at least be aware of the problem and coo

NVIDIA

In the future, we hope a standard API will be available to g

direct instructions to the driver on which render targets ne

doubled for stereoscopic support

D

S E S S I O

N S

What needed fixing#4 - Portraits



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

Different 3D scenes rendered in the same frame usin

scales

Portrait viewport of selected character

Since the scale is different for each scene, a differe

convergence must be used to render each scene

We change the convergence mid frame to render the

panel

D

S E S S I O

N S

What needed fixing but didn’t becaNVIDIA wouldn’t bribe us



G D C

2 0

1 1

N V I D I A

S P O

N S O R E D

Ok, we just ran out of time

StarCraft II‟s “story mode” presented other challeng

didn‟t have time to deal with

Currently StarCraft II just turns off stereo when ente

mode as there are still remaining issues with this mo

D

S E S S I O

N S

Deferred lighting



G D C 2 0

1 1

N V I D I A

S P O

N S O R E D

D

S E S S I O

N S

Deferred lighting



G D C 2 0

1 1

N V I D I A

S P O

N S O R E

Deferred rendering must reconstruct a world position when doing the

This reconstructed position must be offset for each eye to be correct

This is the “unprojection” problem mentioned by Samuel

Must use special stereo texture that the driver will sample differently

Texture is split in two halves – driver will sample left half for left

versa

Use this texture to find which eye is being sampled in the pixel sh

Left half of texture contains -1, right half contains 1

See NVIDIA developer website for details

D

S E S S I O

N S

Points of focus within cinematic sc

Our story mode has a wide variance in the depth ran



G D C 2 0

1 1

N V I D I A

S P O

N S O R E Our story mode has a wide variance in the depth ran

shots (micro and macro shots) along with varying poi

Accordingly, ideally each camera shot would need a convergence values for best effect

Some heuristic sampling of the depth of the scene co

to dynamically find a convergence value

Convergence values must be set so that storytelling

focus within the scene are at natural convergence po More art than science

The best stereoscopic games are likely to have artists manually

convergence value for each shot

E D

S E S S I O

N S

Conclusion

It



G D C 2 0

1 1

N V I D I A

S P O

N S O R E It

Doesn‟t

Suck.

E D

S E S S I O

N S

Game Demo



G D C 2 0

1 1

N V I D I A

S P O

N S O R E

E D

S E S S I O

N S



G D C 2 0

1 1

N V I D I A

S P O

N S O R E

Presentation will be available after the show at http://developer.nvidia.coPing us for any question at [email protected]

Questions

http://developer.nvidia.com/


mailto:[email protected]

mailto:[email protected]



Documents

Stereos Copy