Space Perception and Display of Data in Space

Immersive Interfaces

Ware Chapter 7

University of Texas – Pan AmericanCSCI 6361, Spring 2014

Introduction

• The “best” visualization systems, and all systems, typically find their task utility through engagement appropriate for the task

– Presence• The sense of being in a place – reading works great

– Immersion• Here, physical immersion, i.e., range of senses engaged –

vision, touch, …, smell

• All of the following are interrelated:– Immersion, engagement, presence, virtual environment, virtual

reality, 3D display and interaction devices

• We’ll expand on Ware’s (narrow) discussion of presence

Examples of Immersive Interfaces

Tiled display wall

Head mounted display

Surround screen projection

Spherical projection

Immersion, “Virtual Reality”, and Virtual Environments

• Immersive interfaces– High sensory immersion – visual, auditory, haptic, proprioceptive

• “Virtual reality”, or, virtual environments– “Virtual reality is a technology that is used to generate a simulated environment

in digital form... Using the equipment, users are immersed in a totally virtual world.”

– Working definition – an immersive interactive system

• In context of “virtual reality”, immersion usually = spatial immersion

• Note: “Immersion” (and engagement and presence) is a continuum

– Text ... Visual and 3d .. Stereo ... HMD… “jacked in”– Cyberspace

• Term coined by Gibson in Neuromancer• … and in the 21st century, the Matrix

Terms and such …

• What is “virtual reality”, as used in visualization?

• Narrow:– immersive environment with head tracking, headmounted

display, glove or wand

• Broad:– interactive computer graphics

• Working definition:– an immersive interactive system

• Will see several examples …

Immersion and Virtual Reality

• “The mind has a strong desire to believe that the world it perceives is real” – Jaron Lanier, among others

• For example, “illusion” (perception) of depth (for spatial immersion)• Stereo parallax• Head motion parallax• Object motion parallax• Texture scale

• Interaction: grab and move an object

• Proprioceptive cues: – when you reach out and see a hand where you believe your hand to be,

you accept the hand as your own

• Often you will accept what you see as “real” even if graphics poor

• Constellation of cues

Presence “The Aesthetic Impression of 3D Space”

• Sense of presence – Vividly 3d– Actually present in the world– Sense of being there– Holodeck …

• Presence has to do as much with engagement, as visual information– E.g., one can be “in the world”, when reading– Here, one sees, or visualizes, the world

• 3D depth cues are those elements that enhance feeling of 3 (vs. 2) dimensions in a display, – From occlusion to stereoscopic display

Presence “The Aesthetic Impression of 3D Space”

• Immersive interfaces– term used to describe interfaces/devices which lead toward immersion

(sense of presence, engagement) in the virtual environment presented on the display

• Virtual reality interfaces– term used similarly to immersive interfaces

• Degree of immersion– conventional desktop screen– fishtank virtual reality (semi-immersive workbench)– immersive virtual reality– augmented reality with video or optical blending– … number of cues …

Components of Immersion - Ware

• 3 dimensions– Strongest– Perception of 3d from

depth cues• See figure

• Other elements– Integration important– Visual display types– Stereoscopic display – Head position sensing – Hand-position sensing – Force feedback – Sound input and output – Other sensations

Pictorial Depth Cues

• 3D depth cues – Static / pictorial vs. dynamic– Monocular vs. binocular– Oculomotor

• Static monocular cues– Occlusion– Relative size– Linear perspective– Texture gradient– Aerial perspective– Shading– Relative height

Sutherland’s Sketchpad

• Ivan Sutherland– “Pioneer” of … lots of things– Visualization– Graphics– Interaction– Still around

• Evans and Sutherland graphics

• First truly interactive graphics system, Sketchpad

– A fairly sophisticated “paint” (or drawing) program

• MIT, Ivan Sutherland’s 1963 Ph.D. thesis

– “Sketchpad, A Man-Machine Graphical Communication System”

• Available: www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf

• Video: www.youtube.com/watch?v=mOZqRJzE8xg

• Among most important works in computer science

Ivan Sutherland using Sketchpad in 1963 CRT monitor, light pen and function-key panel

Ivan Sutherland’s Sketchpad, 1963

• Regarded as the first to implement much of what called “visualization”, “immersion”, and “virtual reality” (not to mention cg)

• Some quotes:

– ….. If the task of the display is to serve as a looking-glass into the mathematical wonderland constructed in computer memory, it should serve as many senses as possible.

– ….. By working with such displays of mathematical phenomena we can learn to know them as well as we know our own natural world. Such knowledge is the major promise of computer displays.

– ….. The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.

Sutherland’s 1960’s equipment

• Ultimate display, 1965

• Sword of Damocles – 1st HMD– Actual camera-like metal shutters

Less Profound maybe, but Fun – Sensorama, 1965

• Morton Heilig– cinematographer / director of

documentaries

• Motorcycle simulator - all senses– visual (city scenes)– sound (engine, city sounds)– vibration (engine)– smell (exhaust, food)

• Not a big commercial success, but “immersive”

from Virtual Reality Technology, Burdea & Coiffet

USAF Super Cockpit, 1985

• Wright Patterson Air Force Base

• Visual, auditory, tactile

• Head, eye, speech, and hand input

• Designed to deal with problem of pilot information overload– Flight controls and tasks too

complicated

• Research only– big system, not safe for ejecting

Immersive and 3D Interfaces

• Teleoperation

• Virtual and augmented reality

• Immersion and VR – contribution of components …

• Survey of 3D displays– Surround screen displays - CAVE– Input devices - Data glove– Data walls– Workbenches– Hemispherical display– Head-mounted displays– Arm-mounted displays– Virtual retinal display– Autostereoscopic displays

Visual Displays for VEs(Bowman)

• Types:– Standard monitor (mono/stereo)– Head-mounted/head-referenced– Projected (usually stereo)

• single-screen• multiple, surrounding screens

– Retinal display– Volumetric displays

• Characteristics of visual displays– Field of regard (FOR), field of view (FOV)– Brightness, contrast ratio– Resolution (two definitions)– Screen geometry– Light transfer– Refresh rate– Ergonomics

Remote (or tele-) Operation - Ware

• Combines:– direct manipulation in personal

computers– process control in complex

environments

• Physical operation is remote– Submarines, rovers

• Complicating factors in architecture of remote environments:

– Time delays • transmission delays

• operation delays – Incomplete feedback – Feedback from multiple sources – Unanticipated interferences

Remote (or tele-) Operation

• Combines:– direct manipulation in personal

computers– process control in complex

environments

• Physical operation is remote– Submarines, rovers, operating

rooms

• Complicating factors in architecture of remote environments:

– Time delays • transmission delays

• operation delays – Incomplete feedback – Feedback from multiple sources – Unanticipated interferences

Virtual and Augmented Reality

• Augmented reality shows real world with an overlay of additional overlay

• Knowlton (1975)

• Partially-silvered mirror over keyboard

• Programmable labels

• Tactile feedback

Virtual and Augmented Reality

• Augmented reality shows real world with an overlay of additional overlay

• Knowlton (1975)– Partially-silvered mirror over

keyboard– Programmable labels– Tactile feedback

• Ware, ch. 2

Augmented Reality

• Enables users to see real world with an overlay of additional interaction

– Situational awareness

• See through glasses– E.g., Google Glass

• Typically, add text+images to real world

• Very sensitive to head tracking, when used – the real challenge

Augmented RealityGuidelines - More examples

Ware, Ch 2, p. 45

Augmented RealityGuidelines - More examples

Augmented RealityGuidelines - More examples

“Arm Coupled” Display with Boom

• Variation

“Fish Tank VR” / Immersion

• Stereoscopic viewing– Depth of image appears to

be from about 12” behind and 6” in front of display

– Hence, a fish tank

• Often with “head coupled display”– Position of head is tracked– Image changed to appear

as if “looking around”

LCD Shutter Glasses

• >120 hz monitor refresh• Different images on odd and even• LCD “Shutters” open and close for left and right eyes

– (used to be metal shutters!)

• From Stereographics web site:– Weight: 3.3 oz. (93 grams)– Shutters: Liquid Crystal– Field Rate: From 80 to 160 fields per second– Transmittance: 32% typical– Dynamic Range: 1500:1 typical– Battery Life: >250 hours of continuous operation– Battery Type: Two 3V lithium/manganese dioxide– Emitter

• Designed for PC/Unix desktop workstations using 3 pin mini-DIN connectors.

• Emitter for workstations allowing control over the IR spread (for multi-user environments).

• Connectors: 3-pin mini-DIN

Right!

Workbenches

• Rear stereo projection – fishtank view volume

• UNC NanoManipulator– Below with force feedback to “feel” carbon nanotubes with Atomic Force Microscope

Workbenches

• Immersadesk is best know

Surround screen displays - CAVE

• A room with walls and/or floor formed by rear projection screens– Head tracking– Stereo– Light scattering

problems

• Visual immersion– Field of view is

100% possible, ~200 degrees

Surround screen displays - CAVE

• Typical size: 10’ x 10’ x 10’ room

• 2 or 3 walls are rear projection screens– Floor is projected from above

• User is – tracked (usually magnetically)– He/she also wears stereo shutter goggles…– Carries a wand to manipulate

• Projects 3D scenes for viewer’s point of view on walls

– Walls vanish, user perceives a full 3D scene

– So, view is only correct for that viewer

• Turning head doesn’t necessitate redraw, so latency problems are reduced

• Cost is fairly high

UTPA Immersive Systems Lab~Fall, 2014

Proj.

Proj. Proj.

27’13’

Security area

21’

6’

TV

stor

age

~8’

CAVE

Ph

ysio

log

ica

lM

easu

rem

ent

E

qu

ipm

ent

Computers

Front Projection Screen

Proj.

Develo

pmen

t

Head Mounted Displays

• HMDs– Relatively high field of view (fov)– 90o direct FOV, 140o corneal FOV

• LEEP Optics (1975)– Large Expanse, Extra Perspective (LEEP)– Eric Howlett (Pop-Optix Labs)– Originally for 3D still photo viewing– Reported great realism for still images– Lenses correct for camera distortion

• Display optics matched to camera optics• Often uncorrected distortion for CG images

• And, more current technology:

Head Mounted Displays

• HMDs– Relatively high field of view (fov), ~ 140 x 60

• NASA Ames HMD (1981-1984)

• McGreevy and Humphries– First implemented immersive HMDs– LCD “Watchman” displays

• NASA Ames VIEW or VIVID (1985)

• Virtual Interface Environment Workstation– Polhemus tracker, LEEP-based HMD, 3D audio,

Crystal River’s Convolvotron, Gesture recognition w/ VPL DataGlove, BOOM-mounted CRT (Sterling Software), Remote Camera (Fake Space)

HMDs now

Oculus Rift

• Company founder, Palmer Luckey, well known in “populist vr” community

– USC Institute for Creative Arts– E.g., FOV2GO viewer

• http://projects.ict.usc.edu/mxr/diy/fov2go-viewer/

• Kickstarter project raised $91m

• Acquired by Facebook, $2b ($1.6b stock)

• “Developer version” available

• Accelerometer, gyroscope for tracking– Excellent latency and cost

• Industry support excellent– Unreal, Unity, etc.

Oculus Rift

• ~1mp per eye panels– Short latency

• FOV ~110o x 90o - which is a lot for this class

• Not 100% overlap, so greater fov– Recall, stereoscopic vision best at fovea

• Tracking speed very good for cost– Latency persistent problem in all head tracking

• Two lenses “correct” pin cushion of panels

• Lens to eye distance adjustable

• Weight ~13 ounces

Many Others cf. Virtual Realities, Ltd.

Many Others cf. Virtual Realities, Ltd.

http://www.vrealities.com/proviews035.html

DISPLAY Image Source Type: Single Full-color AMOLED (800x3 pixels) x 600 linesBrightness: 25 foot lamberts Contrast: >50:1

OPTICAL Field of View: 32° Horizontal x24° Vertical (40° Diagonal) @ SVGA resolution (800x600)26° Horiztonal x 19° Vertical (32° Diagonal) @ VGA resolution (640x480)Transmission: Non see-throughOptics: Plastic aspherical lensEye Relief: Eyeglasses compatible, >25 mmExit Pupil: Non-pupil forming

MECHANICAL Mounting: Standard: Clip on to Mich helmet* (Display module); Clip on to belt (Disply Controller)Weight: Display Module: 67 grams (w/out mount); Helmet Mount: 79 grams; Display Controller: 200 grams

DIMENSIONS Display Module (less mount): Height: 0.7"; Width: 1.7"; Depth: 1.2">br> Electronics Box: Height: 3.0"; Width: 2.8"; Depth: 1.2" Cable: Length: 1.2 meters

INTERFACE Plugable Display Module: Display Module includes 51" cable (Interface cable available separately) with water-resistant connectorDisplay Controller: Lemo Connector

ELECTRONIC Analog Video Rates: 800 x600 (SVGA 56 Hz to 85 Hz), 640x480 (VGA 60 Hz to 85 Hz)Analog Video Inputs: NTSC/RS-170 (A); CVBS and Y/C; R,G,B,HS,VSPower: System (VIN 6-24 VDC, 2.5 W nominal)Controls: RS-232 brightness and channel selection

DESIGNED TO MEET THE FOLLOWING PARAMETERS Temperature: Operating: -32° to +55°C; Storage: -32°C to +71°CHumidity: Six 48-hour cycles, 20°C to 55°C, 95% RHSalt Fog: Four 24-hour cycles (two wet, two dry)Vibration: Random vibration, 6 axis, 5 Hz to 2500Hz, up to 40 gsImmersion: Immersion in 1 meter of water for 2 hours

VR Price: $9,995.00

Add a Head Tracker! CLICK HERE Integrated Microphone with Voice Recognition Software Add $125.00

Wanting another payment option? Questions on products? Need help? CLICK HERE

http://www.vrealities.com/addvisor150.html

• Resolution: 1280 x 1024 - SXGAField of View: 60 Degrees DiagonalImage Size: 76" at 13' Color Depth: 24 Bit InputIPD Adjustments: None RequiredEye Relief: Eyeglass compatibleConvergence: 7'10", 100% Overlap, TBRAudio: OptionalWeight: < 1 kgAdjusts to Fit all IndividualsControl Features: On / Off, Volume Control

• The AddVisor 150 personal display system is designed for the high-end professional market for Head-Mounted Displays with AR and VR capability. The AddVisor 150 is a transparent high-resolution Head-Mounted Display. The image is generated on two independent full colour 1280x1024 pixels (SXGA) Liquid Crystal on Silicon microdisplays. A light weight solution for the most demanding of applications the AddVisor 150 is recognized for excellent image quality, high brightness and contrast. The AddVisor 150 is ideal for direct view mono- and stereo (3D) viewing.

• The image can be shown superimposed on the environment with up to 35 % see-through or fully immersed. The AddVisor 150 is designed for a 46 degree diagonal 100% overlap field of view. A 50% overlap can also be used, giving a 54 degree horizontal or 60 degree diagonal field of view. It features a patented optical design that combines a wide field of view with high transparency see-through. Using a patent pending head fitting system with easy adjustments, low weight and eyeglass compatibility, the AddVisor 150 provides hours of easy and comfortable viewing.

http://www.cwonline.com/store/view_product.asp?Product=1151

• Resolution: 800x600 Field of View: 26 Degrees Diagonal Image Size: 76" at 13 Color Depth: 24 Bit Input IPD Adjustments: None Required Focus: 13 TBR Eye Relief: 25mm Exit Pupil: 17mmH x 6mmV Convergence: 710", 100% Overlap, TBR VGA / SVGA / XVGA Input: Scaled to SVGA (800 x 600)

Refresh Rate: Flicker Free 100Hz Audio: Full Stereo Weight: 7 Ounces

• $799.00In Stock The new i-glasses PC models utilize LCOS micro-displays from Brillian Corporation with 800 x 600 resolution. These displays have been designed to achieve a much greater range of contrast and brightness capability. This results in an image substantially more vibrant and vivid. The new models will also reach their optimal operating temperature much faster and offer a very consistent contrast and color fidelity over the entire display. With built on integrated headphones. The Ultimate Portable PC Video System. Available in 3D model too! 

Hemispherical display

• As with mirror stereoscope, high resolution possible

Display Walls – Rear Projection

• Widespread use– Literature of

practical use

• Can use commodity projectors

– E.g., with 27 as 3x9, with 1kx1k each gives

– 9,000 x 3,000

Display Walls – Front Projection

• “Home theater” projectors

– 4mp, 2012

• TACC at right– Texas Advanced

Computing Center• UT - Austin

– 9mp, 4096x2160– 20’ x 11’

• … to … IMAX(ish)

Tiled Display Walls

• Commodity monitors

• For some, bezels disrupt, though progress

– 2011, ¼” bezel, 54” diag., $5k

TACC, 46”, interactive

TACC, ~60’ x 8’, 75 Dell 30’ 4mp monitors, 307mp

3D Televisions, ~2012

• 92” 3D DLP, ~$5k– LCD, etc. more

• Now, 2mp widely available– 1920 x 1080

• 4mp introduced 2011– Lacks commercial content

• Larger is better, stereo is good

3D Televisions, >2014

• 196” = ~2 x 92”

• ~ 8’ x 14’ … or, 1 wall

3D Televisions, >2012

• 92”

• 196” = 1 wall

Virtual Retinal Display

• Eric Seibel, U. Washington Human Interface Technology Lab– http://www.hitl.washington.edu/research/vrd/– www.mvis.com (commercial version)

• Simple enough: shine a laser in your eye and modulate it real fast

• Potential for wearable very high resolution virtual or augmented reality

Virtual Retinal Display

• “Paint” all of retina for immersive display, or part for augmented

• Monochrome 1000x1000– Color 3 lights

Conclusion (still): 3D is better, but only it adds something

End?

Evaluation(Bowman, 2004)

• Case studies - Silva• Evaluate a new bodybased

interaction technique for the desktop game World of Warcraft

• Interviews• Think aloud• Critical incidents• Suggestions for improvement• Silva, M. and Bowman, D.

Body-based interaction for desktop games. Submitted to CHI 2009 Workin-Progress.

Which Visual Display to Use?(Bowman, 2004)

• Consider lists of pros and cons

• Consider depth cues supported

• Consider level of visual immersion

• Hhard question to answer empirically

• Instead of comparing actual displays, compare levels of immersion

Visual Display Types Comparison(Bowman, 2004)

Visual Display Types Comparison(Bowman, 2004)

End

• .