Information Visualization

and

Immersive Interfaces

CSCI 6174: Open Problems in CS

Fall 2013

Richard Fowler

An example

• Hans Rosling's 200 Countries, 200 Years, 4 Minutes • http://www.youtube.com/watch?v=jbkSRLYSojo

To visualize is …

• I see what you mean …

– so, visualization can be considered not just a visual process, but a cognitive (thought) process as well

• …. and a very large part of human brain taken up with visual system

– and that part of brain is useful beyond “simply” getting an

image of the world

Visualization is …

• Visualize:– “To form a mental image or vision of …”– “To imagine or remember as if actually seeing …”– Firmly embedded in language, if you see what I mean

• (Computer-based) Visualization:– “The use of computer-supported, interactive, visual

representations of data to amplify cognition”• Cognition is the acquisition or use of knowledge• Card, Mackinlay Shneiderman ’98

– Scientific Visualization: physical

– Information Visualization: abstract

Visualization is not New

• Cave guys, prehistory, hunting

• Directions and maps

• Science and graphs– e.g, Boyle: p = vt

• … but, computer based visualization is new– Allows interactive exploration and can be embedded

in the work flow– … and the systematic delineation of the design space

of (especially information) visualization systems is growing nonlinearly

Visualization and Insight

• “Computing is about insight, not numbers”– Richard Hamming, 1969– And a lot of people knew that already

• Likewise, purpose of visualization is insight, not pictures– “An information visualization is a visual user

interface to information with the goal of providing insight.”, (Spence, in North)

• Goals of insight– Discovery– Explanation– Decision making

“... insight, not numbers”

• Numbers – states, %college, income:State % college degree income State % college degree income

“... insight, not numbers”

• Insights:– What state has highest income?

– What is relation between education and income?

– Any outliers?

State % college degree income State % college degree income

“… insight, not numbers”

• Insights:– What state has highest income?,

– What is relation between education and income?,

– Any outliers?

Not about Useless Visual Stuff - Clutter

• Here, “3d” adds nothing– (at best)

Detrimental, Useless Stuff

• USA Today• So, what information is this

trying to convey?

An Example, Challenger Shuttle

• Presented to decision makers prior to launch– To launch or not

– Temp in 30’s

• “Chart junk”

• Finding form of visual representation is important– cf. “Many Eyes”

An Example, Challenger Shuttle

• With right visualization, insight (pattern) is obvious– Plot o-ring damage vs. temperature

Insight, through Visualization …

• Some examples ….

Insight, through Visualization …

• What accounts for US deficit?

– Economic recovery measures– TARP, Fannie, and Freddie

“bailouts”– Wars in Iraq and Afghanistan– Bush-era tax cuts– Economic downturn– Other (longer standing) things

For what it’s worth …

A “Taxonomy” of Visualization TopicsCard, Mackinlay, and Schneiderman

SpacePhysical Data1D, 2D, 3DMultiple Dimensions, >3TreesNetworks

InteractionDynamic QueriesInteractive AnalysisOverview + Detail

Focus + ContextFisheye ViewsBifocal LensDistorted ViewsAlternate Geometry

Data Mapping: TextText in 1DText in 2DText in 3DText in 3D + Time

Higher-Level VisualizationInfoSphereWorkspacesVisual Objects

1D - Linear DataSoftware Visualization

2D - Map DataPresidential votes 2008 on Map

3D World Data – Scientific Visualizatione.g., Molecule Modeling – Augmented Physical Representation

Multiple Dimensions > 3What does visual representation for > 3 dimensions look like?

• “Straightforward” 1, 2, 3 dimensional representations– E.g., time and

concrete

• Can extend to more challenging n-dimensional representations– Which is at core of

visualization challenges

• E.g., Feiner et al., “Worlds within Worlds”

Temporal DataBaby names across years

Trees, Networks, and Graphs

• Connections between /among individual entities

• Most generally, a graph is a set edges connected by a set of vertices– G = V(e)– “Most general” data

structure

• Graph layout and display an area of iv

• Trees, as data structure, occur … a lot– E.g., Cone trees

Routes of the Internet, 1/15/05

• Earlier snapshot in permanent collection of NY Museum of Modern Art

Abstract, Non-physical, Network

• E.g., concept map– Graph of

“conceptual” information

• What’s this one?

• From Tim Berners-Lee’s proposal to CERN for what is now called w w w, March 1989

• Manual “graph drawing”

http://www.nic.funet.fi/index/FUNET/history/internet/w3c/proposal.html

Text and Document Collection Structure

• Derivation of relationships upon which display is to be based a challenge

• E.g., Wise et al.

Overview Strategies

• Typically useful, or critical, to have “feel” for all data– Then, system allows closer inspection in “context” of all data– Overview + detail, focus + context

• Known from the outset of visualization– Bifocal Lens

• Database navigation: An Office Environment for the Professional by R. Spence and M. Apperley

• Shneiderman mantra– “overview first, zoom and filter, details on demand”

Focus+Context: Fisheye Views, 1

• Detail + Overview – Keep focus, while remaining aware of context

• Fisheye views– Physical, of course– Also, distance function (based on relevance)– Given a target item (focus)– Less relevant items dropped from display– Classic cover

• New Yorker’s idea of the world

Focus+Context: Fisheye Views, 2• Detail + Overview

– Keep focus while remaining aware of context

• Fisheye views– Physical, of course– Also, a distance function (based on relevance)– Given a target item (focus)– Less relevant items dropped from display – Or, are just physically smaller – distortion

Focus + Context – Spatial Distortion

• Selectively reduce complexity as:

f(user’s viewpoint)

• Spatial distortion– Project network on

distorted space

• Viewing “lens”

Focus + Context – Spatial Distortion

• Selectively reduce complexity as:

f(user’s viewpoint)

• Spatial distortion– Project network on

distorted space

• Viewing “lens”

• Seamless transition

Focus + Context – Hyperbolic View

• Again, selectively reduce complexity as f(user’s viewpoint)• As with distorted lens, space is non-linear, here, hyperbolic

• Smooth change during interaction

3-d Hyperbolic Tree of Web Sites

• Using Prefuse

Visualization Pipeline:Mapping Data to Visual Form, cf. North paper

• Visualizations: – “adjustable mappings from data to visual form to human perceiver”

• Series of data transformations– Multiple chained transformations– Human adjust the transformation

• Entire pipeline comprises an information visualization

RawInformation

VisualFormDataset Views

User - Task

DataTransformations

VisualMappings

ViewTransformations

F F -1

Interaction

VisualPerception

Visualization Stages

• Data transformations:– Map raw data (idiosynchratic form) into data tables (relational descriptions

including metatags)

• Visual Mappings:– Transform data tables into visual structures that combine spatial substrates,

marks, and graphical properties

• View Transformations:– Create views of the Visual Structures by specifying graphical parameters

such as position, scaling, and clipping

RawInformation

VisualFormDataset Views

User - Task

DataTransformations

VisualMappings

ViewTransformations

F F -1

Interaction

VisualPerception

Information Structure

• Visual mapping is starting point for visualization design

• Includes identifying underlying structure in data, and for display– Tabular structure– Spatial and temporal structure– Trees, networks, and graphs– Text and document collection structure– Combining multiple strategies

• Impacts how user thinks about problem - Mental model

RawInformation

VisualFormDataset Views

User - Task

DataTransformations

VisualMappings

ViewTransformations

F F -1

Interaction

VisualPerception

A Recent (2012) System

A Recent (2012) System

• Information visualization for task of document retrieval

• Document collection is organized as clusters (visually represented)– User views and interacts with the cluster representations

• IV Interaction:

– Overview– Flock-based clustering

– Zoom, spatially

– Filter– Text match, query– Markers

– Details on demand– Document itself

– Tunable

Swarm (Flock) based Clusteringfor Information Visualization and Retrieval

• Reynold’s technique for animation of flocks of birds– Computational model

of group motion

• Particle system– Standard part of many

toolkits

• http://www.youtube.com/watch?v=-8KZBGAAM30

• Agent-based approach– Individual elements each apply rules for motion, as particle systems, etc.

Reynolds (1987). Flocks, herds, and schools: A distributed behavioral model,Computer Graphics, 21(4), 25-34.

Flock-based Movement

• Individual agents utilize only three rules– Agents adjust their directions of movement and velocities - steer – Form single group

• Movement perceptually similar to flock of birds, other social biological group

– Alignment: Steer towards average direction of movement of nearby agents– Separation: Steer to avoid being too close to nearby agents– Cohesion: Steer to move toward the center position of nearby agents

• Rules are added for obstacles, …, edge of screen, …

Flock-based Clustering: Multiple Groups – Bird Species, Document Clusters, …

• Rules described so far form a single group– As, e.g., with a single species of birds

• Can add additional rules that consider similarity or a priori categorization among agents to form multiple groups

• “Multi-species flocking”– Birds of same species form groups and moved together– Agents (birds) described by feature vector, e.g., length of beak, color, size

• Flock-based document clustering– Vector Space Model– Similarity of documents determined by weighted number of terms in common– Essentially, a biologically (or animation) inspired “Iterative clustering technique”

System

• Information visualization for task of document retrieval

– Overview– Flock-based clustering

– Zoom, spatially

– Filter– Text match, query– Markers

– Details on demand– Document itself

– Tunable

System’s Flock-based Clustering

• E.g., 500 documents– Clusters apparent

• Uses standard metrics of document similarity with velocity vector change based on similarity

• Also considers thresholds of similarity and dissimilarity – Provides tunable parameters for arranging spatial display (Cui et al., 2006)

– Now,

Identifying Clusters of InterestTerm Match

• Term match to mark documents to guide exploration– Matching documents’

color is changed

– Multiple term matches

Identifying Clusters of InterestTerm Match

• Term match to mark documents to guide exploration– Matching documents’

color is changed

– Multiple term matches

Identifying Clusters of InterestTerm Match

• Term match to mark documents to guide exploration– Matching documents’

color is changed

– Multiple term matches

Identifying Clusters of InterestTerm Match

• Term match to mark documents to guide exploration– Matching documents’

color is changed

– Multiple term matches

Identifying Clusters of InterestQuery Match

• Term match to mark documents to guide exploration– Matching documents’

color is changed

– Multiple term matches

• Query match to mark documents

Viewing Individual Documents

• Zoom, then select individual element– Title is displayed

Viewing Individual Documents

• Select title to display individual document text– Also, can select from list retrieved in query

Interactively Adjusting Clusters

• In practice, many clustering techniques make use of “tuning”

• System allows user to change parameters interactively

– Weights for “basic” rules and other motion parameters

• Avoidance, cohesion, alignment• Velocity minimum, maximum• Limit of steering

– Range for consideration– Thresholds for similarity and dissimilarity

Stereoscopic Display

• Perception / extraction of structure facilitated by stereoscopic display– Reveals z component of

3d arrangement

– Head-tracked stereo.

• Structure from motion

• Challenge of practical stereoscopy in context of task performance Two monitor desktop

Stereoscopic Display

Multi-screen display

• Perception / extraction of structure facilitated by stereoscopic display– Reveals z component of

3d arrangement

– Head-tracked stereo.

• Structure from motion

• Challenge of practical stereoscopy in context of task performance

Immersive Interfaces, Prescence, …

Introduction

• The “best” interfaces, and all systems, typically find their task utility through engagement appropriate for the task

– This idea is at the core of arguments for the use of direct manipulation interfaces

• All of the following are interrelated:– Immersion, engagement, presence, virtual reality– 3D display and interaction devices

• In field of CS and HCI: – “spatial interfaces”, “immersive interfaces”

Examples of Immersive Interfaces

Tiled display wall

Head mounted display

Surround screen projection

Spherical projection

Immersion, “Virtual Reality”, and Virtual Environments

• Immersive interfaces– High sensory immersion – visual, auditory, haptic, proprioceptive

• “Virtual reality”, or, virtual environments– “Virtual reality is a technology that is used to generate a simulated environment in

digital form… users are immersed in a totally virtual world.”– Working definition – an immersive interactive system

• In context of “virtual reality”, immersion = spatial immersion– Different literatures use word in different ways

• Note: “Immersion” (and engagement and presence) is a continuum

– Text ... Visual and 3d .. Stereo ... HMD… “jacked in”– Cyberspace

• Term coined by Gibson in Neuromancer, 1984• … and in the 21st century, the Matrix

Immersion and Virtual Reality

• “The mind has a strong desire to believe that the world it perceives is real” – Jaron Lanier, VR “pioneer”, … among others

• For example, “illusion” (perception) of depth (for spatial immersion)• Stereo parallax• Head motion parallax• Object motion parallax• Texture scale

• Interaction: grab and move an object

• Proprioceptive cues: – When you reach out and see a hand where you believe your hand

should be, you accept the hand as your own

• Often user will accept what see asn “real” even if graphics poor

• Constellation of cues

optional

Components for Perception of 3D for Spatial Immersion, Cutting, 1996

• Perception of 3 (vs. 2) dimensions strongest element of spatial immersion

– Perception of 3d from depth cues

• See figure

• Other elements– Integration important– Visual display types– Stereoscopic display – Head position sensing – Hand-position sensing – Force feedback – Sound input and

output – Other sensations

Presence “The Aesthetic Impression of 3D Space”

• Sense of presence – Actually present in the world– Vividly 3d– Sense of being there– Holodeck …– In game literature, often called “immersion”, which is fine, but confusing

for us

• Presence has to do with engagement, as well as visual information– E.g., one can be “in the world”, when reading– Here, one sees, or visualizes, the world

Immersive and 3D Interfaces

• Degree of immersion– conventional desktop screen

• Some 3D displays:– Data walls– Workbenches– Hemispherical display– Head-mounted displays– Arm-mounted displays– Surround screen displays - CAVE– Virtual retinal display– Autostereoscopic displays

• But first, … about stereoscopic display– A “compelling” 3D depth cue

3D Interfaces, Stereopsis“Discovery” of Stereopsis, 1838

• Charles Wheatstone– Prolific scientist, Wheatstone bridge

• “… the mind perceives an object of three-dimensions by means of the two dissimilar pictures projected by it on the two retinae…”, 1838

• Contributions to the Physiology of Vision.—Part the First. On some remarkable, and hitherto unobserved, Phenomena of Binocular Vision.

• "Philosophical Transactions" of the Royal Society of London, Vol. 128, pp. 371 – 394, 1838.

• http://www.stereoscopy.com/library/wheatstone-paper1838.html

What the left and right eye see (retinal image)

Getting Different Images to Each Eye Wheatstone’s Stereoscope

• “The stereoscope is represented by figs. 8. and 9; the former being a front view, and the latter a plan of the instrument. A A' are two plane mirrors, about four inches square, inserted in frames, …”

• Below, modern mirror stereoscope using computer monitors

Getting Different Images to Each Eyethese days

• Commodity televisions– 120 hz display rate– Alternately, left then right

eye image– Active lcd glasses,

alternately block left eye, then right eye image

• lcd lens is alternately transparent, then opague

• 60 hz left eye, • 60 hz right eye

• Theaters– Polarized glasses

Getting Different Images to Each Eye Sutherland’s 1960’s equipment

• “Ultimate display”, 1965

• Sword of Damocles – 1st HMD– Actual camera-like shutters– Actual camera-like metal shutters

Immersive 3D Displays(head mounted displays)

• Head position is tracked, and display updated for user view– As if “in a virtual (cg) world”

Head mounted display, NASA Ames, Fisher et al., mid-1980’s

Also, Virtual and Augmented Reality

• Augmented reality shows real world with an overlay of additional overlay

• Knowlton (1975)

• Partially-silvered mirror over keyboard

• Programmable labels

• Tactile feedback

Also, Virtual and Augmented Reality, 2

• Enables users to see real world with an overlay of additional interaction– Situational awareness

• See through glasses

• Typically, add text+images to real world

• Very sensitive to tracking

• These days, “mobile apps” are appearing

Immersive 3D DisplaysSurround Screen Systems

• Essentially same multi-surface display and interaction paradigm used today as 1992

• Orders of magnitude less hardware and software cost

Immersive 3D Displays Surround Screen Systems

• Space with walls and/or floor formed by rear projection screens

– Stereoscopic – Head and position tracking

• So can change 3D view (projection) as user moves, or moves head

• Projects 3D scenes for viewer’s point of view on walls

– Walls “vanish”, • User perceives full 3D scene

– View only correct for one viewer

• Visual immersion– Field of view is 100% possible

Immersive 3D Displays Surround Screen Systems

• Typical size: 10’ x 10’ x 10’

• 2 or 3 walls are rear projection screens

– To save space, mirrors often used to “fold” projection

– Floor projected from above

UTPA Immersive Systems Lab~Summer, 2013 … or so

Proj.

Proj. Proj.

27’13’

Security area

21’

6’

TV

stor

age

~8’

CAVE

Ph

ysio

log

ica

lM

easu

rem

ent

E

qu

ipm

ent

Computers

Front Projection Screen

Proj.

Develo

pmen

t

Current Immersive Display Projects

• Information visualization in immersive environments

• Role of stereoscopy, display size, and narrative on presence

• Role of personality factors in presence

• Gallery installation

• Getting things to work …– Clusters, projectors, software

Questions?