Of Spaces and Objects:Re-imagining Visualization

with Interactive Data Editing

Chris WeaverSchool of Computer Science

University of Oklahoma

weaver@cs.ou.edu

how does the visual representation of data affect how we can interact with it?

the visualization community is still lookingat the design of digital tools primarily in terms of

how can interaction allow us to express our ideas as data?

process

representationinteraction

we need to ask two big questions about interaction

A visualization in Google Earth of the correspondences of Benjamin Franklin (green), Voltaire,(orange), and Athanasius Kircher viewed over the Cassini Globe, created in 1790, globe image and georectification courtesy of David Rumsey.

first, a little ancient historybecause Michael asked me to include some!

What is Improvise?

it’s a desktop application for interactivelybuilding and browsing visualizations

it’s different because of howrichly interactive its visualizations can be,

and how multiple views of data allowanalysts to express complex queries

using only simple interactions

rich interaction enables design ofmore useful visual analysis tools

Live Design of Visual Tools for Data Exploration and Analysis

a full information visualization toolkitan architecture for coordinating multiple views

a declarative language for interactive query specificationan integrated development environment for building and browsing

Visualizations in Improvise

a coherent, shared, high-dimensional state space of interactive parameters

a graph of data record transformation statements with parameter references

a set of views that process queries, render results, and map local interactions into parameter value changes.

InputTransformation

Space

Naviga

tion

Selection

Objects

Display

VisualEncoding

Users

Interaction

Views

Query

Design

CoordinatedQuery Graph

are specifications ofhow interactive states determine visual forms and

how visual forms constrain changes to interactive states

Live Properties

1. A control modifies the value of a property in response to interaction.2. The property assigns the new value to its bound variable.3. The variable sends a change notification to all bound properties.4. Properties notify parent controls of the change. Controls update

themselves by accessing variable values (through their properties).

Imagine synchronizing all the clocks in your house by setting the time on any one of them.

Interactive Data Transformations

are called lexicalsare composed of expressions

can be variable valuesto form coordinated query graphs

Projections (encodings) map data attributes into visual attributes or view-specific glyphs

Filters map data attributes into booleans that indicate (in)visibility

Sorts map data attributes into comparators that determine pairwise ordering of records

Infos access and process data by grouping, joining, clustering, projecting, filtering, sorting, etc.

aggregated data set

aggregation function

Coordinated Queries

5. Variables also notify any lexical whose expressions refer to them.

6. Each lexical notifies any variables to which it is assigned as a value.

7. Variable change notifications propagate from properties to their parent controls.

8. When a view receives notification that a lexical has changed, it updates itself by processing the modified query as needed.

Coordination Patterns

Recipes for building visualizations using multiple views

Inspired by software design patterns [Gamma1994]

Narrower focus than infovis software design patterns [Heer2006]

Navigation — Manipulate points, ranges, and other spatial regions.

Selection — Distinguish arbitrary sets of data items.

Ordering — Visually prioritize data items.

Containment — Present additional information in existing views.

Mutation — Radically alter visual encoding in response to navigation.

Compound — Interconnect multiple views of multiple data sets through multiple coordinations to create complex interface behaviors.

evacuation scenarioa visualization of movements of RDIF-carrying

health care workers and visitors

Data Source: VAST 2008 Challenge, Evacuation Mini-Challenge (synthetic)Visualization Design: Chris Weaver and Anthony Robinson

migrant boat interdictionsa visualization of boat landings and interdictions

Data Source: VAST 2008 Challenge, Boats Mini-Challenge (synthetic)Visualization Design: Chris Weaver, Michael Stryker, Ian Turton

Cross-Dimensional Data Dissection

Multiple views support selection over sets of unique attribute values in multiple raw or derived data columns, across one or more tables.

Attributes are displayed in dimensionally-appropriate view(s) that supports a binary categorization of values by selection or navigation.

Users can rapidly toggle dependencies between pairs of views to pose complex drill-down set queries: effect (show or highlight) only those values in view B that co-occur in the data with the values selected in view A

Attributes are also displayed in a entity-relationship view (e.g., a graph) that shows co-occurrences between values.

Users can rapidly toggle visibility of attributes and attribute relationships to dissect the data by slicing in and across data columns

Analysts can form hypotheses and follow chains of evidence by successive selection/deselection and filtering of values.

methods for expressing sequences of multidimensional set queries by interactively

associating unique data values across multiple views

KEDS (Kansas Event Data System)newswire-extracted international events between state-level actors

Collaborator and Data Source: Phil Schrodt,University of Kansas Department of Political Science

Cross-Dimensional Data Dissection

Multiple views support selection over sets of unique attribute values in multiple raw or derived data columns, across one or more tables.

Attributes are displayed in dimensionally-appropriate view(s) that supports a binary categorization of values by selection or navigation.

Users can rapidly toggle dependencies between pairs of views to pose complex drill-down set queries: effect (show or highlight) only those values in view B that co-occur in the data with the values selected in view A

Attributes are also displayed in a entity-relationship view (e.g., a graph) that shows co-occurrences between values.

Users can rapidly toggle visibility of attributes and attribute relationships to dissect the data by slicing in and across data columns

Analysts can form hypotheses and follow chains of evidence by successive selection/deselection and filtering of values.

methods for expressing sequences of multidimensional set queries by interactively

associating unique data values across multiple views

Cinegraphhigh-dimensional drill-down into people, genres, awards, release

dates, and box office characteristics of mainstream movies

Data Sources: www.imdb.com and InfoVis 2007 Contest Co-Chairs

A visualization in Google Earth of the correspondences of Benjamin Franklin (green), Voltaire,(orange), and Athanasius Kircher viewed over the Cassini Globe, created in 1790, globe image and georectification courtesy of David Rumsey.

okay, that’s enough history

Movement

Space

Navigation

Selection

Objects

Display

Highlighting

Users

Interaction

Views

Query

Design

DataTransformation

Pipeline

Developers

the human-computer interaction loop of information visualization

➊expression

➋ ➌

➍

execution exhibition

evaluation

I think a lot about space and objectsin our physical world experience...

we can

see thingstouch them

move around

the things

take up spacehave appearancecan be refashioned

we imagine

Formsobjectsqualities

views

lookingselecting

navigating

items

layoutencoding

???

data

typesrecords

attributes

...and wonder, how do these properties correspondto our visualization world experience?

???

the things

can be refashioned

there are many ways to (re)fashion things in the physical world

move themturn them

stretch themtwist them

mold themflatten them

paint thempolish themscuff them

decorate them

cut them into pieces glue them together

the physical world is based on matter and dynamics

move themturn them

stretch themtwist them

mold themflatten them

paint thempolish themscuff them

decorate them

cut them into pieces glue them together

the visualization world is based on data

and transformation

apply spatial mapping

modify geometry

change color/textureadd marks/labels

compose graphics

the visualization world is based on data

and transformationtransformation produces additional, derived data

we usually treat the results as transient and throw them away (eventually)

that’s okay when transformation is automatic and reproducible

it’s not okay when transformation is manual and not readily reproducible

when a user manipulates items interactively,extrinsic information enters the system

there are many well-known ways to visually encode individual data items, visually represent entire data collections, and interactively gesture to navigate the resulting views

almost all interactionwith data items is either:selecting items (in views)

adjusting values (in widgets)labeling items (using a text box)

richer ways of interacting with data items are app-specific: paint an item with colordrop a pin on a maptype text in a form

how can we generalize this?

how does the visual representation of data affect how we can interact with it?

formfollows function follows form...

objects offer affordances for

sensible interaction

graphical characteristics

constrain effecting movements

bounds of motion determine possible changes to data

but not necessarily via literal geometrydata

attributes

graphicalobjects

visualcharacteristics

shapingactions

decode

gesture

encode

render

ancient citypopulation estimate

size =1+log10(pop)

5

scroll wheel back by one

10,000 1,000

4

primary food crop

color ={red(wheat), yellow(corn)}

‘c’ pressed during hover

wheat corn

A

indirectness provides design flexibility

encoding and decoding need not be exact inverse functions

it’s enough if users can learn and use them together effectively

we’re working on a declarative language for defining both

this is more than literal drawing

inversions are often ambiguous and sometimes impossible to

solve in closed form

automatic algorithms can often solve them in practice by relaxing assumptions (as in the

constraint-based UIs of the 80’s)

actions always change visual characteristics indirectly

through data

to design an action, one must define a decoding function to sensibly “invert” the encoding

how does the visual representation of data affect how we can interact with it?

graphicalobjects

visualcharacteristics

shapingactions

gesturerender

drawing

dataattributes

decodeencode

objects and actions need not have corresponding

geometries

how does the visual representation of data affect how we can interact with it?

designing along this path is increasingly user-centric and domain-specific

well-established declarative languages

exist for specifying this path

graphicalobjects

visualcharacteristics

shapingactions

gesturerender

drawing

dataattributes

decodeencode

the path from data to graphics is well explored in visualization design

visual encoding maps data attributesnames, times, locations, quantities

into aspects of graphical form

designing along this path is inherently user-centric and domain-specific

the path from graphics to data is almost entirely unexplored

it’s a new challenge to create declarative languages for

specifying this path

how does the visual representation of data affect how we can interact with it?

graphicalobjects

visualcharacteristics

shapingactions

gesturerender

drawing

dataattributes

decodeencode

gesture decoding maps gesture attributestouch locations, mouse location, keystrokes,

modifier keys, gaze location, glove orientationinto modifications of data

detect gestures / event pattern matching

accumulate and aggregate events

extract geometry and other essential structure

map features into data modifications

map features into visual feedback

Visual Representation (interactive state, data, visual encoding)

interaction events(keyboard, mouse,

wheel, glove, wand,multitouch, gaze...)

gestures(keypress, click,drag, roll, swipe,pinch, fixation...)

behaviors(navigation, selection,

annotation, composition)decoding encoding

appearances(cursor, rubberband,

lasso, tooltip, menu...)

graphicalobjects

visualcharacteristics

shapingactions

gesturerender

drawing

dataattributes

decodeencode

declarations to specify how to:

with many pre-defined for common gestures

how does the visual representation of data affect how we can interact with it?

how can interaction allow us to express our ideas as data?

Nudge events to reorder them on

a timeline.

Drag things on a map to adjust

location.

Draw, move, erase arrows to edit links

between items.

Press modifier keys to group or ungroup

selected items.

Pinch to associate uncertainty with a

measurement.

Click and type to name, label, or

annotate.

Wheel scroll over an item to classify all items in its group.

Draw a path to sequence a set of

locations over time.

entitiesactions express characteristics

relationshipsactions express connections

loca

l file

proj

ect (

obje

ct)

filte

r (b

oole

an)

sort

(com

para

tor)

Importedf(data source)

Calculatedf(columns)

...data

base

URL

reco

rd ID

(uni

que)

Intrinsicg(defined)

... ...so

rted

dat

a va

lues

in in

dexe

d bu

cket

s

Groupedg(column)

inve

rse

inde

x

enco

de (g

lyph)

... ...

Annotatedh(user data entry)

bool

ean

cate

goric

al

nom

inal

prev

ious

sta

te B

prev

ious

sta

te Z

Variantg(column, h(parameters))

prev

ious

sta

te A

...

Ampliatedf(columns, h-1(events, encoding))

sele

ct (b

oole

an)

orde

r (in

tege

r)

clas

sify

(str

ing)

posit

ion

(floa

t)

quan

titat

ive

com

posit

e

how can interaction allow us to express our ideas as data?

Change column values to record

observations.

Create multiple columns to record

other interpretations.

Define columns that transform, filter and

sort others.

Aggregate and define categories using linked tables.

reasoningactions express comprehension

Associate columns with users, times, and purposes.

Specify access and editing options for

each column.

Define columns that adjudicate multiple

interpretations.

Snapshot copies of columns for recall

and export.

sharingactions express collaboration

...

open-ended editing by defining and populating columns dynamically

Movement

Space

Navigation

Selection

Objects

Display

Highlighting

Users

Interaction

Views

Query

Design

DataTransformation

Pipeline

Developers

Parameter

Setting

ItemGesturesData

Editing

View

Gesture

s

the human-computer interaction loop of information visualization

➊expression

➋ ➌

➍

execution exhibition

evaluation

annotation shouldn’t be an endpoint

but in digital tools,annotations are really datain which writing is hand-formatted text

and symbols encode quantities

and arrows imply relationships

we think of annotation aswriting in the margin

adding a mark to a map

drawing an arrow to connect things

and in a sense,all data is annotation

because whether by writing it down directly

or recording it with designed instruments

it ultimately comes from us...

...interpreting the world

ampliationopen-ended, interpretation-drivenextension and modification of datadirectly within visual representations

www.math.nyu.edu/~crorres/Archimedes/Lever/leverBigCorners.gif

Ampliativefrom Latin ampliare, "to enlarge"

used in logic to mean “extending” or“adding to that which is already known.”

In Norman law,an “ampliation” wasa postponement of a sentenceto obtain further evidence.

annotation as a part of open-ended interactionto question, observe, reason, conclude, and share

shaping actions

decodeencode

render gesture

visual characteristics

old graphics

new graphics

data attributes

old data values new data values

what’s your example

data editinginteraction?

Thanks!

Our software workbenchfor creating visualizationsis called Improvise.

To see more, visitwww.cs.ou.edu/~weaver/improvise

The next major version of Improvise is in planning. New features will include

a data editing specification language and end-user ampliation capabilities.

Stay tuned!