Theory-Driven Design in Theory-Driven Design in HCIHCI

Mary CzerwinskiMary Czerwinski

Microsoft ResearchMicrosoft Research

UIST 2004UIST 2004 22

OverviewOverview

Why theory, especially now?Why theory, especially now? Brief history of HCI, psychology and theoryBrief history of HCI, psychology and theory

The importance/role of theory The importance/role of theory Examples of theory-driven researchExamples of theory-driven research How to do theoryHow to do theory Summary of benefits of theory to HCISummary of benefits of theory to HCI

UIST 2004UIST 2004 33

Why is it a good time to revisit Why is it a good time to revisit theory? Yesterday…theory? Yesterday…

UIST 2004UIST 2004 44

Dichotic Listening Task (Cherry, Dichotic Listening Task (Cherry, 1953)1953)

UIST 2004UIST 2004 55

Cocktail Party PhenomenonCocktail Party Phenomenon

Cocktail party Cocktail party problemproblem How is it that out of a sea of voices we How is it that out of a sea of voices we

can focus on a single conversation?can focus on a single conversation? Cocktail party Cocktail party effecteffect

Moray (1959) Moray (1959) • While you are usually unaware of identity of While you are usually unaware of identity of

words in a non-attended conversation…words in a non-attended conversation…• A notable exception is your A notable exception is your namename

UIST 2004UIST 2004 66

Attenuation Theory of Attention Attenuation Theory of Attention (Treisman, 1960)(Treisman, 1960)

Blocking out Blocking out the irrelevant the irrelevant content easy content easy until….until….

It’s It’s semantically semantically meaningful or meaningful or important to important to youyou

Hey, Mary!

UIST 2004UIST 2004 77

Guidelines for Speech Communication Guidelines for Speech Communication Applications, Cockpits, etc.Applications, Cockpits, etc.

Provide a mechanism to “pull” one voice Provide a mechanism to “pull” one voice into focus into focus Mostly focusing on physical attributes of the Mostly focusing on physical attributes of the

messagemessage Do not present too much information Do not present too much information

simultaneously simultaneously Provide enough time for the user to fully Provide enough time for the user to fully

fuse streams if necessaryfuse streams if necessary

UIST 2004UIST 2004 88

80s and 90s: Personal Computer80s and 90s: Personal Computer

UIST 2004UIST 2004 99

User-Centered Theories in HCIUser-Centered Theories in HCI

The Psychology of Human-Computer The Psychology of Human-Computer InteractionInteraction, by Card, Moran and Newell , by Card, Moran and Newell (1983)(1983)

Simon and Newell’s contributions to Simon and Newell’s contributions to psychology, AI and HCI psychology, AI and HCI

Cognitive and perceptual psychology Cognitive and perceptual psychology theories codified into guidelinestheories codified into guidelines A LOT of theory from this era is used in HCI A LOT of theory from this era is used in HCI

design practice todaydesign practice today

UIST 2004UIST 2004 1010

And Then, Time Stood Still…And Then, Time Stood Still…

Advances to GUI Advances to GUI desktops arguably desktops arguably stalledstalled

Lots of evolution Lots of evolution on designs but on designs but less new theoryless new theory

Do we need it?Do we need it? Where are the Where are the

breakthroughs?breakthroughs?

Evolution

UIST 2004UIST 2004 1111

Today: UbicompToday: Ubicomp

UIST 2004UIST 2004 1212

Theoretical Guidance Missing…Theoretical Guidance Missing…

Social proxemics and etiquette?Social proxemics and etiquette? Multi-cursor interaction?Multi-cursor interaction? New mental models?New mental models? New metrics for productivity and New metrics for productivity and

acceptance?acceptance? New artifacts?New artifacts? Privacy?Privacy? New input approaches?New input approaches?

UIST 2004UIST 2004 1313

Tomorrow TodayTomorrow Today

Brain-computer Brain-computer interaction devices interaction devices like BrainGatelike BrainGate

Controlling objects Controlling objects with thought is with thought is becoming a realitybecoming a reality

Good science—Good science—new theories?new theories?

UIST 2004UIST 2004 1414

UIST 2004UIST 2004 1515

VeriChipVeriChip

FDA approved FDA approved implantable chipsimplantable chips

RFID tagsRFID tags Rooted in the skin for Rooted in the skin for

accessing medical accessing medical recordsrecords

Privacy issues are Privacy issues are becoming pervasive becoming pervasive in our researchin our research

Philosophy->Theory-?Philosophy->Theory-?

UIST 2004UIST 2004 1616

A Tools Stage of HCIA Tools Stage of HCI (Painting by (Painting by ZdenekZdenek

BurianBurian)) Cro Magnons occurred Cro Magnons occurred

~40,000 years ago ~40,000 years ago Hunted mainly with Hunted mainly with

spears, (bow and spears, (bow and arrows were later arrows were later developed). developed).

Made tools from blades Made tools from blades of Flint stone, used for of Flint stone, used for preparing animal skins.preparing animal skins.

UIST 2004UIST 2004 1717

Shneiderman—Why Theories in Shneiderman—Why Theories in HCI?HCI?

DescriptiveDescriptive: clarify terms, key concepts: clarify terms, key concepts ExplanatoryExplanatory: reveal relationships and : reveal relationships and

processesprocesses PredictivePredictive: about performance and situations: about performance and situations PrescriptivePrescriptive: convey guidance for decision : convey guidance for decision

making in design by recording best practicemaking in design by recording best practice GenerativeGenerative: enable practitioners to create, : enable practitioners to create,

invent or discover something newinvent or discover something new

UIST 2004UIST 2004 1818

Examples of Theory-Driven Examples of Theory-Driven ResearchResearch

Just a few examplesJust a few examplesNot exhaustive!Not exhaustive!Still, a tough task to identify well-Still, a tough task to identify well-

known examples across all types known examples across all types of theoryof theory

UIST 2004UIST 2004 1919

Explanatory Theory ExampleExplanatory Theory Example

Norman’s seven stage model from POET (1988)Norman’s seven stage model from POET (1988) An approximate model with a continuous An approximate model with a continuous

feedback loopfeedback loop Forming the goalForming the goal Forming the intentionForming the intention Specifying the actionSpecifying the action Executing the actionExecuting the action Perceiving the state of the worldPerceiving the state of the world Interpreting the state of the worldInterpreting the state of the world Evaluating the outcomeEvaluating the outcome

UIST 2004UIST 2004 2020

Using the model predictively…Using the model predictively… Miyata & Norman (1986)Miyata & Norman (1986)

Predicted interruptions between task Predicted interruptions between task execution and evaluation as less harmful execution and evaluation as less harmful when multitaskingwhen multitasking

UntestedUntested In 2000, we decided to test this using IM In 2000, we decided to test this using IM

and multiple tasksand multiple tasks Examined planning, execution and evaluation Examined planning, execution and evaluation

phases of tasksphases of tasks

UIST 2004UIST 2004 2121

UIST 2004UIST 2004 2222

Attention-Based Principles of Attention-Based Principles of NotificationsNotifications

Early in a task was the worst time to Early in a task was the worst time to interrupt if you want user to rememberinterrupt if you want user to remember

Make notifications situation-awareMake notifications situation-aware Look for cognitive breakpoints in users’ Look for cognitive breakpoints in users’

interactions. interactions. When possible, use smart monitoringWhen possible, use smart monitoring

Monitor the user (what stage in task?)Monitor the user (what stage in task?) Content of interruption—similar is betterContent of interruption—similar is better

• Obvious privacy issues, etc.Obvious privacy issues, etc.

UIST 2004UIST 2004 2323

Predictive Theory ExamplesPredictive Theory Examples

Large display research (Tan, Large display research (Tan, Czerwinski & Robertson, 2001-Czerwinski & Robertson, 2001-2003)2003)• Most early research carried out Most early research carried out

around cockpit design around cockpit design • New hardware often necessitates the New hardware often necessitates the

need for new software/interactionneed for new software/interaction• Serendipitous gender and spatial Serendipitous gender and spatial

cognition findings based on theories cognition findings based on theories of perception and cognitionof perception and cognition

UIST 2004UIST 2004 2424

Dsharp DisplayDsharp Display

43"

11"

UIST 2004UIST 2004 2525

Prescriptive Theory ExamplesPrescriptive Theory Examples Example: Gestalt Theory of Perception Example: Gestalt Theory of Perception

(similarity, closure, good continuity, (similarity, closure, good continuity, proximity/figure-ground)proximity/figure-ground)

Example: Feature Integration Theory Example: Feature Integration Theory (Triesman et al., 80s), guided visual (Triesman et al., 80s), guided visual search and pop out effectssearch and pop out effects

Utilized well in design guidelines todayUtilized well in design guidelines today

UIST 2004UIST 2004 2626

Information Visualizer SystemInformation Visualizer System

Access CostsAccess Costs

Interaction CostsInteraction Costs

Assimilation CostsAssimilation Costs

Larger WorkspaceDenser Workspace

Larger WorkspaceDenser Workspace

Highly InteractiveHighly Interactive

Information Visualization

Information Visualization

COST STRUCTUREOF INFORMATION

INFORMATIONWORKSPACE

Analysis Goals UI Artifacts

CASE STUDIES EXPLOIT HUMANPERCEPTION

INFORMATIONVISUALIZATIONS

ANIMATED GUI

3D/RoomsInteractive Objects

Cognitive Coprocessor

UIST 2004UIST 2004 2727

•ACT-IF (Pirolli & Card ’99)

•Theory based on information foraging, sensemaking and the scatter/gather approach

•Published in Psychology Review

UIST 2004UIST 2004 2828

Scatter/Gather Document BrowserScatter/Gather Document Browser

UIST 2004UIST 2004 2929

UIST 2004UIST 2004 3030

Generative Theory ExamplesGenerative Theory Examples

Shneiderman: “The future of HCI Shneiderman: “The future of HCI must be tied to more effective must be tied to more effective generativegenerative theoriestheories that enable HCI to that enable HCI to become the bright shining source of become the bright shining source of innovation; a much stronger role than innovation; a much stronger role than the usability testers and refiners of the usability testers and refiners of ideas initiated by others.”ideas initiated by others.”

UIST 2004UIST 2004 3131

Buxton’s 3-State Model of Buxton’s 3-State Model of Graphical Input (1990)Graphical Input (1990)

Model inspired Model inspired Mackinlay, Card & Mackinlay, Card & Robertson (1991) to Robertson (1991) to write “A Semantic write “A Semantic Analysis of the Design Analysis of the Design Space of Input Devices”Space of Input Devices”

Hinckley et al. (1998) Hinckley et al. (1998) extended the ideas to extended the ideas to add notation for add notation for continuous properties continuous properties during state transitions during state transitions of devicesof devices

UIST 2004UIST 2004 3232

Pointing and Fitts’ law – The abc’s of user Pointing and Fitts’ law – The abc’s of user interfaces (Predictive and Generative)interfaces (Predictive and Generative)

Fitts’ lawFitts’ law )1(log2 WDbaT

UIST 2004UIST 2004 3333

Zhai et al.: Laws of actionsZhai et al.: Laws of actions

PointingPointingCrossingCrossingSteeringSteering

Thank you Shumin!Thank you Shumin!

UIST 2004UIST 2004 3434

Crossing – more than dotting the i’sCrossing – more than dotting the i’s

Why crossing?Why crossing? increasing interaction increasing interaction

“vocabulary”“vocabulary” Pen based computingPen based computing

How does crossing How does crossing compare with pointing?compare with pointing?

What is the theoretical What is the theoretical foundation of crossing?foundation of crossing?

Accot & Zhai 2002

UIST 2004UIST 2004 3535

Systematic comparison Systematic comparison

Continuity: Continuity: continuous vs discretecontinuous vs discrete

Target type:Target type:pointing vs crossingpointing vs crossing

Constraint direction:Constraint direction:collinear vs orthogonalcollinear vs orthogonal

Accot & Zhai, CHI 2002

UIST 2004UIST 2004 3636

Crossing-based interfacesCrossing-based interfacesAccot & Zhai 2003

UIST 2004UIST 2004 3737W

D

Law of steeringLaw of steering

Rashevsky 1959, Drury 1971; Accot & Zhai, 1997, 1999, 2001

Accot & Zhai 1997, 1999, 2001)

UIST 2004UIST 2004 3838

Law of Steering - General formLaw of Steering - General form

dsdsW(s)W(s)

IDIDCC = =

CC

TTCC = a + b ID = a + b IDCC

dsW(s)

CC

UIST 2004UIST 2004 3939

Computing off the Computing off the desktopdesktop

Desktop computing Desktop computing “workstation” interface “workstation” interface foundationfoundation Large and personal displayLarge and personal display Input device (mouse)Input device (mouse) Typewriter keyboardTypewriter keyboard

HCI Frontier – beyond the HCI Frontier – beyond the desktopdesktop Interfaces without display-Interfaces without display-

mouse-keyboard tripodmouse-keyboard tripod Numerous difficult challengesNumerous difficult challenges

UIST 2004UIST 2004 4040

“Fitts-digraph energy”

27

1

27

12 1

i j i

ijij

W

DLog

IP

Pt

Alphabetically Tuned and Optimized Mobile Interface KeyboardAlphabetically Tuned and Optimized Mobile Interface Keyboard

(ATOMIK)(ATOMIK)Human Movement Study: Fitts’ law

MT = a + b Log2(Dsi/Wi + 1)

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

sp E T A H O N S R I D L U W M C G Y F B P K V J X Q Z

English Letter Corpus(News, chat etc)

01

0

Eif

EifeBAW kTE

Metropolis “random walk” optimization

))(( i

z

aii xyite

Alphabetical tuning

N

i

icifCI1

)()(

Word connectivity

Zhai, Hunter, Smith 2000; Zhai, Sue, Accot 2002, Drews

UIST 2004UIST 2004 4141

Limitations and hints from Limitations and hints from ATOMIKATOMIK

Tapping one key at a time – tedious. The Tapping one key at a time – tedious. The stylus can be more expressive and dexterous.stylus can be more expressive and dexterous.

Does not utilize language Does not utilize language redundancy/statistical intelligence.redundancy/statistical intelligence.

People tend to remember the pattern of a People tend to remember the pattern of a whole word, not individual letters.whole word, not individual letters.

UIST 2004UIST 2004 4242

Zhai et al.--SharkZhai et al.--Shark

UIST 2004UIST 2004 4343

Metrics Development—”Metrics Development—”Subjective Subjective Duration Assessment”Duration Assessment”

HCI and iterative usability metricsHCI and iterative usability metrics Performance (task times, success rates)Performance (task times, success rates) Preference (user satisfaction Preference (user satisfaction

questionnaires)questionnaires) Usually correlated, but not alwaysUsually correlated, but not always Users notoriously “positive” w/ratingsUsers notoriously “positive” w/ratings Neilsen & Levy (1994): need an Neilsen & Levy (1994): need an

average of 5 on a 7 point scaleaverage of 5 on a 7 point scale

UIST 2004UIST 2004 4444

Ziegarnik Effect (1927)Ziegarnik Effect (1927)

People remember uncompleted tasks People remember uncompleted tasks better than completedbetter than completed

Weybrew (1984) used time estimationWeybrew (1984) used time estimation People overestimate time on unfinished tasksPeople overestimate time on unfinished tasks People underestimate time on completed People underestimate time on completed

taskstasks Van Bergen (1968) task has to be Van Bergen (1968) task has to be

engaging or more difficult to get the effectengaging or more difficult to get the effect

UIST 2004UIST 2004 4545

Spool (2001)Spool (2001)

Found a strong correlation between Found a strong correlation between perceived download time and whether perceived download time and whether users successfully completed their tasks users successfully completed their tasks on a siteon a site

When people accomplish task on a site, When people accomplish task on a site, they perceive that site to be fast, and vice they perceive that site to be fast, and vice versaversa

Goes against emphasis on fast download Goes against emphasis on fast download times only, need to support user’s task times only, need to support user’s task mostmost

UIST 2004UIST 2004 4646

67% 83% 100%-50

-40

-30

-20

-10

0

10

20

30

Task Success Rate

MSN Study—SDA Better Predictor of Difficulty than User Satisfaction

UIST 2004UIST 2004 4747

So Are We in Good Shape?So Are We in Good Shape?

Obviously, great work has come Obviously, great work has come out of theory in HCIout of theory in HCI

Do we need more?Do we need more?

UIST 2004UIST 2004 4848

Why I Think Tomorrow’s HCI Why I Think Tomorrow’s HCI Needs TheoryNeeds Theory

We’ve come full circleWe’ve come full circle Many new tools used both singly and by multiple Many new tools used both singly and by multiple

peoplepeople

Theory is one of the most important things we Theory is one of the most important things we

should be doing as a disciplineshould be doing as a discipline Not everyone needs to do it, but some of us have to!Not everyone needs to do it, but some of us have to!

A way to guarantee progressA way to guarantee progress Invite new disciplines to work with usInvite new disciplines to work with us

Cognitive neuroscientists, biologists, ethicists?Cognitive neuroscientists, biologists, ethicists?

UIST 2004UIST 2004 4949

How to Do Theory?How to Do Theory? Whittaker, Terveen & Nardi (2000)Whittaker, Terveen & Nardi (2000)

Adopt a reference task agendaAdopt a reference task agenda A set of well-defined “tasks” that candidate A set of well-defined “tasks” that candidate

systems could be evaluated against like systems could be evaluated against like TRECTREC

Even skeptics (Landauer, Carroll et al., Bannon) Even skeptics (Landauer, Carroll et al., Bannon) agree on the importance of having an adequate  agree on the importance of having an adequate  descriptive base for HCI as a prerequisite for descriptive base for HCI as a prerequisite for more theory development more theory development

UIST 2004UIST 2004 5050

Engestrom’s Activity System Model Engestrom’s Activity System Model (1987)(1987)

Most human activities are collective ones taking Most human activities are collective ones taking place in rich social environments place in rich social environments

Model used to explain collective activities and Model used to explain collective activities and cooperative work, including cultural conventions cooperative work, including cultural conventions and established rulesand established rules

UIST 2004UIST 2004 5151

Activity Theory and HCIActivity Theory and HCI Transformation of tools, rules and divisions of labor is Transformation of tools, rules and divisions of labor is

key key Utilized more in CSCW communityUtilized more in CSCW community Exacty how Activity Theory is operationalized in Exacty how Activity Theory is operationalized in

studies is a common problemstudies is a common problem Can provide a framework to reinterpret the concept of Can provide a framework to reinterpret the concept of

user needs and societyuser needs and society

UIST 2004UIST 2004 5252

Challenges to Theory BuildingChallenges to Theory Building Advancing technologyAdvancing technology Influence of individual differencesInfluence of individual differences Contextual effectsContextual effects Longitudinal effectsLongitudinal effects New metrics neededNew metrics needed

But paradigms exist that can be But paradigms exist that can be leveragedleveraged

UIST 2004UIST 2004 5353

It’s Good for Your Career!It’s Good for Your Career! Top 10 Most Frequently Cited CHI AuthorsTop 10 Most Frequently Cited CHI Authors

Stu Card (484 citations)Stu Card (484 citations) Bill Buxton (351)Bill Buxton (351) Thomas Moran (344)Thomas Moran (344) Ben Shneiderman (322)Ben Shneiderman (322) Hiroshi Ishii (298)Hiroshi Ishii (298) Brad Myers (287)Brad Myers (287) Jakob Nielsen (286)Jakob Nielsen (286) Allen Newell (222)Allen Newell (222) Jock Mackinlay (217)Jock Mackinlay (217) George Robertson (215)George Robertson (215)

UIST 2004UIST 2004 5454

ConclusionConclusion Benefits of a theoretical Benefits of a theoretical

approach to HCI:approach to HCI: Move our community toward a Move our community toward a

solid, scientific disciplinesolid, scientific discipline Identify gaps in the design spaceIdentify gaps in the design space Engage in a communal Engage in a communal

discussion about designdiscussion about design Career advancementCareer advancement InnovationInnovation Holy Grail for

HCI

UIST 2004UIST 2004 5555

Thank You!Thank You!

UIST 2004UIST 2004 5656

Tom Erickson’s PoemTom Erickson’s Poem Theory Theory: A Designer's View Theory Theory: A Designer's View

Theory weary, theory leery, why can't I be theory cheery? Theory weary, theory leery, why can't I be theory cheery? I often try out little bits, wheresoever they might fit. I often try out little bits, wheresoever they might fit. (Affordances are very pliable, though what they add is quite (Affordances are very pliable, though what they add is quite deniable.) deniable.) The sages call this bricolage, the promiscuous prefer menage... The sages call this bricolage, the promiscuous prefer menage...

A savage, I, my mind's pragmatic, I keep what's good, discard A savage, I, my mind's pragmatic, I keep what's good, discard dogmatic. dogmatic.

//// Oddly now, I'm theory cheery -- I find I have a theory theory! Oddly now, I'm theory cheery -- I find I have a theory theory!

Neither holy grail, nor deep disgrace, theory's useful in its Neither holy grail, nor deep disgrace, theory's useful in its place, place, (Framing, talking, predicting, bonding, evoking discourse--(Framing, talking, predicting, bonding, evoking discourse--Others responding) Others responding) Like goals and methods, plans and actions, theory's situated, Like goals and methods, plans and actions, theory's situated, not pure abstraction. not pure abstraction. So make your theory a public way, where passers by may So make your theory a public way, where passers by may pause and staypause and stay