Leave  BlankVisually-enabled Distribution of

Cognition

Brian Fisher

SFU Interactive Arts & Technology / Cognitive ScienceUBC Media & Graphics Interdisciplinary Centre Calit2 Visiting Scholar, University of California 2012

1

Leave  Blank

2

1965 Visual Analytics

Individuals who operate effectively in our culture have already been considerably "augmented." Basic human capabilities for sensing stimuli, performing numerous mental operations, and for communicating with the outside world, are put to work in our society within a system--an H-LAM/T system--the individual augmented by the language, artifacts, and methodology in which he is trained. Furthermore, we suspect that improving the effectiveness of the individual as he operates in our society should be approached as a system-engineering problem--that is, the H-LAM/T system should be studied as an interacting whole from a synthesis-oriented approach.

Doug Engelbart2

Leave  Blank

On the Death of Visualization (2004)

Can It Survive Without Customers?

• Visualization, alone, is not a solution.

• Visualization is a critical part of many applications.

• Visualization, the Community, lacks application domain knowledge.

• Visualization has become a commodity.

• Visualization is not having an impact in applications.

Text

Bill Lorensen

3

Leave  Blank

“This science must be built on integrated perceptual and cognitive theories that embrace the dynamic interaction between cognition, perception, and action. It must provide insight on fundamental cognitive concepts such as attention and memory. It must build basic knowledge about the psychological foundations of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’ and ‘abstraction.’ “

“Illuminating the Path” (IEEE Press)

“The science of analytical reasoning facilitated by interactive visual interfaces”

Visual Analytics

4

Leave  Blank

Cybernetics

• 1948 Hixon Symposium, Macy Conferences

• Use control theory, information theory to explain cognitive and social phenomena

• Gurus: Weiner, McColloch & Pitts, Ashby, Shannon, von Neumann, Turing

• Application people: Bateson, Mead

On the Origins of Cognitive Science: The Mechanization of the Mind by Jean-Pierre Dupuy

5

Leave  Blank

The study of thought, learning, and mental organization, which draws on aspects of psychology, linguistics, philosophy, and computer modelling. (OED)

Cognitive Science Society founded in 1979

Cognitive Science

6

Leave  Blank

• Daniel Bobrow - AI• Eugene Charniak - AI• Allan Collins - Psychology• Edward Feigenbaum - AI• Charles Fillmore - Linguistics• Jerry Fodor - Philosophy• Walter Kintsch - Psychology• Donald Norman - Psychology• Zenon Pylyshyn - Psychology• Raj Reddy - AI• Eleanor Rosch - Psychology• Roger Schank - AI

Founders

7

Leave  BlankKey figures not at

Dallas

AI Linguistics Neurosci Philosophy Psychology

Chomsky

Miller

Minsky

Newell

Simon

✓ ✓✓ ✓ ✓ ✓✓ ✓ ✓✓ ✓✓ ✓ ✓

8

Leave  Blank

9

Leave  Blank

• A division of labour?• Simon’s“nearly decomposable

problems”

• Conceptual & methodological “trading zone”?• Galison: Image and Logic: a Material

Culture of Microphysics• Thagard: Trading Zones in Cognitive

Science

What Happens in the Maze?

10

Leave  Blank

• Unified Theories of Cognition (Newell)• Models = theories• Models should explain a range of

behaviours • “You can’t play 20 questions with

nature and expect to win”

• Fodor “Modularity of Mind” 1980• Egon Brunswick “Conceptual

Framework of Psychology” 1952

Cognitive Architecture

11

Leave  Blank

Time scale (sec)

Psychologicaldomain

• Unit task• Operations• Attention

107

106

105

104

103

102

101

100

10-1

• SOCIAL

• RATIONAL(Adaptive)

• COGNITIVE(Proximal Mechanisms)

• Task

(Newell)Time scales

12

Leave  BlankNew experiences,old

brains

• Transistor density doubles every 24 months

• Disk density doubles every 12 months

• Brain volume doubles every 3 x107 months

Intuitions about architectural processing are inaccurate = “metacognitive gap”

13

Leave  Blank

Controller/display systems in air traffic

control • NextGen ATC

“fishtank” projection • Change camera

position for better view

• How will global motion affect tracking? Liu, G. Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005) Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of Experimental Psychology: Human Perception and Performance. 31(2), Apr 2005, 235-247.

http://www.youtube.com/watch?v=tKJVB4id_TY

14

Leave  BlankFINST theory of spatial indexing

15

Leave  BlankMultiple object

tracking (Pylyshyn)

16

Leave  Blank3-D Projected

display

17

Leave  BlankTest Performance at

Different Speeds

18

Leave  Blank

Fit human tracking function (Lui)

19

Leave  Blank... Then add display

motion

20

Leave  BlankTracking vs object

speed

21

Leave  Blank

Tracking in warped space

22

Leave  BlankTracking in warped

space

23

Leave  BlankConclusion: We track in allocentric space

• Retinal speed of targets does not determine performance

• Motion of targets relative to each other does

• But only if motion preserves good metric characteristics of space

• Explanation is at the level of a human - display cognitive system

24

Leave  Blank

Optimizing analytic systems

• Personal Equation of Interaction• Test users• Model data• Describe individual differences in model parameters

• Customize display for PEI• Attentive systems adapt PEI

• Within a session: fatigue, attentiveness• Between sessions: aging changes

• Personal Equation of Interaction can support training and selection as well as tuning interaction

25

Leave  BlankLimits of

Cyberpsychology

• We show how we can lower the bar for adoption• Reduce attentional demands• Customization for the individual• Adaptation to conditions of use• Support perceptual, cognitive, interactive

expertise

• ... but what if the users are bears?

26

Leave  BlankD-Cog origins

• Cognition in the Wild 1995 • Navy ships and canoes

• Cognition distributed among personnel on the ship

• Cognition in the head in canoe navigation

• System of measurement, communication, confirmation, protocol navigates ship

• Challenge to cogsci: Mind-as-computer misses cognition

27

Leave  BlankAnalytics in the Wild

• Emergency Management (NSERC, DHS)• Mobile analytics / sensor analytics• “Virtual EOC” visual analytic environment

• Aircraft Safety, Reliability (Boeing/MITACS)• “Pair analytics” of complex quant and text data

• Economics and finance (MITACS, NSF)• Behavioural economics (portfolios) • Systemic risk analysis

• Healthcare Monitoring & Management (DHS)• Complex data in health research (CFRI)• Public health management (BC Injury Research & Prevention)

28

Leave  Blank

"Il n'existe pas une catégorie de sciences auxquelles on puisse donner le nom de sciences appliquées. Il y a la science et les applications de la science, liées entre elles comme le fruit à l'arbre qui l'a porté"

Louis Pasteur

Pure Basic Research(Bohr)

Use-inspired Basic

Research (Pasteur)

Sampling,Description,Taxonomy(Audubon)

Pure Applied Research (Edison)

Quest for Fundamental Understanding?

No

Yes

Consideration of Use ?

No

Yes(1822–95)

Pasteur’s Quadrant

(Stokes)

29

Leave  Blank

How can D-Cog help design?

• Perspective for designers• Interpret theories from cognitive sciences for use

in design• Activity Theory-> Michael Cole, Yrjö Engeström• D-Cog -> Yvonne Rogers, Jim Hollan

• Develop new, more targeted theories & methods• Translation from “clinic” to lab and back again

30

Leave  BlankD-Cog at Cogsci• Pluralistic nature of

Cogsci makes it difficult to exclude new approaches

• Also makes it difficult to come to a conclusion

• CogSci 2013 at Humboldt University in Berlin Jul 31 - Aug 3.

31

Leave  Blank

• A division of labour?

• Conceptual & methodological “trading zone”?• Galison & Thagard

• A “Mangle of Practice”?• Pickering (Exeter)

What Happens in the Maze?

32

Leave  Blank

Pickering’s Mangle of Practice

• Mangle 2 |ˈmøŋgəәl| |ˈmaŋg(əә)l| noun

• A large machine for ironing sheets or other fabrics, usually when they are damp, using heated rollers.

• chiefly Brit. a machine having two or more cylinders turned by a handle, between which wet laundry is squeezed (to remove excess moisture) and pressed

33

Leave  Blank

CognitionPerceptual

ScienceMethods

SocialScienceMethods

Computation, Visualization

Methods

Graphic & Interaction

DesignMethods

Cognitionin the wild

Mangle  of    Visual  Analy0cs  Research?

34

34

Leave  Blank

“For five decades I have been driven by an intuitive certainty that computer supported argumentation could increase humankind's collective problem-solving capabilities to a degree that was (is) greatly unappreciated, and that its explicit pursuit should become one of society's high-priority, "grand challenges".

Douglas Engelbart, 2003

35

Leave  Blank

Putting theory into practice (and practice into theory)

36

36

Leave  Blank

CognitionPerceptual

ScienceMethods

SocialScienceMethods

Computation, Visualization

Methods

Graphic & Interaction

DesignMethods

Cognitionin the wild

Mangle  of    Visual  Analy0cs  Research?

37

37

Leave  BlankD-Cog VA lab

• Dr. Richard Arias-Hernández

• Dr. Linda Kaastra• Dr. Nathalie Prevost• Samar Al-Hajj• Nadya Calderón • Tera Marie Green• Ali Khalili• Hon Cheong Lam

• Amanda Pype• Aaron Smith• Numerous SFU &

UBC student analysts

38

Leave  BlankLab Mascot

39

Leave  BlankStudent analysts

• Best debrief VAST 2007

• Discovery Exhibition Best paper 2010 (Andrew Wade) & 2011 (Samar Al-Hajj)

• Analytics are extracted and communicated to tech developers as methods & prototypes

40

Leave  Blank

• Applied Visual Analyst (2013)• Perceptually rich human-information discourse• Tech-mediated social cognition & collaboration• Learning model, personal equation

• Analytic Designer• Develop VA technologies• Customize display & interaction for user & task• Consult on organizational roles, communication

• Analytic Researcher• Advance the field

Educational Programs

41

Leave  BlankAndrew Wade VA

Challenge• Aircraft Birdstrike Incident Data• Aircraft Pitot Tube Icing Data• Immunology - Flow Cytometry Data• Healthcare Records Analysis Data• Domestic Violence Data• Risk Assessment – Financial Data• Aircraft Hazardous Bills of Materials Data• Corporate PC Performance Data• UBC On-line Course Performance Data• Justice System Prisoner Management

42

Leave  Blank

• “Pair analytics” sessions• Student visual analyst &

trained domain expert collaborate on analytic task

• Student “drives”, expert “navigates”

• Video session & capture screen

Setting the Stage

Arias-Hernandez, R, Kaastra, L.T., and Fisher, B. (2011) Joint Action Theory and Pair Analytics: In-vivo Studies of Cognition and Social Interaction in Collaborative Visual Analytics. In L. Carlson, C. Hoelscher, and T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 3244-3249). Austin TX: Cognitive Science Society.

43

Leave  BlankBird Strikes

44

Leave  Blank

45

Leave  Blank

46

Leave  Blank

47

Leave  Blank

Tableau Bird Strike Pair Analysis

48

Leave  Blank

IN-SPIRE Bird Strike Pair Analysis

49

Leave  BlankWade Internship

• Video recorded and screen captured over 10 Paired Analysis sessions using both Tableau and IN-SPIRE

• Influenced design decisions on:• 777• P8-A• 787• 747-8

• Changes to pilot training manual

50

Leave  BlankWade Internship• Presented work to:

• 787 Engineers• Aviation Safety Community of Practice• Aerodynamics, Performance, Stability and Control

flight data recorder analysis group• Advanced Analytics group• UW Aeronautics and Astronautics students • Boeing Educational Network webcast (400+)

• 500+ people exposed to Visual Analytics, Paired Analysis for Aviation Safety

51

Leave  Blank

• Build on social science (GT, JAT) approaches to understand organizations and cognitive work practices.

• The innovation here is in the extension of social science to bridge to the perceptual and cognitive science theories that apply to the use of Cognitive Science in analytical tasks.

Understand technological

distribution of cognition

SocialSciences

52

Leave  BlankJoint Activity Theory

(Clark)

• Language is an essentially collaborative activity, like playing duet or paddling canoe

• We work to build common ground so as to communicate effectively and efficiently

• Clark’s theory:• Defines kinds of common ground• Formalizes the notion of activity as a “joint action”• Describes the processes by which common

ground is developed through joint action

53

Leave  Blank

54

Grounding

• Common ground: set of things mutually believed by both speaker and hearer

• Hearer must ground or acknowledge understanding of speakers utterance.

• Clark (1996):• Principle of closure. Agents performing an action require

evidence, sufficient for current purposes, that they have succeeded in performing it

• Need to know whether an action succeeded or failed

54

Leave  Blank

Metacognition in communication

• Grounding mechanisms to establish common referents• Inflection, gesture, expression• Hesitation, pausing phrases (e.g. “um”)• Misc. visual and auditory cues• Explicit signals

• Speakers change their methods based upon unconscious assessments of these cues

55

Leave  BlankGeneralizing Clark

• Clark’s theory draws upon joint activity metaphors (paddling, music) but focuses on spoken language

• Kaastra applied that framework to musical performance analysis

• Will it work for analytic duet?

56

Leave  BlankJAT Analysis of Pair Session

• Structuring and navigation markers: • “Vertical markers” are verbal gestures, such

as “okay,” and “all right,” that signal transitions between different analytical tasks.

• “Horizontal markers,” such as “yeah” and “mhmm,” are used to signal continuation within a singular analytical task

57

Leave  Blank

• Management of joint attention: • Gaze, finger-pointing and mouse-point are used by a

speaker to direct joint attention, and by a listener to confirm that joint attention is in place.

• Use of “self-talk” with on-screen gesturing to inform about progress on the execution of a cognitively demanding task: Pauses in analysis were accompanied by “self-talk” of participants on-task:

• We are currently studying and categorizing “self-talk” occurrences in PA as indicators of cognitive workload.

JAT Analysis of Pair Session

58

Leave  Blank

Emergency Management

• Coordinated technological, methodological, organizational & training support

• Many technologies w/o rich visualization-- small form factor devices, sensors, data input.

• Example: VA for Emergency Management NSERC SPP (+ 2 SPP companion proposals)• Population: cell phones• First responders: blackberries • Data fusion centres: geotagged sensor networks, big data

processing• Command centres: interactive tabletops, walls, etc.

“Your users are bears”

59

Leave  Blank

Visual analytics in the Wild

• Emergency management for mixed-bear systems• RAH to JIBC, train at Richmond EOC• CREATE-ERE cross-border earthquake

“experiment”• Richmond, Vancouver, Provincial EOC• Field reports, interviews• Build Advanced Interaction EOC• Virtual EOC to come

60

Leave  BlankVA is more than

technology!• VA proposes not just new technologies but

new analytic methods • These new methods will change structures

of communication, selection, & training• New analytic systems need new analysts (Boeing) • Analysts take on new roles, organizations change

• VA must address human, organization, and technology (this includes universities!)

61

Leave  BlankTranslational Science of

Analytical Reasoning

• VA must develop scientific methods that help us to augment analytical reasoning

• VA includes human, organization, & tech• Human: perceptual/cog/motor “cyberpsychology”• Community: social science of information flow• Technology: software design & engineering 1&2

• Methods to bridge these levels of analysis

Fisher, B., Green, T.M., Arias-Hernández, R. (2011) "Visual analytics as a translational cognitive science," Topics in Cognitive Science 3,3 609–625.

62

Leave  BlankInjury Prevention & Response Group

Analysis• BC Injury Research & Prevention Unit (Dr Ian Pike)• Goals:

• Enhance stakeholders’ understanding of injury indicators.

• Support multi-party analysis.• Support multi-party decision-making

• Visual Analysis Expert (Samar Al-Hajj) will:• Manage and guide the group analysis • Offer stakeholders different approaches to

solving analytical problem.63

Leave  Blank

Child Injury Indicators

Indicators Spanning Across All Domains

1. Mortality Rate2. Potential Years of Life Lost3. Hospital Separations Rate

Overall Health Service Implications Indicators

4. Diagnosis-Specific Hospital Separations5. Hospital Admission - Injury Severity 16. Hospital Admission - Injury Severity 27. Length of Stay in Hospital

Motor Vehicle Injury Indicators

8. Cost of Motor Vehicle Injuries9. Crash Rate10. Intersection Crash Rate11. Rural Roadways12. Drunk Driving13. Speed14. Young Drivers15. Graduated Driver Licensing16. Child Restraints17. Unrestrained Injuries18. Child Restraint Laws

Sport, Recreation and Leisure Injury Indicators

19. Bicycle Helmet Laws20. Cost of Sport, Recreation and Leisure Injuries21. Percentage of Sport Specific Injuries (Participation Rates)22. Requirements that Playgrounds Meet CSA Standards23. Legislations Requiring Pool Fencing

Other Policy Indicators 24. Window Guard By-law25. Provincial Standards for Hot Water Tap Temperature

Violence Indicators26. Violent Crime Rate27. Abusive Head Trauma Rate28. Suicide Prevention29. Anti-Violence/Anti-Bullying Policies

Trauma Care, Quality and Outcomes

Indicators

30. Access to Pediatric Trauma Centre (PTC)31. Appropriate Use of Pediatric Trauma Centre (PTC)32. Quality of Trauma System33. Pre-hospital Transport Time34. Presence of a Coordinated Pediatric Trauma System

64

Leave  Blank

65

Leave  Blank

66

Leave  BlankVisual Rhetoric

• Child injury prevention users are creating and interpreting policies

• Can we build a visual rhetoric for them?• Concept mapping & related methods• Diagrammatic reasoning, visual communication• Visualizing argumentation• Modal logics (deontic, temporal,doxastic)

• Can we bridge the analyses?

67

Leave  Blank

• We certainly do not advocate that business schools, in revising MBA curricula, abandon science. Rather, they should encourage and reward research that illuminates the mysteries and ambiguities of today's business practices. Oddly, despite B schools’ scientific emphasis, they do little in the areas of contemporary science that probably hold the greatest promise for business education: cognitive science and neuroscience.

“How Business Schools Lost Their

Way”

Text

Bennis & O’Toole, HBR, 2005

68

Leave  BlankNSF Science of

Interaction

• Mixed-initiative data manipulation and discovery (thinking about data)

• Collaborative analytics • Multi-modal sensemaking • Fluid interaction • Visual discourse (rhetoric)

69

Leave  Blank

Decision Support & Operational

Management Analytics mini-track

70