Brain, Body, and Bytes CHI 2010 Workshop Presentations

BBB Workshop Overview

Audrey Girouard, Erin Treacy Solovey,Regan Mandryk, Desney Tan,

Lennart Nacke, Robert J.K. Jacob

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

BBB Workshop: Bringing Together BCI& Physiological Computing

3


Workshop Goals

1. Provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI).

2. Kick-start BBB research in HCI by identifying key research questions and application areas

3. Enhancing HCI methodologies by adding these new methods and techniques to the toolbox of HCI researchers

4


About Us

5


About you

• 25 Participants

• 9 Brain papers• 19 Body papers

• X EEG papers• Y fNIRS papers• Z EMG papers• A GSR• B Eye tracking

6

32%

68%

Brain Body


Workshop Discussion Questions

7

<on your tables>



What terms/acronyms should we use to describe our work?

Psychophysiological computing seems a bit long… BBB?BBI?

Other suggestions?

8


Today’s Schedule

9

09:00-10:30 Introduction of organizers & participants, CHI madness style

10:30-11:00 Break

11:00-12:30 Group Discussion (assign groups + topic)

12:30-14:00 Lunch

14:00-15:30 Small group discussion (1h of discussion + 30 minutes of reporting)

15:30-16:00 Break

16:00-17:00 Report from small groups, recap, group discussion – barriers to publication in HCI


Staying connected

10

Website http://www.eecs.tufts.edu/~agirou01/workshop/

Twitter http://twitter.com/bbbCHI2010

Twitter Hashtag

#bbbCHI2010

Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and-Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786

Flickr http://www.flickr.com/groups/bbbchi/

Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological-user-interaction


And now… 2 Minute Introductions

• About your research:– Application or domain space? (e.g. video games)

– Signal you are measuring? (e.g. fNIRS)

– Device/hardware you are using? (e.g. ISS, Inc)

• What would you like to get out of workshop?– What problems are you facing?– What do you see in common with others here?– What device would you like to know more about?

11


List of Presentations1. Adriane Randolph2. Krzysztof Z. Gajos3. Saraswathi Bellur4. Danny Plass-Oude Bos5. Kai Kuikkaniemi6. Mitchel Benovoy7. Yee Chieh Chew8. Martha Crosby9. Jan Kallenbach10. Eva Oliveira11. Lucia Filgueiras12. Marc Grootjen13. Melody Moore Jackson14. Johann Schrammel15. Daniel Sjölie

12

16. Katarzyna Wac17. Feng Tian18. Veronica Zammitto19. Johanna Octavia20. Bob Wray21. Marjolein van der Zwaag

Brain-Computer Interfaces as Child’s Play: Using Mental Training to Improve

Motor Skills in Children

Adriane B. Randolph, Ph.D.Ashley Ingraham

Kennesaw State University


Outline• Research Position

– Examine ties between sensorimotor area and physical ability to uncover a training tool to aid children’s motor skill development

• Pilot Study– 3 able-bodied adults (2 Male, 1 Female)– Acquire baseline Wii tennis score– Conduct mental training with BCI2000 mu training protocol for 5 days– Retest Wii tennis score for improvements– Mixed results

• Future Work– Extend study with 20 able-bodied adults (currently in progress)– Analyze link between training and baseline ability– Test with elementary-aged children

14

Participant Session One Session Five

AZ 98 60

CM 70 64

MW 103 118


Screening for MuAshley prepares participant to use mental imagery of hand and foot movements to determine his best channels for use in mu BCI training before he plays a match of Wii Tennis.

15


Thank you

Adriane B. Randolph, Ph.D.Kennesaw State University BrainLab

(770) 423-6083 [email protected]

http://coles.kennesaw.edu/brainlab/

16

Understanding How to Design ComplexBrain-Controlled Applications

Krzysztof Z. Gajos


How to Design Brain-Controlled Applications?• Basic operations

18


How to Design Brain-Controlled Applications?

19


How to Design Brain-Controlled Applications?

20


How to Design Brain-Controlled Applications?• Basic operations

21


How to Design Brain-Controlled Applications?• Interaction between primary task and BCI

performance

22


How to Design Brain-Controlled Applications?• Error prevention, recovery, and actions

with side effects

23


Thank you

Krzysztof [email protected]://www.eecs.harvard.edu/~kgajos/

24

mailto:[email protected]

http://www.eecs.harvard.edu/~kgajos/

Psychophysiological Responsesto Media Interfaces

S. Shyam SundarSaraswathi Bellur

Media Effects Research Labhttp://www.psu.edu/dept/medialab

Pennsylvania State University


Biopac MP 35 & MP 150 ACQknowledge software

26

Skin Conductance (EDA) Heart Rate (ECG) Facial EMG Brain Wave (EEG) Eye Movement (EOG)

Application AreasInterface presentationInformation Processing Usability Testing & Evaluation User Experience

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 27

Physiological Measures: Affect, Arousal & Attention

1. EEG: Bipolar recordings Occipital Alpha Blocking

2. Skin Conductance (EDA) Tonic arousal levels

3. ECG: Heart Rate recording

OrientingResponse viaBPM change

4. Facial EMG:Corrugator &Zygomaticrecordings


Theoretical Frameworks• The Media Equation: Mediated Life = Real Life (Reeves & Nass,

1996). • Structural aspects of media technologies: Automatic or

Controlled processing? Voluntary or Involuntary responses?• Specific cues triggered via interactions with four types of

media affordances, MAIN model (Sundar, 2008)

28

M: MODALITY

I: INTERACTIVITY

A: AGENCY

N: NAVIGABILITY

Animation

Customization

Contingency

PerceptualBandwidth

InformationScent

UGC & Sourceness

Heuristics


Thank you

Saras Bellur ([email protected])S. Shyam Sundar ([email protected])

29

Media Effects Research Labhttp://www.psu.edu/dept/medialab

Pennsylvania State University



BCI for Games

Danny Plass-Oude BosBoris Reuderink, Bram van de Laar, Hayrettin Gürkök, Christian Mühl,

Mannes Poel, Anton Nijholt, Dirk HeylenHMI, University of Twente


Target Group: the general population

31


BCI: issues

32


BCI: new information, new features

33


Acceptance: Usability & Realistic Settings

34


Subjects Users

35


Thank you

[email protected]://hmi.ewi.utwente.nl/~oudebos

36


http://hmi.ewi.utwente.nl/~oudebos

Designing Biofeedback for Games and Playful Applications

Kai KuikkaniemiHelsinki Institute for Information

Technology


Emoshooter

38


Emoshigeru - cinema gamesReal-timePublic displayBiosignal adaptationMassive co-locatedShooter / StrategyPart of Shigeru-games

localwlan

Public displayConnected to game

server


Presemo – presentation engine


Emolisten – biosignal sonification


Research statement

• Social Games and Playful Applications, which Utilize Biosignals

• Grey Area between Adaptation and Biofeedback– Implicit and Explicit Biofeedback

• Biosignal Synchrony (Inter-Personal)

42


Our team and projects

Professor Marko TurpeinenProfessor Timo SaariIlkka Kosunen, Toni Laitinen, Petri Lievonen, Pauli Ojala

Fun of Gaming (EU FP6) (2006-2009)Enactive Social Media and Games (Aalto) (2009)Emokeitai (Tekes – National Innovation Agency) (2009-2011) Thanks:CKIR / Niklas Ravaja

43

Psychophysiological signal analysis and classification

Mitchel BenovoyCentre for Intelligent Machine, McGill University


Outline

• Signals

• Hardware

• Application domains

45


Signals

46

0.5 1 1.5 2 2.5

x 104

-0.5

0

0.5

1

1.5

Sample

∆C

ond.

( µS

)

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000323436384042

Ref

l.

1000 2000 3000 4000 5000 6000 7000 8000 9000 100005

6

7

Am

p.

1000 2000 3000 4000 5000 6000 7000 8000 9000 1000090

95

100

105

HR

(bpm

)

1000 2000 3000 4000 5000 6000 7000 8000 9000 100001.21.41.61.8

22.2

Exp

.

1000 2000 3000 4000 5000 6000 7000 8000 9000 100000.20.40.60.8

1A

mp.

1000 2000 3000 4000 5000 6000 7000 8000 9000 1000015

20

25

30

RR

(bpm

)

Galvanic Skin Response Phalange Temperature

Electromyogram

Blood Volume Pulse – HR & Amp. Respiration – Rate & Amp.


Hardware• Thought Technology ProCom Infinity

47

Images via www.thoughttechnology.com


Application Domains• Affective Computing

– Interaction evaluation (UI, devices, latency)– Music & emotion research– Psychoneurology

48


Thank you

[email protected]

49

BCI and Creativity

Yee Chieh (Denise) ChewGeorgia Institute of Technology


Outline

• fNIR and Creative Expression• EEG and Music

51


Character Templates for fNIRCreated with a mouse, these are templates that subjects traced using fNIR

52


Thank you

[email protected]

53

Physiological Measures Used for Identification of Cognitive States and

Continuous AuthenticationCurtis S. Ikehara & Martha E. Crosby

University of Hawaii at Manoa


Outline

• Overview of our Research Using Physiological sensors– Assessing Cognitive States– Continuous Authentication

• Possible Applications

55


Multi-modal Physiological Sensors System

MTF(Moving Target Fractions) HUMAN

Custom Electronics(Analog to Digital Converter)

Eye Tracker(Gaze Position, Fixation Number, Fixation

Duration, Repeat Fixations, Search Patterns, Pupil Size, Blink Rate, Blink Duration)

Pressure Mouse(Pressures on Mouse Button and Body)

Laptop(Data Collection, Analysis & Display) EDA / TSA Finger Sensors

(Electrodermal and Temperature Sensor Activity)

Oximeter(Pulse, Blood Oxygen)


A Composite of Over 20 Mouse Clicks of Three People (Click Signature)

Subject 5

Subject 4

Subject 7

US Patent # 7,245,218


Possible Applications

• The student’s confidence can be assessed by evaluating the distortions of the mouse click signature.

• High Value Vehicle Control– Pressure and other physiological sensors can be

attached to a steering (e.g. aircraft yoke or steering wheel) .

– Sensors can be used to indicate that:• The authorized pilot or driver is in control.• The cognitive state of the pilot is consistent with the

vehicle’s activity.

58


Thank you

University of Hawaii at Manoa1680 East-West Road - POST 317

Honolulu, Hawaii 96822 USA

59

Martha E. [email protected]

Curtis S. [email protected]

Measuring Interaction Experiences:Integration of Multiple

Psychophysiological Methods

Jan KallenbachAalto UniversityHelsinki, Finland


OutlineUser Experience

=Choice & Interaction

+Cost-Benefit Evaluations

Measuring Interaction Experiences?

Software Framework61


Client Application for UX ResearchSampling frequency: max 1000HzPsychophysiological Measures: Gaze, Pupil Size, EDA, EMG, EEG, …Behavioral Measures: Keyboard, Mouse, Slider, …System Measures: Window Events

62

MultimodalMeasurements


Thank you

Jan KallenbachResearch Scholar

Aalto UniversitySchool of Science & TechnologyDepartment of Media TechnologyP.O. Box 15500FIN - 00076 Aalto

63

Towards enhanced video access and recommendation through emotions

Eva OliveiraNuno Ribeiro

Teresa ChambelIPCA | LASIGE

65Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

We Want

• To access to videos based on emotions– user actual emotional (psychophysiological) state– video emotional content– director’s perspective

• To recommend videos based on emotions– create a set of rules from every affective

classification perspective (user and video, director)


Affective video recommendation system requirements ()• Every video must be classified affectively from the user

and from the content perspective.• Every user has an emotional profile constituted by all his

implicit (physiological) classification of every classified movie.

• User physiological signals can change the way we watch the videos


User Classification Issues

• Automatic Classification– Users emotional state

• emotional model– Classification techniques

• Result output• Accuracy• Lab contexts


Current State

• Capturing users phychophysiological signals (BIOPAC System) while watching movies

• Next step: several classification techniques to convert signal result into emotional categories

This is my exciting picture


Thank you

• Eva Oliveira :

• [email protected]



Your users can’t talk!How can you tell they liked

your system? Lucia Filgueiras and Claudia Tambascia

Escola PolitecnicaUniversidade de São Paulo


What we want to do

• Develop products for cerebral palsy users• Perform participatory design• Understand non-verbal communication

71


Direct measures and usability evaluation

Variabilidade da frequência cardíaca OP13

2,6

8,3

5,2

2,6 2,63,9

4,6 5,0

11,7

2,3

9,1

3,82,9

6,6

4,0 4,4

2,4

14,1

5,3 5,64,1

7,3

3,3

8,5

5,9

0,0

2,0

4,0

6,0

8,0

10,0

12,0

14,0

16,0

18,0

09:35

09:40

09:45

09:50

09:55

10:00

10:05

10:10

10:15

10:20

10:25

10:30

10:35

10:40

10:45

10:50

10:55

11:00

11:05

11:10

11:15

11:20

11:25

11:30

11:35

HORA

Índi

ce d

o B

alan

ço S

impá

tico?

Par

assi

mpá

tico

LF nu / HF nu LSC Média

Alarme de temperarura alta:

10:15

Operador analisa sistema após vários alarmes: temperatura alta (10:29);

nível alto (10:32) e segurança (10:32; 10:37; 10:43)

Ação do operador para correção

Ação do operador para correção

Eye tracking in iTV apps evaluation

NASA TLX, SWAT, Heart Rate Variability

Heart Rate Variability of an operator in a unstable unit situation

Comparison of eye tracker findings to conventional usability tests results

HRV and workload measures in control room operability evaluation


Our questions:• How personalized are body expressions for cerebral palsy patients?

• Are there general body expressions that can mean the same for everyone?

• Is it possible to determine a generic way of measuring these expressions?

• Is there any other liaison besides gestures and facial expressions , like an unconscious connection of limbic systems that can be responsible for the mutual sensitive communication?

• Is there a reference body signal that can be used as a reliable parameter to evaluate body expressions?

• How can we establish a correlation between body expressions and patient´s opinion of approval or disapproval of a design solution?

73


Thank you!Contact us:

Lucia V. L. FilgueirasEscola Politecnica, Universidade de São [email protected]

Claudia A.TambasciaFundação [email protected]

http://lts-i.pcs.usp.br

74



http://lts-i.pcs.usp.br/

Human Computer Symbiosis by Mutual Understanding

Marc GrootjenEagleScience-TNO


Outline

• Symbiosis & mutual understanding• Information models• Interfaces

76


Tasks (goals)

Disturbance

computerMutual

understanding

Symbiotic entity

Maximum task performance

human

Human-computer symbiosis with mutual understanding as key issue for maximum task performance



Domesticating Brain-Computer Interfaces: Systems

for the HomeMelody Moore Jackson

BrainLabSchool of Interactive Computing

Georgia Tech


Extended Home-based BCI study

• fNIR-based BCI for communication• 32 People with ALS (so far) over 1 year• 78% (25) could operate the device on

initial visit with accuracy > 70%• Accuracy to 100%, average 76.5%• Issues:

– Motivation– Data Collection– Remote Support

80


The Kokoro GatariBrain Computer Interface based on Functional Near Infrared imaging

81


Thank you

Melody Moore [email protected]

Ian [email protected]

82



Get up, move on! Using Electromyography to Explore

the Relationship of Experience and Motion

Johann SchrammelCentre for Usability Research and

Engineering


Outline

• Compare experiences and emotions in conditions with and without movements with Electromyography(EMG) and other methods

84


Outline

• Can EMG successfully be used to estimate a user‘s emotional experience in motion contexts (artefacts, feasibility, etc.)?

85


Outline

• Is a more embodied way of controlling interaction actually amplifying experience?

86


Results

• EMG is suitable for motion contexts.• Both EMG and self-report data indicate

that natural body movement intensifies emotions.

• It’s crucial that body movements are in accordance with activity.

87


Thank you

[email protected]

88

Reality-Based Brain-Computer Interaction

Daniel SjölieUmeå University, Umeå, Sweden


Reality-Based Brain-Computer Interaction• Complex, reality-based, interaction with the

ability to adapt to the user in real-time– Using brain measurements to evaluate

• Direct brain-computer interaction– The user can be unaware

• We’re using a combination of virtual reality (VR) and functional MRI (fMRI)

90


HCI and cognition

• An improved understanding of the brain in this interactive and complex context is needed

• Build on theories of cognition as a foundation for the development of advanced HCI– As interaction moves towards being reality-based

we move closer to “interacting with reality”, and thus to general cognition

• Cognitive neuroscience and computational models of the brain are important tools

91


Working with fMRI and Virtual RealityWorking with fMRI requires specialist competence and forces us to deal with powerful magnetic fields and restricted mobility, but the ability to measure the whole brain is an invaluable research tool.

92


Thank you

Daniel Sjölie, [email protected]

93


emoBAN: Improving Quality of Life

via Psychophysiological Mobile Computing

Katarzyna Wac, Anind K. DeyCarnegie Mellon University, HCII


emoBAN idea

• “what always speaks silently is the body” (Brown, 1990)

– measure QoL via psychophysiology

95


emoBAN: Usable Useful Functional

• Body Area Network– sensors

• vital signs, e.g., ECG, temp, resp• context, e.g., location, time, activity

– actuators• feedback via, e.g., AV, tactile, light

96


Thank youDr Katarzyna WacCarnegie Mellon UniversityHuman-Computer Interaction [email protected]/~wac

97

Leveraging Psychophysical Data in Monitoring and Analyzing the States of Badminton Players

Feng Tian, Wencan Luo, Jing Dai, Reza Naima, Yongmin Cheng, HonganWang, Guozhong Dai

Institute of Software, Chinese Academy of SciencesPhysical Education Collage, Beijing Sport University

Berkeley Institute of Design, UC. BerkeleyChina Badminton Team, General Administration of Sport of China

4/20/2010


Traces and physical actions - Camera, accelerometers

Psychophysical data- Berkeley TricorderECG, EMG, Pulse Oximetry, BioImpedance Respiration


Exploring Quantitative Methods for Evaluating Sports Games

Veronica ZammittoSimon Fraser University

Magy Seif El-NasrSimon Fraser University

Paul NewtonElectronic Arts Canada


Current Study: NBA Live 10

• Eye tracking– Eye Tech TM3

• Psycho-physiological data– ProComp5 Infiniti– EMG (corrugator supercilii and zygomatic major),

EKG (heart rate), GSR and BVP• Gameplay Telemetry Data

– event-based RPC telemetry collection system• Triangulation

102


Data Triangulation - NBA Live 10Game user experience

103

What was the user looking at?

Where user’s attention was?

What was the emotional response?

Was positive? Was negative?

Event, when


Data Triangulation - NBA Live 10Game user experience

104


Thank you

Veronica [email protected]

Magy Seif [email protected]

Paul [email protected]

105

Exploring Psycho-physiological Measures for the Design and Behavior of Socially-Aware

Avatars in Ubicomp Environments

Johanna Renny OctaviaHasselt University, Belgium


Avatars in Ubicomp Environment User’s representation Mediator between system and user (companion) To design a socially-aware avatar that adapts its behavior according to user’s emotion Any incorrect actions of the avatar can lead to loss of identification of the user

107


Our informal experiment ProComp Infiniti device, measured GSR, EMG and BVP Higher physiological values when playing against a friend How different social relationships between players influence those values?

108


Thank you

Expertise Centre for Digital Media (EDM)Web: www.edm.uhasselt.beEmail: [email protected]

109

http://www.edm.uhasselt.be


Tailoring Practice for Individualized Learning via

Cognitive StateBob Wray

Soar Technology, Inc.


Why I need Psychophysiological Computing• Practice environments are becoming

common-place• Adaptive tailoring of practice is important for

individualized experience/learning• Need more than traditional adaptations (e.g.,

scaffolding)– Adapting to engage learner / maintain interest – Adaptation to reducing “gaming” in environment– Adaptation to elicit learner affect/arousal (“real”)

111


Research TestbedNarrative, conversational, inter-cultural training application

112


Thank you

Bob WraySoar [email protected], 919.967.5079

Brian MagerkoGeorgia Institute of [email protected]

113

Guidelines to biosignaldriven HCI

Marjolein van der Zwaag

Philips Research Europe, University of Groningen

Guidelines to biosignaldriven HCI

Marjolein van der ZwaagEgon L. van den Broek

Joris H. Janssen


Outline

• Signal(s) measured: psycho-physiological • Application domain: Affective computing• Device used: Nexus 10, mind media

• Guidelines for processing biosignals– Triangulation– Identification of users– Biosignal characteristics – Temporal construction

116


Unobtrusive affective computingMusic direct your mood concept

117


Thank you

[email protected]

118




What is brain/body computing good for?

How can we think systematically about this space?

120



What are the major themes of HCI that can be explored using brain/body interaction?

121



What specific brain & body sensing modalities are interesting and how can we use them?

What are most innovative technologies in this space?

How do these compare with existing practices (NASA-TLX, Likert-style questionnaires, etc)?

122



How can BCI and PC be defined within an HCI context?

How is a HCI context different from neuropsychology, biomedicine or biomedical

engineering?

How does HCI contribute to psychophysiological computing (or more

specifically neural engineering, bio-engineering, etc)?

123



If we could measure anything on brain and body, what would we want to measure and

why?

What cognitive states and emotions have been investigated with sensors?

What methodologies are best for what signals?

124



How can accurate physiological, psychological and neurophysiological

measurements improve HCI?

125



With what software techniques and hardware technologies could these signals

be optimally processed?

How can software account for the combination of user actions taken to accomplish a task versus actions that

would surprise the user, yet optimize their experience?

126



What are the issues that must be considered when using brain/body computing for

direct control, passive sensing, adaptation, etc?

127


Groups

• How to go from raw data to something else– Emotions: TABLE 1– Data Fusion: TABLE 4

• How to use these things in interactive interfaces: TABLE 3

128


BBB Workshop: Bringing Together BCI& Physiological Computing

129


Staying connected

130

Website http://www.eecs.tufts.edu/~agirou01/workshop/

Twitter http://twitter.com/bbbCHI2010

Twitter Hashtag

#bbbCHI2010

Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and-Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786

Flickr http://www.flickr.com/groups/bbbchi/

Grouplist http://groups.google.com/group/brain-body-interfaces/

Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological-user-interaction


BBB at CHI2010Brains and Brawn session: Tuesday 11:30-1pm

Making Muscle-Computer Interfaces More PracticalT. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, James A. LandayUniversity of Washington, USAMicrosoft Corporation, USA

A Novel Brain-Computer Interface Using a Multi-Touch SurfaceBeste F. Yuksel, Michael Donnerer, James Tompkin, Anthony Steed.University College London, UK

The Influence of Implicit and Explicit Biofeedback in First-Person Shooter GamesKai Kuikkaniemi, Toni Laitinen, Marko Turpeinen, Timo Saari, Ilkka Kosunen, Niklas Ravaja, Helsinki Institute for Information Technology, FinlandCenter for Knowledge and Innovation Research, Finland

Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual Environment

Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, Lars NybergUmeå University, Sweden

131


THANK YOU

• Keep in touch!

132