Brain, Body, and Bytes CHI 2010 Workshop Presentations

Preview:

DESCRIPTION

Brain and Body Interfaces (BBI) were discussed in this workshop. Its goal was to provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI). Find out more at the workshop website: http://www.eecs.tufts.edu/~agirou01/workshop/

Citation preview

BBB Workshop Overview

Audrey Girouard, Erin Treacy Solovey,Regan Mandryk, Desney Tan,

Lennart Nacke, Robert J.K. Jacob

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

BBB Workshop: Bringing Together BCI& Physiological Computing

3

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Goals

1. Provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI).

2. Kick-start BBB research in HCI by identifying key research questions and application areas

3. Enhancing HCI methodologies by adding these new methods and techniques to the toolbox of HCI researchers

4

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

About Us

5

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

About you

• 25 Participants

• 9 Brain papers• 19 Body papers

• X EEG papers• Y fNIRS papers• Z EMG papers• A GSR• B Eye tracking

6

32%

68%

Brain Body

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

7

<on your tables>

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

What terms/acronyms should we use to describe our work?

Psychophysiological computing seems a bit long… BBB?BBI?

Other suggestions?

8

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Today’s Schedule

9

09:00-10:30 Introduction of organizers & participants, CHI madness style

10:30-11:00 Break

11:00-12:30 Group Discussion (assign groups + topic)

12:30-14:00 Lunch

14:00-15:30 Small group discussion (1h of discussion + 30 minutes of reporting)

15:30-16:00 Break

16:00-17:00 Report from small groups, recap, group discussion – barriers to publication in HCI

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Staying connected

10

Website http://www.eecs.tufts.edu/~agirou01/workshop/

Twitter http://twitter.com/bbbCHI2010

Twitter Hashtag

#bbbCHI2010

Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and-Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786

Flickr http://www.flickr.com/groups/bbbchi/

Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological-user-interaction

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

And now… 2 Minute Introductions

• About your research:– Application or domain space? (e.g. video games)

– Signal you are measuring? (e.g. fNIRS)

– Device/hardware you are using? (e.g. ISS, Inc)

• What would you like to get out of workshop?– What problems are you facing?– What do you see in common with others here?– What device would you like to know more about?

11

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

List of Presentations1. Adriane Randolph2. Krzysztof Z. Gajos3. Saraswathi Bellur4. Danny Plass-Oude Bos5. Kai Kuikkaniemi6. Mitchel Benovoy7. Yee Chieh Chew8. Martha Crosby9. Jan Kallenbach10. Eva Oliveira11. Lucia Filgueiras12. Marc Grootjen13. Melody Moore Jackson14. Johann Schrammel15. Daniel Sjölie

12

16. Katarzyna Wac17. Feng Tian18. Veronica Zammitto19. Johanna Octavia20. Bob Wray21. Marjolein van der Zwaag

Brain-Computer Interfaces as Child’s Play: Using Mental Training to Improve

Motor Skills in Children

Adriane B. Randolph, Ph.D.Ashley Ingraham

Kennesaw State University

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline• Research Position

– Examine ties between sensorimotor area and physical ability to uncover a training tool to aid children’s motor skill development

• Pilot Study– 3 able-bodied adults (2 Male, 1 Female)– Acquire baseline Wii tennis score– Conduct mental training with BCI2000 mu training protocol for 5 days– Retest Wii tennis score for improvements– Mixed results

• Future Work– Extend study with 20 able-bodied adults (currently in progress)– Analyze link between training and baseline ability– Test with elementary-aged children

14

Participant Session One Session Five

AZ 98 60

CM 70 64

MW 103 118

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Screening for MuAshley prepares participant to use mental imagery of hand and foot movements to determine his best channels for use in mu BCI training before he plays a match of Wii Tennis.

15

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Adriane B. Randolph, Ph.D.Kennesaw State University BrainLab

(770) 423-6083 brainlab@kennesaw.edu

http://coles.kennesaw.edu/brainlab/

16

Understanding How to Design ComplexBrain-Controlled Applications

Krzysztof Z. Gajos

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

How to Design Brain-Controlled Applications?• Basic operations

18

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

How to Design Brain-Controlled Applications?

19

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

How to Design Brain-Controlled Applications?

20

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

How to Design Brain-Controlled Applications?• Basic operations

21

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

How to Design Brain-Controlled Applications?• Interaction between primary task and BCI

performance

22

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

How to Design Brain-Controlled Applications?• Error prevention, recovery, and actions

with side effects

23

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Krzysztof Gajoskgajos@eecs.harvard.eduhttp://www.eecs.harvard.edu/~kgajos/

24

Psychophysiological Responsesto Media Interfaces

S. Shyam SundarSaraswathi Bellur

Media Effects Research Labhttp://www.psu.edu/dept/medialab

Pennsylvania State University

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Biopac MP 35 & MP 150 ACQknowledge software

26

Skin Conductance (EDA) Heart Rate (ECG) Facial EMG Brain Wave (EEG) Eye Movement (EOG)

Application AreasInterface presentationInformation Processing Usability Testing & Evaluation User Experience

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 27

Physiological Measures: Affect, Arousal & Attention

1. EEG: Bipolar recordings Occipital Alpha Blocking

2. Skin Conductance (EDA) Tonic arousal levels

3. ECG: Heart Rate recording

OrientingResponse viaBPM change

4. Facial EMG:Corrugator &Zygomaticrecordings

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Theoretical Frameworks• The Media Equation: Mediated Life = Real Life (Reeves & Nass,

1996). • Structural aspects of media technologies: Automatic or

Controlled processing? Voluntary or Involuntary responses?• Specific cues triggered via interactions with four types of

media affordances, MAIN model (Sundar, 2008)

28

M: MODALITY

I: INTERACTIVITY

A: AGENCY

N: NAVIGABILITY

Animation

Customization

Contingency

PerceptualBandwidth

InformationScent

UGC & Sourceness

Heuristics

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Saras Bellur (saras@psu.edu)S. Shyam Sundar (sss12@psu.edu)

29

Media Effects Research Labhttp://www.psu.edu/dept/medialab

Pennsylvania State University

BCI for Games

Danny Plass-Oude BosBoris Reuderink, Bram van de Laar, Hayrettin Gürkök, Christian Mühl,

Mannes Poel, Anton Nijholt, Dirk HeylenHMI, University of Twente

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Target Group: the general population

31

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

BCI: issues

32

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

BCI: new information, new features

33

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Acceptance: Usability & Realistic Settings

34

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Subjects Users

35

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

d.plass@ewi.utwente.nlhttp://hmi.ewi.utwente.nl/~oudebos

36

Designing Biofeedback for Games and Playful Applications

Kai KuikkaniemiHelsinki Institute for Information

Technology

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Emoshooter

38

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Emoshigeru - cinema gamesReal-timePublic displayBiosignal adaptationMassive co-locatedShooter / StrategyPart of Shigeru-games

localwlan

Public displayConnected to game

server

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Presemo – presentation engine

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Emolisten – biosignal sonification

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Research statement

• Social Games and Playful Applications, which Utilize Biosignals

• Grey Area between Adaptation and Biofeedback– Implicit and Explicit Biofeedback

• Biosignal Synchrony (Inter-Personal)

42

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Our team and projects

Professor Marko TurpeinenProfessor Timo SaariIlkka Kosunen, Toni Laitinen, Petri Lievonen, Pauli Ojala

Fun of Gaming (EU FP6) (2006-2009)Enactive Social Media and Games (Aalto) (2009)Emokeitai (Tekes – National Innovation Agency) (2009-2011) Thanks:CKIR / Niklas Ravaja

43

Psychophysiological signal analysis and classification

Mitchel BenovoyCentre for Intelligent Machine, McGill University

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Signals

• Hardware

• Application domains

45

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Signals

46

0.5 1 1.5 2 2.5

x 104

-0.5

0

0.5

1

1.5

Sample

∆C

ond.

( µS

)

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000323436384042

Ref

l.

1000 2000 3000 4000 5000 6000 7000 8000 9000 100005

6

7

Am

p.

1000 2000 3000 4000 5000 6000 7000 8000 9000 1000090

95

100

105

HR

(bpm

)

1000 2000 3000 4000 5000 6000 7000 8000 9000 100001.21.41.61.8

22.2

Exp

.

1000 2000 3000 4000 5000 6000 7000 8000 9000 100000.20.40.60.8

1A

mp.

1000 2000 3000 4000 5000 6000 7000 8000 9000 1000015

20

25

30

RR

(bpm

)

Galvanic Skin Response Phalange Temperature

Electromyogram

Blood Volume Pulse – HR & Amp. Respiration – Rate & Amp.

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Hardware• Thought Technology ProCom Infinity

47

Images via www.thoughttechnology.com

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Application Domains• Affective Computing

– Interaction evaluation (UI, devices, latency)– Music & emotion research– Psychoneurology

48

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

benovoym@cim.mcgill.ca

49

BCI and Creativity

Yee Chieh (Denise) ChewGeorgia Institute of Technology

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• fNIR and Creative Expression• EEG and Music

51

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Character Templates for fNIRCreated with a mouse, these are templates that subjects traced using fNIR

52

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

ychew@gatech.edu

53

Physiological Measures Used for Identification of Cognitive States and

Continuous AuthenticationCurtis S. Ikehara & Martha E. Crosby

University of Hawaii at Manoa

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Overview of our Research Using Physiological sensors– Assessing Cognitive States– Continuous Authentication

• Possible Applications

55

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 5656

Multi-modal Physiological Sensors System

MTF(Moving Target Fractions) HUMAN

Custom Electronics(Analog to Digital Converter)

Eye Tracker(Gaze Position, Fixation Number, Fixation

Duration, Repeat Fixations, Search Patterns, Pupil Size, Blink Rate, Blink Duration)

Pressure Mouse(Pressures on Mouse Button and Body)

Laptop(Data Collection, Analysis & Display) EDA / TSA Finger Sensors

(Electrodermal and Temperature Sensor Activity)

Oximeter(Pulse, Blood Oxygen)

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 20105757

A Composite of Over 20 Mouse Clicks of Three People (Click Signature)

Subject 5

Subject 4

Subject 7

US Patent # 7,245,218

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Possible Applications

• The student’s confidence can be assessed by evaluating the distortions of the mouse click signature.

• High Value Vehicle Control– Pressure and other physiological sensors can be

attached to a steering (e.g. aircraft yoke or steering wheel) .

– Sensors can be used to indicate that:• The authorized pilot or driver is in control.• The cognitive state of the pilot is consistent with the

vehicle’s activity.

58

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

University of Hawaii at Manoa1680 East-West Road - POST 317

Honolulu, Hawaii 96822 USA

59

Martha E. Crosbycrosby@hawaii.edu

Curtis S. Ikeharacikehara@hawaii.edu

Measuring Interaction Experiences:Integration of Multiple

Psychophysiological Methods

Jan KallenbachAalto UniversityHelsinki, Finland

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

OutlineUser Experience

=Choice & Interaction

+Cost-Benefit Evaluations

Measuring Interaction Experiences?

Software Framework61

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Client Application for UX ResearchSampling frequency: max 1000HzPsychophysiological Measures: Gaze, Pupil Size, EDA, EMG, EEG, …Behavioral Measures: Keyboard, Mouse, Slider, …System Measures: Window Events

62

MultimodalMeasurements

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Jan KallenbachResearch Scholar

Aalto UniversitySchool of Science & TechnologyDepartment of Media TechnologyP.O. Box 15500FIN - 00076 Aalto

63

Towards enhanced video access and recommendation through emotions

Eva OliveiraNuno Ribeiro

Teresa ChambelIPCA | LASIGE

65Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

We Want

• To access to videos based on emotions– user actual emotional (psychophysiological) state– video emotional content– director’s perspective

• To recommend videos based on emotions– create a set of rules from every affective

classification perspective (user and video, director)

66Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Affective video recommendation system requirements ()• Every video must be classified affectively from the user

and from the content perspective.• Every user has an emotional profile constituted by all his

implicit (physiological) classification of every classified movie.

• User physiological signals can change the way we watch the videos

67Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

User Classification Issues

• Automatic Classification– Users emotional state

• emotional model– Classification techniques

• Result output• Accuracy• Lab contexts

68Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Current State

• Capturing users phychophysiological signals (BIOPAC System) while watching movies

• Next step: several classification techniques to convert signal result into emotional categories

This is my exciting picture

69Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

• Eva Oliveira :

• eva.oliveira@gmail.com

Your users can’t talk!How can you tell they liked

your system? Lucia Filgueiras and Claudia Tambascia

Escola PolitecnicaUniversidade de São Paulo

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

What we want to do

• Develop products for cerebral palsy users• Perform participatory design• Understand non-verbal communication

71

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Direct measures and usability evaluation

Variabilidade da frequência cardíaca OP13

2,6

8,3

5,2

2,6 2,63,9

4,6 5,0

11,7

2,3

9,1

3,82,9

6,6

4,0 4,4

2,4

14,1

5,3 5,64,1

7,3

3,3

8,5

5,9

0,0

2,0

4,0

6,0

8,0

10,0

12,0

14,0

16,0

18,0

09:35

09:40

09:45

09:50

09:55

10:00

10:05

10:10

10:15

10:20

10:25

10:30

10:35

10:40

10:45

10:50

10:55

11:00

11:05

11:10

11:15

11:20

11:25

11:30

11:35

HORA

Índi

ce d

o B

alan

ço S

impá

tico?

Par

assi

mpá

tico

LF nu / HF nu LSC Média

Alarme de temperarura alta:

10:15

Operador analisa sistema após vários alarmes: temperatura alta (10:29);

nível alto (10:32) e segurança (10:32; 10:37; 10:43)

Ação do operador para correção

Ação do operador para correção

Eye tracking in iTV apps evaluation

NASA TLX, SWAT, Heart Rate Variability

Heart Rate Variability of an operator in a unstable unit situation

Comparison of eye tracker findings to conventional usability tests results

HRV and workload measures in control room operability evaluation

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Our questions:• How personalized are body expressions for cerebral palsy patients?

• Are there general body expressions that can mean the same for everyone?

• Is it possible to determine a generic way of measuring these expressions?

• Is there any other liaison besides gestures and facial expressions , like an unconscious connection of limbic systems that can be responsible for the mutual sensitive communication?

• Is there a reference body signal that can be used as a reliable parameter to evaluate body expressions?

• How can we establish a correlation between body expressions and patient´s opinion of approval or disapproval of a design solution?

73

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you!Contact us:

Lucia V. L. FilgueirasEscola Politecnica, Universidade de São Paulolucia.filgueiras@poli.usp.br

Claudia A.TambasciaFundação CPqDclaudiat@cpqd.com.br

http://lts-i.pcs.usp.br

74

Human Computer Symbiosis by Mutual Understanding

Marc GrootjenEagleScience-TNO

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Symbiosis & mutual understanding• Information models• Interfaces

76

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 77

Tasks (goals)

Disturbance

computerMutual

understanding

Symbiotic entity

Maximum task performance

human

Human-computer symbiosis with mutual understanding as key issue for maximum task performance

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 78

Domesticating Brain-Computer Interfaces: Systems

for the HomeMelody Moore Jackson

BrainLabSchool of Interactive Computing

Georgia Tech

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Extended Home-based BCI study

• fNIR-based BCI for communication• 32 People with ALS (so far) over 1 year• 78% (25) could operate the device on

initial visit with accuracy > 70%• Accuracy to 100%, average 76.5%• Issues:

– Motivation– Data Collection– Remote Support

80

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

The Kokoro GatariBrain Computer Interface based on Functional Near Infrared imaging

81

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Melody Moore Jacksonmelody@cc.gatech.edu404-277-1324

Ian McClendonimclend@cc.gatech.edu404-641-7712

82

Get up, move on! Using Electromyography to Explore

the Relationship of Experience and Motion

Johann SchrammelCentre for Usability Research and

Engineering

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Compare experiences and emotions in conditions with and without movements with Electromyography(EMG) and other methods

84

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Can EMG successfully be used to estimate a user‘s emotional experience in motion contexts (artefacts, feasibility, etc.)?

85

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Is a more embodied way of controlling interaction actually amplifying experience?

86

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Results

• EMG is suitable for motion contexts.• Both EMG and self-report data indicate

that natural body movement intensifies emotions.

• It’s crucial that body movements are in accordance with activity.

87

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

schrammel@cure.at

88

Reality-Based Brain-Computer Interaction

Daniel SjölieUmeå University, Umeå, Sweden

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Reality-Based Brain-Computer Interaction• Complex, reality-based, interaction with the

ability to adapt to the user in real-time– Using brain measurements to evaluate

• Direct brain-computer interaction– The user can be unaware

• We’re using a combination of virtual reality (VR) and functional MRI (fMRI)

90

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

HCI and cognition

• An improved understanding of the brain in this interactive and complex context is needed

• Build on theories of cognition as a foundation for the development of advanced HCI– As interaction moves towards being reality-based

we move closer to “interacting with reality”, and thus to general cognition

• Cognitive neuroscience and computational models of the brain are important tools

91

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Working with fMRI and Virtual RealityWorking with fMRI requires specialist competence and forces us to deal with powerful magnetic fields and restricted mobility, but the ability to measure the whole brain is an invaluable research tool.

92

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Daniel Sjölie, daniel@cs.umu.se

93

emoBAN: Improving Quality of Life

via Psychophysiological Mobile Computing

Katarzyna Wac, Anind K. DeyCarnegie Mellon University, HCII

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

emoBAN idea

• “what always speaks silently is the body” (Brown, 1990)

– measure QoL via psychophysiology

95

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

emoBAN: Usable Useful Functional

• Body Area Network– sensors

• vital signs, e.g., ECG, temp, resp• context, e.g., location, time, activity

– actuators• feedback via, e.g., AV, tactile, light

96

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank youDr Katarzyna WacCarnegie Mellon UniversityHuman-Computer Interaction Institutekatewac@cmu.cs.eduwww.cui.unige.ch/~wac

97

Leveraging Psychophysical Data in Monitoring and Analyzing the States of Badminton Players

Feng Tian, Wencan Luo, Jing Dai, Reza Naima, Yongmin Cheng, HonganWang, Guozhong Dai

Institute of Software, Chinese Academy of SciencesPhysical Education Collage, Beijing Sport University

Berkeley Institute of Design, UC. BerkeleyChina Badminton Team, General Administration of Sport of China

4/20/2010

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Traces and physical actions - Camera, accelerometers

Psychophysical data- Berkeley TricorderECG, EMG, Pulse Oximetry, BioImpedance Respiration

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Exploring Quantitative Methods for Evaluating Sports Games

Veronica ZammittoSimon Fraser University

Magy Seif El-NasrSimon Fraser University

Paul NewtonElectronic Arts Canada

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Current Study: NBA Live 10

• Eye tracking– Eye Tech TM3

• Psycho-physiological data– ProComp5 Infiniti– EMG (corrugator supercilii and zygomatic major),

EKG (heart rate), GSR and BVP• Gameplay Telemetry Data

– event-based RPC telemetry collection system• Triangulation

102

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Data Triangulation - NBA Live 10Game user experience

103

What was the user looking at?

Where user’s attention was?

What was the emotional response?

Was positive? Was negative?

Event, when

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Data Triangulation - NBA Live 10Game user experience

104

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Veronica Zammittovzammitt@sfu.ca

Magy Seif El-Nasrmagy@sfu.ca

Paul Newtonpnewton@ea.com

105

Exploring Psycho-physiological Measures for the Design and Behavior of Socially-Aware

Avatars in Ubicomp Environments

Johanna Renny OctaviaHasselt University, Belgium

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Avatars in Ubicomp Environment User’s representation Mediator between system and user (companion) To design a socially-aware avatar that adapts its behavior according to user’s emotion Any incorrect actions of the avatar can lead to loss of identification of the user

107

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Our informal experiment ProComp Infiniti device, measured GSR, EMG and BVP Higher physiological values when playing against a friend How different social relationships between players influence those values?

108

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Expertise Centre for Digital Media (EDM)Web: www.edm.uhasselt.beEmail: johanna.octavia@uhasselt.be

109

Tailoring Practice for Individualized Learning via

Cognitive StateBob Wray

Soar Technology, Inc.

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Why I need Psychophysiological Computing• Practice environments are becoming

common-place• Adaptive tailoring of practice is important for

individualized experience/learning• Need more than traditional adaptations (e.g.,

scaffolding)– Adapting to engage learner / maintain interest – Adaptation to reducing “gaming” in environment– Adaptation to elicit learner affect/arousal (“real”)

111

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Research TestbedNarrative, conversational, inter-cultural training application

112

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Bob WraySoar Technologywray@soartech.com, 919.967.5079

Brian MagerkoGeorgia Institute of Technologymagerko@git.edu

113

Guidelines to biosignaldriven HCI

Marjolein van der Zwaag

Philips Research Europe, University of Groningen

Guidelines to biosignaldriven HCI

Marjolein van der ZwaagEgon L. van den Broek

Joris H. Janssen

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Outline

• Signal(s) measured: psycho-physiological • Application domain: Affective computing• Device used: Nexus 10, mind media

• Guidelines for processing biosignals– Triangulation– Identification of users– Biosignal characteristics – Temporal construction

116

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Unobtrusive affective computingMusic direct your mood concept

117

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Thank you

Marjolein.van.der.zwaag@philips.com

118

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

What is brain/body computing good for?

How can we think systematically about this space?

120

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

What are the major themes of HCI that can be explored using brain/body interaction?

121

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

What specific brain & body sensing modalities are interesting and how can we use them?

What are most innovative technologies in this space?

How do these compare with existing practices (NASA-TLX, Likert-style questionnaires, etc)?

122

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

How can BCI and PC be defined within an HCI context?

How is a HCI context different from neuropsychology, biomedicine or biomedical

engineering?

How does HCI contribute to psychophysiological computing (or more

specifically neural engineering, bio-engineering, etc)?

123

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

If we could measure anything on brain and body, what would we want to measure and

why?

What cognitive states and emotions have been investigated with sensors?

What methodologies are best for what signals?

124

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

How can accurate physiological, psychological and neurophysiological

measurements improve HCI?

125

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

With what software techniques and hardware technologies could these signals

be optimally processed?

How can software account for the combination of user actions taken to accomplish a task versus actions that

would surprise the user, yet optimize their experience?

126

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Workshop Discussion Questions

What are the issues that must be considered when using brain/body computing for

direct control, passive sensing, adaptation, etc?

127

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Groups

• How to go from raw data to something else– Emotions: TABLE 1– Data Fusion: TABLE 4

• How to use these things in interactive interfaces: TABLE 3

128

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

BBB Workshop: Bringing Together BCI& Physiological Computing

129

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Staying connected

130

Website http://www.eecs.tufts.edu/~agirou01/workshop/

Twitter http://twitter.com/bbbCHI2010

Twitter Hashtag

#bbbCHI2010

Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and-Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786

Flickr http://www.flickr.com/groups/bbbchi/

Grouplist http://groups.google.com/group/brain-body-interfaces/

Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological-user-interaction

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

BBB at CHI2010Brains and Brawn session: Tuesday 11:30-1pm

Making Muscle-Computer Interfaces More PracticalT. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, James A. LandayUniversity of Washington, USAMicrosoft Corporation, USA

A Novel Brain-Computer Interface Using a Multi-Touch SurfaceBeste F. Yuksel, Michael Donnerer, James Tompkin, Anthony Steed.University College London, UK

The Influence of Implicit and Explicit Biofeedback in First-Person Shooter GamesKai Kuikkaniemi, Toni Laitinen, Marko Turpeinen, Timo Saari, Ilkka Kosunen, Niklas Ravaja, Helsinki Institute for Information Technology, FinlandCenter for Knowledge and Innovation Research, Finland

Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual Environment

Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, Lars NybergUmeå University, Sweden

131

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

THANK YOU

• Keep in touch!

132

Recommended