Using Electrodermal Activity to Recognize Ease of...

Using Electrodermal Activity

to Recognize Ease of

Engagement in Children

during Social Interactions

Javier Hernandez Ivan Riobo

Agata Rozga

Gregory D. Abowd

Rosalind W. Picard

Expeditions in

Computing

3

Can we characterize qualitative

aspects of children’s social

engagement with wearable biosensors?

Contents • Background

• Data Collection

• Response Characterization

• Engagement Recognition

• Conclusions

4

Background

Engagement Measurement

6

Self-reports

Fast

Direct

Subjective

Recall problems

Disruptive

Not adequate for children

Engagement Measurement

7

Self-reports

Fast

Direct

Subjective

Recall problems

Disruptive

Not adequate for children

Continuous

Widely used for faces

Time intensive

Costly

Training

Coders

Engagement Measurement

8

Self-reports Physiology

Less disruptive

Objective

Difficult to analyze

Instrumentation

Unfamiliar

Fast

Direct

Subjective

Recall problems

Disruptive

Not adequate for children

Continuous

Widely used for faces

Time intensive

Costly

Training

Coders

Engagement Measurement

9

Physiology Wedel & Pieters, 2000

LaBarbera, Tucciarone, 1995

Silveira et al, 2013

Hernandez et al 2013

Lang, 1990

Social Engagement

10

Interactional Synchrony “Coordination of behaviors between individuals during social interactions.”

Physiological linkage

Levenson & Gottman , 1983

30 married couples

Marci et al, 2007

20 patient-therapist dyads

Data

Collection

12 Project: http://cbs.gatech.edu

Computational Behavioral Science Modeling, Analysis, and Visualization of Social and Communicative Behavior

Dataset: http://cbi.gatech.edu/mmdb

Good Indicator

• Arousal

• Cognitive Load

• Wireless

• Comfortable

• 32 Hz

• 4 sensors

Limitations

• Specificity

• Artifacts

Electrodermal Activity

Affectiva QTM

Social Interaction Greet Ball Play Book Hat Tickle

5 scripted activities with an adult

External coder

“Amount of effort required to get child’s attention” Ratings for each Activity

• 0 – little effort

• 1 – some effort

• 2 – extensive effort 14

Social Interaction

23 sessions exclude due to artifacts

51 clean sessions 24 males, 27 females

Average age: 21 months (STD: 5.23)

Average duration: 2.72 minutes (STD: 1.02)

Engagement groups

• Easier to engage (N = 29)

• Harder to engage (N = 22)

Greet Ball Play Book Hat Tickle

15

EDA Response

Characterization

Preprocessing 1. Normalize EDA values (Lykken, 1971)

2. Reduce noise (Hanning filter, 1 sec. window)

3. Extract tonic and phasic EDA (Benedek and Kaembach, 2010)

17

EDA

Phasic Tonic

Feature Extraction

18

Ton

ic/P

ha

sic

C

om

po

ne

nts

Time

Child Adult

Synchrony Features

Individual Features

Mean

# Peaks

Area under the curve

Position Maximum

Position Minimum

Standard Deviation

Slope

Average Peak Amplitude

Pearson Product-moment Correlation (standard)

Canonical Correlation (explores different representations)

Dynamic Time Warping (allows for temporal flexibility)

L2-norm between means

L2-norm between #peaks

L2 norm between averages of peak amplitudes

Ton

ic/P

ha

sic

C

om

po

ne

nts

Feature Extraction

19

Ton

ic/P

ha

sic

C

om

po

ne

nts

Time

Child Adult

Synchrony Features

Individual Features

Mean

# Peaks

Area under the curve

Position Maximum

Position Minimum

Standard Deviation

Slope

Average Peak Amplitude

Pearson Product-moment Correlation (standard)

Canonical Correlation (explores different representations)

Dynamic Time Warping (allows for temporal flexibility)

L2-norm between means

L2-norm between #peaks

L2 norm between averages of peak amplitudes

Ton

ic/P

ha

sic

C

om

po

ne

nts

Feature Extraction

20

Ton

ic/P

ha

sic

C

om

po

ne

nts

Time

Child Adult

Synchrony Features

Individual Features

Mean

# Peaks

Area under the curve

Position Maximum

Position Minimum

Standard Deviation

Slope

Average Peak Amplitude

Pearson Product-moment Correlation (standard)

Canonical Correlation (explores different representations)

Dynamic Time Warping (allows for temporal flexibility)

L2-norm between means

L2-norm between #peaks

L2 norm between averages of peak amplitudes

Ton

ic/P

ha

sic

C

om

po

ne

nts

Engagement

Recognition

Experiment Details

22

10 fold cross validation for training and testing

http://www.csie.ntu.edu.tw/~cjlin/libsvm/

Easier to Engage VS Harder to Engage

Misclassification Costs: 2^[ 0:1:18]

Support Vector Machines

Linear Kernel

Easier: 29 samples

Harder: 22 samples

Boser et al., 1992

Cortes and Vapnik, 1995

Validation

Sequential Feature Selection

0 20 40 60 80 100 120 1400.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

Avg. Time (seconds)

Avg

. N

orm

aliz

ed

ED

A

0 20 40 60 80 100 120 140 1600.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

Avg. Time (seconds)

Avg

. N

orm

aliz

ed

ED

A

Avg. Normalized EDA

(±Standard Error)

Easier to Engage Children Harder to Engage Children

Average EDA Response

23

Results

50

55

60

65

70

75

80

85

90

IF+SF IF SF IF SF

IF: Individual Features (only from the child’s EDA) SF: Synchronized Features (from the dyad’s EDA)

Best performance 69% Similar performance for both IF and SF IF better from tonic SF better from phasic Best IF: Position Max. from Tonic Best SF: #Peaks from Phasic (>13% than correlation) Tonic and Phasic decomposition improves >6% and interpretability

One feature at a time

Av

erag

e A

UC

s (R

OC

s an

d P

reci

sio

n/R

ecal

l)

24

Results

50

55

60

65

70

75

80

85

90

IF+SF IF SF IF SF

IF: Individual Features (only from the child’s EDA) SF: Synchronized Features (from the dyad’s EDA)

Feature selection improves >11% SF slightly better than IF (>2%) Tonic and phasic equally represented SF and IF are complementary (81%) Some of most selected features: Difference between peaks from phasic STD and DTW from tonic

Combinations of features

Av

erag

e A

UC

s (R

OC

s an

d P

reci

sio

n/R

ecal

l)

25

Conclusions

• 51 child-adult dyads in a structured social interaction

• Evaluation of individual and synchrony features from

tonic and phasic components

• Promising results using SVMs and feature selection (81%)

• Main benefits: automated and scalable • Biggest challenge: 31% of sessions had to be excluded

• Comparison with other modalities

• Goals: better quantify and understand behavior, and

detect developmental delays

Conclusions

27

javierhr@mit.edu

ivan.riobo@gatech.edu

agata@gatech.edu

abowd@gatech.edu picard@mit.edu

Javier Hernandez

Ivan Riobo

Agata Rozga

Gregory D. Abowd

Rosalind W. Picard

Using Electrodermal Activity to

Recognize Ease of Engagement in

Children during Social Interactions

Expeditions in Computing

Project: http://cbs.gatech.edu Dataset: http://cbi.gatech.edu/mmdb