Computational Physiology LabDepartment of Computer Science

University of HoustonHouston, TX 77004

Eustressed or Distressed?Combining Physiology withObservation in User Studies

Avinash WesleyDr. Peggy Lindner (Co-Advisor)

Dr. Ioannis Pavlidis (Advisor)

2/13

Stress Signs

• Peripheral Physiological Measurement of Stress– Adrenergic response

• Elevates heart rate, respiration rate, and blood pressure – Cholinergic response

• Activates sweat glands on fingers and the perinasal area

• Introduction• Methods• Results and Discussion• Acknowledgements

- Stress Mechanism- Motivation- Background

3/13

Physiology and Observation

• Perspiratory responses are – Sympathetic in nature – Non-specific to positive or negative arousal

• Introduction• Methods• Results and Discussion• Acknowledgements

Distress Eustress

- Stress Mechanism- Motivation- Background

4/13

Emotions vs. Performance

• Introduction• Methods• Results and Discussion• Acknowledgements

- Stress Mechanism- Motivation- Background

Arousal

Performance

LOW MEDIUM HIGH

HIGH

Sleep Disorganization

AnxietyAlertness

Optimal

• An important goal in user studies: Study the role of emotions on human performance

• Emotions can be quantified via physiological response

• Physiological responses can be disambiguated via observation

5/13

Perspiration Signal and Observation

• Physiology– Perspiration extraction method using Thermal Imagery [1]

• Observational Annotation of Emotions– Traditionally done using Visual Imagery

• Manual

• Introduction• Methods• Results and Discussion• Acknowledgements

[1] D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis. Imaging facial signs of neurophysiological responses. IEEE Transactions on Biomedical Engineering, 56(2):477–484, 2009.

Courtesy of Science channel

- Stress Mechanism- Motivation- Background

6/13

Region Tracking

• Seven anatomical regions tracked over time by a dynamic template update tracker [2]

• Introduction• Methods• Results and Discussion• Acknowledgements

[2] Y. Zhou, P. Tsiamyrtzis, and I. Pavlidis. Tissue tracking in thermo-physiological imagery through spatio-temporal smoothing. Proc. of the 12th Int. Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2009),5762:1092–1099, 2009.

- Facial Expression Recognition- Field Study

7/13

Pattern Classification

• Feature Vector

• Classifier– Classify five action units (AU1+2, 4, 9, 10, and 12)– Multilayer Perceptron– 10-fold Cross Validation

• Introduction• Methods• Results and Discussion• Acknowledgements

AU 1+2 Inner + Outer Eyebrow Raise

d(x,5): Euclidean distance between ROI-x and 5, (x 5)

- Facial Expression Recognition- Field Study

8/13

Surgical Training

• Surgeon Pool (n=17)– Novices – Experienced

• Tasks– Running string (Task-1)– Pattern cut (Task-2)– Intracorporeal suture (Task-3)

• Dataset: 977 Thermal Clips

• Introduction• Methods• Results and Discussion• Acknowledgements

- Facial Expression Recognition- Field Study

9/13

Validation Results

• Using Thermal Imagery– 244 Facial Expressions – Ground Truth via Visual annotation– Method Accuracy 81.55%

• Introduction• Methods• Results and Discussion• Acknowledgements

* Use of visual images instead of thermal images for display purpose only

* Confusion matrix

- Quantitative Analysis- Qualitative Analysis- Conclusions

10/13

Results From The Field Study

• Distress is inversely related to experience

– EN (Perinasal perspiratory signal on portions of negative emotions)

• Introduction• Methods• Results and Discussion• Acknowledgements

- Quantitative Analysis- Qualitative Analysis- Conclusions

Novice Experienced

11/13

Example Visualizations

Eustress Distress

• Introduction• Methods• Results and Discussion• Acknowledgements

- Quantitative Analysis- Qualitative Analysis- Conclusions

12/13

Conclusions

• The proposed method is – Comprehensive (quantitative and qualitative) – Economical (single imaging modality with no labor)

• Conducted a study design that is applicable to a broad class of Human Machine Interaction

• Future Work– Expand the facial expression set– Apply the method to more field studies

• Detection of pain onset-offset

• Introduction• Methods• Results and Discussion• Acknowledgements

- Quantitative Analysis- Qualitative Analysis- Conclusions

13/13

• Support provided by NSF award # IIS-0812526

• Introduction• Methods• Results and Discussion• Acknowledgements