ACI/HFES, Baltimore, October 1-3, 2007 Sponsored by US Army Research Office SBIR Phase II: Wearable Physiological Sensor Suite For Early Detection Of Cognitive

ACI/HFES, Baltimore, October 1-3, 2007

Experimental Design and Testing of a Multimodal Cognitive Overload Classifier

Leonard J. Trejo Neil J. McDonald Robert Matthews

Quantum Applied Science and Research

Brendan Z. Allison

The Scripps Research Institute

Sponsored by US Army Research Office SBIR Phase II: “Wearable Physiological Sensor Suite For Early Detection Of Cognitive Overload”US Army Aberdeen Test Center: “Remote Neurological Monitoring Program”


Stress, Overload, and Performance


Mental State Estimation

Work-load

MentalFatigue

Non-specificFactors

Engage-ment

GeneralCognitive

Status

Engage-ment

Non-specificFactors

MentalFatigue

Work-load

BiosignalSources


Definitions

• Engagement: selection of a task as the focus of attention and effort

• Workload: significant commitment of processing resources to an engaged task

•Visual, Auditory, Haptic•Psychomotor•Cognitive (memory, executive)

• Overload: task demands outstrip available processing resources

• Mental Fatigue: desire to withdraw attention and effort from an engaged task associated with extended performance (~45 min)

Work-load

MentalFatigue

Non-specificFactors

Engage-ment

GeneralCognitive

Status


Overload Patterns in Multimodal Signals

0 50 100 150 200 250 300 350 400 450

50

100

RT - blue; HRstd - red

0 50 100 150 200 250 300 350 400 4500

5

10

15

Left temporalis EMG - blue; Right temporalis EMG - red

0 50 100 150 200 250 300 350 400 4500

5

10

15

Fz/theta - blue; Pz/alpha - red

0 50 100 150 200 250 300 350 400 4500

100

200

300

vEOG - blue; hEOG - red


Multimodal Classifier Testbed









C2

C1


Engagement/Workload Related EEG Sources

Passive viewing: theta alpha

Engaged 5: theta alpha

10.55 Hz

10.30 Hz5.79 Hz

5.79 Hz

Anterior Cingulate Inferior Parietal Precuneus


Fatigue-Related EEG Sources Black = Alert Red = Mentally Fatigued

Fz Pz

FrontalTheta

ParietalAlpha


Other Sources

Source Effect of Workload

Heart rate Increase

Heart rate variability (and HFQRS) Decrease

Vertical and horizontal EOG (eye movements) Increase

Blinks May decrease for intake

Pupil diameter Increase

Skin conductance, SCR, GSR Increase

EMG (frontalis, temporalis, trapezius) Increase


Experimental Controls

• Task learning• Time of day and time on task• Test day• Food consumption • Neurotoxic effects • Test environment • Inadequate measurement of physiological variance • Inadequate definition of ground truth workload levels:

− Expert analysis and scoring of replayed videos− Logging all user inputs− Measuring reaction times to probes


Validation of Workload Manipulation

NASA - TLX questionnaires

P300


Within-day Results

• Trials 1 and 5: disengaged.

• Trials 2-4, 6-8: engaged increasing numbers of enemies.

• Trials 5-8: engaged and reported status in response to command tones

Trial 2 3 4 5 6 7 81 99.95 100.00 100.00 97.78 100.00 99.89 99.662 87.28 88.94 100.00 74.06 83.56 82.783 68.56 99.95 82.83 88.89 86.164 99.95 82.50 85.22 82.645 100.00 99.84 99.786 76.00 76.087 82.44

Accuracy (% correct)Trial 2 3 4 5 6 7 81 99.95 100.00 100.00 97.78 100.00 99.89 99.662 87.28 88.94 100.00 74.06 83.56 82.783 68.56 99.95 82.83 88.89 86.164 99.95 82.50 85.22 82.645 100.00 99.84 99.786 76.00 76.087 82.44

Accuracy (% correct)

EEG/ERP/ECGEOG/EMG

Engagement and workload algorithms

Real-time alert or advisory signal

Accuracy of only EEG-based classification of engagement or mental workload levels in 18 human subjects performing a first-person shooter simulation.


Day-to-day Results

Model Type Time Range Su

bje

ct

Tes

t P

rop

ort

ion

Co

rrec

t C

lass

1

Tes

t P

rop

ort

ion

Co

rrec

t C

lass

2

Po

siti

ve P

red

icti

ve V

alu

e=

TP

/(T

P+

FP

)

Neg

ativ

e P

red

icti

ve V

alu

e=

TN

/(T

N+

FN

)

Ave

rag

e P

red

icit

ive

Val

ue

5 0.97 0.97 1.00 0.97 0.98

6 0.34 0.97 0.92 0.34 0.63

7 1.00 0.03 1.00 1.00 1.00

13 0.97 0.91 1.00 0.97 0.98

5 0.77 0.33 0.59 0.77 0.68

6 0.21 0.93 0.54 0.21 0.37

7 0.28 0.81 0.53 0.28 0.40

13 0.64 0.56 0.61 0.64 0.62

EngagmentDay 1 predicts

Day 2

WorkloadDay 1 predicts

Day 2


Stabilizing Classifiers

Spectral Normalization

EEG Bandwidth Limiter

EEG Spectral Features

Classifiers

2s ECG Epoch

MV EOG/ECG Regression

Filters

R-wave Detector

PSDEMG

FeaturesRMS20,

Burst Duration, Burst

Frequency…

Innovations in EEG Algorithm Stabilization

ECG

FeaturesR-wave detectHR,

HRV, STD-IBI,

…

EOG

FeaturesRMS20, Blinks, EMs ,

…

AVAS Thresholds

Filters

4- 20s ECG Epoch

Gauges


Stabilized Day-to-day Results

Model Low (0 + 1 Enemy) High (5 Enemies) Low (0 + 1 Enemy) High (5 Enemies)A 1, 2, 3, 7, 8 5, 6, 10, 11 1, 2, 3, 7, 8 5, 6, 10, 11 NoneB 1, 2, 3 5, 6 7, 8 10, 11 PlaceboD 1, 2, 3, 7, 8 5, 6, 10, 11 Day 2: 1, 2, 3 Day 2: 5, 6 MixedE 1, 2, 3, 7, 8 + Day 2: 1, 2 5, 6, 10, 11 Day 2: 1, 2, 3 Day 2: 5, 6 AlcoholF Day 2: 1, 2, 3, 7, 8 Day 2: 5, 6, 10, 11 Day 2: 1, 2, 3, 7, 8 Day 2: 5, 6, 10, 11G Day 2: 1, 2, 3 Day 2: 5, 6 Day 2: 7, 8 Day 2: 10, 11

LOW HIGHS206-A LOW 0.86 0.14

HIGH 0.12 0.88S206-B LOW 0.87 0.13

HIGH 0.16 0.84S206-D LOW 0.74 0.36

HIGH 0.29 0.71S206-E LOW 0.83 0.17

HIGH 0.16 0.84S206-F LOW 0.87 0.13

HIGH 0.14 0.86S206-G LOW 0.85 0.15

HIGH 0.12 0.88

Training Set Test Set

Day 1+Calib. Day 2 vs Day 2 Early

Day1 Early vs Late (also Alcohol vs No-Alcohol)

Day1 All

Day1 Early vs Late

Day1 All vs. Day 2 Early

Day 2 All


Summary

• Biosignals exhibit high sensitivity to mental states, such as engagement, workload, and fatigue

• Accurate biosignal-based models or “gauges” can be developed under controlled conditions and extended to new conditions

• However, cognitive gauges are not very stable over time, due to behavioral, strategic, and physiological variability

• Multimodal models capture a wide range of behavioral and physiological variability, improving robustness of gauges over time and conditions

• Signal processing and computational methods help, but are not enough to yield stable models

• Some recalibration or model adaptation is currently required• We seek ways to stabilize models with a minimum of recalibration

Documents

ACI/HFES, Baltimore, October 1-3, 2007 Sponsored by US Army Research Office SBIR Phase II: Wearable Physiological Sensor Suite For Early Detection Of Cognitive