EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

Department of Aerospace Vehicles Design and Control

ROY, RAPHAËLLE N.

Associate Professor (tenure track)

EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO NEUROERGONOMICS

Workshop Neurophénoménologie, Imagerie Motrice et Performance Sportive, CLLE, 04/11/19 [email protected]

ISAE-SUPAERO 2019 1

Common measures:

Subjective (questionnaires)

Objective behavioral (performance: accuracy and response time; oculomotor behavior)

Objective peripheral (autonomous nervous system, e.g. cardiac)

INDIRECT

Neuroergonomics

Faros

Tobii

ISAE-SUPAERO 2019 2

New cerebral imaging methods for direct measurements of the CNS, and possibly online in ecological settings/real life.

• Neuroscience applied to the study of humans at work • Direct assessment • New approach

Neuroergonomics

Neuroergonomics

ISAE-SUPAERO 2019 3

Neuroergonomics

TraditionalNeuroimaging

WearableNeuroimaging

MobileNeuroimaging

Ultra-mobileNeuroimaging

Highlycontrolled&restrictedsettings

Low-fidelitysimulator/officesettings

High-fidelitysimulator/realisticsettings

Actualtask/realsettings

NeuroergonomicResearchSpectrum

FutureNeuroimagingContinuous&ubiquitousindailylife

Dehais & Ayaz (2019). In Neuroergonomics.

ISAE-SUPAERO 2019 4

Neuroergonomics

MENTAL STATES

(e.g. mental fatigue, workload, attentional and

affective states)

Fundamental end: Understanding humans at

work

Applied end:

Improve both safety & performance

Fixed offline/online

tool complimentary to subjective and behavioral

measures

Adaptive online

interaction/environment adaptation

2nd focus:

Closing the loop

1st focus:

MSM on single operators & dyads

ISAE-SUPAERO 2019 5

Mental State Monitoring

How to perform mental state monitoring on single operators and dyads?

Mental states: cognitive & affective

Estimation

ISAE-SUPAERO 2019 6


Signal acquisition

Preprocessing Feature extraction Translation

Processing pipeline

Expert, manager

Complimentary tool

EEG, ECG, eye-tracking, …

Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …

Classification, regression, …

ISAE-SUPAERO 2019 7


Signal acquisition


Processing pipeline




1st step: MS characterization based on the

features, using statistical analyses & ‘expert eye’

Expert, manager

Complimentary tool

ISAE-SUPAERO 2019 8

Inattentional deafness

Mental State Monitoring single operator

ISAE-SUPAERO 2019 9



Goal: Elicit inattentional deafness in a full motion simulator due to working memory limit

or visual dominance? & processing stage?

Experimental design:

• Control condition: Level flight with visibility & good weather

• Stress condition: Landing task with no visibility, winds, alarms & smoke in the cockpit

• Secondary task: classical auditory oddball paradigm (280 pure tones; 20% of targets:

70; 1100 Hz vs 100 Hz).

Dehais, Roy & Scannella, S. (2019). Inattentional deafness to auditory alarms: Inter-individual differences,

electrophysiological signature and single trial classification. Behavioural brain research, 360, 51-59.

Frédéric Dehais, Sébastien Scannella

ISAE-SUPAERO 2019 10



Before main task:

N-back task Spatial audiovisual conflict task.




Data acquisition

- 13 healthy male pilots (mean age = 26.3 years, SD = 5.2; flight experience =

81.1 h, SD = 43.8), students at ISAE-SUPAERO

- EEG Biosemi 32 active, Ag/AgCl, 512 Hz, reference & ground on the mastoids




Behavioral results

Alarm detection accuracy: Control 99% / Stress 53.61% inattentional

deafness

Scenario effect over the oddball auditory target detection.

(t = 15.73, p < 0.001,

Cohen’s d=-6.44)


Oddball accuracy function of…


p < 0.05



Target Hit vs Miss Control vs Stress

EEG: Event-related potentials (elicited by the sounds)


Cooperation

Mental State Monitoring operator dyads

Quentin Chenot, Kevin Verdière, Christophe Lounis

• Technical challenge in itself • Interest in aeronautics Identical markers in multi-operator settings? Impact/interaction cooperation & workload



Cooperation & workload, MATB, cardiac synchrony

Verdière, Albert, Dehais & Roy (submitted). IEEE Transactions Human-Machine Systems

156760790.MOV





Results: Subjective, behavioural and cardiac measures

• NASA-TLX : PM load higher in (p <

10−3), same for PF (p<10−3).

• Performance drop in HW for the PF (p < 10−3), and the PM (p < 0.01).

• Less cooperation in HW (number

of cross-clicks) PF (p < 0.05) and PM (p < 0.01).

• HR higher and HRV lower in HW for the PF (resp. p < 0.05 et p < 0.01).



Cardiac synchrony new method (can be computed online with only few data, easy to interpret)

Coincidence detection [Albert_2015] Temporally localized effects

δ = 20 ms

Verdière, Albert, Dehais & Roy (submitted). IEEE Transactions Human-Machine Systems



Signal acquisition


Processing pipeline




2nd step: automated MS characterization based

on the features, using machine learning tools

allows for an unobtrusive & online MSM

Expert, manager

Complimentary tool


Estimation EEG signal during the alarm


Stressing scenario

3 axis motion platform + EEG

Easy & stressing scenarii (25 min each)

Oddball paradigm (2 tones: 75% standards,

25% alarms)

Miss

Hit

Miss

Hit

EEG in flight simulator w/ motion Single-trial classification (500 ms, CCA + LDA)

Hit/Miss: 72,2%, oddball 80%

Dehais, Roy, & Scannella (2019). Behavioural Brain Research.

Frédéric Dehais


Estimation EEG signal before the alarm


Miss Hit

3s 3s

30

ele

ctr

od

es

10 pilots – ENOBIO Dry Electrode system

Source localization: Miss>Hit

Sloreta: Frequency analysis [8-10Hz]

Right Inferior Frontal Gyrus

Right Insula

Right Middle Frontal Gyrus

Attentional Bottleneck Tombu et al (11)

Dehais, Rida, Roy, Iversen, Mullen & Callan (2019). IEEE SMC.

Supervised Dictionary Learning Power in 5 frequency bands (δ, θ, α, β and γ)

Inter-subject classification 67% in average


Closing the loop

How to close the loop by injecting this assessment into a control loop with a decisional system unit in order to enhance the human-system interaction?

Mental states: cognitive & affective

Estimation

Feedback

Decision


Necessary ingredients:

• Machine learning • Planification tools Direct and online measurements of subtle incapacitations Passive Brain-Computer Interfaces / neuroadaptive technology / biocybernetical loop [Fairclough2009; Zander2010]

Caroline Chanel & Nicolas Drougard

Decisional system

System - Human(s) - Artifical agent(s) x Performance

Goal Decision

policy

Closing the loop

Sensors (EEG,

fNIRS, etc)

Feature extraction

Classification


The policy can adapt:

• The human/artificial agent interaction e.g. autonomy level of the artificial agent • The interface (dealt with by an artificial agent) e.g. task difficulty

Closing the loop

Gateau, Chanel, Le & Dehais. (2016) IROS.

Online UAVs behavior adaptation depending on operator’s availability (eye-tracking)

MOMDP model to drive the

adaptive behavior of the entire

artificial agents and system


Closing the loop

Performance increase on the secondary task with online gaze monitoring-based adaptation!

Gateau, Chanel, Le & Dehais. (2016) IROS.


Using physiological data one can perform:

Conclusion

MS characterization

Offline use Closed systems

that adapt to the operator’s state

Online use

Challenges/Studies in progress / Outlook

• Robustness of both markers & pipelines (inter-subject variability, sessions, tasks, and ecological settings)

• Ethics, acceptability, …

28

THANKS

SAVE THE DATE: 19th of December at ISAE-SUPAERO

10:30am: seminars by Pr S. Fairclough, Dr A.-M. Brouwer and Dr F. Lotte

2pm: PhD defense of Kevin Verdière

Documents

EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO