29
Department of Aerospace Vehicles Design and Control ROY, RAPHAËLLE N. Associate Professor (tenure track) EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO NEUROERGONOMICS Workshop Neurophénoménologie, Imagerie Motrice et Performance Sportive, CLLE, 04/11/19 [email protected]

EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

  • Upload
    others

  • View
    4

  • Download
    0

Embed Size (px)

Citation preview

Page 1: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

Department of Aerospace Vehicles Design and Control

ROY, RAPHAËLLE N.

Associate Professor (tenure track)

EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO NEUROERGONOMICS

Workshop Neurophénoménologie, Imagerie Motrice et Performance Sportive, CLLE, 04/11/19 [email protected]

Page 2: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 1

Common measures:

Subjective (questionnaires)

Objective behavioral (performance: accuracy and response time; oculomotor behavior)

Objective peripheral (autonomous nervous system, e.g. cardiac)

INDIRECT

Neuroergonomics

Faros

Tobii

Page 3: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 2

New cerebral imaging methods for direct measurements of the CNS, and possibly online in ecological settings/real life.

• Neuroscience applied to the study of humans at work • Direct assessment • New approach

Neuroergonomics

Neuroergonomics

Page 4: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 3

Neuroergonomics

TraditionalNeuroimaging

WearableNeuroimaging

MobileNeuroimaging

Ultra-mobileNeuroimaging

Highlycontrolled&restrictedsettings

Low-fidelitysimulator/officesettings

High-fidelitysimulator/realisticsettings

Actualtask/realsettings

NeuroergonomicResearchSpectrum

FutureNeuroimagingContinuous&ubiquitousindailylife

Dehais & Ayaz (2019). In Neuroergonomics.

Page 5: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 4

Neuroergonomics

MENTAL STATES

(e.g. mental fatigue, workload, attentional and

affective states)

Fundamental end: Understanding humans at

work

Applied end:

Improve both safety & performance

Fixed offline/online

tool complimentary to subjective and behavioral

measures

Adaptive online

interaction/environment adaptation

2nd focus:

Closing the loop

1st focus:

MSM on single operators & dyads

Page 6: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 5

Mental State Monitoring

How to perform mental state monitoring on single operators and dyads?

Mental states: cognitive & affective

Estimation

Page 7: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 6

Mental State Monitoring

Signal acquisition

Preprocessing Feature extraction Translation

Processing pipeline

Expert, manager

Complimentary tool

EEG, ECG, eye-tracking, …

Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …

Classification, regression, …

Page 8: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 7

Mental State Monitoring

Signal acquisition

Preprocessing Feature extraction Translation

Processing pipeline

EEG, ECG, eye-tracking, …

Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …

Classification, regression, …

1st step: MS characterization based on the

features, using statistical analyses & ‘expert eye’

Expert, manager

Complimentary tool

Page 9: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 8

Inattentional deafness

Mental State Monitoring single operator

Page 10: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 9

Inattentional deafness

Mental State Monitoring single operator

Goal: Elicit inattentional deafness in a full motion simulator due to working memory limit

or visual dominance? & processing stage?

Experimental design:

• Control condition: Level flight with visibility & good weather

• Stress condition: Landing task with no visibility, winds, alarms & smoke in the cockpit

• Secondary task: classical auditory oddball paradigm (280 pure tones; 20% of targets:

70; 1100 Hz vs 100 Hz).

Dehais, Roy & Scannella, S. (2019). Inattentional deafness to auditory alarms: Inter-individual differences,

electrophysiological signature and single trial classification. Behavioural brain research, 360, 51-59.

Frédéric Dehais, Sébastien Scannella

Page 11: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 10

Inattentional deafness

Mental State Monitoring single operator

Before main task:

N-back task Spatial audiovisual conflict task.

Page 12: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 11

Inattentional deafness

Mental State Monitoring single operator

Data acquisition

- 13 healthy male pilots (mean age = 26.3 years, SD = 5.2; flight experience =

81.1 h, SD = 43.8), students at ISAE-SUPAERO

- EEG Biosemi 32 active, Ag/AgCl, 512 Hz, reference & ground on the mastoids

Page 13: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 12

Inattentional deafness

Mental State Monitoring single operator

Behavioral results

Alarm detection accuracy: Control 99% / Stress 53.61% inattentional

deafness

Scenario effect over the oddball auditory target detection.

(t = 15.73, p < 0.001,

Cohen’s d=-6.44)

Page 14: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 13

Oddball accuracy function of…

Mental State Monitoring single operator

p < 0.05

Page 15: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 14

Mental State Monitoring single operator

Target Hit vs Miss Control vs Stress

EEG: Event-related potentials (elicited by the sounds)

Page 16: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 15

Cooperation

Mental State Monitoring operator dyads

Quentin Chenot, Kevin Verdière, Christophe Lounis

• Technical challenge in itself • Interest in aeronautics Identical markers in multi-operator settings? Impact/interaction cooperation & workload

Page 17: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 16

Mental State Monitoring operator dyads

Cooperation & workload, MATB, cardiac synchrony

Verdière, Albert, Dehais & Roy (submitted). IEEE Transactions Human-Machine Systems

Page 18: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 17

Mental State Monitoring operator dyads

Page 19: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 18

Mental State Monitoring operator dyads

Results: Subjective, behavioural and cardiac measures

• NASA-TLX : PM load higher in (p <

10−3), same for PF (p<10−3).

• Performance drop in HW for the PF (p < 10−3), and the PM (p < 0.01).

• Less cooperation in HW (number

of cross-clicks) PF (p < 0.05) and PM (p < 0.01).

• HR higher and HRV lower in HW for the PF (resp. p < 0.05 et p < 0.01).

Page 20: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 19

Mental State Monitoring operator dyads

Cardiac synchrony new method (can be computed online with only few data, easy to interpret)

Coincidence detection [Albert_2015] Temporally localized effects

δ = 20 ms

Verdière, Albert, Dehais & Roy (submitted). IEEE Transactions Human-Machine Systems

Page 21: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 20

Mental State Monitoring

Signal acquisition

Preprocessing Feature extraction Translation

Processing pipeline

EEG, ECG, eye-tracking, …

Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …

Classification, regression, …

2nd step: automated MS characterization based

on the features, using machine learning tools

allows for an unobtrusive & online MSM

Expert, manager

Complimentary tool

Page 22: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 21

Estimation EEG signal during the alarm

Mental State Monitoring single operator

Stressing scenario

3 axis motion platform + EEG

Easy & stressing scenarii (25 min each)

Oddball paradigm (2 tones: 75% standards,

25% alarms)

Miss

Hit

Miss

Hit

EEG in flight simulator w/ motion Single-trial classification (500 ms, CCA + LDA)

Hit/Miss: 72,2%, oddball 80%

Dehais, Roy, & Scannella (2019). Behavioural Brain Research.

Frédéric Dehais

Page 23: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 22

Estimation EEG signal before the alarm

Mental State Monitoring single operator

Miss Hit

3s 3s

30

ele

ctr

od

es

10 pilots – ENOBIO Dry Electrode system

Source localization: Miss>Hit

Sloreta: Frequency analysis [8-10Hz]

Right Inferior Frontal Gyrus

Right Insula

Right Middle Frontal Gyrus

Attentional Bottleneck Tombu et al (11)

Dehais, Rida, Roy, Iversen, Mullen & Callan (2019). IEEE SMC.

Supervised Dictionary Learning Power in 5 frequency bands (δ, θ, α, β and γ)

Inter-subject classification 67% in average

Page 24: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 23

Closing the loop

How to close the loop by injecting this assessment into a control loop with a decisional system unit in order to enhance the human-system interaction?

Mental states: cognitive & affective

Estimation

Feedback

Decision

Page 25: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 24

Necessary ingredients:

• Machine learning • Planification tools Direct and online measurements of subtle incapacitations Passive Brain-Computer Interfaces / neuroadaptive technology / biocybernetical loop [Fairclough2009; Zander2010]

Caroline Chanel & Nicolas Drougard

Decisional system

System - Human(s) - Artifical agent(s) x Performance

Goal Decision

policy

Closing the loop

Sensors (EEG,

fNIRS, etc)

Feature extraction

Classification

Page 26: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 25

The policy can adapt:

• The human/artificial agent interaction e.g. autonomy level of the artificial agent • The interface (dealt with by an artificial agent) e.g. task difficulty

Closing the loop

Gateau, Chanel, Le & Dehais. (2016) IROS.

Online UAVs behavior adaptation depending on operator’s availability (eye-tracking)

MOMDP model to drive the

adaptive behavior of the entire

artificial agents and system

Page 27: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 26

Closing the loop

Performance increase on the secondary task with online gaze monitoring-based adaptation!

Gateau, Chanel, Le & Dehais. (2016) IROS.

Page 28: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

ISAE-SUPAERO 2019 27

Using physiological data one can perform:

Conclusion

MS characterization

Offline use Closed systems

that adapt to the operator’s state

Online use

Challenges/Studies in progress / Outlook

• Robustness of both markers & pipelines (inter-subject variability, sessions, tasks, and ecological settings)

• Ethics, acceptability, …

Page 29: EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO

28

THANKS

SAVE THE DATE: 19th of December at ISAE-SUPAERO

10:30am: seminars by Pr S. Fairclough, Dr A.-M. Brouwer and Dr F. Lotte

2pm: PhD defense of Kevin Verdière