Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Department of Aerospace Vehicles Design and Control
ROY, RAPHAËLLE N.
Associate Professor (tenure track)
EEG-BASED MENTAL STATE MONITORING: INTRODUCTION TO NEUROERGONOMICS
Workshop Neurophénoménologie, Imagerie Motrice et Performance Sportive, CLLE, 04/11/19 [email protected]
ISAE-SUPAERO 2019 1
Common measures:
Subjective (questionnaires)
Objective behavioral (performance: accuracy and response time; oculomotor behavior)
Objective peripheral (autonomous nervous system, e.g. cardiac)
INDIRECT
Neuroergonomics
Faros
Tobii
ISAE-SUPAERO 2019 2
New cerebral imaging methods for direct measurements of the CNS, and possibly online in ecological settings/real life.
• Neuroscience applied to the study of humans at work • Direct assessment • New approach
Neuroergonomics
Neuroergonomics
ISAE-SUPAERO 2019 3
Neuroergonomics
TraditionalNeuroimaging
WearableNeuroimaging
MobileNeuroimaging
Ultra-mobileNeuroimaging
Highlycontrolled&restrictedsettings
Low-fidelitysimulator/officesettings
High-fidelitysimulator/realisticsettings
Actualtask/realsettings
NeuroergonomicResearchSpectrum
FutureNeuroimagingContinuous&ubiquitousindailylife
Dehais & Ayaz (2019). In Neuroergonomics.
ISAE-SUPAERO 2019 4
Neuroergonomics
MENTAL STATES
(e.g. mental fatigue, workload, attentional and
affective states)
Fundamental end: Understanding humans at
work
Applied end:
Improve both safety & performance
Fixed offline/online
tool complimentary to subjective and behavioral
measures
Adaptive online
interaction/environment adaptation
2nd focus:
Closing the loop
1st focus:
MSM on single operators & dyads
ISAE-SUPAERO 2019 5
Mental State Monitoring
How to perform mental state monitoring on single operators and dyads?
Mental states: cognitive & affective
Estimation
ISAE-SUPAERO 2019 6
Mental State Monitoring
Signal acquisition
Preprocessing Feature extraction Translation
Processing pipeline
Expert, manager
Complimentary tool
EEG, ECG, eye-tracking, …
Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …
Classification, regression, …
ISAE-SUPAERO 2019 7
Mental State Monitoring
Signal acquisition
Preprocessing Feature extraction Translation
Processing pipeline
EEG, ECG, eye-tracking, …
Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …
Classification, regression, …
1st step: MS characterization based on the
features, using statistical analyses & ‘expert eye’
Expert, manager
Complimentary tool
ISAE-SUPAERO 2019 8
Inattentional deafness
Mental State Monitoring single operator
ISAE-SUPAERO 2019 9
Inattentional deafness
Mental State Monitoring single operator
Goal: Elicit inattentional deafness in a full motion simulator due to working memory limit
or visual dominance? & processing stage?
Experimental design:
• Control condition: Level flight with visibility & good weather
• Stress condition: Landing task with no visibility, winds, alarms & smoke in the cockpit
• Secondary task: classical auditory oddball paradigm (280 pure tones; 20% of targets:
70; 1100 Hz vs 100 Hz).
Dehais, Roy & Scannella, S. (2019). Inattentional deafness to auditory alarms: Inter-individual differences,
electrophysiological signature and single trial classification. Behavioural brain research, 360, 51-59.
Frédéric Dehais, Sébastien Scannella
ISAE-SUPAERO 2019 10
Inattentional deafness
Mental State Monitoring single operator
Before main task:
N-back task Spatial audiovisual conflict task.
ISAE-SUPAERO 2019 11
Inattentional deafness
Mental State Monitoring single operator
Data acquisition
- 13 healthy male pilots (mean age = 26.3 years, SD = 5.2; flight experience =
81.1 h, SD = 43.8), students at ISAE-SUPAERO
- EEG Biosemi 32 active, Ag/AgCl, 512 Hz, reference & ground on the mastoids
ISAE-SUPAERO 2019 12
Inattentional deafness
Mental State Monitoring single operator
Behavioral results
Alarm detection accuracy: Control 99% / Stress 53.61% inattentional
deafness
Scenario effect over the oddball auditory target detection.
(t = 15.73, p < 0.001,
Cohen’s d=-6.44)
ISAE-SUPAERO 2019 13
Oddball accuracy function of…
Mental State Monitoring single operator
p < 0.05
ISAE-SUPAERO 2019 14
Mental State Monitoring single operator
Target Hit vs Miss Control vs Stress
EEG: Event-related potentials (elicited by the sounds)
ISAE-SUPAERO 2019 15
Cooperation
Mental State Monitoring operator dyads
Quentin Chenot, Kevin Verdière, Christophe Lounis
• Technical challenge in itself • Interest in aeronautics Identical markers in multi-operator settings? Impact/interaction cooperation & workload
ISAE-SUPAERO 2019 16
Mental State Monitoring operator dyads
Cooperation & workload, MATB, cardiac synchrony
Verdière, Albert, Dehais & Roy (submitted). IEEE Transactions Human-Machine Systems
ISAE-SUPAERO 2019 17
Mental State Monitoring operator dyads
ISAE-SUPAERO 2019 18
Mental State Monitoring operator dyads
Results: Subjective, behavioural and cardiac measures
• NASA-TLX : PM load higher in (p <
10−3), same for PF (p<10−3).
• Performance drop in HW for the PF (p < 10−3), and the PM (p < 0.01).
• Less cooperation in HW (number
of cross-clicks) PF (p < 0.05) and PM (p < 0.01).
• HR higher and HRV lower in HW for the PF (resp. p < 0.05 et p < 0.01).
ISAE-SUPAERO 2019 19
Mental State Monitoring operator dyads
Cardiac synchrony new method (can be computed online with only few data, easy to interpret)
Coincidence detection [Albert_2015] Temporally localized effects
δ = 20 ms
Verdière, Albert, Dehais & Roy (submitted). IEEE Transactions Human-Machine Systems
ISAE-SUPAERO 2019 20
Mental State Monitoring
Signal acquisition
Preprocessing Feature extraction Translation
Processing pipeline
EEG, ECG, eye-tracking, …
Filtering, signal conditioning, … HR, HRV, blinks, band power, ERPs, …
Classification, regression, …
2nd step: automated MS characterization based
on the features, using machine learning tools
allows for an unobtrusive & online MSM
Expert, manager
Complimentary tool
ISAE-SUPAERO 2019 21
Estimation EEG signal during the alarm
Mental State Monitoring single operator
Stressing scenario
3 axis motion platform + EEG
Easy & stressing scenarii (25 min each)
Oddball paradigm (2 tones: 75% standards,
25% alarms)
Miss
Hit
Miss
Hit
EEG in flight simulator w/ motion Single-trial classification (500 ms, CCA + LDA)
Hit/Miss: 72,2%, oddball 80%
Dehais, Roy, & Scannella (2019). Behavioural Brain Research.
Frédéric Dehais
ISAE-SUPAERO 2019 22
Estimation EEG signal before the alarm
Mental State Monitoring single operator
Miss Hit
3s 3s
30
ele
ctr
od
es
10 pilots – ENOBIO Dry Electrode system
Source localization: Miss>Hit
Sloreta: Frequency analysis [8-10Hz]
Right Inferior Frontal Gyrus
Right Insula
Right Middle Frontal Gyrus
Attentional Bottleneck Tombu et al (11)
Dehais, Rida, Roy, Iversen, Mullen & Callan (2019). IEEE SMC.
Supervised Dictionary Learning Power in 5 frequency bands (δ, θ, α, β and γ)
Inter-subject classification 67% in average
ISAE-SUPAERO 2019 23
Closing the loop
How to close the loop by injecting this assessment into a control loop with a decisional system unit in order to enhance the human-system interaction?
Mental states: cognitive & affective
Estimation
Feedback
Decision
ISAE-SUPAERO 2019 24
Necessary ingredients:
• Machine learning • Planification tools Direct and online measurements of subtle incapacitations Passive Brain-Computer Interfaces / neuroadaptive technology / biocybernetical loop [Fairclough2009; Zander2010]
Caroline Chanel & Nicolas Drougard
Decisional system
System - Human(s) - Artifical agent(s) x Performance
Goal Decision
policy
Closing the loop
Sensors (EEG,
fNIRS, etc)
Feature extraction
Classification
ISAE-SUPAERO 2019 25
The policy can adapt:
• The human/artificial agent interaction e.g. autonomy level of the artificial agent • The interface (dealt with by an artificial agent) e.g. task difficulty
Closing the loop
Gateau, Chanel, Le & Dehais. (2016) IROS.
Online UAVs behavior adaptation depending on operator’s availability (eye-tracking)
MOMDP model to drive the
adaptive behavior of the entire
artificial agents and system
ISAE-SUPAERO 2019 26
Closing the loop
Performance increase on the secondary task with online gaze monitoring-based adaptation!
Gateau, Chanel, Le & Dehais. (2016) IROS.
ISAE-SUPAERO 2019 27
Using physiological data one can perform:
Conclusion
MS characterization
Offline use Closed systems
that adapt to the operator’s state
Online use
Challenges/Studies in progress / Outlook
• Robustness of both markers & pipelines (inter-subject variability, sessions, tasks, and ecological settings)
• Ethics, acceptability, …
28
THANKS
SAVE THE DATE: 19th of December at ISAE-SUPAERO
10:30am: seminars by Pr S. Fairclough, Dr A.-M. Brouwer and Dr F. Lotte
2pm: PhD defense of Kevin Verdière