Upload
matthew-gray
View
82
Download
2
Embed Size (px)
Citation preview
1
Project Echo and Speech
Recognition Software
Matthew N. GrayNIFS Student Intern – Summer
2016Airspace Operations and
Safety ProgramCrew Systems and Aviation
Operations Branch Mentors: Dr. Angela Harrivel,
Chad Stephens
2
Crew State Monitoring (CSM) – Motivation
• National airspace is becoming increasingly busy and more complex.
• If we could stop just one fatal commercial aviation accident, we could save hundreds of lives and a billion dollars.
• Loss of Control – Inflight (LOC-I)o Largest Category of aircraft-related Fatal Events
• CSM aims to improve pilot operational efficiency during safety-critical operations by: o Improving the human-machine interface in
aircrafto Aiding in pilot attention training 3 Boeing Statistical Summary of Commercial Worldwide Jet
Transport Accidents, 2011. Includes only accidents involving turbofan or turbojet airplanes with max takeoff weight > 60,000 lbs., referenced in the CAST Airplane State Awareness Joint Safety Analysis Team Final Report, June 17, 2014
3
Background – Crew State Monitoring (CSM)
• Data Collectiono fNIRS, EEG, EKG, Resp., GSR, Eye-
tracking, etc.
• Data Synchronizationo MAPPS Software
• Signal Pre-Processingo Filtering, Feature Extraction
• Machine Learningo Classification Algorithmo Model Evaluation
• Real-time State Indicator Displayo High/Low Workload, Distraction,
Inattentional Blindness, Confirmation Bias, etc. Figure courtesy of Charles Liles
and the LaRC Big Data and Machine Information Team
4
Background – AFDC 2.0 and SHARP 1.0
• Augmented Flight Deck Countermeasures (AFDC 2.0)o Seeks to improve human-
machine interaction among safety critical operations of airplanes
• Scenarios for Human Attention Recovery using Psychophysiology (SHARP 1.0)o Supports CAST goals by
measuring crew cognitive states in simulators during safety critical operations SHARP Display
Muse
SmartEye
Spire
Empatica E4
BIOPAC fNIR100B
EEG
fNIRS
Heart Rate (PPG)GSR
Temp.Accel.
Respiratory Rate
Eye-tracking
Project Echo
Sensors
Data Acquisitio
n
Data Synchronizat
ion
MAPPS or MAPPS Equivalent:• Lab Streaming Layer (LSL)• iMotions• XTObserver
Signal Processin
g
Classification
Algorithm
Hidden Markov Models (HMMs)
Neural Netowrks (NNs)
Machine Learning Display
Indicate Cognitive State:• High/Nominal Workload• Channelized Attention• Diverted Attention• Confirmation Bias• Inattentional Blindness
6
Project Echo – SmartEye® Eye-Tracking System
• Hardware Setup• Camera Calibration• Building World Model• Head Profile Creation• Gaze Calibration• Real-time data transfer
(future)
SmartEye® Setup with corresponding World Model
7
Project Echo – Empatica E4 Wristband
• Measures oHeart Rate – HR (PPG)oGalvanic Skin Response – GSRo Skin Temperatureo Acceleration/Movement
• Lightweight, non-invasive, portable
• Real-time data streaming through Matlabo TCP/IP Connectiono Stored in text file
8
Speech Recognition Software – Motivation
• Talking could affect classification accuracy during runsoMore/varying cognitive
activationo Irregular breathingoMovement
• Automatic labeling of speech vs. other noises for future analysis
Talking
9
Speech Recognition – MFCCs
• Mel Frequency Cepstral Coefficients (MFCCs)oFeature derived from audio signal oRepresents shape of vocal tract
through short-time frequency analysis
oApplies filter to frequency response of signal to emulate human hearing
Humans can distinguish low frequencies easier than high frequencies
Cross-sectional shapes of vocal tracts
Cochlear frequency map showing logarithmic frequency resolution of human hearing
10
Speech Recognition – MFCCs (cont.)
Steps:• Calculate moving window PSD
o Assume ‘stationarity’ at 25ms window sizeso Apply Hamming filter to windowed signal
• Create Mel filter bank (shown to the right)• Multiply each windowed PSD by each
filter in filter bank• Sum powers in each binned and filtered
frequency for each frame• Take log and discrete cosine transform of
summed powers• Produces array of 12 coefficients for
each windowed time series
×
=
11
Future Work
• Acquire data real-time from SmartEye® system
• Parse out data stream from Empatica in Matlab into separate text files
• Synchronize data streams from all devices with MAPPS or MAPPS equivalent (LSL, etc.)
• Incorporate in-house preprocessing and machine learning scripts
• Input MFCCs from audio signals into Hidden Markov Model machine learning classifier
12
Acknowledgements
• Angela Harrivel, PhD• Chad Stephens, PhD Candidate• Kyle Ellis, PhD• Ray Comstock, PhD• Kellie Kennedy, PhD Candidate• Nick Napoli• Katrina Colucci-Chang• Will Hollingsworth• Alex Liang