Human Cognition: Decoding Perceived, Attended, Imagined
Acoustic Events and Human-Robot Interfaces
Slide 2
The Team Adriano Claro Monteiro Alain de Cheveign Anahita Mehta
Byron Galbraith Dimitra Emmanouilidou Edmund Lalor Deniz Erdogmus
Jim OSullivan Mehmet Ozdas Lakshmi Krishnan Malcolm Slaney Mike
Crosse Nima Mesgarani Jose L Pepe Contreras- Vidal Shihab Shamma
Thusitha Chandrapala
Slide 3
The Goal To determine a reliable measure of imagined audition
using electroencephalography (EEG). To use this measure to
communicate.
Slide 4
What types of imagined audition? Speech: Short (~3-4s)
sentences The whole maritime population of Europe and America.
Twinkle-twinkle little star. London bridge is falling down, falling
down, falling down. Music Short (~3-4s) phrases Imperial March from
Star Wars. Simple sequence of tones. Steady-State Auditory
Stimulation 20 s trials Broadband signal amplitude modulated at 4
or 6 Hz
Slide 5
The Experiment 64 channel EEG system (Brain Vision LLC thanks!)
500 samples/s Each trial consisted of the presentation of the
actual auditory stimulus (perceived condition) followed (2 s later)
by the subject imagining hearing that stimulus again (imagined
condition).
Slide 6
The Experiment Careful control of experimental timing.
Perceived...2s... Imagined...2 s x 5... Break... next stimulus 4,
3, 2, 1, +
Slide 7
Data Analysis - Preprocessing Filtering Independent Component
Analysis (ICA) Time-Shift Denoising Source Separation (DSS) Looks
for reproducibility over stimulus repetitions
Slide 8
The hypothesis: EEG recorded while people listen to (actual)
speech varies in a way that relates to the amplitude envelope of
the presented (actual) speech. EEG recorded while people IMAGINE
speech will vary in a way that relates to the amplitude envelope of
the IMAGINED speech. Data Analysis: Hypothesis-driven.
Slide 9
Phase consistency over trials... EEG from same sentence
imagined over several trials should show consistent phase
variations. EEG from different imagined sentences should not show
consistent phase variations. Data Analysis: Hypothesis-driven.
Slide 10
Actual speechImagined speech Consistency in theta (4-8Hz) band
Consistency in alpha (8-14Hz) band
Slide 11
Data Analysis: Hypothesis-driven.
Slide 12
Red line perceived music Green line imagined music
Slide 13
Data Analysis - Decoding
Slide 14
r = 0.30, p = 3e-5r = 0.19, p = 0.01 Londons BridgeTwinkle
Original Reconstruction
Slide 15
Data Analysis - SSAEP
Slide 16
4Hz 6Hz Perceived Imagined Data Analysis - SSAEP
Slide 17
Data Analysis Data Mining/Machine Learning Approaches:
Slide 18
Data Analysis Data Mining/Machine Learning Approaches:
Slide 19
SVM Classifier Input EEG data (channels time) : Concatenate
channels: Group N trials: Input covariance matrix: Class Labels
Predicted Labels
Slide 20
SVM Classifier Results Mean DA = 90% Decoding imagined speech
and music: Mean DA = 90%Mean DA = 87%
Slide 21
Raw EEG Signal (500Hz data) DSS Output (Look for repeatability)
DCT Output (Reduce dimensionality) DCT Processing Chain
Data Analysis Data Mining/Machine Learning Approaches: Linear
Discriminant Analysis on Different Frequency Bands Music vs Speech
Speech 1 vs Speech 2 Music 1 vs Music 2 Speech vs Rest Music vs
Rest - results ~ 50 66%
Slide 24
Summary Both hypothesis drive and machine-learning approaches
indicate that it is possible to decode/classify imagined audition
Many very encouraging results that align with our original
hypothesis More data needed!! In a controlled environment!! To be
continued...