View
11
Download
0
Category
Preview:
Citation preview
Brain Signal Processing and Applications in Brain Machine Interface (BMI)
Tassos Bezerianos, BS,MS,PhD
Head of Biosignal Lab, Dept of Medical Physics
School of Medicine, University of PatrasPATRAS, GREECE
1
1st PhD School on Complexity Sciences July 18‐29, 2011
ContentsBrain Computer Interface (BCI) (Definitions, Features and Limitations)BCI ArchitectureBCI CategoriesBCI ApplicationsBCI at Game MarketFuture Developments
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 2
IntroductionInterface Between Brain and computer
Definition:Wolpaw et al.: “A direct brain‐computer interface is a device that provides the brain with a new, non‐muscular communication and control channel”
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, GreeceBiosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 3
What is BCI (in General)A BCI enables communication without movement Most BCIs translate your brain’s electrical activity (EEGs) into messages or commands.
BCI Rely on
Mental Activities
Imagine Movement and Emotional
Imaginery
Selective attention (P300 and Steady
State Evoked
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, GreeceBiosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 4
What is NOT a BCIBCI cannot read mindsInterpret mental activityWrite to the brainBiofeedbackProstheticsRetinal or cochlear implantsMedical EEGsEEG or fMRI Lie DetectionNeuromarketingEmployee screeningAttention or fatigue monitors
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, GreeceBiosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 5
Who can use BCIWhy use a BCI if you’re healthy?BCIs:Only provide communication.Provide the same information available via
conventional interfaces.Are exclusive interfaces.Thus are of no practical value to people who canotherwise communicate.
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece6Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 6
Who can use BCIWho are able to use a BCI? People with disabilities of sensorimotor system .Due to a phenomenon named by the most of research groups as “BCI Illiteracy”, about 20% of people are not able to control a conventional BCI.
We Conclude that:Will not attain wider adoption without dramaticimprovements in information transfer rate (ITR).
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece7Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 7
A Definition for BCI and Neuroprosthetics
BCI NEUROPROSTHETICS
A speficificated device directly connected to any part of nervous
system
Connect the Brain with a Computer
System
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, GreeceBiosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 8
Different Kind of Subjects
Different Kind of
Recordings
One or more signal Processing
Unit (or Combination)
Feature/Classifier Selection
ArchitectureFocus on the Basic Components
9Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 9
Stimulus, Visual/Auditory Evoked Potentials
Command Execution
A BCI Example
Diagram of the BCI developed by Miguel Nicolelis and colleagues for use on Rhesus monkeys
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 10
How BCI works IThere are several brain areas with distinct functionality which is known.
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece
PRESS cCLICK HERE
ToPreview The Video
11Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 11
How BCI works IIFor each BCI Implementation we are rely on the corresponding Brain region.
z
Imagine Movement and Emotional
Imaginery
Selective attention (P300 and Steady
State Evoked
Mental Activities
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece12Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 12
How BCI works IIIBrain Anatomy and Fuctions
For each BCI Implementation we rely on the corresponding Brain region.E.g. Finger Movement results in Event Related Desynchronization, while foot movement Event Related Synchronization at specific Frequency band (mu Rhythm) over the hand area
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece13Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 13
The beginning of the BCI
14
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 51, NO. 6, JUNE 2004
Toward a Direct Brain Interface Based on HumanSubdural Recordings and Wavelet‐Packet Analysis
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 14
Functionally event results
ERD(Event-related
desynchronization)
ERS(Event-related
synchronization)
ERP(Event-related
potentials)
not phase-locked not phase-locked phase-locked
decreaseHand movement: in Mu rhythm
(9-13 Hz)
Closing eyes and relaxation: 12 Hz)-in alpha (9increase
15Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 15
Event related Synchronization/ Desynchronization
Nitish Thakor 16Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 16
Blind Source Separation‐Motor Imaginary Enhacement
17Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 17
BCI Categories ‐Formally
18Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 18
BCI Categories
ECOG Microelectrodes Intracortical
SUA, MUA, LFPsEEG
NON INVASIVE SYSTEMS
INVASIVE SYSTEMS
19Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 19
Brief Introduction to Electroencephalography (EEG)
20
Electroencephalography (EEG)EEG Records the current flow in cortical areas, tracking tiny electrical impulses that caused by brain cells communication.
21Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 21
EEG Recording
Frequency Rage 0‐ 40Hz (Due to lowpass filter effect of the scalpand intervening tissues)
No Localized (spatial Resolution of few centimeters)
Amplitude Signals 10–20 microVolts
NON‐ INVASIVE
SUMMARYBOARD
22
Objectives of EEG studyThe EEG is used in the evaluation of brain disorders. Most commonly it is used to show the type and location of the activity in the brain during a seizure. It also is used to evaluate people who are having problems associated with brain function. These problems might include confusion, coma, tumors, long‐term difficulties with thinking or memory, or weakening of specific parts of the body (such as weakness associated with a stroke).An EEG is also used to determine brain death. It may be used to prove that someone on life‐support equipment has no chance of recovery.
23Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 23
CharacteristicsThe EEG is typically described in terms of rhythmic activity. The rhythmic activity is divided into bands by frequency: Delta, Theta, Alpha, Beta and Gamma.
Frequency Location
Delta < 4Hz Frontally in adults, posteriorly in children; high amplitude waves Adults slow wave sleep; Babies
Theta 4 – 7Hz Found in locations not related to task at hand
Young children; Drowsiness or arousal in older childen and adults; idling
Alpha 8 – 12Hz
posterior regions of head, both sides, higher in amplitude on dominant side. Central sites (c3‐c4) at rest .
Relaxed/reflecting; Closing the eyes
Beta 12 – 30 Hzboth sides, symmetrical distribution, most evident frontally; low amplitude waves
Alert/Working; Active busy or anxious thinking, active concertration
Gamma > 30Hz Somatosensory cortexShort term memory matching of recognized objects, sounds or tactile sensations; Cross‐modal sensory processing
24Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 24
Delta waves < 4Hz
Theta waves 4 – 7Hz
Alpha waves 8 – 12Hz
Beta waves 12 – 30 Hz
Gamma waves > 30Hz
EEG rhythmic activity
25Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 25
RecordingsEEG
An EEG recording net (Electrical Geodesics, Inc. ) being used on a participant in a brain wave study.
Montages•Bipolar •Referential•Average reference•Laplacian
EEG Electrode Positions
26Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 26
1 second, 1 Channel Recording
EEG Recording
An EEG recording net (Electrical Geodesics, Inc. ) being used on a participant in a brain wave study.
EEG Electrodes The Neuroscan SynAmps2amplifier, power supply,
and 70‐channel headbox (left)
27Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 27
EEG BCI – Recent Progress
Brice Rebsamen, et al., Controlling a wheelchair indoors using thought, Intelligent Systems, 2007.
Wireless EEG systemBCI controlled wheelchair
Chin‐Teng Lin, et al., Noninvasive Neural Prostheses Using Mobile and Wireless EEG, Proceedings of the IEEE, 2008
Gerolf Vanacker et al., Context‐based filtering for assisted brain‐actuated wheelchair driving EEG, Computational Intelligence and Neuroscience, 2007
28Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 28
EEG BCI ‐ Non‐invasiveBCI controlled FES
BCI combined with Virtual Reality
Leeb, R., et al., A tetraplegic patient controls a wheelchair in virtual reality, BRAINPLAY 2007
Pfurtscheller, et al., EEG‐based asynchronous BCI controls functional electrical stimulation in a tetraplegic patient, Eurasip Journal on Applied Signal Processing, 2005.
29Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 29
EEG BCI Visual Evoked Potentials Applications
Brain to Brain Communication
1010101110011 1010101110011
30Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 30
EEG BCIApplication for control a robotic hand
31Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 31
BCI Categories
ECOG Microelectrodes IntracorticalSUA, MUA
EEG
NON INVASIVE SYSTEMS
INVASIVE SYSTEMS
32Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 32
ECoG Recording
Frequency Rage 0‐ 200 Hz (reported usefull information 300Hz‐6Khz
More Localized than EEG (spatial Resolution of milimeters)
ECoG provides higher amplitude signals with values in the rangeof 50–100 microVolts compared to 10–20microVolts for EEG
INVASIVE
SUMMARYBOARD
33
ECoG electrodes‐ Platinum‐ Diameter: 4mm‐ Exposed electrode area diameter: 2.3mm‐ Grids of 8x8, 2x8, 4x5, etc.‐ Strips of 2‐16‐ Depth electrodes reach deeper structures (e.g.,
hippocampus)
34Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 34
MicroECoG‐ Higher spatial resolution‐ 0.04mm microECoG depth electrodes
35Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 35
ECoG InstrumentationHeadbox:
70 input channels24 bit A/D conversionActive noise cancellation110 dB CMRR10 GΩ input impedance
‐ System Unit: Synchronization of samplingTransmits data and trigger information to computer20KHz maximum sampling rate
‐ Power Unit: Isolated power supply necessaryfor patient protection
36Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 36
Use of ECoG for identification of functional brain areas
ECoG stimulations: determine critical location by disrupting the function.
ECoG recordings: mapping endogenous cortical function, reflecting normal cortical function.
37Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 37
ECoG Recording
38Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 38
Low frequency decrease in power (inhibitory) and high frequency increase with activity
Kai Miller et.al., 2007, J Neuroscience
Functional Brain mapping using ECoG
39Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 39
Functional Brain mapping using ECoG
Spectogram from a single electrode for 15s hand movement vs. 10s baseline. Decrease in mu rhythm in hand movement
Kai Miller et.al., 2007, J Neuroscience40Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 40
Offline hand motor area mapping. The bar plots indicate the sum ofsuprathreshold activity for each electrode.
ECoG Based BCI system
41Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 41
Example of ECoG based BCI system
Finger flexion
ECoG features related to finger flexion
Brain signal changes between rest and thumb movement for subjects
J Kub´anek et.al., 2009, J Neural Eng 42Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 42
Example of ECoG based BCI system
Actual and decoded movement trajectories
Relationship of brain areas with thumb flexion
J Kub´anek et.al., 2009, J Neural Eng
Interdependence of actual and decoded finger flexion. (thumb: first row/column, little finger: last row/column)
Discrete classification of finger movements. (from left to right: thumb, index, middle, ring and little finger)
43Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 43
Event Related Task Detected by ECoGMotor Imaginery
44Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 44
BCI Categories
ECOGMicroelectrodes IntracorticalSUA, MUA
EEG
NON INVASIVE SYSTEMS
INVASIVE SYSTEMS
45Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 45
Microelectrode Recording
Local Field Potentials Frequency Rage 0‐ 300 Hz. Well Spatial Localized.
Single and Multi Unit Activity (usually threshold 350Hz‐3500Hz)
INVASIVE
SUMMARYBOARD
46
Overview of all Methods
Gerwin Schalk et al: Sensor Modalities for Brain‐Computer Interfacing. 47Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 47
Microelectrode RecordingsMicroelectrode recordings from within the brain represent activity from one or multiple neurons.These systems use:1. firing rates of individual or multiple neurons or2. or the overall neuronal activity (local field potentials (LFPs)) of multiple neurons recorded within the brain.
48Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 48
Microelectrode RecordingsHowever, the stability of recordings from electrodesimplanted within the brain is currently uncertain because electrodes are subject to different tissue responses that lead to encapsulationAlso, tiny movements of theelectrodes can move them away from individual neurons
MOSTLY LFPs ARE USED
49Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 49
Microelectrodes
50Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 50
Microelectrodes
51Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 51
Cell RecordingWhat we measure?
52Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 52
Multi‐ Single Unit Activity and Local Field Potentials
Spike ClusteringSpike Sorting
Spike Sorting
53Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 53
Typical Example of Animal SUA Activity Recording
~15msecond
EEG
Stimulus
Spikes‐MUA
54Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 54
Decoding spoken words using local field otentials recorded from the cortical surface
Spencer Kellis1, Kai Miller2, Kyle Thomson3 Richard Brown1, Paul House and Bradley Greger
55Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 55
An example of BCIThe four components of a closed‐loop, neural interface system: (1) a recording array that extracts neuralsignals, (2) a decoding algorithm that translates these neural signals into a set of command signals, (3) anoutput device that is controlled by these command signals, and (4) sensory feedback in the form of vision andpotentially other sensory modalities. Transparent head image is courtesy of c iStockphoto.com/KiyoshiTakahase Segundo.
56Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 56
Recording Technology
SfN Spike and ECoG Workshop g.tec
57Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 57
BCI based on Implants
58Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 58
BCI Implant and Muscle Stimulator
Mini‐Symposium Biomimetic Brain Machine Interfaces for the Controlof Movement Andrew H. Fagg,1 Nicholas G. Hatsopoulos,3,4,5 Victor de Lafuente,6 Karen A. Moxon,7 Shamim Nemati,2 James M. Rebesco,8 Ranulfo Romo,6 Sara A. Solla,8,9 Jake Reimer,3 Dennis Tkach,4 Eric A. Pohlmeyer,8,10 and Lee E. Miller8,10a
59Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 59
SUA, MUA and LFP RecordingsA real recording from Dr. N. Hatsopoulos Laboratory during an experiment similar to that is depicted in the figure above. PRESS
CLICK HERE
ToPreview The Video
60Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 60
Hardware and recording technology
Neurochip, Wikipedia
A wireless multi‐channel neural amplifier for freely moving animals
61Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 61
Use Dissolvable films
Dissolvable films of silk fibroin for ultrathinconformal bio‐integrated electronicsDae‐Hyeong Kim and Jonathan Viventi et al.*
62Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 62
Bio Integrated Electronics
Dissolvable films of silk fibroin for ultrathinconformal bio‐integrated electronicsDae‐Hyeong Kim and Jonathan Viventi et al.*
63Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 63
Other BCI Applications
Rats implanted with BCIs in Theodore Berger's experiments
64Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 64
Applications‐MultimediaRecently a system for rapid Image retrieval were proposed.
65Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 65
Other BCI ApplicationsDirect Neural Control of Anatomically Correct Robotic Hands‐Interface Technologies‐‐Control Strategies: Population Decoding
66Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 66
Games based on EEG BCI
67Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 67
The Future In BCIThe question now is:
“What would be the futureFor Brain Computer and Brain Machine Interface?”
New approachesFully implantable BMI.
68Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 68
Memristors A new approach to artificial Inteligence?
An array of 17 purpose-built oxygen-depleted titanium dioxide memristors built at HP Labs, imaged by an atomic force microscope. The wires are about 50 nm, or 150 atoms, wide.[1] Electric current through the memristors shifts the oxygen vacancies, causing a gradual and persistent change in electrical resistance.[2]
69Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 69
MoNETA: A Mind Made from Memristors
70Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 70
MoNETA: A Mind Made from Memristors
71Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 71
Hybrid BCIHYBRID BCI SYSTEMSSIMULTANEOUS ERD/SSVEP BCI TO IMPROVE ACCURACY
COMBINING EYE GAZE AND ERD BCI
“Is this Enough?”
72Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 72
Recent Activities of our Laboratory
October 20105th International Summer School on Emerging Technologies in Biomedicine “High Throughput Communication between Brain and Machines(http://heart.med.upatras.gr/school2010)
73Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 73
Summer School,Invited Speakers
Fabio BabiloniDepartment of Human Physiology and Pharmacology,Biophysics Interest Group, University of Rome "La Sapienza", Italy
Anastasios BezerianosDepartment of Medical Physics, School of Medicine,University of Patras, GreeceNicho Hatsopoulos
Dept. of Organismal Biology & Anatomy, Committees on Computational Neuroscience and Neurobiology, Center for Integrative Neuroscience and Neuroengineering Research (CINNR) University of Chicago , US
74Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 74
Summer School,Invited Speakers
Andreas IoannidesAAI Scientific Cultural Services Ltd, Nicosia, Cyprus
Christopher J. JamesProfessor of Healthcare Technology and Director of a the Institute of Digital Healthcare, University of Warwick, UK
Jürgen KurthsProfessor and Chair of Nonlinear Dynamics, Institute for Physics, University of Potsdam, DE
Pedro LarrañagaProfessor at the Department of Artificial Intelligence at the Polytechnic University of Madrid.
75Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 75
Summer School,Invited Speakers
Nikos LogothetisDirector of the Dept. "Physiology of Cognitive Processes" at the Max Planck Institute for
Biological Cybernetics (MPIK), in Tübingen, DEFivos PanetsosProfessor, Escuela Universitaria de Óptica, Universidad Complutense de MadridPanos M. Pardalos
Distinguished Professor Director, Center for Applied Optimization, Department of Industrialand Systems Engineering, University of Florida
76Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 76
Summer School,Invited Speakers
Selma SupekProfessor of Physics, Department of Physics, University of Zagreb, Croatia
Nitish ThakorThe JHU Biomedical InstrumentationLabSchool of Medicine Johns Hopkins University, Baltimore, US
Marcin ByczukInstitute of Electronics, Medical ElectronicsDivision, Technical University of Lodz, PolandGunther Krausz – g‐tec
Guger Technologies OG, Graz, Austria
77Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 77
Experiments‐ Demonstration
Two Experiments Took Place1. Steady state Evoked Potentials2. Visual Evoked Potentials P300 Based SpellerThe procedure is aparted from the three folowing steps
78
TRAININGPREPARATION
OF THE SUBJECT
TESTING
Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 78
Expetiments‐ Demonstration
79Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 79
P300 TRAIN P300 TEST and SSEP TRAIN
SSEP TEST
80
Local OrganizingCommittee
Chairman,T. Bezerianos
SUMMER SCHOOL 2010 FAMILY
THANK YOU
Biosignal Lab Dept of Medical Physics School of Medicine, University of Patras2010
81
Recommended