Upload
zenda
View
61
Download
0
Tags:
Embed Size (px)
DESCRIPTION
BMI Principles . Jose C. Principe University of Florida Adapted from Hayrettin Gürkök , U. of Twente , NL. Literature. Difficulties in Invasive BMIs. BCIs offer an easy entry to research Non invasiveness straight forward data collection Closer to cognition - PowerPoint PPT Presentation
Citation preview
BMI Principles
Jose C. PrincipeUniversity of Florida
Adapted from Hayrettin Gürkök, U. of Twente, NL
Literature
Difficulties in Invasive BMIs
• BCIs offer an easy entry to research – Non invasiveness straight forward data collection– Closer to cognition– Conventional signal processing
• BMIs research infrastructure is much harder– Work with animals (ethics)– Difficult instrumentation– Unclear signal processing
Choice of Scale for Neuroprosthetics
Bandwidth (approximate)
Localization
Scalp Electrodes
0 ~ 80 Hz Cortical SurfaceVolume Conduction3-5 cm
Electro-corticogram (ECoG)
0 ~ 500Hz Cortical Surface0.5-1 cm
Micro Electrodes
0 ~ 500Hz
500 ~ 7kHz
Local Fields1mm
Single Neuron200 mm
footing
two polyimide cables
Electrode Arrays
0 0.01 0.02 0.03 0.04-40
-30
-20
-10
0
10
20
30
40
50
Time (s)
Micr
ovol
ts
J. C. Sanchez, N. Alba, T. Nishida, C. Batich, and P. R. Carney, "Structural modifications in chronic microwire electrodes for cortical neuroprosthetics: a case study," IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2006
Utah array
Brain Gate
Michigan probes
Technical Issues with BMIs
• An implantable BMI requires beyond of state of the art technology:– Ultra low power– Ultra miniaturized– Huge data bandwidth/power form factor– Packaging
28mm
15mm
12mm Thru vias to RX/Power Coil
+
12.5 mm
Coil winding
3.5 mm
50µm pitch Electrodes
Coin Battery(10 x 2.5 mm)
Thru vias to Battery
Supportingscrews
Flexiblesubstrate
TX antenna
ModularElectrodes
Electrodeattachment
sites
IF-IC
RFIC
18 mm
Coil
Battery
PatternedSubstrate
SupportingSubstrate
ElectrodeArray
IC
Flip-chipconnection
Specifications:16 flexible microelectrodes (40 dB, 20 KHz)Wireless (500 Kpulse/sec)2mW of power (72-96 hours between charges)
FWIRE: Florida Wireless Implantable Recording Electrodes
RatPack Low-Power, Wireless, Portable BMIs
• Requirements– Total Weight: < 100g– Battery Powered: Run for 4 hours
• Implantable– Biocompatible– Heat flux: < 50 mW/cm2
– Power dissipation should not exceed a few hundred milliwatts
• Backpack– Small form factor– Speed vs. Low Power
UF PICO SystemPICO system = DSP + Wireless
Generation 3
J.R. Wolpaw et al. 2002
BCI (BMI) bypasses the brain’s normal pathways of peripheral nerves (and muscles)
General Architecture
BMIs: How to put it together?• NeoCortical Brain Areas Related to
Movement Posterior Parietal (PP) – Visual to motor transformation
Premotor (PM) and Dorsal Premotor (PMD) -
Planning and guidance (visual inputs)
Primary Motor (M1) – Initiates muscle contraction
Motor Tasks Performed
-40 -30 -20 -10 0 10 20 30 40-40
-30
-20
-10
0
10
20
30
40
Task
1Ta
sk 2
Data• 2 Owl monkeys – Belle, Carmen• 2 Rhesus monkeys – Aurora, Ivy• 54-192 sorted cells• Cortices sampled: PP, M1, PMd, S1, SMA• Neuronal rate (100 Hz) and behavior is time synchronized and downsampled to 10Hz
100 msec Binned Counts Raster of 105 neurons (spike sorted)
Firing Rates
Time
Neu
ron
Num
ber
200 400 600 800 1000 1200 1400 1600 1800 2000
10
20
30
40
50
60
70
80
90
100
Ensemble Correlations – Local in Time – are Averaged with Global Models
Computational Models of Neural Intent
• Three different levels of neurophysiology realism
– Black Box models – function relation between input - desired response (no realism!)
– Generative Models –state space models using neuroscience elements (minimal realism).
– White models – significant realism (wish list!)
Optimal Linear Model
• The Wiener (regression) solution
• Normalized LMS with weight decay is a simple starting point.
• Four multiplies, one divide and two adds per weight update
• Ten tap embedding with 105 neurons• For 1-D topology contains 1,050
parameters (3,150)
)()()(
)()1( 2 nxnenx
nwnw
pw 1)( IR
Z-1 delay of 1 sampleS adderwi(n) parameter i at time n
w0
w9
3-D, 2-D Trajectory Modeling and Robot Control
• Collaboration with Miguel Nicolelis, Duke University
• Sponsored by DARPA
Time-Delay Neural Network (TDNN)
• The first layer is a bank of linear filters followed by a nonlinearity.
• The number of delays to span I second
• y(n)= Σ wf(Σwx(n))• Trained with
backpropagation• Topology contains a ten tap
embedding and five hidden PEs– 5,255 weights (1-D)
Principe, UF
Multiple Switching Local Models
• Multiple adaptive filters that compete to win the modeling of a signal segment.
• Structure is trained all together with normalized LMS/weight decay
• Needs to be adapted for input-output modeling.• We selected 10 FIR experts of order 10 (105 input channels)
d(n)
Recurrent Multilayer Perceptron (RMLP) – Nonlinear “Black Box”
• Spatially recurrent dynamical systems
• Memory is created by feeding back the states of the hidden PEs.
• Feedback allows for continuous representations on multiple timescales.
• If unfolded into a TDNN it can be shown to be a universal mapper in Rn
• Trained with backpropagation through time
))1()(()( 1111 byWxWy ttft f
2122 )()( byWy tt
Generative Models for BMIs
• Use partial information about the physiological system, normally in the form of states.
• They can be either applied to binned data or to spike trains directly.
• Here we will only cover the spike train implementations.
Difficulty of spike train Analysis: Spike trains are point processes, i.e. all the information is
contained in the timing of events, not in the amplitude of the signals!
Particle Filters for Point Processes
)(iitt
Xkf
Kinematic State
Neural Tuning function spike trains
Prediction
it
itt
it vXFX 11
Updating
)|( )(1
it
jt
it
it Npww
)( jtN
NonGaussian
P(state|observation)
N
i
itt
it
jtt xxkwNxp
1:0:0
)(:1:0 )()|(
N
i
ikk
ikkk kWNp
1:1 )()|( xxx
Linear filter
nonlinearity f
Poisson model
kinematics
spikes
)( tt Poissonspike
)( lagtt kf x
Instantaneous tuning modelt
-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.250
0.05
0.1
0.15
0.2
0.25
KX
(K
X)
neuron 80 nonlinear estimation
optimum delay
[-300, 500] ms
[-250, 450] ms
[-200, 400] ms
[-150, 350] ms
[-100, 300] ms
[-50, 250] ms
[0,200] ms
-50 -40 -30 -20 -10 0 10 20 30 40-50
-40
-30
-20
-10
0
10
20
30
40
x
y
position
desiredKalmanPPMonte Carlo PP
Generative Data Modeling
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
Neu
ral C
hann
els
Time
ObservableProcesses
(probed neurons)
HiddenProcesses
(Brain areas)
BMI lessons learnedBMIs are beyond the Proof of Concept
stage, but….Present systems are signal translators
and will not be the blue print for clinical applications
Current decoding methods use kinematic training signals - not available in the paralyzed
I/O models cannot contend with new environments without retraining
BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user
BMI lessons learnedBMIs are beyond the Proof of
Concept stage, but….Present systems are signal
translators and will not be the blue print for clinical applications
Current decoding methods use kinematic training signals - not available in the paralyzed
I/O models cannot contend with new environments without retraining
BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user
A Paradigm Shift for BMIs!
• During training the user actions create a desired response to the DSP algorithm.
• During testing the DSP algorithm creates an approximation to the desired response.
DSP algorithm
Desired response
Neural Signal Processing
• The control algorithm learns through reinforcement to achieve common goals in the environment.
• Shared control with user to enhance learning in multiple scenarios and acquire the net benefits of behavioral, computational, and physiological strategies
X
Control Algorithm
Learning Algorithm
Neural Signal Processing
A Paradigm Shift for BMIs!
Construction of a New FrameworkHow to capitalize on the perception-action cycle?
• The brain is embodied and the body is embedded
• Need to quantify Brain State at different time resolutions
• Intelligent behavior arises from the actions of an individual seeking to maximize received reward in a complex and changing world.
• The BMI must engage and dialogue with the user:– Exploits better engineering knowledge– Utilizes cognitive states– Dissects behavior top-down– Exploits rewards– Learns with use
• Propose Reinforcement Learning to train the BMI.
FUTUREPAST
INTERNALREPRESENTATION
EXTERNALWORLD
LIMBICSYSTEM
ORGANIZED PASTEXPERIENCE
PREDICTIVEMODELING
DOES ACTIONMEET
FUTURE REALITY?
SENSORYSTIMULUS
Causality line
Body line
Reward Learning Involves a Dialogue
• Relation between the agent and its environment.
• Environment: You are in state 14. You have 2 possible actions.
• Agent: I'll take action 2.• Environment: You received a
reinforcement of 17.8 units. You are now in state 13. You have 2 possible actions.
• Agent: I'll take action 1.
• repeat
AGENT
ENVIRONMENT
actions
rew
ards
stat
es
Goal
Start
CABMI involves TWO intelligent agents in a cooperative dialogue!!!
stat
es
ROBOT
actions
rew
ards
RAT’S BRAIN
environment
RAT’S BRAIN
COMPUTER AGENTUser’s
neuromodulationsets the value function for the
CA
Both the CA and the user have the same reward in 3D space
Features of co-adaptive BMI
• Enables intelligent system design in BMIs• Both systems adapt in close loop in a very tight
coupling between brain activity and computer agent ( CA states are specified by brain activity).
• User must incorporate the CA in its world (can a rat learn this?)
• CA must decode brain activity for its value function (can
it model the signature of behavior?). • In fact CABMI is a “symbiotic” biological-computer
hybrid system.
31
Experiment workspace [top view]
The user learns first to associate levers with water reward in a training phase.
In brain control, it progressively associates the blue guide LED of the robotic arm with the target lever LEDs. Only when the robot presses the target lever it will get reward.
Experiment workspace [top view]
Experimental CABMI Paradigm
-3 -2 -1 0 1 2 3
012
3
0
1
2
3
IncorrectTarget
CorrectTarget
StartingPosition
Match LEDs
Grid-space
Match LEDs
Rat’s PerspectiveWater Reward
Map workspace to grid
Rat
Robot Arm
Left Lever Right Lever
27 discrete actions 26 movements
1 stationary
Experimental CABMI Paradigm
• CA rewards are defined in 3D at the target lever positions.
• RL is used to train the CA in brain control (tabula rasa, i.e. no a priori information).
• During brain control, shaping of the reward field increases the level of difficulty across multiple sections with an adjustable threshold target.
35
-0.1-0.05
0
0.150.2
0.250.3
0.35
0.15
0.2
0.25
0.3
z
Left Target
xy
00.05
0.10.15
0.20.25
0.30.35
0.15
0.2
0.25
0.3
z
Right Target
xy
00.05
0.10.15
0.20.25
0.30.35
0.15
0.2
0.25
0.3
z
Right Target
xy-0.1
-0.050
0.150.2
0.250.3
0.35
0.15
0.2
0.25
0.3
z
Left Target
xy
Neuromodulation defines the States
Sampling rate 24.4 kHz
Hall, Brain Research (1974)
32 channelsSpike sorted data
Bilateral Premotor/motorAreas
Performance metrics
Performance metrics:1. Percentage of trials earning reward2. Average control time required to reach a target
4 sessions were simulated using random action selection to estimate chance performance for the CABMI in increasing difficulty tasks.
% trials earning reward time to achieve reward
Performance in 4 tasks of increasing difficulty
Closed-Loop RLBMI
Non-functional levers
Functional levers
Robot workspace in rat visual field of view.
BLUE – Robot
GREEN - Lever
Top-view of the rat behavioral cage.
• It is well established that preparation, execution, and also imagination of movement produce an event-related desynchronization (ERD) over the sensorimotor areas, with maxima in the alpha band (mu rhythm, 10 Hz) and beta band (20 Hz).
• The mu ERD is most prominent over the contralateral sensorimotor areas during motor preparation and extends bilaterally with movement initiation
• ERD during hand motor imagery is very similar to the pre-movement ERD, i.e., it is locally restricted to the contralateral sensorimotor areas
Event Related Desynchronization (ERD) and synchronization (ERS)
• During movement preparation and execution, an increase of synchronization (ERS) in the 10-Hz band normally appears over areas not engaged in the task (idling)
• ERS can also be observed after the movement, over the same areas that had displayed ERD earlier
Event Related Desynchronization (ERD) and synchronization (ERS)
Beta rebound following movement and somatosensory stimulation
• The general finding is that beta oscillations are desynchronized during preparation, execution, and imagination of a motor act
• After movement offset, the beta band activity recovers very fast (<1 s) and short-lasting beta bursts appear.
• The occurrence of a beta rebound related to mental motor imagery implies that this activity does not necessarily depend on motor cortex output.
• A number of experiments have also shown beta oscillations to be sensitive to somatosensory stimulation
ERS (Blue) and ERD (Red)
ERD
ERS 12.0 Hz +/- 1.0
Pfurtscheller
10.9 Hz +/- 0.9
ERS (Blue) and ERD (Red)
Pfurtscheller
Beta ERS
Pfurtscheller
Alpha and Beta ERS
Pfurtscheller
Signal Processing for ERD/ERS
• Bandpass filtering between 9-13 Hz will emphasize this component.
• Estimate the power • Place a statistical threshold for detection.
• Alternatively use PSD and threshold the appropriate frequency band.
Paradigm 1
(http://www.dcs.gla.ac.uk/~rod/Videos.html)
Paradigm 2
Event Related Potentials
• ERPs are a signature of cognition. They signal a massive communication amongst brain areas (kind of the brain’s impulse response to an internal stimulus).
• This is very good, but the problem is that it is normally much smaller than the ongoing EEG activity (i.e. the SNR is negative).
Event Related Potentials
• The ERP shape is well known and pretty stable across individuals, and has a known distribution across the channels.
• The P300 is the most used for BMIs because it is task relevant
N100-P200 complex is pre-attentive response appearing over sensory areasP300 signals a rare tasks relevant event (Cz)N400 signals an unexpected event (Cz)
Event Related Potentials
• In order to deal with the negative SNR, we use averaging of the stimulus.
• If you have a transient that appears in white Gaussian noise, align the transient and average across trielas you obtain an increase of SNR by , where N is the number of trials.
• This is normally done but has three shortcomings:– It is not real time– It assumes that the shape of the ERP is the same– It assumes that the latency is constant
N
P300 Event Related Potentials
P300 Event Related Potentials
Negative SNR so need averaging (i.e. repeated presentation of stimuli)
P300 Paradigm
P300 Paradigm
P300 Paradigm 2
The Cortical Mouse In 1990 the CNEL proposed a new computer interface that would control cursor movement in the screen using directly brain activity (EEG) and implemented in a NeXT Computer
• Decision based on single ERPs (N400) in real time • Neural network classifier implemented in DSP chip• Overall control (synch, screen, data flow) by the OS
Left/RightYES NO
Konger, C., Principe, J., ANN classification of ERPs for a new computer interface IEEE IJCNN, 1990Sina Eatemadi, A new computer interface using event related potentials University of Florida, 1992.
4.5bits/min
Slow Cortical Potentials
SCP Paradigm
Steady State Evoked Potential (ssEP)
ssVEP Paradigms
ssVEP Paradigms
ssVEP Paradigms
• One of the most reliable effects. • Need to do FFT of occipital channels and pick
the highest frequency.• Car race (winner of the first BCI competition)
Taxonomy of BCI paradigms
Taxonomy of BCI paradigms
Taxonomy of BCI paradigms
Taxonomy of BCI paradigms
Mu Rhythm
• When a subject imagines movement or sees movement made by others a burst of activity in the 8-12Hz range appears over the sensorimotor areas in the brain
• The subject can synchronize the rhythm and by moving desynchronize it, hence it ia good signal to be used for motor BMI tasks.