Learning and Synaptic Plasticity. Molecules Levels of Information Processing in the Nervous System...

Learning and Synaptic Plasticity

Molecules

Levels of Information Processing in the Nervous System

0.01mm

Synapses1mm

Neurons100mm

Local Networks1mm

Areas / „Maps“ 1cm

Sub-Systems10cm

At the dendrite the incomingsignals arrive (incoming currents)

Molekules

Synapses

Neurons

Local Nets

Systems

At the soma currentare finally integrated.

At the axon hillock action potentialare generated if the potential crosses the membrane threshold

The axon transmits (transports) theaction potential to distant sites

At the synapses are the outgoing signals transmitted onto the dendrites of the target neurons

Structure of a Neuron:

Receptor ≈ Channel

Vesicle

TransmitterAxon

Dendrite

Schematic Diagram of a Synapse:

Transmitter, Receptors, Vesicles, Channels, etc.synaptic weight: 𝒘

Machine Learning Classical Conditioning Synaptic Plasticity

Dynamic Prog.(Bellman Eq.)

REINFORCEMENT LEARNING UN-SUPERVISED LEARNINGexample based correlation based

d-Rule

Monte CarloControl

Q-Learning

TD( )often =0

TD(1) TD(0)

Rescorla/Wagner

Neur.TD-Models(“Critic”)

Neur.TD-formalism

DifferentialHebb-Rule

(”fast”)

STDP-Modelsbiophysical & network

EVALUATIVE FEEDBACK (Rewards)

NON-EVALUATIVE FEEDBACK (Correlations)

Correlationbased Control

(non-evaluative)

ISO-Learning

ISO-Modelof STDP

Actor/Critictechnical & Basal Gangl.

Eligibility Traces

Hebb-Rule

(”slow”)

supervised L.

Anticipatory Control of Actions and Prediction of Values Correlation of Signals

Neuronal Reward Systems(Basal Ganglia)

Biophys. of Syn. PlasticityDopamine Glutamate

LTP(LTD=anti)

ISO-Control

Overview over different methods

Different Types/Classes of Learning

Unsupervised Learning (non-evaluative feedback)• Trial and Error Learning.

• No Error Signal.

• No influence from a Teacher, Correlation evaluation only.

Reinforcement Learning (evaluative feedback)• (Classic. & Instrumental) Conditioning, Reward-based Lng.

• “Good-Bad” Error Signals.

• Teacher defines what is good and what is bad.

Supervised Learning (evaluative error-signal feedback)• Teaching, Coaching, Imitation Learning, Lng. from examples and more.

• Rigorous Error Signals.

• Direct influence from a teacher/teaching signal.

Basic Hebb-Rule: = m ui v m << 1dwi

For Learning: One input, one output.

An unsupervised learning rule:

A supervised learning rule (Delta Rule):

! i ! ! i à ör ! iE

No input, No output, one Error Function Derivative,where the error function compares input- with output-examples.

A reinforcement learning rule (TD-learning):

One input, one output, one reward.

wi ! wi + ö[r(t + 1) + í v(t + 1) à v(t)]uà(t)

The influence of the type of learning on speed and autonomy of the learner

Correlation based learning: No teacher

Reinforcement learning , indirect influence

Reinforcement learning, direct influence

Supervised Learning, Teacher

Programming

Learning Speed Autonomy

Hebbian learning

When an axon of cell A excites cell B and repeatedly or persistently takes part in firing it, some growth processes or metabolic change takes place in one or both cells so that A‘s efficiency ... is increased.

Donald Hebb (1949)

d-Rule

Monte CarloControl

Q-Learning

TD( )often =0

TD(1) TD(0)

Rescorla/Wagner

Neur.TD-formalism

(”fast”)

(non-evaluative)

ISO-Learning

ISO-Modelof STDP

Eligibility Traces

Hebb-Rule

(”slow”)

supervised L.

LTP(LTD=anti)

ISO-Control

You are here !

Hebbian Learning

…Basic Hebb-Rule:

…correlates inputs with outputs by the…

= m v u1 m << 1dw1

Vector Notation

Cell Activity: v = w . u

This is a dot product, where w is a weight vector and uthe input vector. Strictly we need to assume that weightchanges are slow, otherwise this turns into a differential eq.

= m v u1 m << 1dw1

dtSingle Input

= m v u m << 1dw

dtMany Inputs

As v is a single output, it is scalar.

Averaging Inputs= m <v u> m << 1

We can just average over all input patterns and approximate the weight change by this. Remember, this assumes that weight changes are slow.

If we replace v with w . u we can write:

= m Q . w where Q = <uu> is the input correlation matrix

Note: Hebb yields an instable (always growing) weight vector!

Synaptic plasticity evoked artificially

Examples of Long term potentiation (LTP)and long term depression (LTD).

LTP First demonstrated by Bliss and Lomo in 1973. Since then induced in many different ways, usually in slice.

LTD, robustly shown by Dudek and Bear in 1992, in Hippocampal slice.

Why is this interesting?

LTP and Learninge.g. Morris Water Maze

platform

Morris et al., 1986

Control Blocked LTP

2 3 41 321 4

Learn the position of the platform

Before learning

After learning

Receptor ≈ Channel

Vesicle

TransmitterAxon

Dendrite

Transmitter, Receptors, Vesicles, Channels, etc.synaptic weight: 𝒘

Schematic Diagram of a Synapse:

LTP will lead to new synaptic contacts

Synaptic Plasticity:Dudek and Bear, 1993

LTD(Long-Term Depression)

LTP(Long-TermPotentiation)

Conventional LTP = Hebbian Learning

Symmetrical Weight-change curve

Synaptic change %

The temporal order of input and output does not play any role

Spike timing dependent plasticity - STDP

Markram et. al. 1997

+10 ms

-10 ms

Synaptic Plasticity: STDP

Makram et al., 1997Bi and Poo, 2001

SynapseNeuron BNeuron A

Pre follows Post:Long-term Depression

Synaptic

change %

Spike Timing Dependent Plasticity: Temporal Hebbian Learning

Pre precedes Post:Long-term Potentiation

Causal

(possibly)

Time difference T [ms]

= m v u1 m << 1dw1

dtSingle Input

= m v u m << 1dw

dtMany Inputs

As v is a single output, it is scalar.

Averaging Inputs= m <v u> m << 1

We can just average over all input patterns and approximate the weight change by this. Remember, this assumes that weight changes are slow.

If we replace v with w . u we can write:

= m Q . w where Q = <uu> is the input correlation matrix

Note: Hebb yields an instable (always growing) weight vector!

Back to the Math. We had:

= m (v - Q) u m << 1dw

Covariance Rule(s)

Normally firing rates are only positive and plain Hebb would yield only LTP.Hence we introduce a threshold to also get LTD

Output threshold

= m v (u - Q) m << 1dw

dtInput vector threshold

Many times one sets the threshold as the average activity of somereference time period (training period)

Q = <v> or Q = <u> together with v = w . u we get:

= m C . w, where C is the covariance matrix of the input

C = <(u-<u>)(u-<u>)> = <uu> - <u2> = <(u-<u>)u>

v < Q: homosynaptic depression

u < Q: heterosynaptic depression

The covariance rule can produce LTD without (!) post-synaptic input.This is biologically unrealistic and the BCM rule (Bienenstock, Cooper,Munro) takes care of this.

BCM- Rule

= m vu (v - Q) m << 1dw

Experiment BCM-Rule

pre ≠Dudek and Bear, 1992

The covariance rule can produce LTD without (!) post-synaptic input.This is biologically unrealistic and the BCM rule (Bienenstock, Cooper,Munro) takes care of this.

BCM- Rule

= m vu (v - Q) m << 1dw

As such this rule is again unstable, but BCM introduces a sliding threshold

= n (v2 - Q) n < 1dQ

Note the rate of threshold change n should be faster than then weight

changes (m), but slower than the presentation of the individual inputpatterns. This way the weight growth will be over-dampened relative to the (weight – induced) activity increase.

open: control conditionfilled: light-deprived

less input leads to shift of threshold to enable more LTP

BCM is just one type of (implicit) weight normalization.

Kirkwood et al., 1996

Evidence for weight normalization:Reduced weight increase as soon as weights are already big(Bi and Poo, 1998, J. Neurosci.)

Problem: Hebbian Learning can lead to unlimited weight growth.

Solution: Weight normalizationa) subtractive (subtract the mean change of all weights from each individual weight).b) multiplicative (mult. each weight by a gradually decreasing factor).

Examples of Applications • Kohonen (1984). Speech recognition - a map of

phonemes in the Finish language• Goodhill (1993) proposed a model for the

development of retinotopy and ocular dominance, based on Kohonen Maps (SOM)

• Angeliol et al (1988) – travelling salesman problem (an optimization problem)

• Kohonen (1990) – learning vector quantization (pattern classification problem)

• Ritter & Kohonen (1989) – semantic maps

OD ORI

Program

Differential Hebbian Learning of SequencesLearning to act in response to

sequences of sensor events

d-Rule

Monte CarloControl

Q-Learning

TD( )often =0

TD(1) TD(0)

Rescorla/Wagner

Neur.TD-formalism

(”fast”)

(non-evaluative)

ISO-Learning

ISO-Modelof STDP

Eligibility Traces

Hebb-Rule

(”slow”)

supervised L.

LTP(LTD=anti)

ISO-Control

You are here !

I. Pawlow

History of the Concept of TemporallyAsymmetrical Learning: Classical Conditioning

I. Pawlow

Correlating two stimuli which are shifted with respect to each other in time.

Pavlov’s Dog: “Bell comes earlier than Food”

This requires to remember the stimuli in the system.

Eligibility Trace: A synapse remains “eligible” for modification for some time after it was active (Hull 1938, then a still abstract concept).

Sw0 = 1

Unconditioned Stimulus (Food)

Conditioned Stimulus (Bell)

Response

Dw1+Stimulus Trace E

The first stimulus needs to be “remembered” in the system

Classical Conditioning: Eligibility Traces

I. Pawlow

Eligibility Traces

Note: There are vastly different time-scales for (Pavlov’s) behavioural experiments:

Typically up to 4 seconds

as compared to STDP at neurons:

Typically 40-60 milliseconds (max.)

Defining the TraceIn general there are many ways to do this, but usually one chooses a trace that looks biologically realistic and allows for some analytical calculations, too.

EPSP-like functions:a-function:

Double exp.:

This one is most easy to handle analytically and, thus, often used.

DampenedSine wave:

Shows an oscillation.

h(t) =n

0 t<0hk(t) tõ 0

h(t) = teà atk

h(t) = b1sin(bt) eà at

h(t) = î1(eà at à eà bt)

d-Rule

Monte CarloControl

Q-Learning

TD( )often =0

TD(1) TD(0)

Rescorla/Wagner

Neur.TD-formalism

(”fast”)

(non-evaluative)

ISO-Learning

ISO-Modelof STDP

Eligibility Traces

Hebb-Rule

(”slow”)

supervised L.

LTP(LTD=anti)

ISO-Control

Mathematical formulation of learning rules is

similar but time-scales are much different.

wEarly: “Bell”

Late: “Food”

)( )( )( tytutdt

Differential Hebb Learning Rule

Simpler Notationx = Inputu = Traced Input

V’(t)

Convolution used to define the traced input,

Correlation used to calculate weight growth.

)()()()()()()( xfxgxgxfduuxgufxh

)()()()()()()( xgxfxfxgduxugufxh

Produces asymmetric weight change curve(if the filters h produce unimodal „humps“)

)(' )( )( tvtutdt

Derivative of the Output

Filtered Input

)( )()( tuttv ii

Output

Differential Hebbian Learning

Conventional LTP

Symmetrical Weight-change curve

Synaptic change %

The temporal order of input and output does not play any role

Produces asymmetric weight change curve(if the filters h produce unimodal „humps“)

)(' )( )( tvtutdt

Derivative of the Output

Filtered Input

)( )()( tuttv ii

Output

Differential Hebbian Learning

47Weight-change curve

(Bi&Poo, 2001)

T=tPost - tPrems

Pre follows Post:Long-term Depression

Synaptic change %Pre

Pre precedes Post:Long-term Potentiation

Spike-timing-dependent plasticity(STDP): Some vague shape similarity

d-Rule

Monte CarloControl

Q-Learning

TD( )often =0

TD(1) TD(0)

Rescorla/Wagner

Neur.TD-formalism

(”fast”)

(non-evaluative)

ISO-Learning

ISO-Modelof STDP

Eligibility Traces

Hebb-Rule

(”slow”)

supervised L.

LTP(LTD=anti)

ISO-Control

You are here !

PlasticSynapse

NMDA/AMPA

Postsynaptic:Source of Depolarization

The biophysical equivalent of Hebb’s postulate

Presynaptic Signal(Glu)

Pre-Post Correlation,but why is this needed?

Plasticity is mainly mediated by so calledN-methyl-D-Aspartate (NMDA) channels.

These channels respond to Glutamate as their transmitter andthey are voltage depended:

Biophysical Model: Structure

x NMDA synapse

Hence NMDA-synapses (channels) do require a (hebbian) correlation between pre and post-synaptic activity!

Source of depolarization:

1) Any other drive (AMPA or NMDA)

2) Back-propagating spike

Local Events at the Synapse

Current sources “under” the synapse:• Synaptic current

Isynaptic

GlobalIBP

• Influence of a Back-propagating spike

• Currents from all parts of the dendritic tree

IDendritic

Pre-syn. Spike

BP- or D-Spike

0 2 4 6 8 10

0 40 80 t [ms]

g [nS ]NM DA

On „Eligibility Traces“

Membrane potential:

Weight Synaptic input

Depolarization source

deprest

tVVVEttV

))((g )()(

ISO-Learning

• Dendritic compartment

• Plastic synapse with NMDA channels Source of Ca2+ influx and coincidence detector

PlasticSynapse NMDA/AMPA

ii IVEt~dt

dV ))((g

NMDA/AMPAgBP spike

Source of Depolarization

Dendritic spike

• Source of depolarization: 1. Back-propagating spike 2. Local dendritic spike

Model structure

Plasticity Rule(Differential Hebb)

NMDA synapse -Plastic synapse

ii IVEtdt

dV ))((g ~

NMDA/AMPAg

NMDA/AMPA

Source of depolarization

Instantenous weight change:

)(' )( )( tFtctdt

Presynaptic influence Glutamate effect on NMDA channels

Postsynaptic influence

0 40 80 t [ms]

g [nS ]NM DA

Normalized NMDA conductance:

NMDA channels are instrumental for LTP and LTD induction (Malenka and Nicoll, 1999; Dudek and Bear ,1992)

Pre-synaptic influence

ii IVEtdt

dV ))((g ~

NMDA/AMPAg

NMDA/AMPA

)(' )( )( tFtctdt

20V [m V]

20 t [m s]

20V [m V]

20 t [ms]

20V [m V]

20 t [m s]

20V [m V]

20 t [m s]

Dendriticspikes

Back-propagating spikes

(Larkum et al., 2001

Golding et al, 2002

Häusser and Mel, 2003)

(Stuart et al., 1997)

Depolarizing potentials in the dendritic tree

ii IVEtdt

dV ))((g ~

NMDA/AMPAg

NMDA/AMPA

Postsyn. Influence

)(' )( )( tFtctdt

For F we use a low-pass filtered („slow“) version of a back-propagating or a dendritic spike.

20V [m V]

20 t [m s]

20V [m V]

20 t [m s]

0 50 150 t [ms]100

V [mV]

0 5 0 1 5 0 t [ m s ]1 0 0

V [ m V ]

0 20 80 t [ms]40 60

V [mV]

0 20 80 t [ms]40 60

V [mV]

20V [m V]

20 t [ms]

20V [m V]

20 t [m s]

BP and D-Spikes

20V [m V]

20 t [m s]

20V [m V]

20 t [m s]

0-20 40 T [ms]-40 20

Back-propagating spike

Weight change curve

NMDAr activation

Back-propagating spike

T=tPost – tPre

Weight Change Curves

Source of Depolarization: Back-Propagating Spikes

PlasticSynapse

NMDA/AMPA

Postsynaptic:Source of Depolarization

The biophysical equivalent of Hebb’s PRE-POST CORRELATION postulate:

THINGS TO REMEMBER

Presynaptic Signal(Glu)

Possible sources are: BP-SpikeDendritic SpikeLocal Depolarization

Slow-Acting NMDA

Signal as presynatic

influence

One word about

Supervised Learning

d-Rule

Monte CarloControl

Q-Learning

TD( )often =0

TD(1) TD(0)

Rescorla/Wagner

Neur.TD-formalism

(”fast”)

(non-evaluative)

ISO-Learning

ISO-Modelof STDP

Eligibility Traces

Hebb-Rule

(”slow”)

supervised L.

LTP(LTD=anti)

ISO-Control

Overview over different methods – Supervised Learning

!And many more

Supervised learningmethods are mostlynon-neuronal andwill therefore not

be discussedhere.

So Far:

• Open Loop Learning

All slides so far !

CLOSED LOOP LEARNING

• Learning to Act (to produce appropriate behavior)

• Instrumental (Operant) Conditioning

All slides to come now !

This is an open-loopsystem

Sensor 2

conditionedInput

Bell Food Salivation

Pavlov, 1927

Temporal Sequence

This is an Open Loop System !

Adaptable

Neuron

Closed loop

Sensing Behaving

Learning and Synaptic Plasticity. Molecules Levels of Information Processing in the Nervous System...

Documents

Physiology of synapsesphysiology.med.uoa.gr/fileadmin/physiology.med.uoa.gr/...Physiology of synapses An understanding of synaptic transmission is necessary to understand the operations

Memristors - fp7-nanotec.eu€¦ · Memristors directly implement the synaptic plasticity • v = M(q) i • sub-µm size Memristors : artificial synapses 4. speed, low energy consumption,

Synaptic Transmission Synapse – specialized junction where an axon terminal contacts another neuron or cell type Types of synapses –Electrical synapses

Synaptic Transmission and Integrationusers.ece.cmu.edu/~byronyu/teaching/nsp_sp10/coursemats/lecture… · Synaptic Transmission at Chemical Synapses 5) Receptors cause ion channels

The ‘Life Cycle’ of Neuromuscular Synapses · the safety factor for neuromuscular transmission 1. Synaptic transmission, safety factor and size-strength relationships at NMJ 2

NeurobiologyofDisease ObesityElicitsInterleukin1 ...faculty.bennington.edu/~sherman/neuro/Obesity - Deficits in Hippocampal Synaptic...medial perforant path synapses were functionally

Chapter 45 Hormones and the Endocrine System...Synaptic and Neuroendocrine Signaling •In synaptic signaling, neurons form specialized junctions with target cells, called synapses

Biology Specification 13.6 – 13.8 13.6 Nervous coordination The mammalian eye Rods and Cones The nerve impulse Synapses and synaptic transmission Drugs

Chemical synapses: post-synaptic mechanisms. Postsynaptic Membranes and ion channels Ligand gated ion channels – a review a. Resting K + channels: responsible

The Effects of Short-Term Synaptic Depression at ...The Effects of Short-Term Synaptic Depression at Thalamocortical Synapses on Orientation Tuning in Cat V1 Aylin Cimenser1,2*, Kenneth

2B or not 2B that is the question about the synaptic · 2B or not 2B that is the question about the synaptic proteasome in glutamatergic synapses . Ter 2B ou não ter 2B: esta é

ΣυναπτικήΔιαβίβαση (Synaptic Transmition) 6.pdf · Συνάψεις (Synapses) Axon •Ένας νευρώνας εννευρώνει (innervates), δηλαδή

Synapses and Synaptic Transmission Dr Fawzia ALRoug, MBBS, Master, Ph.D Assistant Professor, Department of Physiology, College of Medicine, King Khalid

Neural Networks. Molecules Levels of Information Processing in the Nervous System 0.01 m Synapses 1m1m Neurons 100 m Local Networks 1mm Areas

Synaptic integration, Types of synapses, EPSP and IPSP

Review Neuronal networks and synaptic plasticity ...jeb.biologists.org/content/jexbio/209/12/2312.full.pdfSynaptic and network plasticity Synapses have innate propensity to alter the

PHYSIOLOGY 1 LECTURE 14 SYNAPTIC TRANSMISSION. n Objectives: The student should know –1. The types of synapses, electrical and chemical –2. The structure

ELECTRON MICROSCOPE OBSERVATIONS ON SYNAPTIC VESI- · electron microscope observations on synaptic vesi- cles in synapses of the retinal rods and cones* by eduardo de robertis, m.d.,

Glycolytic Enzymes Localize to Synapses under Energy ... manuscripts pdfs/2016 Jang... · Glycolytic Enzymes Localize to Synapses under Energy Stress to Support Synaptic Function

Synaptic transmission Module 725 Lecture 2. Aim nWhy do we need synapses? nTo know about chemical synapses…