Week lecture 4

Rescorla-Wagner Model; Neurobiology of Prediction Error

Surprisingness of the US

Robert Rescorla & Allan Wagner The model is a mathematical expression of surprise:

Learning will occur only when the subject is surprised - that is, when what happens is different from what the subject expected to happen

Blocking (Leon Kamin)

Group Phase 1 Phase 2 Test Result Blocking L-US L & T-US T no CR

Control L & T-US T CR

This experiment is important because it shows that:

1. Conditioning is not an automatic result of CS-US pairings

2. For conditioning to occur, the CS must be informative and the US surprising

V = associative strength between CS and US

Vmax = maximum associative strength

V = change in associative strength

on each conditioning trial

Measure of size of CR during CS-US conditioning trials

Reality

Theory

Quantification of surprisingness of the US

More surprise

Less surprise

V = Vmax - Vn Vn = strength of the association at the beginning of trial n

Vn = change is the strength of the association produced by trial n

Learning curves can differ in terms of: 1. Vmax

2. Rate of acquisition

Vn = (Vmax - Vn)

Vmax is determined by the magnitude of the US

and relate to the salience of the CS and the US, respectively. Their values are between 0 and 1.

Vn = (Vmax -Vn)

Rescorla-Wagner model: valuable predictions

This model precludes quantitative predictions but allows interesting qualitative predictions (increases, decreases, and more).

Vn = (Vmax -Vn)

ACQUISITION Assume = 0.3 and Vmax = 1

Trial Vn Vn = (Vmax - Vn) 1 0.00 V1 = 0.3 (1 - 0.00) = 0.30

2 0.00 + 0.30 V2 = 0.3 (1 - 0.30) = 0.21

3 0.00 + 0.30 + 0.21 V3 = 0.3 (1 - 0.51) = 0.15

4 0.00 + 0.30 + 0.21 + 0.15 V4 = 0.3 (1 - 0.66) = 0.10

Trial Vn Vn = (Vmax - Vn) 1 0.00 V1 = 0.3 (1 - 0.00) = 0.30

2 0.00 + 0.30 V2 = 0.3 (1 - 0.30) = 0.21

3 0.00 + 0.30 + 0.21 V3 = 0.3 (1 - 0.51) = 0.15

4 0.00 + 0.30 + 0.21 + 0.15 V4 = 0.3 (1 - 0.66) = 0.10

1 Ext 0.00 + 0.30 + 0.21 + 0.15 + 0.10 V5 = 0.3 (0 - 0.76) = - 0.22

Vmax = 0

CONDITIONED INHIBITORS have negative associative strength

EXTINCTION The weakening of a conditioned response when a CS is presented by itself

BLOCKING

When two CSs are used (a & b), the association or expectation at the beginning of a trial would be the sum of the strengths of each of the stimuli present

Vab = Va + Vb

Therefore, the amount of conditioning on a compound trial in which a and b occur together would be

Va = Vb = (Vmax - Vab)

In the blocking group, if the Vmax for the light (L) = 1.0, then:

VL = 1.0 at the end of Phase 1 (because of extensive L conditioning)

when the light and the tone (T) are presented in combination on trial 1 of Phase 2

VLT = VL + VT = 1.0 + 0 = 1.0

Therefore, the amount of conditioning to the T in the blocking group after 1 trial of conditioning with the LT compound is:

VT = (Vmax - VLT) = 0.3 (1.0 - 1.0) = 0

Tone ------>shock Light ------>shock

Tone + Light-------> shock

Tone ?

Assume that only few trials were given before the compound trial, and that Vmax = 1 and = 0.3

VT = 0.2 and VL = 0.2, thus VTL = 0.4

VT = VL = 0.3 (1.0 - 0.4) = 0.18

The model predicts an increase in associative strength for both T and L when presented during the compound trial

But, if there was extensive conditioning before the compound trial such that:

VT = 0.9 and VL = 0.9, thus VTL = 1.8

VT = VL = 0.3 (1.0 - 1.8) = - 0.24

Therefore, the model predicts a decrease associative strength for both T and L when presented during the compound trial

OVEREXPECTATION EFFECT

Rescorla (1970) - Extensively trained rats

Tone ---------->shock

Light ---------->shock

Experimental group (E) Tone + Light----------> shock

Control group (C) Nothing

No conditioning to the CS does not mean no conditioning at all

Contextual stimuli

Context

Trial 1 CS + Context ------> US = + associative strength to compound Trial 2 Context alone ------>US = + associative strength to context

Trial 20 CS + Context ------> US = - associative strength to compound

Trial 21 Context alone ------> US = + associative strength to context

When the US is not contingent to the CS, conditioning will be strong to contextual (background) stimuli but not to the CS

= context

Some problems with the Rescorla-Wagner model

1. Exclusive focus on the surprisingness of the US Nicholas Mackintosh John Pearce & Geoffrey Hall

The Mackintosh Model The Pearce-Hall Model

It is important to consider how the salience of the CS () changes

during conditioning

2. The conclusion that extinction destroys the original learning

2. Spontaneous Recovery: the reappearance of a CR to a CS after a period of time following the last extinction trial

Renewal: the reappearance of a CR to a CS due to return to the training environment, instead of the environment used during extinction Reinstatement: the reappearance of a CR to a CS due to a brief presentation of the US

Rapid Reacquisition: rapid return of a CR to a previously extinguished CS

Surprise and Prediction Error

US not predicted and therefore very surprising

US somehow predicted and therefore less surprising

US more predicted and therefore much less surprising

V = Prediction Error

Neurobiology of Prediction Error

DOPAMINE A neurotransmitter involved in learning, motivation and a variety of psychobiological functions

Agonist: Cocaine, Amphetamine, Methylphenidate

Antagonists: Chlorpromazine, Haloperidol (Anti-Psychotic drugs)

US = Drops of juice UR = Lick

CSs

DA neurons VTA

Wolfram Schultz

Prior to CS-US conditioning

After CS-US conditioning

During extinction

Dopaminergic neurons in the VTA encode a Prediction Error

CS

Auto-shaping

US

NAC = nucleus accumbens

Homeostatic hypothesis of learning

Optimal level

Disturbance Actual level

Receptors

Activation of learning systems in the brain

Past knowledge

New event

Current knowledge

DA in VTA Error Signal

Response system

DA neurons VTA Prediction Error Signal

Learning

Learning

Learning