Embed Size (px)
Citation preview
OverviewWe studied how the brain represents, organizes, and retrieves conceptual (semantic) information.
We examined recordings from neurosurgical patients as they studied and freely recalled lists of words.
Whereas previous studies have attempted to decode stimulus identities from neural patterns, our goal was to infer how stored representations of stimuli are organized in participants’ memories.
Methods
Cour
tesy
Mic
hael
Jaco
bs
Figure 2. Our recording setup. Patients are implanted with subdural and depth electrodes. Experiments are administered on a bedside laptop computer.
Results
For reprints contact [email protected]
ConclusionsWe identi!ed components of brain activity that varied with the meanings of words during study. We examined these same “semantic features” just prior to recall.
The correlation between neural and semantic similarity was preserved in prefrontal, occipital, and lateral temporal cortex, indicating that these regions are involved with representing the meanings of words.
We also found that the correlation between neural and semantic similarity predicted semantic clustering. The prefrontal cortex, lateral temporal cortex, and hippocampus exhibited this e"ect, indicating that these regions are involved with organizing the memories of the words.
We also identi!ed components of brain activity that evolved gradually during study. These “contextual features” were reinstated just prior to recall.
The magnitude of the neural signature of contextual reinstatement predicted participants’ tendencies to temporally cluster their recalls.
Figure 3. Analysis. a. Participants study and freely recalls lists of 15 or 20 common nouns. b. For each electrode we compute mean power contained in 50 frequencies (2 - 99 Hz) during each study and recall event. c. We apply PCA and select features that either vary with the meanings of studied words or vary gradually during the study interval. We examine the selected features during the recall interval.
Study Recall
10
BOWL
12
LEG BREAD
14 15
LAMP “truck”
1
“bread”
4
Spec
tra
PCA
“lamp”
3
“bowl”
2
Featureselection
}
Elec
trod
e
#4
7 Norm
alized power
#149
16
Norm
alized PCAscores
ELF
2
DRUM
11
BENCH
131
NOSE
Frequency
a.
b.
c.
Figure 1. Representing and retrieving conceptual memories. a. Distributed memory hypothesis. Concepts are broken down into constituent semantic features; each feature is represented by a distributed pattern of brain activity. b. Contextual reinstatement hypothesis. Context drifts gradually over time and is associated with each experienced event.
}
a. b.
Figure 5. Clustering during free recall. a. Semantic clustering. The semantic clustering score measures a participant’s tendancy to successively recall semantically related words. b. Temporal clustering. The temporal clustering score measures a participant’s tendancy to successively recall words that appeared at nearby positions on the studied list.
“...oar...boat....uh... glove...ball...”
“...whale...kite...fudge... umm...road...”
1.00.5 Semantic clustering score
“...leg...bench....uh... lamp...bread...”
“...nose...drum...bread... umm...road...”
1.00.5 Temporal clustering score
Semantic similarity
Neu
ral s
imila
rity
Semantic similarity
Neu
ral s
imila
rity
Semantic similarity
Neu
ral s
imila
rity
0.45 0.5 0.55 0.6 0.65-2
-1
0
1
2
3
Semantic clustering
Neu
ral c
lust
erin
g
Figure 9. Neural vs. semantic clustering. Each dot represents a single participant. Neural clustering indicates the t-value from the correlation between neural and semantic similarity. (Data are from PFC.)
All FR PFC Temp MTL Hippo Occ Par��
��
��
�
���
���
���
���
���
���
���
Correlation
*** ** #
Figure 10. Regions that predict semantic clustering. Each bar indicates the correlation between neural and semantic clustering from the indicated region of interest.
Figure 8. Neural vs. semantic similarity. Neural activity is recorded just prior to each recall. The dots indicate the means of 30 equally sized bins.
0.1 0.11 0.12 0.13 0.14
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
Semantic similarity
Neu
ral s
imila
rity
ï� ï� 0 � �ï����
0
����
����
����
Lag
Neur
al s
imila
rity
ï� ï� 0 � �0
0.05
0.1
0.15
���
����
0.3
Lag
Prob
abilit
y
0.4 0.5 0.6 0.7 0.8ï�
0
�
4
Temporal clustering score
t-val
ue
a. b. c.r = 0.42
p = 0.007
Figure 7. Neural signature of contextual reinstatement. a. Observed neural similarity between the feature vector corresponding to recall of a word from serial position i and study of a word from serial position i+lag. b. Probability of recalling an item from serial position i+lag immediately following an item from serial postion i, conditional on the availability of an item in that list position for recall. c. Participants exhibiting greater context reinstatement also exhibited more pronounced contiguity e"ects.
4 2 0 2 4
0.12
0.14
0.16
0.18
0.2
Lag
Neur
al s
imila
rity
�4 �2 0 2 40.1
0.2
0.3
0.4
0.5
0.6
Lag
Similarity
�4 �2 0 2 4
0.3
0.4
0.5
0.6
0.7
0.8
Lag
Similarity
Autocorrelated noise
Feature number
Tim
e (e
vent
num
ber)
5 10 15 ��
5
10
15
��
��0
���
���
0.6
0.8
b. Content reinstatement
Feature number
Tim
e (e
vent
num
ber)
5 10 15 ��
5
10
15
��
��0
0.1
���
0.3
���
0.5
0.6
0.7
c. Context reinstatement
Feature number
Tim
e (e
vent
num
ber)
5 10 15 ��
5
10
15
��
��0
0.1
���
0.3
���
0.5
0.6
d.
Figure 6. Predicted neural similarity as a function of lag according to three models. a. In each simulation, a single neuron is activated during each experimental event, i. Once activated, a neuron’s activity decays gradually according to equations (1) and (2). b. Autocorrelated noise. Each experimental event activates a random neuron, irrespective of which item is being presented or recalled. c,d. Each neuron is activated by a single item or distractor during study. c. Content reinstatement. During recall of the jth presented item we set fi = wj. d. Context reinstatement. During recall of the jth presented item we set fi = fj.
f i = $i fi -1 + % w i(1)
$i = 1 + %2 [( fi -1 · wi )2 # 1] # % ( fi -1 · w i)(2) wi =
fi =
NOSE ELF
1 2 43 5 6 87 9 10 1211 13 14 1615 17 18 2019
3+5+2 = ?
Itemfeatures
Distractorfeatures
... ...
a.
How does the brain represent and retrieve word meanings?Jeremy R. Manning & Michael J. KahanaPrinceton University, University of Pennsylvania
Figure 4. Feature selection. a. Contextual feature selection. We compute the neural similarity between each pair of studied words, for each neural component. Correlations between neural and temporal similarity imply that the given neural component evolves gradually during study. b. Semantic feature selection. We compute the neural and semantic similarity between each pair of studied words. (A seperate neural similarity matrix must be constructed for each neural component.) Correlations between neural and semantic similarity imply that the given neural component represents the meanings of the studied words.
b.
#0.4 #0.2 0.0 0.2 0.4 0.6
#0.5
#0.4
#0.3
#0.2
#0.1
0.0
0.1
0.2
0.3
0.4
0.5
CATDOG
GOAT
SEA
SHEEP
SHELLTRUCK
WAVE
Dimension 1
Dim
ensio
n 2
CATDOG
GOATSEA
SHEEPSHELL
TRUCK
WAVE
CAT
DOG
GOAT
SEA
SHEEP
SHELL
TRUCK
WAVE 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0similar
not similar
0.0
a.
1 � 3 � 50.08
0.10
����
����
0.16
0.18
Study distance
Neur
al s
imila
rity
Figure 11. Interpreting components of neural activity. The box represents the full pattern of brain activity exhibited by participant i during the experiment. Our feature selection framework identi!es neural patterns in segments C, E, G, and H. Neural patterns in segment A, D, F, and G contain information about the order in which participants will recall the words.
E
G
Cselected semanticfeatures
selected contextualfeatures
E
G H
neural basis ofsemantic clustering
D
GF
neural basis oftemporal clustering
D
G
A
behaviorally relevantneural activity capturedby analyses
G (Hippocampus)
Courtesy Talia Manning
CG
G
semanticcontext
behavior capt
ured
B
D E
G
CA
F H
I
participant i’sneural activity
