Identification of peaks in the ERPs - uni-hamburg.de€¦ · Web viewAll word types alike were assumed to activate areas in left ... Polk & Farah, 2002) and, finally, there is the

MirrorBot

IST-2001-35282

Bio-mimetic multi-modal learning in a mirror neuron-based robot

Category Specificity In The Processing Of Colour And Form Related Words

Authors: Friedemann Pulvermüller, Fermin Moscoso-del-Prado-Martin, Olaf HaukCovering period 1.9.2003-25.5.2004

MirrorBot Report 13

Report Version: 1

Report Preparation Date: 25. May. 2002

Classification: Restricted

Contract Start Date: 1st June 2002 Duration: Three Years

Project Co-ordinator: Professor Stefan Wermter

Partners: University of Sunderland, Institut National de Recherche en Informatique et en Automatique at Nancy, Universität Ulm, Medical Research Council at Cambridge, Università degli Studi di Parma

Project funded by the European Community under the “Information Society Technologies Programme“

0. Overview 3

1. Background 5

2. Method 8

3. Results 11

3. Discussion 14

4. Summary and Outlook 16

5. Implications for Computational Neuroscience and 18Brain-Inspired Robotics – Future Interaction with MirrorBot Partners

5. References 19

2

0 OVERVIEWThe Cambridge group in the Mirrobot project investigates the brain correlates of words and

their semantic and conceptual relationships to referent actions and objects. Multimodal

imaging techniques, including functional magnetic resonance imaging (fMRI), electro-

encephalography (EEG), magneto-encephalography (MEG) and trans-craneal magnetic

stimulation (TMS), were used to investigate words from different lexico-semantic categories.

In the first year of the project, action words were in the focus. We could demonstrate that

aspects of the meaning of action words are reflected in specific activation along the motor

strip and the premotor cortex. For example, words related to the hand (e.g., “pick”) activated

hand motor cortex, whereas words related to the leg (e.g., “kick”) activated leg motor cortex

(Hauk, Johnsrude, & Pulvermüller, 2004). This brain correlate of word meaning is evoked

already within 200 ms after information in the input is sufficient for word recognition (Hauk

& Pulvermüller, 2004). The patterns of semantic activation distinguishing between arm- and

leg-related words were present even if subjects did not attend to the stimulus words,

indicating a high degree of automaticity of semantic processes at the single word level

(Shtyrov, Hauk, & Pulvermüller, 2004). These findings are best explained by the existence of

widespread cortical cell assemblies that bind the neuron populations that process language

with those processing perception and actions, in which mirror neurons play a crucial role

(Pulvermüller, 2002; Rizzolatti, Fogassi, & Gallese, 2002). We envisage these cell assemblies

to include neurons in perisylvian cortex along with neurons in different parts of the motor

system. The proposal that category specific associative networks in the brain are the basis of

word, action and perceptual processing receives support from different lines of modern

neuroimaging research (cf., Pulvermüller, 2003) and could be implemented in

neurocomputational models (Wermter et al., 2004). In collaboration with Parma, TMS

experiments were performed to further study the functional relationships between language

and action. These efforts are detailed in Mirrobot Report 4 of WP2, Reports 7 and 8 of WP1,

Report 11 of WP 10 and mentioned in several others.

In the second year, the Cambridge group continued their research efforts to uncover the

cortical basis of action word processing, an area which had proven to be extremely fruitful

during the first year of the project. In parallel, the group investigated the processing of fine-

grained subcategories of words whose meanings relate to different types of visual

information. Words related knowledge about colors and shapes, for example “brown” and

“round”, were investigated using EEG, MEG, and fMRI. We expected word category

differences to be present in the inferior temporal “what”-stream of visual processing (Ishai,

Ungerleider, Martin, Schouten, & Haxby, 1999; Martin & Chao, 2001b; Mishkin,

Ungerleider, & Macko, 1983). In particular, the middle temporal gyrus has been described to

be involved in the association of objects with their particular colours (Zeki & Marini, 1998).

3

Therefore we expected an greater activation of middle temporal gyrus upon presentation of

colour-related words than following presentation of form-related words. In a similar manner,

because shapes, but not colours, relate to abstract action knowledge – an abstract shape can

always be sketched by distinctive movements which can be performed with different parts of

the body – we expected higher prefrontal and premotor activation during the processing of

form-related words than in the processing of colour-related words. The evaluation of fMRI

and MEG results is still in progress. We present here results of an EEG study and a modeling

effort undertaken to calculate the cortical activation underlying the processing of colour and

form related words. Further work was done to study the neural basis of serial order and

syntax. These research efforts will be discussed in a later report.

4

Arm word Leg word

Figure 1: Model of the cortical distribution of neuron ensembles linking word and action representations; possible cortical networks for arm and leg related wirds are illustrated (top). Sources of magnetoencephalographic activity seen for words semantica

1. BACKGROUND

Among the most intensely debated issues in the cognitive neuroscience of language is the

question of the cortical “seat” of word meaning (Martin & Chao, 2001a; Pulvermüller, 1999).

Although there is little doubt that areas in left inferior frontal and superior temporal cortex –

sometimes referred to as Broca’s and Wernicke’s regions – play a major role in language

processing, the location of additional areas possibly contributing to semantic processing

remains controversial. Most theories localize meaning-related mechanisms in areas anterior,

inferior, and posterior to Wernicke’s area in the left temporal lobe (Hickok & Poeppel, 2000;

Hodges, 2001; Price, Warburton, Moore, Frackowiak, & Friston, 2001; Scott & Johnsrude,

2003). However, since most studies investigating the issue have focussed on the cortical

processing of highly imageable concrete nouns and concepts related to their meaning, it is

possible that other word types engage semantic representations in other cortical regions.

When haemodynamic and neurophysiological imaging studies compared words referring to

objects with words that have a clear semantic relationship to actions, typically action verbs

(Dehaene, 1995; Kellenbach, Wijers, Hovius, Mulder, & Mulder, 2002; Preissl, Pulvermüller,

Lutzenberger, & Birbaumer, 1995; Pulvermüller, Preissl, Lutzenberger, & Birbaumer, 1996),

or nouns referring to tools (Chao, Haxby, & Martin, 1999; Ishai et al., 1999; Martin, Wiggs,

Ungerleider, & Haxby, 1996), the latter elicited strong frontal activation including premotor

cortex, suggesting that the frontal activation might reflect aspects of the action-related

meaning of action words (Martin & Chao, 2001a; Pulvermüller, 1996). If so, the cortical

locus of meaning processing could be, in part, determined by the general neuroscientific

principle of Hebbian learning according to which neuronal correlation is mapped onto

connection strength: (Hebb, 1949; Tsumoto, 1992). If word forms frequently co-occur with

visual perceptions (object words), their meaning-related activity may be found in temporal

visual areas, whereas action words frequently encountered in the context of body movements

may produce meaning-related activation in the fronto-central motor areas (Braitenberg, 1992;

Pulvermüller, 1996; Martin, 2001; Pulvermüller, 2003). In earlier studies (Hauk et al., 2004;

Hauk & Pulvermüller, 2004; Pulvermüller, Härle, & Hummel, 2000; Shtyrov et al., 2004), we

provided the first compelling evidence that word-meaning processing elicits specific action-

related activity patterns in fronto-central areas, including motor and premotor cortex. Action

words referring to different parts of the body lighted up fronto-central sensorimotor areas that

were also active when actions were performed with the word-related body parts. This

demonstrates a close link between the cortical substrate of language and action in the

processing of semantic meaning of words that refer to human actions.

5

Whether or not frontal areas also contribute to the processing of other word categories is a

matter of debate. According to the cell assembly model of Pulvermüller and Preissl (1991),

words of all kinds are organized by distributed neuron populations distributed over superior

temporal and inferior frontal perisylvian cortex, which processes acoustic and articulatory

information. This view, although well-supported by empirical data (Pulvermüller, 2003), is

questioned by some traditional models from aphasiology, according to which separate

modular processors in frontal and temporal lobe selectively contribute to language

comprehension and production, respectively (Hickok & Poeppel, 2000; Wernicke, 1874).

Word reading would, accordingly, only activate posterior areas but leave frontal areas

unaffected. This type of model is falsified by findings about frontal cortex activation elicited

by action words. However, action words form a small subset of the vocabulary and it may be

asked whether activation of specific frontal lobe areas is a more general phenomenon shared

by other word types, too. Here, we investigate word categories that relate to visual

perceptions: Words that were rated by experimental subjects to have a strong semantic

association to a colour, and words rated to be semantically related to a distinctive form or

shape. We hypothesized that, although both word categories would activate specific inferior

temporal areas in the ventral where-stream of visual processing, form-related words would

elicit additional activity in frontal cortex while colour-related words should elicit a higher

activation in the middle temporal gyrus. The additional activation in frontal lobe would reflect

the fact that an abstract form is always related to a distinct action pattern, a sequence of

movements that can be performed with different body parts to outline the shape. Similarly,

the higher involvement of the middle temporal gyrus in the processing of words with strong

associations to a colour would relate to the involvement of this area in the perception of

colours.

To investigate whether such a differentiation could be observed we performed an EEG

experiment similar to the one described by Hauk and Pulvermüller (2004). The usage of event

related potentials (ERPs) extracted from multi-channel EEG recording allows us to

investigate the precise time course of the differentiation of the words according to their

semantic categories. Source localisation techniques were used to pinpoint the cortical origin

of word-evoked neurophysiological activity.

All word types alike were assumed to activate areas in left inferior temporal lobe. It has been

claimed that this activation could reflect the engagement of a centre processing the meaning

of words, similar to Lichtheim’s “concept area” (Price, 2000). However, it has also been

postulated that an area in posterior inferior temporal lobe specialises in processing visual

word forms (VWF area; Cohen et al., 2000; 2002; Dehaene et al., 2002; Fiez & Pedersen,

1998; McCandliss et al., 2003; Polk & Farah, 2002) and, finally, there is the claim that

inferior temporal areas house word category specific processes (Martin & Chao, 2001). Given

6

the limited spatial resolution of EEG, we expected a general word related activation in

inferior temporal lobe initially, followed by spreading of activity either in temporal lobe

exclusively (colour words) or from temporal to frontal cortex (form words). Interestingly, the

activation of the VWF area was reported to occur between 150 ms and 200 ms following the

presentation of visual stimuli (Bentin et al., 1997; Helenius et al., 1999; McCandliss et al.,

2002; Tarkianen et al., 2002), which temporally overlaps with the activation of category

specific cortical areas reported by Hauk and Pulvermüller (2004) to happen at around 200 ms

from stimulus onset. In fact, even earlier activation of areas related to the meanings of words

has previously been reported by our group (Pulvermüller et al. 2001). This suggests that the

early activation of inferior temporal areas, might in fact correspond not only to the activation

of the orthographic features of the words, but also to their meanings. In this respect, the ERP

methodology allows us to investigate whether there are distinct temporal windows in which

semantic and specifically orthographic processes occur.

As made evident by Hauk and Pulvermüller (2004) and many others, Mininum Norm Current

Estimates calculated on the basis of EEG recordings are a suitable method to detect

differences in cortical activation elicited by words belonging to different semantic categories.

In the same manner, these topographical estimates should allow us to investigate whether the

some of early peaks in the ERPs and MEG signals do indeed correspond to the area where the

VWFA is generally located.

7

2. METHOD

Subjects

Ten monolingual native speakers of English participated in the study. Their age varied

between 18 and 31 years . They spent a minimum of 13 years on basic and higher level

education. All had normal or corrected-to-normal vision and reported no history of

neurological illness or drug abuse. Neuropsychological testing (Oldfield, 1971) revealed that

all of them were right-handed. Informed consent was obtained from all subjects and they were

paid for their participation. This study was approved by the Cambridge Psychology Research

Ethics Committee.

Stimuli

Stimuli were selected from databases using psycholinguistic criteria. A preliminary list of 403

words was evaluated in a behavioral study to assess the words’ cognitive, emotional, and

referential-semantic properties. This is necessary because words differing on corresponding

dimensions are known to elicit different neurophysiological responses (Kounios and

Holcomb, 1992; Pulvermüller, 1999; Skrandies, 1998). Native speakers of English (N = 15,

different from those participating in the EEG study) gave ratings on a 7-point scale answering

the following questions

· “How easily does this word evoke an image or any other sensory impression?”

(Imageability)

· “Do you evaluate this word or its meaning as pleasant or unpleasant?” (Valence)

· “How arousing is this word or its meaning?” (Arousal)

Ratings were given on a scale from 1 to 7. Additionally, participants were asked to answer on

similar 7-point scales whether the words reminded them of a particular colour or of a

particular shape. The results were evaluated statistically using F-tests. On the basis of this

evaluations, we selected 50 colour-related and 50 form-related words for presentation in the

EEG experiment, interspersed with 150 distractor words not related to colours or forms. The

word groups were matched with respect to mean word length, word form frequency according

to the CELEX database (Baayen et al., 1993), imageability (according to the ratings), valence

(according to the ratings), and arousal (according to the ratings). As an additional condition, a

8

sequence of hash marks matched to the length and luminance of each of the words was

included in the materials.

Procedure

Stimuli were presented for 100 ms each in white capital letters on a gray background in the

middle of a computer screen, subtending a horizontal visual angle smaller than 5 degrees. A

fixation cross was always present in the center of the screen between stimulus presentations.

Subjects were instructed to observe the stimuli silently, i.e., no overt response was required.

Stimuli were presented in pseudorandom order with a stimulus onset asynchrony randomly

varying between 2 and 3 sec. Two pseudo-randomized lists of stimuli were created, each

including all stimuli, which were alternated between subjects. Subjects were instructed to

reduce eye-blinks and movements as far as possible, and to restrict unavoidable movements to

the breaks within the experiment. The experimental session contained five breaks of 10 sec

duration.

Data recording

Electrophysiological data were collected in an electrically and acoustically shielded chamber

at the EEG laboratory of the MRC Cognition and Brain Sciences Unit in Cambridge, UK. The

EEG was recorded at a sampling rate of 500 Hz (0.1–30 Hz band-pass filter) from 64

Ag/AgCl electrodes mounted on an electrode cap (QuickCap, Neuromedical Supplies,

Sterling, VA) using SynAmps amplifiers (Neuro-Scan Labs, Sterling, VA). Electrodes were

placed according to the extended 10/20 system. EEG data were recorded against a reference at

AFz and converted off-line to average reference. The EOG was recorded bipolarly through

electrodes placed above and below the left eye (vertical) and at the outer canthi (horizontal).

After the actual experiment, subjects were instructed to blink and to move their eyes to the

left, right, up, and down, as indicated by symbols appearing on the computer screen. Average

responses to these eye movements were used for the correction of corresponding artifacts in

the EEG data (Berg and Scherg, 1994).

Data analysis

The continuously recorded neurophysiological data were divided into epochs of 1 ms length,

starting 200 ms before stimulus onset. Trials with voltage variations larger than 100μV in at

least one channel were rejected, and an eye artifact correction algorithm (Berg and Scherg,

1994) was applied. Data were band-pass filtered between 1–20 Hz. By averaging over

corresponding trials, event-related potentials (ERPs) were computed for every subject,

electrode, and stimulus category (HASH, COLOUR or FORM).

9

Source estimation

We calculated Minimum Norm Current Estimates using the method described by Hauk and

Pulvermüller (2004). We used a three-dimensional source space consisting of four concentric

equidistant “shells” (0.8–0.2 of electrode radius), with 1,965 current sources equally

distributed over these shells. At each location, three orthogonal sources were placed. Their

strengths were estimated separately, and then combined as the Euclidean vector length to

yield the intensity of activity at the corresponding location. For our analysis, we selected only

the uppermost shell at excentricity 0.8, containing 996 current sources (cf., Hauk et al., 1999).

The corresponding resolution vectors focus mostly on superficial cortical sources in this case

(Grave de Peralta et al., 1997). Tikhonov regularization (Bertero et al., 1988) was applied

such that the mean residual variance over data sets was 5%.

The left and right pre-auricular points and the nasion were determined for each subject and

used as landmarks for both the standard electrode configuration and the segmented skin

surface of the average magnetic resonance image (MRI) of the Montreal Neurological

Institute. These landmarks were used to co-register both modalities in the software package

ASA. In the spherical source model, the current sources that entered the statistical analysis

were located on a sphere below the electrodes, with 80% of the electrode radius. Co-

registration of the electrodes, therefore, also implied co-registration of the current sources

with the average MRI. The amplitudes of these current sources were then spherically

projected on the surface of the average brain.

Statistics

Peaks in the activation values were identified by plotting the root mean squares (RMS) of the

amplitudes for each of the conditions (see Figure 2).

In order to investigate the topographical differences elicited by the different types of stimuli,

we selected 4 regions of interest (ROIs) for the ERP analyses. These regions of interest were

chosen to provide contrasts in hemisphere (left or right) and frontality (anterior or posterior).

Each ROI included 7 electrode positions (left anterior: AF3, AF7, F3, F5, F7, FC3, and FC5;

right anterior: AF4, AF8, F4, F6, F8, FC4, and FC6; left posterior: P1, P3, P5, P7, PO3, PO9,

and O1; right posterior: P2, P4, P6, P8, PO4, PO10, and O2). For each of the 4 RMS peaks,

mean amplitudes were subjected to by-participant ANOVAs including the factors Lexicality,

Hemisphere, and Frontality. We only report significant interactions including the factor

Lexicality (word or hash marks). In a similar manner, the mean amplitudes to word stimuli

(colour- and form-related words, excluding the sequences of hash marks) were subjected to

ANOVAs including the factors Category (colour- or form-related), Hemisphere, and

Frontality. Once again, we only report those significant interactions of theoretical relevance.

10

3. RESULTS

Identification of peaks in the ERPs

The bottom part of Figure 2 shows the Root Mean Squares (RMS) of the amplitudes at all

recorded electrodes for each of the three types of stimuli in our experiment: hash marks

(green line), colour-related words (red), and form-related words (blue). The graph shows three

clear peaks in the RMS, centred approximately at 106 ms, 154 ms, and 254 ms from stimulus

onset. An additional peak centred at around 202 ms from stimulus onset is also apparent in the

case of the form-related words, and to a lesser degree in the colour-related words. The upper

panel in Figure 1 summarizes the scalp distribution of the amplitudes for the different types of

stimulus at each of the four peaks mentioned above.

96 to 116 ms time window

11

x

y

106.0 ms106.0 ms

Use cursor keys to rotate

5.20

-5.20EEG (uV)

x

y

154.0 ms154.0 ms


5.20

-5.20EEG (uV)

x

y

202.0 ms202.0 ms


5.20

-5.20EEG (uV)

x

y

254.0 ms254.0 ms


5.20

-5.20EEG (uV)

x

y

106.0 ms106.0 ms


5.37

-5.37EEG (uV)

x

y

154.0 ms154.0 ms


5.37

-5.37EEG (uV)

x

y

202.0 ms202.0 ms


5.37

-5.37EEG (uV)

x

y

254.0 ms254.0 ms


5.37

-5.37EEG (uV)

x

y

106.0 ms106.0 ms


5.19

-5.64EEG (uV)

x

y

154.0 ms154.0 ms


5.19

-5.64EEG (uV)

x

y

202.0 ms202.0 ms


5.19

-5.64EEG (uV)

x

y

254.0 ms254.0 ms


5.19

-5.64EEG (uV)

Colourwords

Formwords

Hashmarks

106 ms. 154 ms. 202 ms. 254 ms.

Figure 2. RMS peaks and their scalp distributions.

There was a significant main effect for the factor Frontality [F(1,9) = 10.28, p = 0.0101] and a

marginally significant interaction between the factors Frontality and Word Category [F(1,9) =

3.57, p = 0.0915].


There was a significant main effect of the factor Hemisphere [F(1,9) = 9.15, p = 0.0001], and

significant interactions of the factors Hemisphere and Lexicality [F(1,9) = 23.06, p = 0.0010],

Frontality, Hemisphere, and Lexicality [F(1,9) = 13.44, p = 0.0052].

Figure 3 compares the Minimum Norm Current Estimates of words (colour-related and form-

related) with hash marks. Both in the case of colour-related and form-related words, the area

where the difference between the estimated currents between words (red) and hash marks

(blue) is greatest in left posterior temporal areas. In both cases, the difference in favour of

words peaks at a point whose estimated Tallairach coordinates are x = -46, y= -41, z = -13,

located in the left fusiform gyrus. In turn, the difference in favour of the hash marks is

maximal at the right posterior cingulate, with Tallairach coordinates x = 25, y = -69, z = 10.


There was a significant main effect of the factor Frontality [F(1,9) = 12.40, p = 0.0065] and a

significant interaction for the factors Frontality and Word Category [F(1,9) = 6.96, p =

0.0270].

12

y

z

154.0 ms154.0 ms


0.14

-0.14EEG (uV)

y

z

154.0 ms154.0 ms


0.12

-0.12EEG (uV)

y

z

154.0 ms154.0 ms


0.14

-0.14EEG (uV)

y

z

154.0 ms154.0 ms


0.12

-0.12EEG (uV)

x

z

154.0 ms154.0 ms


0.14

-0.14EEG (uV)

x

z

154.0 ms154.0 ms


0.12

-0.12EEG (uV)

Colour - Hash

Form - Hash

Figure 3: Average Minimum Norm Current Estimates comparing words with hash marks in the 144-164 ms time window. Red corresponds to the points where the estimated current for words is greater than for hash marks, blue corresponds to the areas with a hig

Figure 4 compares the Minimum Norm Current Estimates for colour-related and form related

words. The areas where the activation elicited by form-related words is greater than that

elicited by colour-related words. Red indicates the areas with a higher current estimate for

colour-related than for form-related words.

Notice in Figure 4 that the difference in estimated currents in favour of colour-related words

peaks in the left temporal lobe, with approximate Tallairach coordinates x = -58, y = 1, z = -

14, corresponding to the left middle temporal gyrus, while the difference in favour of form-

related words is maximal at Tallairach coordinates x = -45, y = 48,z = -5 corresponding to the

left middle frontal gyrus, and also at x =55, y = -29, z = -9 which corresponds to the right

middle temporal gyrus.


The were no significant effects in the ANOVAs for this time window.

13

y

z

202.0 ms202.0 ms


0.13

-0.14EEG (uV)

x

z

202.0 ms202.0 ms


0.13

-0.14EEG (uV)

y

z

202.0 ms202.0 ms


0.13

-0.14EEG (uV)

Figure 4: Average Minimum Norm Current Estimates comparing colour-related words with form-related in the 192-212 ms time window. Red corresponds to the points where the estimated current for colour-related words is greater than for form-related, blue corresponds to the points where the activation for form words is greater than for colour-words

4. DISCUSSION

The present study investigated the ERPs and Minimum Norm Estimates correlating visual

lexical processing. Our analyses concentrated in the contrast between existing words and

sequences of hash marks, and in the contrast between words with strong colour associations in

their meanings, and words with stron form associations in their meanings.

Differences between Words and Sequences of Hash Marks

We found a deflection on the ERP peaking at around 154 ms. According to the ANOVAs on

the amplitudes, this peak differentiates reliably between words and sequences of hash marks.

The differences in the Minimum Norm Current Estimates indicate that words show a higher

current estimate than the sequences of hash marks. This difference in favor of words peaks at

the left fusiform gyrus, while the hash marks show a maximum advantage on the right

posterior cingulate. The estimated Tallairach coordinates of the area where the activation was

superior for words (both colour-related and form-related) is remarkably close to what has

been reported in functional magnetic resonance (fMRI) experiments to be the centre of

activation of the so-called Visual Word Form Area (VWFA; Cohen et al., 2000; 2002;

Dehaene et al., 2002; Fiez & Pedersen, 1998; McCandliss et al., 2003; Polk & Farah, 2002)

which appears to respond selectively to visually presented words. This same area also appears

to be damaged in patients that suffer from word-form dyslexia (Binder & Mohr, 1992;

Warrington and Shallice, 1980). The deficits suffered by these patients include the impaired

ability to read words (but see also Behrmann et al., 1998). In particular, based on a meta-

analysis of 25 group experiments, Cohen et al. (2002) report that the peak of the VWFA is

located at approximately at x = -43, y = -54, z = -12 in Tallairach coordinates. According to

McCandliss et al. (2003) 90% of the individual subject scans locate the peak to the VWFA

within 5 mm of this location. In our analyses, the peak of the difference between words and

sequences of hash marks was estimated to lie at x = -46, y= -41, z = -13, which is about 13

mm away from the reported peak of the VWFA (and 9 mm away from the area where it is

found in 90% of the cases). Our estimate represents the centre of a larger region and it was

obtained using a Minimum Norm Current Estimate on the EEG amplitudes, which assumes a

single layer of possible activation, not including all ventral cortical areas. Therefore we

consider our estimate to be sufficiently close to correspond to the VWFA. Previous evidence

using ERPs in left posterior areas (Bentin et al., 1997, McCandliss et al., 2002),

magnetoencephalography (MEG; Helenius et al., 1999; Tarkianen et al., 2002), and

intracranial electrical recordings (Nobre et al., 1994), indicates that activation which is

specific to written words peaks between 150 ms and 200 ms from stimulus onset. This is also

14

in line with Assadollahi and Pulvermüller (2001), who reported access to cognitive

representations of words, including sensitivity to word length and word frequency, at the 120-

160 ms interval. Again, our results fit well into this general pattern, with the main peak of

activation at the VWFA occurring at around 154 ms. From this we conclude that the

deflection that we found at 154 ms, corresponds to the VWFA previously reported in the

literature. In our experiments, we found that the activation at this peak is not sensitive to

category specific information, thus providing support for the word-form specificity of this

area. This result is also of interest in that it provides an idea of the reliable spatial resolution

of the Minimum Norm Current Estimation technique by relating it to a previously identified

cortical area.

Differences between Colour-related and Form-related Words

We found a deflection in the ERP peaking at around 202 ms that, according to the ANOVAs,

differentiated significantly between colour-related and form-related words. The differences in

the Minimum Norm Current Estimates between colour- and form-related words showed an

advantage for colour-related words peaking around left middle temporal gyrus. At the same

time, the advantage form-related over colour-related words appeared to peak around left

middle frontal and right middle temporal gyri. This specific difference between words

belonging to different semantic categories occurs early in the processing, barely 50 ms after

the peak of activation of the VWFA, and at around the same time interval where Hauk and

Pulvermüller (2004) reported category specific differences between different types of action

words. Early differentiation with respect to semantic factors is in line with theories that posit

automatic activation of all the cortical sub-networks related to the processing of a word,

inmediately following the activation of the sub-networks that encode orthographic and/or

phonological forms of the words (e.g., Pulvermüller, 2001; 2003). In addition, the possible

implication of the right middle temporal gyrus in differentiating between colour-related and

form-related words, as revealed by the comparision of the Minimum Norm Current estimates

points in the direction of distributed, bi-hemispheric networks being involved in the

representation of the meanings of words (Neininger & Pulvermüller, 2001).. Finally, the

marginally significant interaction between the factors Frontality and Word Category at the

100 ms peak provides an indication of a possible earlier differentiation in the cortical

activation elicited by these two word-categories, even 50 ms before the time when activation

peaks in the VWFA. Activation of semantic properties of words at such an early interval is in

line with the MEG results of Pulvermüller et al. (2001), who reported a differentiation between

words with strong multimodal semantic associations occurring already at 100 ms after stimulus

presentation.

15

4. SUMMARY AND OUTLOOK

Taken together, these results support the view that words semantically related to visual

information do not only differentially activate the temporal lobe, but selectively activate

frontal cortical areas as well. In particular, form-related words were found to elicit higher

prefrontal and/or premotor activation than did colour-related words. This differential

activation distinguishing between semantic word categories followed ~50 ms upon activation

of an inferior temporal area by all words under study. This area may be identified as an area

specialized in processing visual word forms, or, as an alternative, as an area generally

contributing to semantic processing of words. The time course of cortical activation seen here

is therefore compatible with models postulating sequential processing of word form and

semantics, but does not necessarily exclude other models. The present spatio-temporal

activation pattern – general word-related activation at 150 ms and category specific activation

at 200 ms – is also compatible with the view that these partly overlapping and partly distinct

patterns reflect aspects of semantic processing.

Our preliminary fMRI results are in line with the EEG source localisation results and support

the view that colour-related words primarily activate inferior and ventral temporal areas

whereas form-related words lead to stronger activation in left frontal areas, including

premotor and prefrontal sites. It remains to be investigated from which particular aspect of the

meaning of form-related words does their premotor and prefrontal activation arise. As a

candidate process, we propose action programs associated with distinct visual shapes.

16

Figure 5: fMRI activation (p < 0.0001) to words related to colour and form relative to a baseline where strings of hash marks matched in lengths to the words were presented. Methods were the same as those described in Hauk et al. (2004).

The present results provide further support for the view that information about words and

their referent objects and actions is laid down in cortex by distributed cortical cell assemblies

whose cortial topography reveals features of the information stored. Among the cortical areas

participating in lexical and semantic storage, frontal areas seem to be of particular interest,

because, as our earlier results suggest, these areas may contribute to the specific linkage of

knowledge about actions, perceptions and language. The present data now indicate that

prefrontal areas may play a specific role in binding information about abstract action related

concepts possibly tied to words that name abstract forms.

17

5. IMPLICATIONS FOR COMPUTAIONAL NEUROSCIENCE AND

BRAIN-INSPIRED ROBOTICS – FUTURE INTERACTIONS WITH

MIRRORBOT PARTNERS

IMPLICATIONS FOR COMPUTAIONAL NEUROSCIENCE AND BRAIN-INSPIRED

ROBOTICS – FUTURE INTERACTIONS WITH MIRRORBOT PARTNERS

This research provides an additional piece of evidence supporting the view that words are

represented by networks with very specific and sometimes clearly different distributions over

the cortex, and that what information is stored about a word is partially revealed by where it is

stored, and when it is being activated. This “topography of meaning” (Pulvermüller, 2001)

can be captured by models in which language is intimately tied to representations of actions

and perceptions. Such models have been further developed and implemented in WP14, based

on close interactions between the groups in Ulm, Sunderland and Cambridge (see Mirrobot

Report 12.2; see Wermter et al., 2004).

New implications of the present research on visually-related words for the

neurocomputational and rebotics work in the Mirrobot project are the following: Sometimes,

words have, according to empircal results, no semantic relationship to specific body actions,

but nevertheless activate frontal lobe, in particular prefrontal cortex. It is possible that this

activation – here reported for abstract form words – relates to abstract action plans associated

with these lexical items. In a neural model implemented on a robot, this could inspire

“abstract action concept” areas (AACA) that would model processes possibly taking place in

prefrontal cortex. Such models could serve the same role as the arm-, face- and leg-action

modules in Pulvermüller’s (2001) and Wermter et al.’s (2004) model serve for action verbs.

An AACA could thus represent and process aspects of the meaning of abstract form related

words and possibly other abstract word categories. Clearly, such a model, if correct, has

implications for views on the nature of abstract concepts and the way they are laid down in

our brains and minds and, more generally, on the cortical relationship between language,

concepts and actions (Rizzolatti et al., 2002; Gallese, 2003).

18

6. REFERENCES

Assadollahi, R. and Pulvermüller, F. (2001) Neuromagnetic evidence for early access to

cognitive representations. Neuroreport 12, 207-213

Baayen, R.H. et al. (1993) The CELEX lexical database (CD-ROM). Linguistic Data

Consortium, University of Pennsylvania, Philadelphia, PA.

Behrmann, M. et al. (1998) Visual complexity in letter-by-letter reading: ‘pure’ alexia is not

pure. Neuropsychologia 36, 1115–1132

Bentin, S. et al. (1999) ERP manifestations of processing printed words at different

psycholinguistic levels: time course and scalp distribution. Journal of Cognitive Neuroscience

11, 235–260

Berg, P. and Scherg, M. (1994): A multiple source approach to the correction of eye artifacts.

Electroencephalography Clinical Neurolorgy 90, 229 –241

Bertero, M. et al. (1988) Linear inverse problems with discrete data. II. Stability and

regularisation. Inverse Problems 4, 573–594.

Binder, J.R. and Mohr, J.P. (1992) The topography of callosal reading pathways. a case-

control analysis. Brain 115, 1807–182

Braitenberg, V. and Schüz, A. (1998) Anatomy of the cortex. Statistics and geometry. Berlin:

Springer

Chao, L. L., Haxby, J. V., & Martin, A. (1999). Attribute-based neural substrates in temporal

cortex for perceiving and knowing about objects. Nature Neuroscience, 2(10), 913-919.

Cohen, L. et al. (2000) The visual word form area: spatial and temporal characterization of an

initial stage of reading in normal subjects and posterior split-brain patients. Brain 123, 291-

307

Cohen, L. et al. (2002) Language-specific tuning of visual cortex? Functional properties of

the visual word form area. Brain 125, 1054–1069

19

Dehaene, S. (1995). Electrophysiological evidence for category-specific word processing in

the normal human brain. Neuroreport 6(16), 2153-2157.

Dehaene, S. et al. (2002) The visual word form area: a prelexical representation of visual

words in the fusiform gyrus. Neuroreport 13, 321–325

Fiez, J.A. and Pedersen, S.E. (1998) Neuroimaging studies of word reading. Proceedings of

the National Academy of Sciences U.S.A. 95, 914-921

Fuster, J.M. (1998): Linkage at the top. Neuron 21, 1223–1224.

Gallese, V. (2003). The manifold nature of interpersonal relations: the quest for a

common mechanism. Philosophical Transactions of the Royal Society London, B

Biological Sciences, 358(1431), 517-528.

Grave de Peralta Menéndez, R. et al. (1997): Linear inverse solutions with optimal resolution

kernels applied to the electromagnetic tomography. Human Brain Mapping 5, 454–467.

Hauk O. et al. (2004) Somatotopic Representation Of Action Words In Human Motor And

Premotor Cortex. Neuron 41, 301-307

Hauk, O. and Pulvermüller, F. (2004) Neurophysiological Distinction of Action Words in the

Fronto-Central Cortex. Human Brain Mapping 21, 191–201

Hauk, O. et al. (1999): The minimum norm method as an effective mapping tool for MEG

analysis in T. Yoshimoto, M. Kotani, S. Kuriki, H. Karibe, & N. Nakasato (eds.) Recent

advances in biomagnetism (Proceedings of the 11th conference on biomagnetism). Sendai:

Tohoku University Press, p 213–216

Hebb, D. O. (1949). The organization of behavior. A neuropsychological theory. New York:

John Wiley.

Helenius, P. et al. (1999) Dissociation of normal feature analysis and deficient processing of

letter-strings in dyslexic adults. Cerebral Cortex 9, 476–483

Hickok, G., and Poeppel, D. (2000). Towards a functional neuroanatomy of speech

perception. Trends in Cognitive Sciences 4(4), 131-138.

Hodges, J. R. (2001). Frontotemporal dementia (Pick's disease): clinical features and

assessment. Neurology 56(11 Suppl 4), S6-10.

20

Humphreys, G.W. and Forde, E.M.E. (2001) Hierarchies, similarity, and interactivity in

object recognition: “Category-specific” neuropsychological deficits. Behavioral and Brain

Sciences 24, 453–509.

Ishai, A. et al. (1999). Distributed representation of objects in the human ventral visual

pathway. Proceedings of the National Academy of Sciences U S A 96(16), 9379-9384.

Kellenbach, M. L. et al. (2002). Neural differentiation of lexico-syntactic categories or

semantic features? event-related potential evidence for both. Journal of Cognitive

Neuroscience 14(4), 561-577.

Kiefer, M. (2001) Perceptual and semantic sources of category-specific effects: event-related

potentials during picture and word categorization. Memory & Cognition 29, 100–116.

Kounios, J. and Holcomb, P.J. (1992): Structure and process in semantic memory: Evidence

from event-related brain potentials and reaction times. Journal of Experimental Psychology:

General 121, 459-479.

Martin, A. et al. (1996): Neural correlates of category-specific knowledge. Nature 379, 649–

652.

Martin, A. and Chao, L. L. (2001a). Semantic memory and the brain: structure and processes.

Current Oppinion in Neurobiology 11(2), 194-201.

Martin, A. and Chao, L. L. (2001b). Semantic memory and the brain: structure and processes.

Current Opinion in Neurobiology 11(2), 194-201.

McCandliss, B.D. et al. (1997) Brain plasticity in learning visual words. Cognitive

Psychology 33, 88–110

McCandliss, B.D. et al.(2002) The visual word form area: expertise for reading in the

fusiform gyrus.Trends in the Cognitive Sciences 7, 293-299

Mishkin, M. et al. (1983). Object vision and spatial vision: two cortical pathways. Trends in

Neurosciences 6 414-417.

Neininger, B. and Pulvermüller, F. (2001) The right hemisphere's role in action word

processing: A double case study. Neurocase 7, 303-317

Nobre, A.C. et al. (1994) Word recognition in the human inferior temporal lobe. Nature 372,

260–263

21

Oldfield, R.C. (1971): The assessment and analysis of handedness: The Edinburgh inventory.

Neuropsychologia 9, 97–113.

Perani, D., et al. (1999): Word and picture matching: a PET study of semantic category

effects. Neuropsychologia 37, 293–306

Polk, T.A. and Farah, M.J. (2002) Functional MRI evidence for an abstract, not perceptual,

word-form area. Journal of Experimental Psychology: General 131, 65-72

Preissl, H. et al. (1995). Evoked potentials distinguish nouns from verbs. Neuroscience

Letters 197, 81-83.

Price, C. J. et al. (2001). Dynamic diaschisis: anatomically remote and context-sensitive

human brain lesions. Journal of Cognitive Neuroscience, 13(4), 419-429.

Pulvermüller, F. (2001) Brain reflections of words and their meaning. Trends in Cognitive

Sciences 5, 517-524

Pulvermüller, F. (1996). Hebb's concept of cell assemblies and the psychophysiology of word

processing. Psychophysiology, 33, 317-333.

Pulvermüller, F. (1999). Words in the brain's language. Behavioral and Brain Sciences, 22,

253-336.

Pulvermüller, F. (2002). A brain perspective on language mechanisms: from discrete neuronal

ensembles to serial order. Progress in Neurobiology, 67, 85-111.

Pulvermüller, F. (2003). The neuroscience of language. Cambridge: Cambridge University

Press.

Pulvermüller, F. et al. (1996). Brain rhythms of language: nouns versus verbs. European

Journal of Neuroscience, 8, 937-941.

Pulvermüller, F. et al. (2001) Neuromagnetic evidence for early semantic access in word

recognition, European Journal of Neuroscience 13, 201-205

Pulvermüller, F., & Preissl, H. (1991). A cell assembly model of language. Network:

Computation in Neural Systems, 2, 455-468.

Pulvermüller, F. et al. (2000). Neurophysiological distinction of verb categories. Neuroreport,

11(12), 2789-2793.

22

Rizzolatti, G. et al. (2002). Motor and cognitive functions of the ventral premotor cortex.

Curr Opin Neurobiol, 12(2), 149-154.

Scott, S. K. and Johnsrude, I. S. (2003). The neuroanatomical and functional organization of

speech perception. Trends in Neurosciences, 26(2), 100-107.

Shtyrov, Y. et al. (2004). Distributed neuronal networks for encoding category-specific

semantic information: the mismatch negativity to action words. European Journal of

Neuroscience, 19(4), 1083-1092.

Skrandies, W. (1998): Evoked potential correlates of semantic meaning: a brain mapping

study. Cognitive Brain Research 6, 173–183.

Tarkiainen, A. et al. (2002) Dynamics of visual feature analysis and object-level processing in

face versus letter-string perception. Brain 125, 1125–1136

Tsumoto, T. (1992). Long-term potentiation and long-term depression in the neocortex.

Progress in Neurobiology, 39, 209-228.

Warrington, E.K. and McCarthy, R.A. (1987): Categories of knowledge: further

fractionations and an attempted integration. Brain 110, 1273–1296

Warrington, E.K. and Shallice, T. (1980) Word-form dyslexia. Brain 103, 99–112

Wermter, S. et al. (2004). Towards multimodal neural network robot learning. Robotics and

Autonomous Systems, in press.

Wernicke, C. (1874). Der aphasische Symptomencomplex. Eine psychologische Studie auf

anatomischer Basis. Breslau: Kohn und Weigert.

Zeki, S. and Marini, L. (1998). Three cortical stages of colour processing in the human brain.

Brain 121, 1669-1685.

23

Documents

Identification of peaks in the ERPs - uni-hamburg.de€¦ · Web viewAll word types alike were assumed to activate areas in left ... Polk & Farah, 2002) and, finally, there is the