Exp Brain Res (2009) 198:153–164

DOI 10.1007/s00221-009-1881-7

RESEARCH ARTICLE

Multisensory functional magnetic resonance imaging: a future perspective

Rainer Goebel · Nienke van Atteveldt

Received: 24 March 2009 / Accepted: 25 May 2009 / Published online: 17 June 2009© The Author(s) 2009. This article is published with open access at Springerlink.com

Abstract Advances in functional magnetic resonanceimaging (fMRI) technology and analytic tools provide apowerful approach to unravel how the human brain com-bines the diVerent sensory systems. In this perspective, weoutline promising future directions of fMRI to make opti-mal use of its strengths in multisensory research, and tomeet its weaker sides by combining it with other imagingmodalities and computational modeling.

Introduction

The potential to measure whole-brain volumes in about 2 sand its non-invasiveness make functional magnetic reso-nance imaging (fMRI) indispensible in human cognitiveneuroscientiWc research. fMRI enables mapping large-scalebrain activation (Huettel et al. 2004) as well as interactionpatterns (Friston et al. 1997; Roebroeck et al. 2005), which

yield essential exploratory knowledge on brain functioningat the systems level (Logothetis 2008). Interestingly, thesemappings are not restricted to the cortex, but may forinstance include cortical–subcortical interaction patterns.This is highly relevant since in addition to the superior col-liculi (Stein and Meredith 1993), the thalamus and thalamo-cortical interactions in particular may be important formultisensory integration (Schroeder et al. 2003; Hackettet al. 2007; Cappe et al. 2009).

A weak point is that the hemodynamic nature of thefMRI signal (the “Blood Oxygenation Level Dependent” orBOLD signal) makes it an indirect and relatively sluggishmeasure, and therefore inadequate to capture fast dynamicneural processes. Methods measuring human brain activitymore directly and with temporal resolution in the range ofneural dynamics, such as scalp and intracranial electro-encephalography (EEG) and magneto-encephalography(MEG), provide only limited coverage and have lower spa-tial accuracy (except intracranial EEG, but this method isinvasive and depends on patient populations). Therefore,the advantages of fMRI should be optimally exploited andcombined with these complementary methods. In the pres-ent perspective, we will outline recent advancements infMRI technology, design and analytical approaches that canpromote a deeper understanding of our brain’s ability tocombine diVerent sensory systems.

First, we will discuss why multisensory research posesextra challenges on how to interpret fMRI results at theneuronal level. A major challenge is the problem ofchoosing appropriate (statistical) criteria for deciding towhat extent a voxel or region is involved in integration.Typically, fMRI studies on multisensory integrationcompare fMRI responses to multisensory stimulation(e.g., audiovisual) to their unisensory counterparts (sepa-rate auditory and visual stimuli), using univariate General

R. Goebel (&) · N. van AtteveldtDepartment of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlandse-mail: R.Goebel@maastrichtuniversity.nl

N. van Atteveldte-mail: N.vanAtteveldt@maastrichtuniversity.nl

R. GoebelNetherlands Institute for Neuroscience, Meibergdreef 47, 1105 BA Amsterdam, The Netherlands

N. van AtteveldtDepartment of Psychiatry, Columbia University College of Physicians and Surgeons and New York State Psychiatric Institute, 1051 Riverside Drive, 10032 New York, NY, USA

123

154 Exp Brain Res (2009) 198:153–164

Linear Models (GLMs) at each voxel (Friston et al. 1994).Because none of the proposed metrics of multisensoryintegration (see below) that can be applied to estimatedbeta values is ideal, alternative designs and analyticaltools need to be explored. One alternative type of designis to make use of repetition suppression eVects, as is donein fMRI-adaptation (Grill-Spector and Malach 2001).Other designs that can oVer more Xexibility are multifac-torial designs in which multiple factors can be simulta-neously manipulated, e.g., semantic and temporalcorrespondence of multisensory inputs (Van Atteveldtet al. 2007), or both within- and between-group factors(Blau et al. 2009). While these approaches are based onvoxel-wise estimates, multi-voxel pattern analysis(MVPA) approaches (Haynes and Rees 2005; De Martinoet al. 2008) jointly analyze data from multiple voxelswithin a region. By focusing on distributed activity pat-terns, this approach opens the possibility to separate andlocalize spatially distributed patterns, which would bepotentially too weak to be detected by single-voxel (uni-variate) analysis. As recently applied to classify sensory-motor representations (Etzel et al. 2008), it will be veryinteresting to apply MVPA analogously to sensory-sen-sory representations to test whether representations ofevents in one modality generalize to other modalities.

Besides methodological improvements, the potentialbeneWts of technological advancements will be discussed.Scanners with ultra-high magnetic Weld strengths (¸7Tesla) provide enough signal-to-noise for functional scan-ning at sub-millimeter spatial resolution, which mayallow to directly map distributed representations at thecolumnar level (Yacoub et al. 2008). To exploit higherspatial resolution scans also at the group-level, we willdiscuss the use of advanced, cortex-based, multi-subjectalignment tools, which match corresponding macro-ana-tomical structures (gyri and sulci) across subjects. Finally,since the eventual goal is to understand dynamic pro-cesses and neuronal interactions, which take place in amillisecond time-scale, advancements in combining fMRIwith more “temporal” methods will be outlined. Forexample, because of its whole-brain coverage and non-invasive nature, fMRI can be used to raise speciWc newpredictions that can be veriWed by other methods such ashuman intracranial recordings which have both spatialand temporal resolution, but limited coverage and practi-cal restraints. A relevant example is a recent intracranialstudy demonstrating the cortical dynamics of audiovisualspeech processing (Besle et al. 2008), testing predictionsraised by previous fMRI (lacking temporal precision) andscalp EEG studies (lacking spatial precision). We willconclude by discussing the role of computational model-ing in integrating results from multisensory neuroimagingexperiments in a common framework.

Statistical inference in multisensory fMRI: how to deWne integration?

Deciding whether a neuron is “multisensory” on basis ofsingle cell recordings is relatively straightforward, usingdirectly acquired data on how the recorded neuron respondsto diVerent types of stimulation (unisensory, multisensory).Integration is thought to occur when the response to a com-bined stimulus (e.g., audiovisual) is diVerent from theresponse predicted on basis of the separate responses (e.g.,auditory and visual). The initially employed criterion is thata neuron’s spike count during multisensory stimulationshould exceed that to the most eVective unisensory stimulus(Stein and Meredith 1993). An interesting observation isthat some multisensory neurons respond super-additively:the response to multisensory stimuli not only exceeds themaximal unisensory response, but even the summed (oradditive) response to both (or multiple) sensory modalities(Wallace et al. 1996).

When dealing with fMRI data, the decision of when avoxel or region is multisensory is far more complicated. Animportant reason is that instead of single neurons (or smallunits), the responses of several hundred thousand neuronsare combined in the signal of one fMRI unit (voxel). This isa problem because voxels are quite unlikely to consist ofhomogeneous neural populations. Instead, the large sampleof neurons can be made up of mixed unisensory and multi-sensory sub-populations (Laurienti et al. 2005), and multi-sensory sub-populations on their turn can consist ofmultisensory neurons with very diverse response properties(additive, super- or sub-additive; Perrault et al. 2005).Therefore, the voxel-level response can have many diVerentorigins at the neural level. For example, an enhanced BOLDresponse for multisensory relative to unisensory stimulationcan be due to “true” multisensory neurons integrating stimu-lation from two or more sensory modalities, but it can just aswell be explained by driving two unisensory sub-popula-tions instead of one. If the latter scenario would be true, onemight wrongly infer multisensory integration at the neuronallevel. A super-additive BOLD response is less prone to suchfalse inferences (Calvert 2001), but it is unlikely to beobserved because of the same heterogeneity of responsetypes (unisensory, super-additive, sub-additive) that maycancel each other out at the voxel level (Beauchamp 2005b;Laurienti et al. 2005). Observation of an enhanced (whethersuper-additive or not) BOLD response during multisensorystimulation, therefore, has to be carefully interpreted andwill most likely be based on a mixture of multisensory andunisensory responding neurons.

Moreover, the BOLD response does not increase linearlywith increasing neuronal population activity but reaches aceiling level, i.e., it saturates (Buxton et al. 2004; Halleret al. 2006). Whereas the dynamic range of single neurons

123

Exp Brain Res (2009) 198:153–164 155

can have intrinsic functional properties (Perrault et al.2005), the limited dynamic range of the BOLD response isa characteristic of the vascular system (for instance, limitedcapability of vessel dilation) and therefore confounds neu-rofunctional interpretations. In other words, BOLD satura-tion might conceal increased neuronal population responsesto multisensory stimulation, especially when unisensorystimuli already evoke substantial responses (Fig. 1a). Thismay result in false negatives, since integration at the neuro-nal level is not well reXected at the voxel level.

Statistical criteria: diVerent classiWcations in diVerent situations

When using fMRI studies to identify multisensory integra-tion regions in the human brain, we have to seek for means

to objectively deWne integrative fMRI responses. Severalstatistical criteria have been suggested to infer multisensoryintegration from fMRI data (Calvert 2001; Beauchamp2005b; Laurienti et al. 2005; Driver and Noesselt 2008;Stevenson et al. 2008) ranging from stringent to liberal,respectively: the criterion of super-additivity, the max crite-rion and the mean criterion. The super-additivity criterionstates that the multisensory response should exceed the sumof the unisensory responses to be deWned as integrative.The max criterion is deWned in analogy to the criterion usedto infer multisensory enhancement or suppression on thesingle neuron level (Stein and Meredith 1993) and statesthat the multisensory fMRI response should be strongerthan the most eVective unimodal response. The most liberalcriterion is the mean criterion, stating that the multisensoryresponse should exceed the mean of the unimodalresponses. Typically, integration is deWned by a positive

Fig. 1 ClassiWcation by diVerent statistical criteria (columns) forhypothetical brain regions with diVerent unisensory (fMRI) responseproWles (a–c). a Heteromodal response: a signiWcant response to bothunisensory stimulation modalities (auditory and visual). b Auditory-speciWc response and a weak visual response. c Auditory-speciWcresponse and a negative visual response. Bars indicate the fMRI acti-vation level for diVerent unisensory and multisensory stimulationconditions: visual (V red), auditory (A green), and two diVerent audio-visual/multisensory conditions (M1 dark blue; M2 light blue). The dotted

line in the Wrst column (“BOLD max”) represents the maximal fMRIresponse due to hemodynamic saturation. The solid lines in columns2–4 represents the classiWcation criterion: summed unisensory activationlevel (A + V) for the super-additivity criterion, maximal unisensoryactivation level ([A, V]max) for the “Max” criterion, and mean unisen-sory activation level (A + V)/2 for the “Mean” criterion. Plus and mi-nus symbols indicate classiWcation type (super-additivity/enhancementvs. sub-additivity/suppression) and strength

123

156 Exp Brain Res (2009) 198:153–164

outcome using any of the criteria (super-additivity orenhancement); in this case the stimuli are assumed to“belong together”. A negative outcome is typically inter-preted as inhibited processing (sub-additivity or suppres-sion), which can be viewed as another type/direction ofintegration, for stimuli that are assumed to “not belongtogether”. No diVerence between multi- and unisensoryresponses (additivity, no interaction) is interpreted as nointegration, in case two inputs do not inXuence each other’sprocessing in that voxel or region.

To gain insight in how the diVerent criteria reach theirclassiWcations, Fig. 1 illustrates their outcomes with respectto speciWc multisensory (“M”) responses, in regions withdiVerent unisensory (visual and auditory) response proWles.Region “A” shows a heteromodal response, i.e., the arearesponds signiWcantly to both unisensory stimulation types.This proWle is, for instance, typical for regions in the poster-ior superior temporal sulcus (STS; see Amedi et al. 2005;Beauchamp 2005a). Regions “B” and “C” show a sensory-speciWc (auditory) response (typical for areas in auditorycortex), with a weak visual response in “B” and a negativevisual response in “C”. The three integration criteria areapplied to the fMRI activity for two diVerent multisensory(audiovisual) stimulus types “M1” and “M2”. M1 evokes astrong fMRI response, higher than each of the unisensoryresponses, whereas M2 evokes a much weaker response thatdoes not exceed either of the unisensory responses. As willbe discussed below, the Wgure shows that unisensoryresponse proWles as well as the BOLD response saturationlevel (“BOLD max”) both aVect classiWcation of the fMRIresponse to M1 and M2 diVerently for the three criteria.

In region “A”, the sum of the two unisensory responsesexceeds the BOLD saturation level implying that nomultisensory response can show super-additivity. As aconsequence, both M1 and M2 responses are classiWed assub-additive (¡), and hence as not or negatively “integra-tive”, even though M1 is clearly boosted and M2 is not. Boththe max- and the mean criteria classify the response to M1 asenhanced (+) and M2 as suppressed (¡) relative to unisen-sory responses. In region “B”, the summed response does notexceed the BOLD saturation level and hence the response toM1 is now classiWed as super-additive (+), and the weak M2response as sub-additive (¡). The max criterion again classi-Wes the M1 response as enhanced (+) and the M2 response assuppressed (¡). In contrast, the mean criterion classiWes bothM1 and M2 as enhanced (+), while the M2 response does notexceed the auditory response. In region “C”, the summedresponse is actually lower than each of the unisensoryresponses because one of the responses is negative. Thesuper-additivity criterion classiWes both M1 and M2responses as super-additive (+), but it is questionable howmeaningful it is to sum the responses when one is negative(Calvert 2001). The mean criterion also classiWes both M1

and M2 as enhanced, whereas the max criterion still classiWesM1 as enhanced (+) and M2 as suppressed (+).

The super-additivity criterion seems to be prone to falsenegatives in regions such as “A” due to BOLD saturation, andpossibly to false positives in “C” due to a negative response inone of the modalities. The Wrst bias can be limited by using“weak” stimuli to prevent BOLD saturation (Calvert 2001;Stevenson et al. 2008). Note that stimuli at detection thresholdare also recommended from a neural perspective (inverseeVectiveness; Stein and Meredith 1993), since such stimuliincrease the need for integration. On the other extreme, themean criterion seems to be too liberal especially when one ofthe unisensory responses is weak (“B”) or negative (“C”),which reduces the mean in such a way that a multisensoryresponse exceeds the mean even when weaker than the largestunisensory response. Therefore, the mean criterion can bemisleading, especially when examining low-level sensoryregions such as the auditory cortex.

In sum, whereas saturation confounds can be avoided bypresenting weak stimuli, both super-additivity and mean cri-teria seem biased toward classifying a multisensory responseas integrative in sensory-speciWc brain regions (like “B” or“C”). This is problematic because many recent studies sup-port involvement of low-level sensory-speciWc brain regionsin multisensory integration (reviewed in Schroeder and Foxe2005; Ghazanfar and Schroeder 2006; Macaluso 2006;Kayser and Logothetis 2007; Driver and Noesselt 2008). Asargued in the introduction, an asset of fMRI is that functionalmaps can be created over the whole brain. In such whole-brain analyses, identical statistical tests are performed in allsampled voxels. Therefore, it is important that a criterion formultisensory integration is suitable in all voxels, regardlessof diVerent unisensory response proWles. The classiWcationbased on the max criterion seems most robust to diVerentunisensory response proWles. It can also be argued that this isa disadvantage, because the classiWcation by itself does notgive any insight in the response in the least-eVective modal-ity. However, although the other criteria are based on a com-bination of both unisensory responses, diVerent combinationscan lead to the same threshold. This points out that no matterwhich criterion is used, it is of utmost importance to inspectand report the unisensory response levels (% signal change,averaged time-courses, or b-estimates) in addition to show-ing maps for a certain test (Beauchamp 2005b). This isnecessary to fully understand why a criterion has been met bya voxel or region, and to judge the meaningfulness of acertain classiWcation.

Because all above discussed criteria for comparingmultisensory to unisensory responses have limitations, aninteresting alternative is to manipulate the congruency ofthe diVerent inputs (Doehrmann and Naumer 2008), forinstance with regard to stimulus identity (Van Atteveldtet al. 2004) or statistical relation (Baier et al. 2006). In this

123

Exp Brain Res (2009) 198:153–164 157

type of analysis, two bimodal conditions are contrastedwith each other (congruent vs. incongruent), which elimi-nates the unimodal component and its accompanying com-plications from the metric. This comparison follows theassumption that a distinction between congruent and incon-gruent cross-modal stimulus pairs can not be establishedunless the unimodal inputs have been integrated success-fully; therefore the congruency contrast can be used as asupplemental criterion for multisensory integration. Anadditional advantage of using congruency manipulations isthat it facilitates inclusion of diVerent factors within thesame design, e.g., temporal, spatial and/or semantic relationbetween the cross-modal inputs. Such multi-factorialdesigns allow to directly addressing questions regardingrelative contributions and interactions between diVerent(binding) factors (Sestieri et al. 2006; Van Atteveldt et al.2007; Blau et al. 2008; Noppeney et al. 2008). Interest-ingly, between-group factors can also be included in suchmodels to assess group diVerences in integration, as in arecent study that revealed defective multisensory integra-tion of speech sounds and written letters in developmentaldyslexia (Blau et al. 2009).

Unsolved issues and suggested approaches

Whichever statistical criterion applied, the subsequentinterpretation will always be limited because of the hetero-geneity of the measured voxels. As already pointed out inthe introduction, an observed voxel-level response can havemany diVerent origins at the neuronal level because voxelsare likely to consist of mixed populations of uni- and multi-sensory neurons (Laurienti et al. 2005; Driver and Noesselt2008). As a consequence, interpretations following theabove outlined statistical criteria are never exclusive, andalternative designs and analytical tools need be explored. Inthe following, we will discuss alternative designs andanalytical approaches that might circumvent some of theproblems caused by heterogeneity within voxels (fMRI-adaptation), or make optimal use of spatially heterogeneousresponse patterns (MVPA). Finally, technical advance-ments pushing the limits of high-resolution fMRI will beconsidered. As we will see, resolution in the scale of corti-cal columns is in reach, which might drastically reduceundesired heterogeneity within the units of measurement.

Alternative fMRI design: fMRI-adaptation

The fMRI-adaptation paradigm (fMRI-A) is based on thephenomenon of reduced neural activity for repeated stimuli(repetition suppression) and hypothesizes that by targetingspeciWc neuronal populations within voxels, their func-

tional properties can be measured beyond the voxel resolu-tion (Grill-Spector and Malach 2001; Grill-Spector 2006).The typical procedure is to adapt a neuronal population byrepeated presentation of the same stimulus in a control con-dition (fMRI signal reduces), and to vary one stimulusproperty and assess recovery from adaptation in the maincondition(s). If adaptation remains (fMRI signal stays low),the adapted neurons respond invariantly to the manipulatedproperty, whereas a recovered (i.e., increased) fMRI signalindicates sensitivity to that property, i.e., that at least par-tially, a diVerent set of neurons is responding within thevoxel. Within sensory systems, there are many examples inwhich fMRI-A revealed organizational structures that couldnot be revealed using more standard stimulation designs.Since (presumably) only the targeted neural populationadapts, its functional properties can be investigated withoutbeing mixed with responses of other neural populationswithin the same voxel. In the visual system for example,heterogeneous clusters of feature-selective neurons (e.g.,for diVerent object orientations) within voxels wererevealed using fMRI-A (Grill-Spector et al. 1999), whereasin a more standard stimulation design, the averaged voxel-response was not diVerent for the diVerent features since allof them activated a neural population within that voxel.Interestingly, fMRI-A has also been used to investigatesub-voxel level integration of features within the visualmodality (Self and Zeki 2005; Sarkheil et al. 2008). Note,however, that the exact neuronal mechanism underlyingBOLD adaptation is still uncertain (Grill-Spector et al. 2006;Krekelberg et al. 2006; Sawamura et al. 2006; Bartels et al.2008.

Human “multisensory” cortex is most likely composedof a mixture of unisensory and multisensory subpopula-tions. In a high-resolution fMRI study, Beauchamp and col-leagues demonstrated that human multisensory STSconsists of mixed visual, auditory and audiovisual subpopu-lations (Beauchamp et al. 2004). These diVerent neurontypes were organized in clusters on a millimeter scale,which might indicate an organizational structure similar tothat of cortical columns, as is also indicated by anatomicalwork in macaques (Seltzer et al. 1996). Cortical columnsconsist of about hundred thousand neurons with similarresponse speciWcity, for example, orientation columns inV1 (Hubel and Wiesel 1974), or feature-selective columnsin inferotemporal visual cortex (Fujita et al. 1992). As out-lined above, such a heterogeneous organization of unisen-sory and multisensory neuronal populations below thevoxel resolution (typically around 3 £ 3 £ 3 = 27 mm3),limits the certainty with which voxel-level responses can beinterpreted in terms of neuronal processes. A clear exampleis that an enhanced BOLD response to multisensory stimu-lation can be due to integration at the neuronal level, but itcan be explained equally well by a mix of two separate

123

158 Exp Brain Res (2009) 198:153–164

unisensory populations. fMRI-A might be helpful to distin-guish between voxels in multisensory cortex containingonly unisensory neuronal subpopulations and voxels com-posed of a mixture of uni- and multisensory populations.DiVerent adaptation and recovery responses could shedlight on the sub-voxel organization: multisensory neuronsshould adapt to cross-modal repetitions (alternating modali-ties, e.g., A-V), while unisensory neurons should not or atleast less (see below). This might be used to disentangleunisensory and multisensory neural populations. Anotherapproach is to present repetitions of multisensory stimuliand vary the (semantic or other) relation between them(e.g., congruent vs. incongruent pairs), to test whether ornot voxels contain neurons that are sensitive to this relation,assuming that these should be multisensory (Van Atteveldtet al. 2008).

Unfortunately, there are several potential pitfalls forsuch designs. Whereas in the example from the visual sys-tem, diVerent neuronal subpopulations are selective to one-speciWc feature (e.g., maximum response to an orientationof 30°) or are not selective (e.g., orientation-invariant);diVerent populations in multisensory cortex can be selec-tive to “features” in diVerent conditions: visual repetitionsmay adapt visual and audiovisual neurons, auditory repeti-tions may adapt auditory and audiovisual neurons. This canbe problematic because neurons are shown to adapt despiteintervening stimuli (Grill-Spector 2006), so stimulus repeti-tions in alternating modalities will also adapt unisensoryneurons (although probably to a weaker extent). Anotherproblem is that a cross-modal repetition (e.g., visual–audi-tory) may suppress activity of multisensory neurons, butwill also activate new pools of unisensory neurons (in thisexample: auditory) in the same voxel with mixed neuronalpopulations, which may counteract the cross-modal sup-pression. In sum, fMRI-A designs to investigate multisen-sory integration may help interpreting representationalcoding at the neuronal level, but great caution is warranted.

Alternative analytical approach: multivariate statistics

While standard hypothesis-driven fMRI analyses using theGLM process the time-course of each voxel independently,and data-driven methods, such as independent componentanalysis search for functional networks in the wholefour-dimensional data set, several new analysis approachesfocus on rather local, MVPA methods (Haxby et al.2001; Haynes and Rees 2005; Kamitani and Tong 2005;Kriegeskorte et al. 2006; De Martino et al. 2008) In theseapproaches, data from individual voxels within a region arejointly analyzed. An activity pattern is represented as a fea-ture vector where each feature refers to an estimatedresponse measure of a speciWc voxel. The dimension N of

an fMRI feature vector, thus, corresponds to the number ofincluded voxels in the analysis. Using standard statisticaltools, distributed activity patterns corresponding to diVerentconditions may be compared using multivariate approaches(e.g., MANOVA). Alternatively, machine learning tools[(e.g., support vector machines (SVMs)] are trained on asubset of the data while leaving some data aside for testinggeneralization performance of the trained “machine” (clas-siWer). More robust estimates of the generalization perfor-mance of a classiWer are obtained by cross-validationtechniques involving multiple splits of the data in trainingand test sets. To solve diYcult classiWcation problems, non-linear classiWers may be used but for problems with high-dimensional feature vectors and a relatively small numberof training patterns, non-linear kernels are usually notrequired. This is important for fMRI applications, becauseonly linear classiWers allow to use the obtained weight val-ues (one per voxel) for direct visualization of the voxels’contribution to the classiWcation performance. Linear clas-siWers, thus, allow to perform “multivariate brain mapping”by localizing discriminative voxels. By properly weightingthe contribution of individual voxels across a region (local),multivariate pattern analyses approaches open the possibil-ity to separate spatially distributed patterns, which wouldbe potentially too weak to be discovered by univariate(voxel wise) analysis. Note that the joint analysis of weaksignals from multiple voxels does not require that voxelswithin a region behave in the same way, since it extractsdiscriminative information within the multivariate signal.If, for example, some voxels in a local neighborhood showa weak increase and other voxels a weak decrease whencomparing two conditions, these opposing eVects wouldcancel out in a regional average using a standard GLManalysis, but would well contribute to a measure of multi-variate information. The gained higher sensitivity allows toseparate similar distributed representations from each otherdue to the integration of weak information diVerencesacross voxels. Note that a high sensitivity for distinguishingdistributed representations would be important even if acolumnar-level resolution would allow a more direct map-ping of representational units since representations ofexemplars (e.g., two faces) might only slightly diVer intheir distributed code across the same basic features.

A recent publication (Formisano et al. 2008) has shownthat distributed patterns extending from early auditory cor-tex into STS/STG contain enough information to reliablyseparate responses evoked by individual vowels spoken bydiVerent speakers from each other; performance of trainedclassiWers was indicated by successful generalization tonew exemplars of learned vowels even if they were spokenby a novel speaker. Note that this discriminative powercould not be observed when analyzing responses fromsingle voxels. Another relevant recent fMRI study

123

Exp Brain Res (2009) 198:153–164 159

demonstrated that classiWers (SVMs) that were trained toseparate sensory events using activation patterns in premo-tor cortex, could also reliably separate the correspondingmotor events (Etzel et al. 2008).

We propose to follow a similar approach to investigatethe multisensory nature of sensory-sensory representationsin multisensory cortex. It would, for example, be interest-ing to teach classiWers to discriminate responses from mul-tisensory (e.g., audiovisual) stimuli in order to obtain moreinformation about how speciWc stimulus combinations arerepresented in sensory-speciWc cortex (e.g., visual and audi-tory association cortex) and multisensory cortex (e.g.,STS); diVerent training signals (classiWcation labels) couldbe used for the same fMRI data by either learning labelsrepresenting the full cross-modal pairs or by using only thevisual or auditory component of a pair. These diVerenttraining tasks should reveal which parts in the cortical net-work would more prominently code for visual, auditory orthe combination of both stimuli; furthermore, if learningand generalization would be successful, the identiWed rep-resentations would allow predicting from a single trialresponse which speciWc audio-visual combination was pre-sented to the subject. Such knowledge would be highly rel-evant for building computational models of multisensoryprocessing, which will be discussed in the Wnal section.

Increased spatial resolution

Increasing spatial resolution is an obvious approach toobtain more detailed fMRI data, which might additionallyhelp to shed some light on the Wne-grained functional orga-nization of small areas in the human brain (Logothetis2008). High-resolution functional imaging beneWts fromhigher MRI Weld strength since small (millimeter or evensub-millimeter) voxels still possess a reasonable signal-to-noise ratio. As an example, a 7 Tesla fMRI study showedtonotopic, mirror-symmetric maps within early humanauditory cortex (Formisano et al. 2003). The describedtonotopic maps were much smaller than the large retino-topic maps in human visual cortex, which could thereforebe observed already 10 years earlier with 1.5 Tesla scan-ners (Sereno et al. 1995). Another example is the study ofBeauchamp and colleagues (2004), in which the authorsused parallel imaging to achieve a spatial resolution of1.6 £ 1.6 £ 1.6 mm3, providing insight in the moredetailed organization of uni- and multisensory clusters inposterior STS.

Despite progress in high-resolution functional imaging,it is unclear what level of eVective spatial resolution can beachieved with fMRI since the ultimate spatial (and tempo-ral) resolution of fMRI is not primarily limited by technicalconstraints but by properties of the vascular system. The

spatial resolution of the vascular system, and hence fMRI,seems to be in the order of 1 millimeter since relevant bloodvessels run vertically through cortex in a distance of about amillimeter (Duvernoy et al. 1981). An achievable resolu-tion of 0.5–1 mm might be just enough to resolve corticalcolumns (Mountcastle 1997). According to theoretical rea-soning and empirical data (e.g., orientation columns in V1;Hubel and Wiesel 1974), a cortical column is assumed tocontain about hundred thousand neurons with similarresponse speciWcity. A conventional brain area, such as thefusiform face area, could contain a set of about 100 corticalcolumns, each coding a diVerent elementary (e.g., face) fea-ture. Cortical columns could, thus, form the basic buildingblocks of complex distributed representations (Fujita et al.1992). DiVerent entities within a speciWc area would becoded by a speciWc distributed activity pattern across corti-cal columns, like letters or vowels in superior temporal cor-tex (Formisano et al. 2008). If this reasoning is correct, animportant research strategy would aim to unravel the spe-ciWc building blocks in various parts of the cortex, includ-ing individual representations of letters, speech sounds andtheir combinations in auditory cortex and heteromodalareas in the STS/STG. Since neurons within a cortical col-umn code for roughly the same feature, measuring the brainat the level of cortical columns promises to provide a rele-vant level for revealing meaningful distributed neuronalcodes. Recently it became indeed possible to reliably mea-sure orientation columns in the human primary visual cor-tex using high-Weld (7 Tesla) fMRI (Yacoub et al. 2008).This study clearly indicates that columnar-resolution fMRIis indeed possible—at least when using high-Weld fMRIcombined with spin-echo MRI pulse sequences. If the orga-nization of uni- and multisensory neuronal populations isorganized at a columnar level, which is hinted at by fMRI(Beauchamp et al. 2004) and animal work (Seltzer et al.1996), this would strongly increases the feasibility ofhigh-Weld fMRI to provide insight in neuronal multisensoryintegration because putative multisensory and unisensorycolumns could separately be measured, and thus distributedactivity patterns across them could be analyzed.

Making optimal use of spatial resolution in group analyses: cortex-based alignment

Inspection and reporting of individual activation eVects isvery important, but some eVects may only reach signiW-cance when performing group analyses. Moreover, ran-dom-eVects group analysis are essential for assessingconsistency of eVects within groups, and to reveal diVer-ences between groups. Unfortunately the typically usedcoarse brain normalization in volume space (e.g., Talairachor MNI space) compromises the gain of high-resolution

123

160 Exp Brain Res (2009) 198:153–164

imaging since suYcient spatial correspondence can only beachieved through substantial spatial smoothing (e.g., with aGaussian kernel with a FWHM of 8–12 mm). Since manyinteresting multisensory eVects may be observed only inWne-grained activity patterns, spatial smoothing should beminimal or completely avoided. Moreover, standard volu-metric Talairach or MNI template brain matching tech-niques may lead to suboptimal multi-subject results due topoor spatial correspondence of relevant areas (Van Essenand Dierker 2007). Surface-based techniques aligning gyriand sulci across subjects (Fischl et al. 1999; Van Atteveldtet al. 2004; Goebel et al. 2006) may substantially improvespatial correspondence between homolog macro-anatomi-cal brain structures such as STS/STG across subjects. Suchan improved alignment may provide more sensitive statisti-cal results under the assumption that functional regions“respect” macro-anatomical landmarks (Spiridon et al.2005; Hinds et al. 2009); a systematic and large-scale func-tional-anatomical correspondence project is currently inprogress to verify this assumption (Frost and Goebel 2008).

The eVectiveness of surface-based alignment procedureson group statistical maps have been reported in severalrecent studies (reviewed in Van Essen and Dierker 2007).Here, we show its eVectiveness also for multisensory corti-cal areas, such as those demonstrated in STS/STG. Figure 2shows a direct comparison of surface-based and volume-based (Talairach) registration of group data for a multisensoryinvestigation of letter-sound integration (Van Atteveldtet al. 2004). The Wgure illustrates that the analysis with cor-tex-based aligned data improved the statistics and providedmore accurate localization of multisensory eVects in audi-tory cortex and STS. In Fig. 2a, the max criterion resultedin a more robust map (higher threshold) that is much moreclearly localized on STS (and additional clusters on STG)using cortex-based alignment. It is important to verify thisstatement with regard to localization by comparing thegroup maps with individual localizations. The details ofindividual STS ROIs (reported in Van Atteveldt et al. 2004)show variability in Talairach coordinates: average §standard deviation (x, y, z) = (¡54 § 4, ¡33 § 11, 7 § 6),

so most variability in the y coordinate (=anterior–posterioraxis). These individual ROIs were selected based on indi-vidual anatomy, i.e., they were all located on the STS.Importantly, comparison of the Talairach and cortex-basedgroup statistical maps (Fig. 2a) indicates that the averagedTalairach coordinates do not correspond to the location ofthe individual ROIs on STS (Fig. 2a, left: cluster is locatedon STG overlapping partly with auditory cortex), whereasthe cortex-based aligned map shows the dominant clusterclearly localized on STS (Fig. 2a, right). Figure 2b showsthat despite a variable individual anatomy of the auditorycortex indicated in the top row (5 diVerent subjects), thecortex-based aligned group maps accurately locate the

multisensory congruency eVect on Heschl’s sulcus and Pla-num Temporale.

As functional-anatomical correspondence may vary fordiVerent brain regions and functions, a complementaryapproach to account for individual variability is the use offunctional localizers, which allow to “functionally align”brains (Saxe et al. 2006). Future multisensory fMRI studiescould use this approach to functionally localize integrationareas, e.g., by using the max criterion. Group statistics cansubsequently be performed using each subject’s fMRI time-series from the functionally deWned ROI in that subject.Note, however, that there are also pitfalls regarding the useof (separate) functional localizers (Friston and Henson2006; Friston et al. 2006) and intra-subject consistency ofcertain localizers was recently reported to be very low(Duncan et al. 2009). Experiments incorporating functionallocalizers should therefore be designed with care, forinstance, it might be best to embed localizer contrasts infactorial designs (i.e., orthogonal to the main manipulationof interest; Friston et al. 2006).

Dynamic processes and neuronal interactions

Several valuable approaches exist to study temporal charac-teristics of neural processing from fMRI time-series. Forexample, Granger Causality Mapping (GCM, Goebel et al.2003; Roebroeck et al. 2005) is an eVective connectivitytool with the potential to estimate the direction of inXu-ences between brain areas directly from the voxel’s time-course data, which we recently applied to assess inXuencesto/from STS during letter-sound integration (Van Atteveldtet al. 2009). Another dynamic eVective connectivity tool isdynamic causal modeling (DCM; Friston et al. 2003),which was recently used to investigate the neural mecha-nism of visuo-auditory incongruency eVects for objects andspeech (Noppeney et al. 2008). Furthermore, informationabout the onset of the fMRI response (BOLD latency map-ping; Formisano and Goebel 2003) can reveal insight in thetemporal sequence of neural events, and has successfullybeen applied to multisensory research (Martuzzi et al.2007; Fuhrmann Alpert et al. 2008).

Still, using fMRI alone it is diYcult to learn about fastdynamic processes in “real-time” since successive temporalevents lead to an integrated BOLD response. In multisen-sory research, this severely limits the certainty with whichcross-modal eVects observed with fMRI can be interpretedin terms of processing stage (early vs. late, or feedforwardvs. feedback), which is therefore often a matter of heavydebate. A prevailing example concerns the modulation ofauditory cortex activity by visual speech cues (Calvert et al.1997, 1999; Paulesu et al. 2003; Pekkola et al. 2005;reviewed in Campbell 2008), which is typically interpreted

123

Exp Brain Res (2009) 198:153–164 161

as being the result of feedback projections from the hetero-modal STS/STG (reviewed in Calvert 2001). As stated inthe introduction, because of its whole-brain coverage andnon-invasive nature, fMRI can be used to raise speciWc newpredictions that can be tested by intracranial human record-ings, which have both spatial and temporal resolution. Thishas been done recently for the case of audiovisual speechprocessing. Besle and colleagues (2008) recorded ERP’sintracranially from precise locations in the temporal lobeduring audiovisual speech processing, and demonstratedthat visual inXuences in (secondary) auditory cortexoccurred earlier in time than the eVects in STS/STG. TheseWndings advocate a direct feedforward activation of audi-tory cortex by visual speech information.

Another very promising direction is to directly inte-grate fMRI and EEG/MEG. While progress has beenmade in recent years (Lin et al. 2006; Goebel and Esposito

2009), it remains diYcult to reliably separate closelyspaced electrical sources from each other. This is of rele-vance for multisensory research in case nearby locatedauditory and multisensory superior temporal regions needto be separated. The combination of EEG and fMRI datasets into one unique data model is still a focus of intensiveresearch and requires enormous eVorts to integrate inde-pendent Welds of knowledge such as physics, computerscience and neuroscience. In fact, besides the classicalproblems of head modeling in EEG, and hemodynamicmodeling in fMRI, an additional diYculty is given by theneed of understanding and modeling the ongoing correla-tions of EEG and fMRI data. Some of these problems aresolved by direct intra-cranial electrical recordings fromthe human brain (e.g., inverse modeling), but such studiesare limited because they are invasive and depend onpatient populations. Despite the diYculties in properly

Fig. 2 fMRI group analysis results using volumetric normalization(Talairach space) and cortex-based alignment. a Random-eVects statis-tical maps of two diVerent contrasts: audiovisual congruent versusaudiovisual incongruent (orange) and the max criterion expressed asthe conjunction (intersection) of audiovisual versus auditory & audio-visual versus visual (green). The maps show that at higher t values, thecortex-based aligned data still provide a better group map (i.e., locationof the clusters correspond best to the activations in individual subjects).b Individual (top row) and group (bottom row) statistical maps of the

contrasts audiovisual congruent versus audiovisual incongruent (darkblue) and auditory versus baseline (light blue). Top row shows thereconstructed and Xattened cortical sheets of the left temporal lobe inWve representative individual subjects, the bottom row shows the cor-tex-based aligned group statistics (of 16 subjects) on a representativeleft and right temporal lobe. White lines indicate the diVerent sulci andborders between the gyri, from anterior to posterior: FTS Wrst trans-verse sulcus, HG Heschl’s gyrus, HS Heschl’s sulcus, PT planum tem-porale, STS superior temporal sulcus

123

162 Exp Brain Res (2009) 198:153–164

integrating detailed temporal information from (simulta-neously) recorded EEG signals into fMRI analysis, theexpected insights will be essential to build advanced spa-tio-temporal models of multisensory processing in thehuman brain.

Toward computational modeling of multisensory processing

Data from a series of fMRI experiments from our grouphave provided insight in the likely role of brain areasinvolved in letter-speech sound integration as well as theinformation Xow between these areas (reviewed in VanAtteveldt et al. 2009). As has been highlighted in theprevious sections, data might soon be available provid-ing further constraints at the representational level ofindividual visual, auditory and audiovisual entities suchas letters, speech sounds and letter-sound combinations.In light of the richness of present and potential futuremultisensory data, it seems to become feasible to buildcomputational process models to further stimulate dis-cussions of neuronal mechanisms. In the future, we aimto implement large-scale recurrent neural network mod-els because they (1) allow to clearly specify structuralassumptions in the connection patterns within andbetween simulated brain areas and (2) allow to preciselypredict the implications of structural assumptions byfeeding the networks with relevant unimodal and bimo-dal stimuli (e.g., visual letters, speech sounds and theiraudiovisual combinations). Running spatio-temporalsimulations may help to understand how “emergent”phenomena, such as audiovisual congruency eVects inauditory cortex (Van Atteveldt et al. 2004), result frommultiple simultaneously operating synaptic inXuencesfrom diVerent modeled brain regions. Such neural mod-els may also help to link results from electrophysiologi-cal animal studies and fMRI studies. The fMRI BOLDdata reXect a mix of (suprathreshold) spiking activity,hemodynamic spread, and neural spread of subthresholdneural activity (Logothetis et al. 2001; Logothetis andWandell 2004; Oeltermann et al. 2007; Maier et al.2008). To investigate discrepancies between spiking,LFP and BOLD data, these signals must be modeled sep-arately in an environment that permits a comparisonwith matching empirical data (Goebel and De Weerd2009). To compare activity patterns in simulated neuralnetworks with empirical data, modeled cortical columnscan be linked to topographically matching voxels; theselinks implement spatial hypotheses and are obtainedfrom structural brain scans and functional mapping stud-ies in human subjects, thereby establishing a commonrepresentational space for simulated and measured data.

This allows to “run” large-scale neural network models“in the brain” and to analyze predicted fMRI data usingthe same analysis tools as used for the measured data(e.g., GLM, MVPA, GCM). Such a tight integration ofcomputational modeling and fMRI data may help to testand compare the implications of speciWed neuronal cod-ing principles and to study the evolution of dynamicinteractions (Goebel and De Weerd 2009). If these prin-ciples are applied to multisensory experiments, assump-tions about the proportion of unisensory andmultisensory neurons in voxels in diVerent brain areascan be explicitly explored and derived predictions can betested by conducting theory-guided neuroimagingstudies.

Conclusion

When using fMRI at standard resolution (·3T, single-coil imaging) and (mass-) univariate statistical analysis(e.g., using the GLM), the statistical criteria for “multi-sensory integration” should be selected with care, andresponse characteristics of all uni- and multisensoryconditions should always be inspected and reported.Alternative designs such as fMRI-adaptation may pro-vide additional insights in multisensory integrationbeyond the voxel level. In addition to analyzing single-voxel responses, MVPA is an important new statisticalapproach to jointly analyze locally distributed activitypatterns of multiple voxels. Because of the putative het-erogeneous organization of multisensory brain areas,distinct integrative states may be expressed in such dis-tributed activity patterns rather than in the response ofseparate voxels. Increased spatial resolution might ulti-mately lead to columnar-level resolution. If multisen-sory cortex is organized at the columnar level, fMRImight be able to separate uni- and multisensoryresponses, making the application of MVPA even moreinteresting. When data are aligned based on individualcortical anatomy, the high spatial resolution of fMRI canalso be fully exploited at the group-level. Although alsoproWting from higher Weld strength and imaging technol-ogy, the ultimate temporal resolution is limited by thevascular system (and neuro-vascular coupling) and willnot be suYcient to capture fast dynamic neural processesand interactions. Additional information from comple-mentary imaging modalities (EEG, MEG) should there-fore be acquired and integrated. Furthermore, directcomparison (in a common representational space) ofmodeled neural activity and the corresponding BOLDactivity (predicted fMRI data) with real fMRI data willhelp interpreting multisensory hemodynamic data interms of neural mechanisms.

123

Exp Brain Res (2009) 198:153–164 163

Acknowledgments This work is supported by the Dutch Organiza-tion for ScientiWc Research (NWO, VENI grant # 451-07-020 to NvA)and the European Community’s Seventh Framework Programme ([FP/2007-2013] grant # 221187 to NvA).

Open Access This article is distributed under the terms of the Crea-tive Commons Attribution Noncommercial License which permits anynoncommercial use, distribution, and reproduction in any medium,provided the original author(s) and source are credited.

References

Amedi A, von Kriegstein K, Van Atteveldt NM, Beauchamp MS, Nau-mer MJ (2005) Functional imaging of human crossmodal identi-Wcation and object recognition. Exp Brain Res 166:559–571

Baier B, Kleinschmidt A, Muller NG (2006) Cross-modal processingin early visual and auditory cortices depends on expected statisti-cal relationship of multisensory information. J Neurosci26:12260–12265

Beauchamp M (2005a) See me, hear me, touch me: multisensory inte-gration in lateral occipital-temporal cortex. Curr Opin Neurobiol15:1–9

Beauchamp M (2005b) Statistical criteria in fMRI studies of multisen-sory integration. Neuroinformatics 3:93–113

Beauchamp MS, Argall BD, Bodurka J, Duyn J, Martin A (2004)Unraveling multisensory integration: patchy organization withinhuman STS multisensory cortex. Nat Neurosci 7:1190–1192

Besle J, Fischer C, Bidet-Caulet A, Lecaignard F, Bertrand O, GiardMH (2008) Visual activation and audiovisual interactions in theauditory cortex during speech perception: intracranial recordingsin humans. J Neurosci 28:14301–14310

Blau V, Van Atteveldt N, Formisano E, Goebel R, Blomert L (2008)Task-irrelevant visual letters interact with the processing ofspeech sounds in heteromodal and unimodal cortex. Eur J Neuro-sci 28:500–509

Blau V, Van Atteveldt N, Ekkebus M, Goebel R, Blomert L (2009) Re-duced neural integration of letters and speech sounds links phono-logical and reading deWcits in adult dyslexia. Curr Biol 19:503–508

Buxton RB, Uludaf K, Dubowitz DJ, Liu TT (2004) Modeling the hemo-dynamic response to brain activation. Neuroimage 23:S220–S233

Bartels A, Logothetis NK, Moutoussis K (2008) fMRI and its interpre-tations: an illustration on directional selectivity in area V5/MT.Trends Neurosci 31(9):444–453

Calvert GA (2001) Crossmodal processing in the human brain: insightsfrom functional neuroimaging studies. Cereb Cortex 11:1110–1123

Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC,McGuire PK, WoodruV PW, Iversen SD, David AS (1997) Activa-tion of auditory cortex during silent lipreading. Science 276:593–596

Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD,David AS (1999) Response ampliWcation in sensory-speciWc cor-tices during crossmodal binding. Neuroreport 10:2619–2623

Campbell R (2008) The processing of audio-visual speech: empirical andneural bases. Philos Trans R Soc Lond B Biol Sci 363:1001–1010

Cappe C, Morel A, Barone P, Rouiller EM (2009) The thalamocorticalprojection systems in primate: an anatomical support for multi-sensory and sensorimotor interplay. Cerebral Cortex, Jan 15[Epub ahead of print]

De Martino F, Valente G, Staeren N, Ashburner J, Goebel R,Formisano E (2008) Combining multivariate voxel selection andsupport vector machines for mapping and classiWcation of fMRIspatial patterns. Neuroimage 43:44–58

Doehrmann O, Naumer MJ (2008) Semantics and the multisensorybrain: how meaning modulates processes of audio-visual integra-tion. Brain Res 1242:136–150

Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodalinXuences on ‘sensory-speciWc’ brain regions, neural responses,and judgments. Neuron 57:11–23

Duncan KJ, Pattamadilok C, Knierim I, Devlin JT (2009) Consistency andvariability in functional localisers. Neuroimage 46(4):1018–1026

Duvernoy HM, Delon S, Vannson JL (1981) Cortical blood vessels ofthe human brain. Brain Res Bull 7:519–579

Etzel JA, Gazzola V, Keysers C (2008) Testing simulation theory withcross-modal multivariate classiWcation of fMRI data. PLoS ONE3:e3690

Fischl B, Sereno MI, Tootel RBH, Dale AM (1999) High-resolutionintersubject averaging and a coordinate system for the corticalsurface. Hum Brain Mapp 8:272–284

Formisano E, Goebel R (2003) Tracking cognitive processes withfunctional MRI mental chronometry. Curr Opin Neurobiol13:174–181

Formisano E, Kim D-S, Di Salle F, van de Moortele P-F, Ugurbil K,Goebel R (2003) Mirror-symmetric tonotopic maps in humanprimary auditory cortex. Neuron 40:859–869

Formisano E, De Martino F, Bonte M, Goebel R (2008) “Who” is say-ing “what”? Brain-based decoding of human voice and speech.Science 322:970–973

Friston KJ, Henson RN (2006) Commentary on: divide and conquer; adefence of functional localisers. Neuroimage 30:1097–1099

Friston KJ, Holmes AP, Worsley KJ, Poline JB, Frith CD, FrackowiakRS (1994) Statistical parametric maps in functional imaging: ageneral linear approach. Hum Brain Mapp 2:189–210

Friston KJ, Buchel C, Fink GR, Morris J, Rolls E, Dolan RJ (1997)Psychophysiological and modulatory interactions in neuroimag-ing. Neuroimage 6:218–229

Friston K, Harrison L, Penny W (2003) Dynamic causal modelling.Neuroimage 19:1273–1302

Friston KJ, Rotshtein P, Geng JJ, Sterzer P, Henson RN (2006) A cri-tique of functional localisers. Neuroimage 30:1077–1087

Frost M, Goebel R (2008) The structural–functional correspondenceproject. Neuroimage 41

Fuhrmann Alpert G, Hein G, Tsai N, Naumer MJ, Knight RT (2008)Temporal characteristics of audiovisual information processing. JNeurosci 20:5344–5349

Fujita I, Tanaka K, Ito M, Cheng K (1992) Columns for visual featuresof objects in monkey inferotemporal cortex. Nature 360:343–346

Ghazanfar AA, Schroeder C (2006) Is neocortex essentially multisen-sory? Trends Cogn Sci 10:278–285

Goebel R, De Weerd P (2009) Perceptual Wlling-in: from experimentaldata to neural network modeling. In: Gazzaniga MS (ed) The cog-nitive neurosciences IV. MIT Press, Cambridge

Goebel R, Esposito F (2009) The added value of EEG-fMRI in imag-ing neuroscience. In: Mulert C, Lemieux L (eds) CombinedfMRI-EEG data analysis. Springer, Berlin

Goebel R, Roebroeck A, Kim DS, Formisano E (2003) Investigatingdirected cortical interactions in time-resolved fMRI data usingvector autoregressive modeling and Granger causality mapping.Magn Reson Imaging 21:1251–1261

Goebel R, Esposito F, Formisano E (2006) Analysis of FunctionalImage Analysis Contest (FIAC) data with BrainVoyager QX: fromsingle-subject to cortically aligned group general linear modelanalysis and self-organizing group independent component anal-ysis. Hum Brain Mapp 27:392–402

Grill-Spector K (2006) Selectivity of adaptation in single units: impli-cations for fMRI experiments. Neuron 49:170–171

Grill-Spector K, Malach R (2001) fMR-adaptation: a tool for studyingthe functional properties of human cortical neurons. Acta Psychol107:293–321

Grill-Spector K, Kushnir T, Edelman S, Avidan G, Itzchak Y, Malach R(1999) DiVerential processing of objects under various viewing con-ditions in the human lateral occipital complex. Neuron 24:187–203

123

164 Exp Brain Res (2009) 198:153–164

Grill-Spector K, Henson R, Martin A (2006) Repetition in the brain: neu-ral models of stimulus-speciWc eVects. Trends Cogn Sci 10:14–23

Hackett TA, De La Mothe LA, Ulbert I, Karmos G, Smiley J,Schroeder CE (2007) Multisensory convergence in auditorycortex, II thalamocortical connections of the caudal superiortemporal plane. J Comp Neurol 502:924–952

Haller S, Wetzel SG, Radue EW, Bilecen D (2006) Mapping continu-ous neuronal activation without an ON–OFF paradigm: initialresults of BOLD ceiling fMRI. Eur J Neurosci 24:2672–2678

Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P(2001) Distributed and overlapping representations of faces andobjects in ventral temporal cortex. Science 293:2425–2430

Haynes JD, Rees G (2005) Predicting the orientation of invisible stim-uli from activity in human primary visual cortex. Nat Neurosci8:686–691

Hinds O, Polimeni JR, Rajendran N, Balasubramanian M, Amunts K,Zilles K, Schwartz EL, Fischl B, Triantafyllou C (2009) Locatingthe functional and anatomical boundaries of human primary visu-al cortex. Neuroimage 46(4):915–922

Hubel DH, Wiesel TN (1974) Sequence regularity and geometry oforientation columns in the monkey striate cortex. J Comp Neurol158:267–293

Huettel SA, Song AW, McCarthy G (2004) Functional magneticresonance imaging. Sinauer Associates, Inc, Sunderland

Kamitani Y, Tong F (2005) Decoding the visual and subjectivecontents of the human brain. Nat Neurosci 8:679–685

Kayser C, Logothetis NK (2007) Do early sensory cortices integratecross-modal information? Brain Struct Funct 212:121–132

Krekelberg B, Boynton GM, van Wezel RJA (2006) Adaptation: fromsingle cells to BOLD signals. Trends Neurosci 29:250–256

Kriegeskorte N, Goebel R, Bandettini P (2006) Information-based func-tional brain mapping. Proc Natl Acad Sci USA 103:3863–3868

Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005)On the use of superadditivity as a metric for characterizing multi-sensory integration in functional neuroimaging studies. Exp BrainRes 166:289–297

Lin FH, Witzel T, Ahlfors SP, StuZebeam SM, Belliveau JW,Hämäläinen MS (2006) Assessing and improving the spatialaccuracy in MEG source localization by depth-weighted mini-mum-norm estimates. Neuroimage 31:160–171

Logothetis NK (2008) What we can do and what we cannot do withfMRI. Nature 453:869–878

Logothetis NK, Wandell BA (2004) Interpreting the BOLD signal.Annu Rev Physiol 66:735–769

Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001)Neurophysiological investigation of the basis of the fMRI signal.Nature 412:150–157

Macaluso E (2006) Multisensory processing in sensory-speciWc corti-cal areas. Neuroscientist 12:327–338

Maier A, Wilke M, Aura C, Zhu C, Ye F, Leopold DA (2008) Diver-gence of fMRI and neural signals in V1 during perceptual sup-pression in the awake monkey. Nat Neurosci 11:1193–1200

Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP, Clarke S,Meuli RA (2007) Multisensory interactions within human primarycortices revealed by BOLD dynamics. Cereb Cortex 17:1672–1679

Mountcastle VB (1997) The columnar organization of the neocortex.Brain 120:701–722

Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) TheeVect of prior visual information on recognition of speech andsounds. Cereb Cortex 18:598–609

Oeltermann A, Augath MA, Logothetis NK (2007) Simultaneousrecording of neuronal signals and functional NMR imaging.Magn Reson Imaging 25:760–774

Paulesu E, Perani D, Blasi V, Silani G, Borghese NA, De Giovanni U,Sensolo S, Fazio F (2003) A functional-anatomical model forlipreading. J Neurophysiol 90:2005–2013

Pekkola J, Ojanen V, Autti T, Jaaskelainen IP, Mottonen R, TarkiainenA, Sams M (2005) Primary auditory cortex activation by visualspeech: an fMRI study at 3 T. Neuroreport 16:125–128

Perrault TJJ, Vaughan JW, Stein BE, Wallace MT (2005) Superior col-liculus neurons use distinct operational modes in the integrationof multisensory stimuli. J Neurophysiol 93:2575–2586

Roebroeck A, Formisano E, Goebel R (2005) Mapping directed inXu-ence over the brain using Granger causality and fMRI. Neuroim-age 25:230–242

Sarkheil P, Vuong QC, BülthoV HH, Noppeney U (2008) The integra-tion of higher order form and motion by the human brain. Neuro-image 42:1529–1536

Sawamura H, Orban GA, Vogels R (2006) Selectivity of neuronaladaptation does not match response selectivity: a single-cell studyof the fMRI adaptation paradigm. Neuron 49:307–318

Saxe R, Brett M, Kanwisher N (2006) Divide and conquer: a defenseof functional localizers. Neuroimage 30:1088–1096

Schroeder CE, Foxe JJ (2005) Multisensory contributions to low-level,‘unisensory’ processing. Curr Opin Neurobiol 15:1–5

Schroeder CE, Smiley JF, Fu KG, McGinnis T, O’Connel MN, HackettTA (2003) Anatomical mechanisms and functional implicationsof multisensory convergence in early cortical processing. Int JPsychophysiol 50:5–17

Self M, Zeki S (2005) The integration of colour and motion by the hu-man visual brain. Cereb Cortex 15:1270–1279

Seltzer B, Cola MG, Gutierrez C, Massee M, Weldon C, Cusick CG(1996) Overlapping and nonoverlapping cortical projections tocortex of the superior temporal sulcus in the rhesus monkey: dou-ble anterograde tracer studies. J Comp Neurol 370:173–190

Sereno MI, Dale AM, Reppas JB, Kwong KK, Belliveau JW, BradyTJ, Rosen BR, Tootell RB (1995) Borders of multiple visual areasin humans revealed by functional magnetic resonance imaging.Science 268:889–893

Sestieri C, Di Matteo R, Ferretti A, Del Gratta C, Caulo M, Tartaro A,Olivetti Belardinelli M, Romania GL (2006) “What” versus“Where” in the audiovisual domain: an fMRI study. Neuroimage33:672–680

Spiridon M, Fischl B, Kanwisher N (2005) Location and spatial proWleof category-speciWc regions in human extrastriate cortex. HumBrain Mapp 27:77–89

Stein BE, Meredith MA (1993) The merging of the senses. MIT Press,Cambridge

Stevenson RA, Geoghegan ML, James TW (2008) SuperadditiveBOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179:85–95

Van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration ofletters and speech sounds in the human brain. Neuron 43:271–282

Van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) TheeVect of temporal asynchrony on the multisensory integration ofletters and speech sounds. Cereb Cortex 17:962–974

Van Atteveldt N, Blau V, Blomert L, Goebel R (2008) fMR-adaptationreveals multisensory integration in human superior temporal cor-tex. In: Annual meeting of the international multisensory researchforum, Hamburg, Germany

Van Atteveldt NM, Roebroeck A, Goebel R (2009) Interaction ofspeech and script in human auditory cortex: insights from neuro-imaging and eVective connectivity. Hear Res, June 2 [Epub aheadof print]

Van Essen DC, Dierker DL (2007) Surface-based and probabilisticatlases of primate cerebral cortex. Neuron 56:209–225

Wallace MT, Wilkinson LK, Stein BE (1996) Representation and inte-gration of multiple sensory inputs in primate superior colliculus.J Neurophysiol 76:1246–1266

Yacoub E, Harel N, Ugurbil K (2008) High-Weld fMRI unveils orien-tation columns in humans. Proc Natl Acad Sci USA 105:10607–10612

123