Experimental setup

During scanning, the participant lay comfortably in a supine position on the bed of the MR scanner. To ensure that multisensory stimulation of the hand was as natural as possible, we used a setup that gave the subjects direct view of their hand and that employed tactile stimulation conducted with real objects (Fig. 1, left).

Open in a separate window

Fig. 1.

Experimental setup and design. Photograph of the experimental setup and diagram of the experimental design. The participant's head was tilted to provide direct vision of the right hand, which was placed on an angled support. Inset: how the hand looked from the participant's perspective and illustrates the approximate position of the fixation point, above the right hand. The experimental design was a 2 × 2 factorial, manipulating the modality of stimulation and the presence thereof. Rest_EC, rest condition with eyes closed. T, tactile; V, visual; VT, visuotactile.

This was achieved by tilting the participant's head downward and placing his or her right hand on a custom-made support positioned above the waist. The required tilt of the head was achieved by slanting the head coil with the help of a custom-made wooden wedge (through an angle of ~11°). In addition, the participant's head was tilted by another 20° using pillows and foam pads. The participant wore MR-compatible headphones to dampen the scanner noise as well as to receive the appropriate set of instructions throughout the acquisition sessions (see following text). The resting left hand was placed in a comfortable position alongside the participant's body. Instructions were transmitted to the experimenter and the participant via a computer running Presentation 14.1 (Neurobehavioral Systems, Albany, CA) and connected to an MR-compatible sound delivery system (Nordic Neuro Lab, Bergen, Norway).

Experimental design

A 2 × 2 factorial design was used, with the sensory modality of stimulation (visual, tactile) and presence thereof (present, absent) as the experimental factors (Fig. 1, right). A block design was used to maximize the efficiency of the experimental design (Friston et al. 1999). Each stimulation or baseline epoch lasted 16 s and was followed by a 3 s intertrial interval during which both experimenter and participant received a brief audio instruction. The latter identified the upcoming stimulation epoch for the experimenter and cued the participant as to whether keep his or her eyes open or closed. Two baseline conditions were included, with eyes open and closed, respectively. This made it possible to use separate baselines for the visual and tactile unisensory conditions, thereby ensuring that the conjunction analysis was based on orthogonal contrasts (Beauchamp et al. 2008; Bremmer et al. 2001; Friston et al. 2005; Nichols et al. 2005; Price and Friston 1997). The participant was instructed to keep his or her eyes closed during purely tactile stimulation and during the corresponding baseline or open during all visual conditions including the relative baseline. During all epochs involving visual (-tactile) stimulation, the participant was further instructed to fixate on a point ~2 cm above his or her right hand. The fixation point was marked and visible during all conditions carried out with the participant's eyes open. All stimulations were delivered using a spherical object made of a soft material attached to a 50-cm-long stick, which allowed the experimenter to hide his hand from the participant's view while still permitting carefully controlled stimulation. We used real physical objects rather than computer generated stimuli because the former are known to be more potent in activating multisensory neurons in experiments in non-human primates (Graziano and Gandhi 2000). During tactile stimulation, the participant's right index finger was touched by the same object used in the visual condition. A trained experimenter moved the ball on the skin from the distal to the proximal phalanx, yielding a discrete number of events in the 16-s-long epoch. In total, 10 brush strokes were applied to the index finger during each tactile or visuotactile stimulation epoch. To ensure that the same number of stimuli was presented during each epoch, the experimenter followed a prerecorded metronome. Visual stimulation close to the hand was delivered by moving the object at ~2 cm from the participant's right index finger following a sequence of events identical to the one applied during the tactile epochs. During bisensory (visuotactile) stimulation, the experimenter repeated the tactile stimulation pattern while the subject fixed his or her gaze just above his or her own right hand. All participants underwent three sessions of functional acquisition, each composed of five blocks in which all epochs (including the baselines) were repeated once. The baseline condition made with eyes closed always followed the tactile epoch, while the baseline condition made with eyes open always followed the last of the visual epochs, the order of which was randomized across sessions.

fMRI acquisition

fMRI acquisition was performed using a Siemens TIM Trio 3T scanner equipped with a 12-channel head coil (Siemens, Erlangen, Germany). We acquired gradient echo T2*-weighted echo-planar images with blood-oxygenation-level-dependent (BOLD) contrast as an index of brain activity (Logothetis et al. 2001). A functional image volume comprised 47 continuous near-axial slices of 3-mm thickness (with a 0.3-mm interslice gap), which ensured that the whole brain was within the field of view (58 × 76 matrix, 3.0 × 3.0 mm in-plane resolution, TE = 40 ms). One complete volume was collected every 3 s (TR = 3,000 ms). A total of 513 functional volumes were collected for each participant (171 in each run). In each run, an initial baseline of 18 s and a final baseline of 20 s were included. To facilitate anatomical localization of statistically significant activations, a high-resolution structural image was acquired for each participant at the beginning of the experiment (3D MPRAGE sequence, voxel size 1 × 1 × 1 mm, FOV 250 × 250 mm, 176 slices, TR = 1,900 ms, TE = 2.27 ms, flip angle = 9°).

Additive and superadditive responses

At the level of a single neuron, the firing rates induced by uni- and multisensory stimuli can be directly compared and classified into additive and sub- and superadditive (Stein and Stanford 2008; Wallace et al. 1996). All three types of interaction are known to contribute to behavior and perception (Stanford and Stein 2007). In fact, linear, additive responses are the most common mode of response of individual multisensory neurons to non near-threshold stimuli (Stanford and Stein 2007; Stanford et al. 2005). Therefore in the present study, we interpret both linear and nonlinear responses in the hypothesized areas as reflecting biologically meaningful multisensory integration.

This approach calls for some further critical discussion when employed with BOLD-based fMRI as this neuroimaging technique measures signals from a large population of (possibly heterogeneous) neurons within a voxel (Logothetis 2008; Logothetis et al. 2001). In other words, a voxel could contain both uni- and multisensory neuronal populations. This has sparked an important discussion in the field of multisensory brain imaging concerning which criteria should be used to identify multisensory responses. One highly influential study argued that superadditivity should be taken as the “golden standard” (Calvert 2001). More recent reviews have, however, challenged this claim and have suggested that other types of response should also be carefully investigated (Goebel and van Atteveldt 2009; Laurienti et al. 2005; Stein et al. 2009). The central point in this discussion is that fMRI studies cannot directly discriminate between areal and neuronal convergence (Calvert 2001; Goebel and van Atteveldt 2009; Stevenson et al. 2007). In principle, a linear BOLD response could be produced by two separate populations of unisensory cells co-existing within the same voxel. The other side of this coin, though, is that a nonlinear, superadditive response at the voxel level cannot be simply explained away as convergence of sensory inputs onto different, noninteracting neuronal populations (Calvert 2001; Goebel and van Atteveldt 2009; Laurienti et al. 2005; Stevenson et al. 2007). Thus a superadditive BOLD response must reflect a neuronal interaction of some sort, be it expressed at the level of discharge rates or local field potentials (Logothetis et al. 2001). It is important to note that there are several arguments that can be called on to explain why the superadditive criterion may turn out to be too conservative, two such being: the tendency of neurons to respond nonlinearly only for weak, near-threshold stimuli (Laurienti et al. 2005; Stanford et al. 2005; Stevenson et al. 2007) and vascular ceiling effects for the BOLD response (Buxton et al. 2004), which might prevent detection of nonlinear interactions (Calvert 2001; Goebel and van Atteveldt 2009; Stevenson et al. 2007). In light of this, the linear responses we observed in the left ventral premotor cortex and the left supramarginal gyrus, where we had particularly strong a priori anatomo-functional hypotheses (see introduction), most likely reflect biologically relevant evidence of the occurrence of multisensory integration in these areas.

Posterior parietal cortex

The posterior parietal cortex is generally considered to be a key region for the fusion of signals from different sensory modalities. In our study, we have found clear evidence for the convergence of visual and somatic signals on and their integration in several key areas in this part of the brain. The anatomical locations of these response patterns in the parietal lobe fit well with earlier electrophysiological studies on non-human primates (Avillac et al. 2005; Colby and Duhamel 1991; Colby et al. 1993; Duhamel et al. 1998; Hyvärinen 1981; Hyvärinen and Poranen 1974), as will be discussed in some detail in this section.

It is particularly important that we were able to detect a zone of convergence of visual and tactile signals on and their integration in the anterior part of the intraparietal sulcus. This activation exhibited a superadditive response profile and was located on the medial bank of the anterior segment of the intraparietal sulcus, close to the junction with the postcentral sulcus; the cluster of active voxels extended into the superior parietal gyrus as well. The overlap between the active cluster obtained in the conjunction analysis of the two unisensory conditions and the cluster identified by the positive interaction contrast in the factorial analysis was highly convincing. The location of these activations on the medial bank of the intraparietal sulcus fits very well with the activations reported in this region during the rubber hand illusion (Ehrsson et al. 2004). This illusion is associated with perceptual binding of tactile and visual stimuli, consistent with the evidence presented here that this area is a site of visuotactile convergence and integration. Furthermore, our results confirm and go beyond the data presented by Makin and colleagues (2007), showing evidence not only of multisensory convergence, but also of integration of visual and tactile hand-related signals in the anterior intraparietal area.

It is not straightforward to relate the anatomical location of the activated areas directly to specific functional regions defined in and around the intraparietal sulcus in the macaque brain, as homological relations have not been fully established (Culham et al. 2006; Grefkes and Fink 2005). In the monkey brain, the anterior intraparietal area (AIP) lays on the lateral bank of the sulcus, while areas VIP (ventral intraparietal area) and MIP (medial intraparietal area) are found on the medial bank (Grefkes and Fink 2005). Thus it is possible that the activation under discussion was driven by active neurons in the two latter areas rather than in the former. Taken together, the results presented here provide compelling support for the notion that the anterior intraparietal cortex contributes to the multisensory representation of the arm in space (Makin et al. 2007, 2008), in addition to its well-established role in the sensory guidance of grasp-related action (Binkofski et al. 1998; Culham et al. 2003, 2006; Ehrsson et al. 2000, 2001).

Anterior to the intraparietal sulcus, we observed multisensory responses in the form of distinct peaks of activation in the cortex lining the postcentral sulcus. The anterior bank of this sulcus corresponds to area 2 of the primary somatosensory cortex, the posterior bank being associated with area 5 of the posterior parietal cortex instead. Area 2, although traditionally considered part of the primary somatosensory cortex, is thought to be involved in higher order somatosensory processing (Bodegård et al. 2001; Iwamura 1998; Roland et al. 1998), with an emphasis on proprioceptive signals (Naito et al. 2005). Area 5 is generally considered to be a somatosensory association area (Bodegård et al. 2000; Duffy and Burchfiel 1971; Mountcastle et al. 1975; Sakata et al. 1973; Taoka et al. 2000). Based on research on nonhuman primates, areas 2 and 5 are not taken to be the primary candidate sites for visuotactile integration in the parietal lobe (the intraparietal and inferior posterior parietal cortex being stronger candidates; see introduction). However, in area 5 of the monkey brain, neurons with both visual and tactile receptive fields have been described (Graziano et al. 2000; Iriki et al. 1996; Sakata et al. 1973). Graziano and colleagues showed how these cells integrate visual and somatic information for the representation of arm position in space (Graziano et al. 2000), although the greatest proportion of cells of this type was observed in the part of area 5 located in the medial bank of the intraparietal sulcus (MIP) (Graziano and Botvinick 2000). Nevertheless, our human data support the existing evidence in non-human primates that the cortex lining the postcentral sulcus is involved in the integration of visual and somatic information from the hand. Anatomically, area 5 receives prominent inputs from the primary somatosensory cortex, and in particular from area 2 (Pandya and Kuypers 1969; Pearson and Powell 1985). Additionally, area 5 is also connected to the primary motor cortex and premotor areas and is involved in reaching movements (Andersen et al. 1997; Johnson et al. 1996; Mountcastle et al. 1975; Scott et al. 1997). Therefore it is possible that the activations observed in passive participants under discussion here reflect a multisensory mechanism for the localization of the arm in space, a prerequisite for potential hand-object interactions.

The inferior posterior parietal cortex was also identified as a site of multisensory convergence and visuotactile integration in the present study. The corresponding activation was located in the left supramarginal gyrus. Although the precise homology between cytoarchitectonical areas of human and monkey inferior parietal lobes is not fully known (Eidelberg and Galaburda 1984), it has been suggested that the inferior parietal cortex in monkeys (area 7b/PF) likely corresponds to the supramarginal gyrus in the human brain (Krams et al. 1998). In monkeys, this region has anatomical connections to areas 2, 4, 5, and 45, the parietal operculum (Hyvärinen 1982; Neal et al. 1987 1990b), ventral premotor cortex (Ghosh and Gattera 1995; Godschalk et al. 1984; Neal et al. 1990a), and insula (Neal 1990; Neal et al. 1987), all of which are areas that displayed evidence of multisensory integration in the present study. Neurons with bimodal visuotactile responses have been found in this region (Hyvärinen 1981; Hyvärinen and Poranen 1974; Ishida et al. 2010). The inferior parietal cortex is an important node in the fronto-parietal circuits involved in sensory-guided hand actions (Luppino and Rizzolatti 2000); in human neuroimaging studies, it is consistently activated during hand-object interactions (Binkofski et al. 1998; Ehrsson et al. 2000, 2001; Kuhtz-Buschbeck et al. 2001). The supramarginal gyrus is also activated by proprioceptive stimulation inducing sensations of limb movement (Naito et al. 2005), demonstrating that this region receives significant proprioceptive inputs. The data reported here are interesting in this respect as they show that the involvement of this region in the sensory representation of the hand is fundamentally multisensory in nature, continuously combining visual and somatic data from the hand in space.

Parietal operculum and insula

We also found superadditive responses in clusters located in the contralateral (left) insula and in the right parietal operculum; the left parietal operculum exhibited additive multisensory responses. The parietal operculum is part of the secondary somatosensory area (SII) in humans, a relatively early processing node in the somatosensory system (Eickhoff et al. 2006). In human brain imaging studies, it is consistently activated by a wide range of somatic stimuli (Banati et al. 2000; Bodegård et al. 2000; Ledberg et al. 1995; Naito et al. 1999; O'Sullivan et al. 1994; Roland et al. 1998). Furthermore, activations in the parietal operculum are induced by passive movements (Weiller et al. 1996) and illusory arm movements elicited by tendon vibration (Naito et al. 2005). In monkeys, the parietal operculum receives prominent tactile and deep receptor inputs (Robinson and Burton 1980) and is anatomically connected to the inferior posterior parietal cortex (Disbrow et al. 2003) and to the ventral premotor cortex (Cipolloni and Pandya 1999; Disbrow et al. 2003; Ghosh and Gattera 1995; Godschalk et al. 1984; Matelli et al. 1986) and thus probably works in concert with these areas to represent the spatial location of the arm and an object touching it in space. Moreover, a recent study offered further evidence that regions within the human parietal operculum are anatomically connected with key multisensory areas in the frontal and parietal lobes (Eickhoff et al. 2010). In the light of this, the parietal operculum could be an important processing node in the network involved in the multisensory perception of the hand in space as suggested by our results.

In addition, we observed multisensory effects in the left insular cortex. The insula, together with the adjacent claustrum, has been suggested to be an important cortical site for multisensory integration (Bushara et al. 2001a; Hadjikhani and Roland 1998; Naghavi et al. 2007). In particular, the insula/claustrum region has been associated with cross-modal visuotactile processing for recognition (Banati et al. 2000) and discrimination (Hadjikhani and Roland 1998) tasks. The finding that we report here provide new evidence that the basic integration of visual and somatic signals from a static hand and an object touching it is performed in this multisensory area.

Frontal responses

The premotor cortex is a site of convergence of visual, tactile, and proprioceptive information (Rizzolatti et al. 1981; Graziano et al. 1997). It is also involved in the control of goal-directed hand actions (Rizzolatti et al. 1998) and defensive movements (Graziano et al. 2002). In the present study, we found clear evidence in the left (contralateral) premotor cortex for the convergence of visual and somatic signals from the hand as well as of integration of these signals within this area. In the ventral premotor cortex, multisensory enhancement was additive; in addition to this, in the dorsal part of the premotor cortex, we found evidence for superadditive effects. These results fit well with electrophysiological studies on nonhuman primates, which have stressed the importance of the premotor cortex in processing visual and tactile sensory signals from the body and its surrounding space.

Bimodal visuotactile neurons have been described in the ventral premotor cortex of the macaque (Graziano and Gandhi 2000; Graziano et al. 1997; Rizzolatti et al. 1981) with overlapping tactile and visual receptive fields anchored to specific body parts. In particular, Graziano and colleagues described bisensory neurons that are sensitive to somatosensory and visual stimulation of the monkey's hand (Graziano and Botvinick 2002; Graziano and Gandhi 2000; Graziano et al. 1997). These responses are also present in anesthetized monkeys (Graziano and Gandhi 2000), which suggests that premotor neurons perform multisensory integration even when the monkey is not planning or performing an action. In the present study, participants held their arm still and did not perform any motor tasks, so the measured premotor responses most likely reflect basic visuotactile integration mechanisms. This interpretation fits with earlier fMRI studies that reported premotor activations during visuotactile stimulation of the arm in passive subjects (Ehrsson et al. 2004; Lloyd et al. 2003).

Besides the ventral premotor cortex, we also found evidence of visuotactile integration in the dorsal premotor cortex. Although this region of the premotor cortex has not been as extensively characterized as its ventral neighbor in terms of its multisensory properties, visuotactile neurons have been described in the caudal part of this area (Fogassi et al. 1999; Graziano and Gandhi 2000). The convergence of proprioceptive, tactile, and visual information, as well as its connectivity pattern with several regions in the posterior parietal lobe, in particular area 5 (Rizzolatti et al. 1998), has led scientists to assign to the dorsal premotor cortex an important role in monitoring and controlling arm and hand movements in space (Rizzolatti et al. 1998; Scott et al. 1997). Integration of visual, tactile, and proprioceptive information in the dorsal premotor cortex might therefore be important for calibrating the representation of limb position (Ehrsson et al. 2004) as part of a circuit devoted to the control of arm and hand movements (Kertzman et al. 1997).

Putamen

The evidence provided here for visuotactile integration in the human putamen was particularly noteworthy. In the macaque putamen, bisensory, visuotactile neurons with body-centered receptive fields, similar to those found in frontal and parietal regions, were identified in a classic paper by Graziano and Gross (1993). In this report, the authors described how neurons with tactile receptive fields organized somatotopically into arm, hand, and face regions were also responsive to visual stimulation and in particular visual stimuli originating from the space close to the body (Graziano and Gross 1993). Given the tight anatomical connections with multisensory frontal and parietal regions (Cavada and Goldman-Rakic 1991; Graziano and Gross 1993; Künzle 1978; Parthasarathy et al. 1992; Weber and Yin 1984), the putamen is considered part of the visuotactile network responsible for motor interactions with environmental objects (Graziano and Gross 1993). To the best of our knowledge, the evidence we have found of superadditive BOLD responses in the contralateral (left) putamen provides the most conclusive evidence today that the human putamen performs integration of visual and somatic signals from the hand, possibly in hand-centered reference frames. The previously mentioned fMRI studies, which included unisensory visual and tactile stimulation conditions (Makin et al. 2007) and bisensory stimulation (Ehrsson et al. 2004; Lloyd et al. 2003), failed to identify consistent activation in the putamen. Our stimulation paradigm using robust natural bisensory stimulation of the hand and functional imaging at a higher field strength of 3T, rather than the 1.5T in the reported studies, are most likely responsible for the increase in sensitivity that allowed us to describe activations in this small subcortical structure.

Cerebellum

In the study being presented here, the cerebellum displayed evidence of convergence of visual and tactile hand-related signals as well as of multisensory integration of both the additive and superadditive type. All activations were localized in the ipsilateral (right) cerebellum. Although the cerebellum has long been considered to be primarily involved in the control of action, recent developments have suggested that this structure also plays an important role in perceptual and cognitive functions (Gao et al. 1996; Ramnani 2006; Schmahmann and Sherman 1998; Strick et al. 2009). The possibility that there is convergence of signals from different sensory modalities in cerebellar regions is supported by extensive knowledge of the functional connectivity of this structure in nonhuman primates. Visual (Glickstein et al. 1994; Sultan and Glickstein 2007), tactile (Bloedel 1973; Bushara et al. 2001b; Snider and Stowell 1944), and kinesthetic (Murphy et al. 1973; van Kan et al. 1993) information is conveyed to the cerebellum either directly or indirectly, reinforcing the notion that this structure could be an important node in cortico-cerebellar networks for the processing of multisensory information. In humans, neuroimaging studies have demonstrated the involvement of the cerebellum in multisensory processing (Bushara et al. 2001a; Hagura et al. 2009; Kavounoudias et al. 2008; Naumer et al. 2010; Stevenson et al. 2009) as well as cerebellar activation following tactile stimulation of the hand in passive participants (Bushara et al. 2001b; Fox et al. 1985), which is consistent with our results. This accumulating body of evidence suggests that the cerebellum may play a role in the multisensory network devoted to the representation of the limb across sensory modalities, although the present data cannot substantiate any speculation on the functional nature of such involvement.

Multisensory circuit for body ownership

Our results are also relevant for studies of body ownership, where the integration of vision and touch is assumed to be the fundamental mechanism behind the self-attribution of limbs (Ehrsson et al. 2004, 2005, 2007; Makin et al. 2008) and entire bodies (Petkova and Ehrsson 2008). Crucial fMRI evidence for this idea came from the observation that touches sensed on an owned rubber hand were associated with activations in the ventral premotor and intraparietal cortices (Ehrsson et al. 2004). However, in this earlier study and similar ones conducted subsequently, the authors did not include a condition where the real hand was stimulated to demonstrate that these areas are robustly activated when one's own real hand is stimulated naturally in full view (Ehrsson et al. 2004, 2005, 2007). In light of this, the present findings that tactile stimulation of a real hand being watched by its “owner” activates the ventral premotor and intraparietal cortices, regions recognized to be sites of convergence and integration of visual and tactile signals, provide an important validation of the multisensory hypothesis of body ownership (Ehrsson et al. 2004; Makin et al. 2008; Petkova and Ehrsson 2008; Tsakiris 2010).

The authors thank U. Keiller, T. Jonsson, and T.-Q. Li for help with the setup. The authors also thank three anonymous reviewers, whose comments helped improve this manuscript.