Skip to main content
SpringerLink
Log in

Neuroimaging of multisensory processing in vision, audition, touch, and olfaction

  • Research Report
  • Published:
Cognitive Processing Aims and scope Submit manuscript

Abstract

The development of neuroimaging methods has had a significant impact on the study of the human brain. Functional MRI, with its high spatial resolution, provides investigators with a method to localize the neuronal correlates of many sensory and cognitive processes. Magneto- and electroencephalography, in turn, offer excellent temporal resolution allowing the exact time course of neuronal processes to be investigated. Applying these methods to multisensory processing, many research laboratories have been successful in describing cross-sensory interactions and their spatio-temporal dynamics in the human brain. Here, we review data from selected neuroimaging investigations showing how vision can influence and interact with other senses, namely audition, touch, and olfaction. We highlight some of the similarities and differences in the cross-processing of the different sensory modalities and discuss how different neuroimaging methods can be applied to answer specific questions about multisensory processing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3a,b

Similar content being viewed by others

References

  • Anderson AK, Christoff K, Stappen I, Panitz D, Ghahremani DG, Glover G et al (2003) Dissociated neural representations of intensity and valence in human olfaction. Nat Neurosci 6:196–202

    Article  CAS  PubMed  Google Scholar 

  • Barbas H (1988) Anatomic organization of basoventral and mediodorsal visual recipient prefrontal regions in the rhesus monkey. J Comput Neurol 276:313–342

    CAS  Google Scholar 

  • Barbas H (1993) Organization of cortical afferent input to orbitofrontal areas in the rhesus monkey. Neuroscience 56:841–864

    Article  CAS  PubMed  Google Scholar 

  • Benevento LA, Fallon J, Davis BJ, Rezak M (1977) Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Exp Neurol 57:849–872

    CAS  PubMed  Google Scholar 

  • Bruce C, Desimone R, Gross CG (1981) Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. J Neurophysiol 46:369–384

    CAS  PubMed  Google Scholar 

  • Calvert GA, Thesen T (2004) Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol Paris (in press)

    Google Scholar 

  • Calvert GA, Bullmore ET, Brammer MJ, Campbell R et al (1997) Activation of auditory cortex during silent lipreading. Science 276:593–596

    Article  CAS  PubMed  Google Scholar 

  • Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio–visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438

    Article  CAS  PubMed  Google Scholar 

  • Christensen CM (1985) Effect of color on judgments of food aroma and flavour intensity in young and elderly adults. Perception 14:755–762

    CAS  PubMed  Google Scholar 

  • Clark VP, Fan S, Hillyard SA (1995) Identification of early visual evoked potential generators by retinotopic and topographic analyses. Hum Brain Mapp 2:170–187

    Google Scholar 

  • Cooper HM, Parvopassu F, Herbin M, Magnin M (1994) Neuroanatomical pathways linking vision and olfaction in mammals. Psychoneuroendocrinology 19:623–639

    Article  CAS  PubMed  Google Scholar 

  • Driver J, Spence C (1998) Attention and the crossmodal construction of space. Trends Cogn Sci 2:254–262

    Article  Google Scholar 

  • Duhamel JR, Colby CL, Goldberg ME (1991) Brain and space. In: Paillard J (ed). Oxford University Press, New York, pp 223–236

  • Engen T (1972) The effect of expectation on judgments of odor. Acta Psychol 36:450–458

    Article  CAS  Google Scholar 

  • Falchier A, Clavagnier S, Barone P, Kennedy H (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22:5749–5759

    CAS  PubMed  Google Scholar 

  • Fort A, Delpuech C, Pernier J, Giard MH (2002) Dynamics of cortico-subcortical cross-modal operations involved in audio–visual object detection in humans. Cereb Cortex 12:1031–1039

    Article  PubMed  Google Scholar 

  • Giard MH, Peronnet F (1999) Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    CAS  PubMed  Google Scholar 

  • Gottfried JA, O’Doherty J, Dolan RJ (2002) Appetitive and aversive olfactory learning in humans tudied using event-related functional magnetic resonance imaging. J Neurosci 22:10829–10837

    CAS  PubMed  Google Scholar 

  • Grigor J (1995) Do the eyes see what the nose knows? An investigation of the effects of olfactory priming on visual event-related potentials. Chem Senses 20:163

    Google Scholar 

  • Grigor J, Van TS, Behan J, Richardson A (1999) The effect of odour priming on long latency visual evoked potentials of matching and mismatching objects. Chem Senses 24:137–144

    Article  CAS  PubMed  Google Scholar 

  • Haxby JV, Grady CL, Ungerleider LG, Horwitz B (1991) Mapping the functional neuroanatomy of the intact human brain with brain work imaging. Neuropsychologia 29:539–555

    Article  CAS  PubMed  Google Scholar 

  • Jezzard P, Matthews P, Smith S (2001) Functional MRI: an introduction to methods. Oxford University Press, Oxford

    Google Scholar 

  • Kennett S, Eimer M, Spence C, Driver J (2001a) Tactile–visual links in exogenous spatial attention under different postures: convergent evidence from psychophysics and ERPs. J Cogn Neurosci 13:462–478

    CAS  PubMed  Google Scholar 

  • Kennett S, Taylor Clarke M, Haggard P (2001b) Noninformative vision improves the spatial resolution of touch in humans. Curr Biol 11:1188–1191

    Article  CAS  PubMed  Google Scholar 

  • King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500

    CAS  PubMed  Google Scholar 

  • Lloyd DM, Shore DI, Spence C, Calvert GA (2003) Multisensory representation of limb position in human premotor cortex. Nat Neurosci 6:17–18

    Article  CAS  PubMed  Google Scholar 

  • Macaluso E, Driver J (2001) Spatial attention and crossmodal interactions between vision and touch. Neuropsychologia 39:1304–1316

    Article  CAS  PubMed  Google Scholar 

  • Macaluso E, Frith CD, Driver J (2000) Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206–1208

    CAS  PubMed  Google Scholar 

  • MacLeod A, Summerfield Q (1987) Quantifying the contribution of vision to speech perception on noise. Br J Audiol 21:131–141

    CAS  PubMed  Google Scholar 

  • MacSweeney M, Amaro E, Calvert GA, Campbell R, David AS, McGuire P et al (2000) Silent speechreading in the absence of scanner noise: an event-related fMRI study. Neuroreport 11:1729–1733

    CAS  PubMed  Google Scholar 

  • Maravita A, Spence C, Sergent C, Driver J (2002) Seeing your own touched hands in a mirror modulates cross-modal interactions. Psychol Sci 13:350–355

    Article  PubMed  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    CAS  PubMed  Google Scholar 

  • Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cognit Psychol 14:247–279

    CAS  PubMed  Google Scholar 

  • Moir HC (1936) Some observations on the appreciation of flavour in foodstuffs. J Soc Chem Industry Industry Industry Rev 14:145–148

    Google Scholar 

  • Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res 14:115–128

    Article  Google Scholar 

  • Morrot G, Brochet F, Dubourdieu D (2001) The color of odors. Brain Lang 79:309–320

    Article  CAS  PubMed  Google Scholar 

  • Ongur D, Price JL (2000) The organization of networks within the orbital and medial prefrontal cortex of rats, monkeys and humans. Cereb Cortex 10:206–219

    CAS  PubMed  Google Scholar 

  • Osterbauer RA, Matthews PM, Calvert GA (2002) Crossmodal interaction of color and smell. In: Proceedings of the international society for magnetic resonance imaging in medicine.

  • Pavani F, Spence C, Driver J (2000) Visual capture of touch: out-of-the-body experiences with rubber gloves. Psychol Sci 11:353–359

    CAS  PubMed  Google Scholar 

  • Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625

    CAS  PubMed  Google Scholar 

  • Regan D (1989) Human brain electrophysiology. Elsevier, New York

  • Rockland K, Ojima H (2001) Calcarine area V1 as a multimodal convergence area. Abstract Soc Neurosci 27:511.20

    Google Scholar 

  • Rockland KS, Ojima H (2002) Multimodal convergence in calcarine visual areas in macaque monkey. In: International multisensory research forum. Geneva, Switzerland

  • Rolls BJ, Rowe EA, Rolls ET (1982) How sensory properties of foods affect human feeding behavior. Physiol Behav 29:409–417

    Article  CAS  PubMed  Google Scholar 

  • Rolls ET (2001) The rules of formation of the olfactory representations found in the orbitofrontal cortex olfactory areas in primates. Chem Senses 26:595–604

    Article  CAS  PubMed  Google Scholar 

  • Rolls ET, Baylis LL (1994) Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex. J Neurosci 14:5437–5452

    CAS  PubMed  Google Scholar 

  • Royet JP, Zald D, Versace R, Costes N, Lavenne F, Koenig O et al (2000) Emotional responses to pleasant and unpleasant olfactory, visual, and auditory stimuli: a positron emission tomography study. J Neurosci 20:7752–7759

    CAS  PubMed  Google Scholar 

  • Sams M, Aulanko R, Hamalainen M, Hari R, Lounasmaa OV, Lu ST et al (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141–145

    Article  CAS  PubMed  Google Scholar 

  • Shams L, Kamitani Y, Shimojo S (2000) Illusions. What you see is what you hear. Nature 408:788

    Article  CAS  PubMed  Google Scholar 

  • Shams L, Kamitani Y, Thompson S, Shimojo S (2001) Sound alters visual evoked potentials in humans. Neuroreport 12:3849–3852

    Article  CAS  PubMed  Google Scholar 

  • Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586

    CAS  PubMed  Google Scholar 

  • Spence C, Shore DI, Klein RM (2001) Multisensory prior entry. J Exp Psychol 130:799–832

    CAS  PubMed  Google Scholar 

  • Stein BE, Meredith, MA (1993) Merging of the senses. MIT Press, Cambridge, Mass.

  • Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

    Google Scholar 

  • Taylor Clarke M, Kennett S, Haggard P (2002) Vision modulates somatosensory cortical processing. Curr Biol 12:233–236

    Article  CAS  PubMed  Google Scholar 

  • Tipper SP, Lloyd D, Shorland B, Dancer C, Howard LA, McGlone F (1998) Vision influences tactile perception without proprioceptive orienting. Neuroreport 9:1741–1744

    CAS  PubMed  Google Scholar 

  • Van Riper C (1935) An experimental study of the Japanese illusion. Am J Psychol 47:252–263

    Google Scholar 

  • Welch RB, Warren DH (1986) Intersensory interactions. In: Kaufman KR, Thomas JP (eds) Handbook of perception and human performance, vol 1. Sensory processes and perception. Wiley, New York, pp 1–36

  • Wheately J (1973) Putting color into marketing. Marketing 67:24–29

    Google Scholar 

  • Zald DH (2003) The human amygdala and the emotional evaluation of sensory stimuli. Brain Res Brain Res Rev 41:88–123

    Article  PubMed  Google Scholar 

  • Zald DH, Donndelinger MJ, Pardo JV (1998) Elucidating dynamic brain interactions with across-subjects correlational analyses of positron emission tomographic data: the functional connectivity of the amygdala and orbitofrontal cortex during olfactory tasks. J Cereb Blood Flow Metab 18:896–905

    Article  CAS  PubMed  Google Scholar 

  • Zellner DA, Kautz MA (1990) Color affects perceived odor intensity. J Exp Psychol Hum Percept Perform 16:391–397

    Article  CAS  PubMed  Google Scholar 

  • Zellner DA, Bartoli AM, Eckard R (1991) Influence of color on odor identification and liking ratings. Am J Psychol 104:547–561

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

G. A. Calvert and T. Thesen are supported by The Wellcome Trust, R. Osterbauer by the J. S. McDonnell Foundation, and J. F. Vibell by the Medical Research Council UK. We thank A. Fort for helpful comments on the manuscript and for Fig. 2. We thank M. Kringelbach for help with Fig. 1.

Author information

Authors and Affiliations

  1. University Laboratory of Physiology, University of Oxford, Oxford, OX1 3PT, UK

    Thomas Thesen & Gemma A. Calvert

  2. Department of Experimental Psychology, University of Oxford, Oxford, OX1 3UD, UK

    Jonas F. Vibell

  3. Centre for Functional Magnetic Resonance Imaging of the Brain (FMRIB), University of Oxford, Oxford, OX3 9DU, UK

    Thomas Thesen, Gemma A. Calvert & Robert A. Österbauer

Authors
  1. Thomas Thesen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonas F. Vibell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gemma A. Calvert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert A. Österbauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Thesen.

Additional information

Edited by: Marie-Hélène Giard and Mark Wallace

Rights and permissions

Reprints and permissions

About this article

Cite this article

Thesen, T., Vibell, J.F., Calvert, G.A. et al. Neuroimaging of multisensory processing in vision, audition, touch, and olfaction. Cogn Process 5, 84–93 (2004). https://doi.org/10.1007/s10339-004-0012-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10339-004-0012-4

Keywords

Search

Navigation