Skip to main content
SpringerLink
Log in

Cross-Modal Learning in the Auditory System

  • Chapter
  • First Online:
Multisensory Processes

Part of the book series: Springer Handbook of Auditory Research ((SHAR,volume 68))

Abstract

Unisensory auditory representations are strongly shaped by multisensory experience, and, likewise, audition contributes to cross-modal learning in other sensory systems. This applies to lower-level sensory features like spatial and temporal processing as well as to higher-level features like speech identification. Cross-modal learning has particularly profound influences during development, but its effects on unisensory processing are ubiquitous throughout life. Moreover, influences of cross-modal learning on unisensory processing have been observed at various timescales, ranging from long-term structural changes over months to short-term plasticity of auditory representations after minutes or only seconds of cross-modal exposure. This chapter focuses particularly on cross-modal learning and its underlying neural mechanisms in the healthy adult auditory system. Recent findings suggest that cross-modal learning operates in parallel on different neural representations and at different timescales. With an increasing amount of exposure to new cross-modal associations, cross-modal learning seems to progress from higher level multisensory representations to lower level modality-specific representations, possibly even in primary auditory cortex. In addition to cortically mediated learning mechanisms, auditory representations are shaped via subcortical multisensory pathways including the superior colliculi in the midbrain. The emerging view from these findings is that auditory-guided behavior is jointly shaped by cross-modal learning in distinct neural systems. To fully understand the dynamic nature of the auditory system, it will be important to identify how short-term and long-term learning processes interact in the mature brain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262.

    CAS  PubMed  Google Scholar 

  • Bergan, J. F., Ro, P., Ro, D., & Knudsen, E. I. (2005). Hunting increases adaptive auditory map plasticity in adult barn owls. The Journal of Neuroscience, 25, 9816–9820.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Bertelson, P., & Aschersleben, G. (1998). Automatic visual bias of perceived auditory location. Psychonomic Bulletin & Review, 5, 482–489.

    Google Scholar 

  • Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62, 321–332.

    CAS  Google Scholar 

  • Bertelson, P., Vroomen, J., & de Gelder, B. (2003). Visual recalibration of auditory speech identification: A McGurk aftereffect. Psychological Science, 14, 592–597.

    PubMed  Google Scholar 

  • Bertelson, P., Frissen, I., Vroomen, J., & de Gelder, B. (2006). The aftereffects of ventriloquism: Patterns of spatial generalization. Perception & Psychophysics, 68, 428–436.

    Google Scholar 

  • Bertini, C., Leo, F., Avenanti, A., & Làdavas, E. (2010). Independent mechanisms for ventriloquism and multisensory integration as revealed by theta-burst stimulation. European Journal of Neuroscience, 31, 1791–1799.

    PubMed  Google Scholar 

  • Bonath, B., Noesselt, T., Martinez, A., Mishra, J., Schwiecker, K., Heinze, H. J., & Hillyard, S. A. (2007). Neural basis of the ventriloquist illusion. Current Biology, 17, 1697–1703.

    CAS  PubMed  Google Scholar 

  • Bruns, P., & Röder, B. (2010). Tactile capture of auditory localization: An event-related potential study. European Journal of Neuroscience, 31, 1844–1857.

    PubMed  Google Scholar 

  • Bruns, P., & Röder, B. (2015). Sensory recalibration integrates information from the immediate and the cumulative past. Scientific Reports, 5, 12739.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Bruns, P., Liebnau, R., & Röder, B. (2011a). Cross-modal training induces changes in spatial representations early in the auditory processing pathway. Psychological Science, 22, 1120–1126.

    PubMed  Google Scholar 

  • Bruns, P., Spence, C., & Röder, B. (2011b). Tactile recalibration of auditory spatial representations. Experimental Brain Research, 209, 333–344.

    PubMed  Google Scholar 

  • Bruns, P., Maiworm, M., & Röder, B. (2014). Reward expectation influences audiovisual spatial integration. Attention, Perception, & Psychophysics, 76, 1815–1827.

    Google Scholar 

  • Caclin, A., Soto-Faraco, S., Kingstone, A., & Spence, C. (2002). Tactile “capture” of audition. Perception & Psychophysics, 64, 616–630.

    Google Scholar 

  • Chen, L., & Vroomen, J. (2013). Intersensory binding across space and time: A tutorial review. Attention, Perception, & Psychophysics, 75, 790–811.

    Google Scholar 

  • Dahmen, J. C., & King, A. J. (2007). Learning to hear: Plasticity of auditory cortical processing. Current Opinion in Neurobiology, 17, 456–464.

    CAS  PubMed  Google Scholar 

  • Dahmen, J. C., Keating, P., Nodal, F. R., Schulz, A. L., & King, A. J. (2010). Adaptation to stimulus statistics in the perception and neural representation of auditory space. Neuron, 66, 937–948.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Dean, I., Harper, N. S., & McAlpine, D. (2005). Neural population coding of sound level adapts to stimulus statistics. Nature Neuroscience, 8, 1684–1689.

    CAS  PubMed  Google Scholar 

  • Eramudugolla, R., Kamke, M. R., Soto-Faraco, S., & Mattingley, J. B. (2011). Perceptual load influences auditory space perception in the ventriloquist aftereffect. Cognition, 118, 62–74.

    PubMed  Google Scholar 

  • Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162–169.

    PubMed  Google Scholar 

  • Frissen, I., Vroomen, J., de Gelder, B., & Bertelson, P. (2003). The aftereffects of ventriloquism: Are they sound-frequency specific? Acta Psychologica, 113, 315–327.

    PubMed  Google Scholar 

  • Frissen, I., Vroomen, J., de Gelder, B., & Bertelson, P. (2005). The aftereffects of ventriloquism: Generalization across sound-frequencies. Acta Psychologica, 118, 93–100.

    PubMed  Google Scholar 

  • Frissen, I., Vroomen, J., & de Gelder, B. (2012). The aftereffects of ventriloquism: The time course of the visual recalibration of auditory localization. Seeing and Perceiving, 25, 1–14.

    PubMed  Google Scholar 

  • Fujisaki, W., Shimojo, S., Kashino, M., & Nishida, S. (2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7, 773–778.

    CAS  PubMed  Google Scholar 

  • Gutfreund, Y., & King, A. J. (2012). What is the role of vision in the development of the auditory space map? In B. E. Stein (Ed.), The New Handbook of Multisensory Processing (pp. 573–587). Cambridge, MA: The MIT Press.

    Google Scholar 

  • Heron, J., Roach, N. W., Whitaker, D., & Hanson, J. V. M. (2010). Attention regulates the plasticity of multisensory timing. European Journal of Neuroscience, 31, 1755–1762.

    PubMed  Google Scholar 

  • Heron, J., Roach, N. W., Hanson, J. V. M., McGraw, P. V., & Whitaker, D. (2012). Audiovisual time perception is spatially specific. Experimental Brain Research, 218, 477–485.

    PubMed  PubMed Central  Google Scholar 

  • Ikumi, N., & Soto-Faraco, S. (2014). Selective attention modulates the direction of audio-visual temporal recalibration. PLoS One, 9, e99311.

    PubMed  PubMed Central  Google Scholar 

  • Kim, R. S., Seitz, A. R., & Shams, L. (2008). Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One, 3, e1532.

    PubMed  PubMed Central  Google Scholar 

  • Kim, R., Peters, M. A. K., & Shams, L. (2012). 0 + 1 > 1: How adding noninformative sound improves performance on a visual task. Psychological Science, 23, 6–12.

    PubMed  Google Scholar 

  • King, A. J. (2009). Visual influences on auditory spatial learning. Philosophical Transactions of the Royal Society, B: Biological Sciences, 364, 331–339.

    Google Scholar 

  • Knudsen, E. I., & Brainard, M. S. (1991). Visual instruction of the neural map of auditory space in the developing optic tectum. Science, 253, 85–87.

    CAS  PubMed  Google Scholar 

  • Knudsen, E. I., & Knudsen, P. F. (1990). Sensitive and critical periods for visual calibration of sound localization by barn owls. The Journal of Neuroscience, 10, 222–232.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Kopčo, N., Lin, I.-F., Shinn-Cunningham, B. G., & Groh, J. M. (2009). Reference frame of the ventriloquism aftereffect. The Journal of Neuroscience, 29, 13809–13814.

    PubMed  PubMed Central  Google Scholar 

  • Leo, F., Bolognini, N., Passamonti, C., Stein, B. E., & Làdavas, E. (2008). Cross-modal localization in hemianopia: New insights on multisensory integration. Brain, 131, 855–865.

    PubMed  Google Scholar 

  • Lewald, J. (2002). Rapid adaptation to auditory-visual spatial disparity. Learning & Memory, 9, 268–278.

    Google Scholar 

  • Linkenhoker, B. A., & Knudsen, E. I. (2002). Incremental training increases the plasticity of the auditory space map in adult barn owls. Nature, 419, 293–296.

    CAS  PubMed  Google Scholar 

  • Lüttke, C. S., Ekman, M., van Gerven, M. A. J., & de Lange, F. P. (2016). McGurk illusion recalibrates subsequent auditory perception. Scientific Reports, 6, 32891.

    PubMed  PubMed Central  Google Scholar 

  • Maiworm, M., Bellantoni, M., Spence, C., & Röder, B. (2012). When emotional valence modulates audiovisual integration. Attention, Perception, & Psychophysics, 74, 1302–1311.

    Google Scholar 

  • McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.

    CAS  PubMed  Google Scholar 

  • Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154–163.

    PubMed  Google Scholar 

  • Murray, M. M., Lewkowicz, D. J., Amedi, A., & Wallace, M. T. (2016). Multisensory processes: A balancing act across the lifespan. Trends in Neurosciences, 39, 567–579.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Navarra, J., Soto-Faraco, S., & Spence, C. (2007). Adaptation to audiotactile asynchrony. Neuroscience Letters, 413, 72–76.

    CAS  PubMed  Google Scholar 

  • Navarra, J., Hartcher-O’Brien, J., Piazza, E., & Spence, C. (2009). Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proceedings of the National Academy of Sciences of the United States of America, 106, 9169–9173.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Noel, J.-P., De Niear, M., Van der Burg, E., & Wallace, M. T. (2016). Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PLoS One, 11, e0161698.

    PubMed  PubMed Central  Google Scholar 

  • Pages, D. S., & Groh, J. M. (2013). Looking at the ventriloquist: Visual outcome of eye movements calibrates sound localization. PLoS One, 8, e72562.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Passamonti, C., Frissen, I., & Làdavas, E. (2009). Visual recalibration of auditory spatial perception: Two separate neural circuits for perceptual learning. European Journal of Neuroscience, 30, 1141–1150.

    PubMed  Google Scholar 

  • Polley, D. B., Steinberg, E. E., & Merzenich, M. M. (2006). Perceptual learning directs auditory cortical map reorganization through top-down influences. The Journal of Neuroscience, 26, 4970–4982.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Powers, A. R., III, Hillock, A. R., & Wallace, M. T. (2009). Perceptual training narrows the temporal window of multisensory binding. The Journal of Neuroscience, 29, 12265–12274.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Powers, A. R., III, Hevey, M. A., & Wallace, M. T. (2012). Neural correlates of multisensory perceptual learning. The Journal of Neuroscience, 32(18), 6263–6274.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Radeau, M., & Bertelson, P. (1974). The after-effects of ventriloquism. Quarterly Journal of Experimental Psychology, 26, 63–71.

    CAS  PubMed  Google Scholar 

  • Radeau, M., & Bertelson, P. (1977). Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception & Psychophysics, 22, 137–146.

    Google Scholar 

  • Radeau, M., & Bertelson, P. (1978). Cognitive factors and adaptation to auditory-visual discordance. Perception & Psychophysics, 23, 341–343.

    CAS  Google Scholar 

  • Recanzone, G. H. (1998). Rapidly induced auditory plasticity: The ventriloquism aftereffect. Proceedings of the National Academy of Sciences of the United States of America, 95, 869–875.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Recanzone, G. H. (2009). Interactions of auditory and visual stimuli in space and time. Hearing Research, 258, 89–99.

    PubMed  PubMed Central  Google Scholar 

  • Recanzone, G. H., & Sutter, M. L. (2008). The biological basis of audition. Annual Review of Psychology, 59, 119–142.

    PubMed  Google Scholar 

  • Renzi, C., Bruns, P., Heise, K.-F., Zimerman, M., Feldheim, J. F., Hummel, F. C., & Röder, B. (2013). Spatial remapping in the audio-tactile ventriloquism effect: A TMS investigation on the role of the ventral intraparietal area. Journal of Cognitive Neuroscience, 25, 790–801.

    PubMed  Google Scholar 

  • Rohe, T., & Noppeney, U. (2015). Cortical hierarchies perform Bayesian causal inference in multisensory perception. PLoS Biology, 13, e1002073.

    PubMed  PubMed Central  Google Scholar 

  • Roseboom, W., & Arnold, D. H. (2011). Twice upon a time: Multiple concurrent temporal recalibrations of audiovisual speech. Psychological Science, 22, 872–877.

    PubMed  Google Scholar 

  • Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16, 1422–1427.

    CAS  PubMed  Google Scholar 

  • Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.

    PubMed  Google Scholar 

  • Shams, L., Wozny, D. R., Kim, R., & Seitz, A. (2011). Influences of multisensory experience on subsequent unisensory processing. Frontiers in Psychology, 2, 264.

    PubMed  PubMed Central  Google Scholar 

  • Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12, 7–10.

    CAS  PubMed  Google Scholar 

  • Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255–266.

    CAS  PubMed  Google Scholar 

  • Stein, B. E., Huneycutt, W. S., & Meredith, M. A. (1988). Neurons and behavior: The same rules of multisensory integration apply. Brain Research, 448, 355–358.

    CAS  PubMed  Google Scholar 

  • Strelnikov, K., Rosito, M., & Barone, P. (2011). Effect of audiovisual training on monaural spatial hearing in horizontal plane. PLoS One, 6, e18344.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14, 400–410.

    PubMed  PubMed Central  Google Scholar 

  • Tzounopoulos, T., & Leão, R. M. (2012). Mechanisms of memory and learning in the auditory system. In L. O. Trussel, A. N. Popper, & R. R. Fay (Eds.), Synaptic Mechanisms in the Auditory System (pp. 203–226). New York: Springer-Verlag.

    Google Scholar 

  • Van der Burg, E., Alais, D., & Cass, J. (2013). Rapid recalibration to audiovisual asynchrony. The Journal of Neuroscience, 33, 14633–14637.

    PubMed  PubMed Central  Google Scholar 

  • Van der Burg, E., Alais, D., & Cass, J. (2015). Audiovisual temporal recalibration occurs independently at two different time scales. Scientific Reports, 5, 14526.

    PubMed  PubMed Central  Google Scholar 

  • Vatakis, A., Navarra, J., Soto-Faraco, S., & Spence, C. (2007). Temporal recalibration during asynchronous audiovisual speech perception. Experimental Brain Research, 181, 173–181.

    PubMed  Google Scholar 

  • Von Kriegstein, K., & Giraud, A.-L. (2006). Implicit multisensory associations influence voice recognition. PLoS Biology, 4, e326.

    Google Scholar 

  • Vroomen, J., & de Gelder, B. (2004). Temporal ventriloquism: Sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception and Performance, 30, 513–518.

    PubMed  Google Scholar 

  • Vroomen, J., & Keetels, M. (2010). Perception of intersensory synchrony: A tutorial review. Attention, Perception, & Psychophysics, 72, 871–884.

    Google Scholar 

  • Vroomen, J., Bertelson, P., & de Gelder, B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception & Psychophysics, 63, 651–659.

    CAS  Google Scholar 

  • Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research, 22, 32–35.

    PubMed  Google Scholar 

  • Wallace, M. T., & Stein, B. E. (1997). Development of multisensory neurons and multisensory integration in cat superior colliculus. The Journal of Neuroscience, 17, 2429–2444.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Wallace, M. T., & Stein, B. E. (2007). Early experience determines how the senses will interact. Journal of Neurophysiology, 97, 921–926.

    PubMed  Google Scholar 

  • Woods, T. M., & Recanzone, G. H. (2004). Visually induced plasticity of auditory spatial perception in macaques. Current Biology, 14, 1559–1564.

    CAS  PubMed  Google Scholar 

  • Wozny, D. R., & Shams, L. (2011). Recalibration of auditory space following milliseconds of cross-modal discrepancy. The Journal of Neuroscience, 31, 4607–4612.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Yu, L., Rowland, B. A., & Stein, B. E. (2010). Initiating the development of multisensory integration by manipulating sensory experience. The Journal of Neuroscience, 30, 4904–4913.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Zaidel, A., Turner, A. H., & Angelaki, D. E. (2011). Multisensory calibration is independent of cue reliability. The Journal of Neuroscience, 31, 13949–13962.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Zwiers, M. P., van Opstal, A. J., & Paige, G. D. (2003). Plasticity in human sound localization induced by compressed spatial vision. Nature Neuroscience, 6, 175–181.

    CAS  PubMed  Google Scholar 

Download references

Acknowledgments

The authors’ work was supported by German Research Foundation (DFG) Grants BR 4913/2-1 and TRR 169 Subproject A1 and the City of Hamburg Grant “Crossmodal Learning.”

Compliance with Ethics Requirements

Patrick Bruns declares that he has no conflict of interest.

Brigitte Röder declares that she has no conflict of interest.

Author information

Authors and Affiliations

  1. Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany

    Patrick Bruns & Brigitte Röder

Authors
  1. Patrick Bruns
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brigitte Röder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick Bruns .

Editor information

Editors and Affiliations

  1. Department of Speech and Hearing Sciences, Institute for Learning and Brain Sciences, University of Washington, Seattle, WA, USA

    Adrian K. C. Lee

  2. Departments of Hearing and Speech Sciences, Psychiatry, Psychology and Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN, USA

    Mark T. Wallace

  3. Department of Integrative Physiology and Neuroscience, Washington State University Vancouver, Vancouver, WA, USA

    Allison B. Coffin

  4. Department of Biology, University of Maryland, College Park, MD, USA

    Arthur N. Popper

  5. Department of Psychology, Loyola University Chicago, Chicago, IL, USA

    Richard R. Fay

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Bruns, P., Röder, B. (2019). Cross-Modal Learning in the Auditory System. In: Lee, A., Wallace, M., Coffin, A., Popper, A., Fay, R. (eds) Multisensory Processes. Springer Handbook of Auditory Research, vol 68. Springer, Cham. https://doi.org/10.1007/978-3-030-10461-0_11

Download citation

Publish with us

Policies and ethics

Search

Navigation