Audiovisual Integration: The Brain’s Multisensory Approach to Speech Perception

“Do we hear with our ears alone?”

At first glance this question seems quite simple, yet research in auditory neuroscience over the past three decades has substantially changed the answer. For many years, speech perception was largely regarded as an output of the auditory system, and the quality of the auditory signal was accepted as the main determinant of successful communication. Today, however, we know that understanding speech is a complex process made possible by the joint work of not only the ear but also the visual system, attentional mechanisms, working memory and multisensory brain networks.

Studies on audiovisual integration in recent years have shown that the visual cues obtained from a speaker’s lip movements, facial expressions and articulatory gestures play an important role in understanding speech. Integrating auditory and visual information within milliseconds not only increases the accuracy of speech but also reduces listening effort, helps speech be interpreted more accurately when the auditory information is insufficient, and contributes to the sustainability of communication. For this reason, audiovisual integration has today become an important field of study not only for experimental psychology but also for audiology, neuroscience and speech-perception research.

From a clinical perspective, the knowledge accumulated in this field offers important implications for audiology practice. In individuals with hearing loss, and especially in cochlear implant users, explaining speech understanding solely through audiogram results is no longer sufficient. One of the reasons why two individuals with similar auditory performance differ in everyday communication success may be a difference in their ability to use multisensory information. Taking multisensory processing mechanisms into account in auditory rehabilitation is therefore becoming increasingly important.

This article discusses the neurobiological basis of audiovisual integration, its role in speech perception and its possible implications for audiology practice in light of the current literature.

How Does Our Brain Combine Speech?

Although we tend to think that we perceive our surroundings through a single sense, in reality our brain continuously processes information from different sensory sources at the same time. Hearing an approaching car while crossing the street, seeing its motion and estimating its speed; understanding a person’s emotion not only from their tone of voice but also from their facial expression; or keeping up a conversation in a crowded restaurant by turning to the speaker’s lip movements are all natural examples of this process.

This mechanism is defined as multisensory integration. In the simplest terms, multisensory integration is the combination of information from different sensory systems into a single, meaningful perceptual representation. Thanks to this process, environmental events are noticed more quickly, decision accuracy increases and behavioral responses become more effective. Particularly under ambiguous or noisy conditions, incomplete information from one sense can be complemented by cues from other sensory modalities.

Speech perception is one of the most successful examples of multisensory integration. During speech, the speaker’s lip movements, facial expressions and other visual speech cues continuously accompany the auditory signal. The brain does not evaluate these two sources of information independently; on the contrary, it integrates them to form a single speech percept. Understanding speech is therefore a far more multisensory process than we might think.

Why Does Audiovisual Integration Concern Audiologists?

Until recently, speech-understanding performance was largely interpreted through auditory thresholds, speech discrimination scores and auditory processing skills. Today, however, it is known that two individuals with the same audiogram can differ markedly in their everyday communication performance. This shows that the variables determining speech perception are not limited to the auditory system alone.

Cochlear implant users are one of the best examples of this. Electrical stimulation provides more limited spectral and temporal information than natural hearing. Nevertheless, many users can communicate more successfully in daily life than expected. One important reason for this is their effective use of visual speech cues. The effective use of lip-reading and visual speech cues helps compensate for missing auditory information and makes speech easier to understand.

Clinically, this knowledge not only explains how multisensory processing occurs but also brings a new perspective to audiological assessment and rehabilitation. It is not always possible to explain difficulty in understanding speech through auditory thresholds or speech discrimination scores alone. The ability to benefit from visual speech cues is also one of the important variables affecting an individual’s real-life communication performance. Assessing audiovisual integration may therefore contribute to a more comprehensive understanding of speech perception, especially in individuals with hearing loss.

Networks in the Brain: Integration Does Not Happen in a Single Center

The neurobiology of audiovisual integration has been studied for many years. Current neuroimaging studies show that this process does not take place in a single anatomical center; it is carried out through the simultaneous interaction of many cortical and subcortical structures.

While it was initially thought that auditory and visual information were integrated only in higher-order association areas, today it is accepted that this integration is carried out through multiple neural networks that also include subcortical structures such as the sensory cortices, the thalamus and the superior colliculus. Nevertheless, the superior temporal cortex stands out as one of the regions most consistently activated during audiovisual integration. In particular, the superior temporal sulcus (STS) is considered among the structures that play a critical role in integrating auditory and visual information related to speech.

These findings show that audiovisual integration is not a fixed process occurring in a single anatomical center; rather, it is a dynamic processing mechanism to which different brain networks contribute depending on the properties of the stimulus, attention and task demands.

The McGurk Effect: One of the Brain’s Most Intriguing Illusions

One of the best-known examples of audiovisual integration is the McGurk effect. When the syllable /ba/ is presented auditorily while the articulation of /ga/ is shown at the same time, many people perceive the syllable /da/, which is not actually presented.

This shows that the brain does not evaluate auditory and visual information separately but forms a new, integrated perceptual representation. Today the McGurk paradigm is widely used not only in experimental psychology research but also in assessing multisensory integration in cochlear implant users, older adults and various neurological groups.

A Recent Study on Audiovisual Integration

Research on audiovisual integration is also increasing in Turkey. In a recently published study conducted as part of my doctoral work, audiovisual integration was assessed behaviorally using the McGurk paradigm in normal-hearing adults and in prelingual and postlingual cochlear implant users. In addition, Acoustic Change Complex (ACC) responses were recorded in order to examine the relationship between audiovisual integration performance and auditory cortical processing. In this way, behavioral assessments were supported by objective electrophysiological measures, and the possible relationship between the two methods was investigated.

The findings showed that normal-hearing individuals performed better than cochlear implant users on auditory and audiovisual speech-perception tasks. In addition, significant relationships were found between ACC latencies and auditory speech-perception performance in prelingual cochlear implant users. In contrast, ACC responses were not directly related to McGurk performance. These results suggest that the ACC may be a complementary electrophysiological measure reflecting auditory discrimination capacity rather than a direct indicator of multisensory integration.

Conclusion

Work on audiovisual integration shows that speech perception is not merely a product of the peripheral auditory system; it is a dynamic process shaped by the joint work of visual speech cues, cognitive processes and multisensory brain networks. This body of knowledge makes important contributions to many clinical areas — from the assessment of individuals with hearing loss to cochlear implant rehabilitation, and from the communication strategies of older adults to the development of new objective assessment methods.

In audiology practice, evaluating speech understanding solely through the audiogram or speech discrimination scores may not always be sufficient. Understanding how an individual makes use of visual speech cues, especially in noisy listening environments, can contribute to a more accurate interpretation of communication difficulties. As current research on audiovisual integration grows, this knowledge is expected to be reflected more and more in audiological assessment and rehabilitation approaches.

Author

Dr. Ahsen Kartal Özcan

Audiologist · Audiology PhD

She completed her undergraduate and master’s education in Audiology at İstanbul Aydın University and received her PhD in 2026 from the Audiology and Speech Disorders program at Marmara University, with a thesis on memory function, listening effort and multisensory integration in cochlear implant users. Since 2020 she has worked as a research assistant in the Department of Audiology at the Hamidiye Faculty of Health Sciences, University of Health Sciences. Her work focuses on audiovisual integration, cochlear implants, auditory electrophysiology and vestibular assessment; her articles have been published in journals such as International Journal of Audiology, Ear and Hearing, American Journal of Audiology and European Archives of Oto-Rhino-Laryngology.