Why can't I hear clearly when I take off my glasses?

Some people with myopia may find a strange phenomenon: when talking to others, if they take off their glasses, their vision becomes blurred and their voices seem to become unclear. Does this mean that human vision can also "hear" things?

In fact, this is almost true! Our eyes can help us "hear" things, that is, vision can assist hearing.

Our brain is actually a system in which multiple sensory systems work together. Different sensory channels (such as vision, hearing, touch, smell, and taste) are not isolated from each other. The neural networks in the brain are highly connected, and there are a large number of cross-connections between these sensory areas. For example, the temporal lobe is responsible for auditory processing, and the occipital lobe is responsible for visual processing. There are abundant nerve fibers between these areas that can transmit information and coordinate activities. They provide their own information, and the brain integrates this information to enhance the accuracy and richness of perception. [1]

Using brain imaging techniques (such as functional magnetic resonance imaging (fMRI) and electrophysiological recordings, researchers have observed the dynamic activity of these areas under multisensory stimulation. Let's take the audio-visual scene as an example:

When visual and auditory information appear simultaneously, the visual cortex located in the occipital lobe will be significantly active, which is the first stop for visual information processing and is responsible for processing basic visual features such as shape, color, and movement. The auditory cortex located in the temporal lobe will also be significantly active, which is responsible for processing the primary features of auditory information, such as frequency and volume. fMRI shows that the activity of the auditory cortex is not only processing sound information, but also responding to stimuli that appear simultaneously with visual information.

In this process, the superior temporal sulcus (STS) is an important multisensory integration area, especially in audiovisual integration. The superior temporal sulcus receives input from both the primary visual cortex and the primary auditory cortex, and is connected to other higher-level cortical areas. Through neuroimaging studies (such as functional magnetic resonance imaging fMRI and positron emission tomography PET), it was found that the superior temporal sulcus exhibits high neural activity when processing visual and auditory information that appears simultaneously. In other words, when visual information appears synchronously with auditory signals, the superior temporal sulcus is able to integrate the two senses by comparing these signals. For example, when a person sees and hears a person speaking, the superior temporal sulcus combines the visual mouth movement with the auditory speech signal to enhance the understanding of the language.

The amazing phenomenon of the brain "synthesizing" hearing (McGurk effect)

This integration of vision and hearing also leads to a classic psychological phenomenon called the McGurk effect. This phenomenon was first discovered by Harry McGurk and John MacDonald in 1976. In this experiment, participants saw a person's mouth make the "ga" sound, but heard the "ba" sound. They often reported hearing "da" or another third sound. [2][3]

This effect shows that when visual and auditory information are inconsistent, people will have a fusion of perceptual experience, and sometimes people's hearing will be affected by vision, resulting in erroneous cognitive results. This effect reveals the important role of vision in speech perception and the complex mechanism of the brain integrating different sensory information.

The McGurk effect also arises from the multisensory integration mechanism. The superior temporal sulcus (STS) and prefrontal cortex in the brain are responsible for integrating information from the visual and auditory channels. The McGurk effect shows that when there is conflict between two sensory information (visual and auditory), the brain will try to rationalize and integrate this information to form a consistent perceptual experience. [4]

This phenomenon clearly shows that our hearing does not come purely from the auditory channel, but is the result of "synthesis" by the brain. Specifically, when the eyes see a speech action and the ears hear different speech signals, the brain will mix this information, resulting in the perceived speech being neither entirely based on vision nor entirely based on hearing, but a combination of the two. This process of fusion perception shows that vision can, in some cases, influence our perception of speech more than hearing.

So, does talking to different people affect the audio-visual effect?

In fact, it does. For example, the McGurk effect produced by talking to familiar people and talking to unfamiliar people is different.

The familiarity of the speaker's face can also affect our cognition. Studies have found that if the experimenter is familiar with the speaker's face, the McGurk effect is weaker, which means that our hearing is less likely to be affected by vision. On the contrary, if we are speaking to an unfamiliar person, we need to mobilize our vision to understand what the other person is saying, and the McGurk effect is stronger. [5]

The McGurk effect is different in conversations between people who speak different languages. Because the phoneme structure and pronunciation rules of different languages ​​are different, this may also affect the McGurk effect. For example, some languages ​​may have more double consonants or complex consonant pronunciations. These complex speech components may make visual information more significant in speech perception. For example, compared with Mandarin, Cantonese has a stronger McGurk effect, which means that language users rely more on visual information.

The communication culture between people in different countries can also affect the McGurk effect. For example, many Asian cultures (such as Japan and South Korea) emphasize the interpretation of facial expressions and body language during conversations, which may make Asian speakers more dependent on visual information for speech understanding. In contrast, some Western cultures may focus more on accurate speech communication, thus relying less on visual information in McGurk effect experiments.

What is the significance of understanding the phenomenon of audiovisual integration?

Understanding the mechanism behind the human brain's audiovisual integration for hearing can also help to exploit this in the field of artificial intelligence. For example, developing a multimodal speech recognition system that combines visual information with speech recognition can improve the system's recognition accuracy in noisy environments.

At the same time, this can also be applied in hearing aids designed for the hearing impaired. If it can be combined with visual information, it will greatly improve the user's speech comprehension ability. For example, by combining lip reading technology, the lack of some auditory information can be compensated.

Of course, understanding audio-visual integration can also help us solve daily problems. For example, it can help us understand the complex mechanisms behind "hard hearing". Next time when we can't hear others speak clearly, we don't have to doubt that we are "deaf" (it may be because we need a new pair of glasses).

References:

[1] Stein BE, Stanford T R. Multisensory integration: current issues from the perspective of the single neuron[J]. Nature reviews neuroscience, 2008, 9(4): 255-266.

[2] https://www.youtube.com/watch?v=jtsfidRq2tw

[3] McGurk H, MacDonald J. Hearing lips and seeing voices[J]. Nature, 1976, 264(5588): 746-748.

[4] Nath AR, Beauchamp M S. A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion[J]. Neuroimage, 2012, 59(1): 781-787.

[5] Walker S, Bruce V, O'Malley C. Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect[J]. Perception & Psychophysics, 1995, 57(8): 1124-1133.

[6] Combining Behavioral and ERP Methodologies to Investigate the Differences Between McGurk Effects Demonstrated by Cantonese and Mandarin Speakers

Author: Wu Qiong (popular science creator)

Reviewer: Pan Chunchen, deputy chief physician of the Department of Otolaryngology, First Affiliated Hospital of University of Science and Technology of China

The article is produced by Science Popularization China-Creation Cultivation Program. Please indicate the source when reprinting.