(Don’t) Watch that Mouth: Listen Up by Looking Away

A dish clatters to the floor, and you spin around to view the damage. A friend calls out from beyond your line of sight, and you turn toward the sound. We’re instinctively aware that looking at the source of a sound makes it easier to understand—except when your eyes trick your brain into hearing things.

In a phenomenon known as the McGurk illusion, the syllables you hear sound different if you simultaneously watch a person’s mouth moving in the shape of another syllable. Being aware of this audio-visual trick doesn’t stop your brain from falling for it over and over again, though watching subtitled movies can help a little.

Recently published PLOS ONE research shows that the illusion is caused by visual signals reaching the auditory cortex in the brain faster than the sounds processed by your ears. Researchers analyzed brain signals in the auditory cortex, the part of the brain that processes sound, when volunteers were given a combination of videos and sounds to watch and hear.

The sound clips were of the syllables “ba,” “ga,” “va,” and “tha,” but physical mouth movements in the concurrent video weren’t always the same. In some tests, movements in the video matched the spoken sound perfectly, but in others, the sounds were either completely mismatched, like watching a poorly dubbed movie, or just slightly mismatched. Listeners had no trouble identifying the sound they heard in the extreme case of an absolute mismatch, such as a video of “ba” paired with the sound of “tha”, and they did just as well when sound and video lined up perfectly.   However, when the mismatch was less obvious, such as “ba” with “va,” participants “heard” what they saw (va), and not what was played for them (ba).

When sounds and videos were perfectly matched or mismatched, participants’ brain activity corresponded to the auditory signals. But, when the mismatch wasn’t as obvious, activity in the auditory cortex increased in response to what participants saw, rather than what they heard. More simply, their brains ‘heard’ what they saw, not the sound that was played.

Understanding why the McGurk illusion occurs in the brain isn’t likely to change how we experience the effect (no really, try it for yourself), but the results take us a step closer to learning how we really hear voices in our heads.

Citation: Smith E, Duede S, Hanrahan S, Davis T, House P, et al. (2013) Seeing Is Believing: Neural Representations of Visual Stimuli in Human Auditory Cortex Correlate with Illusory Auditory Perceptions. PLoS ONE 8(9): e73148. doi:10.1371/journal.pone.0073148

Image: Dog looking at and listening to a phonograph, from Wikimedia Commons