Perception of temporal synchrony not a prerequisite for multisensory integration

Abstract

AbstractTemporal alignment is often viewed as the most essential cue the brain can use to integrate information from across sensory modalities. However, the importance of conscious perception of synchrony to multisensory integration is a controversial topic. Conversely, the influence of cross-modal incongruence of higher level stimulus features such as phonetics on temporal processing is poorly understood. To explore the nuances of this relationship between temporal processing and multisensory integration, we presented 101 participants (ranging from 19 to 73 years of age) with stimuli designed to elicit the McGurk/MacDonald illusion (either matched or mismatched pairs of phonemes and visemes) with varying degrees of stimulus onset asynchrony between the visual and auditory streams. We asked them to indicate which syllable they perceived and whether the video and audio were synchronized on each trial. We found that participants often experienced the illusion despite not perceiving the stimuli as synchronous, and the same phonetic incongruence that produced the illusion also led to significant interference in simultaneity judgments. These findings challenge the longstanding assumption that perception of synchrony is a prerequisite to multisensory integration, support a more flexible view of multisensory integration, and suggest a complex, reciprocal relationship between temporal and multisensory processing.